Reliable and economical inspection of rail tracks is paramount to ensure the safe and timely operation of the railway network. Automated vision based track inspection utilizing computer vision and pattern recognition techniques have been regarded recently as the most attractive technique for track surface defect detection due to its low-cost, high-speed, and appealing performance. However, the different modes of failures along with the immense range of image variations that can potentially trigger false alarms makes the vision based track inspection a very challenging task. In this paper, a multiphase deep learning based technique which initially performs segmentation, followed by cropping of the segmented image on the region of interest which is then fed to a binary image classifier to identify the true and false alarms is proposed. It is shown that the proposed approach results in improved detection performance by mitigating the false alarm rate.