This paper presents a robust outlier elimination approach for multimodal retina image registration application. Our proposed scheme is based on the Scale-Invariant Feature Transform (SIFT) feature extraction and Partial Intensity Invariant Feature Descriptors (PIIFD), and we combined with a novel outlier elimination approach to robustly eliminate incorrect putative matches to achieve better registration results. Our proposed approach, which we will henceforth refer to as the residual-scaled-weighted Least Trimmed Squares (RSW-LTS) method, has been designed to enforce an affine transformation geometric constraint to solve the problem of image registration when there is very high percentage of incorrect matches in putatively matched feature points. Our experiments on registration of fundus-fluorescein angiographic image pairs show that our proposed scheme significantly outperforms the Harris-PIIFD scheme. We also show that our proposed RSW-LTS approach outperforms other outlier elimination approaches such as RANSAC (RANdom SAmple Consensus) and MSAC (M-estimator SAmple and Consensus).