Spatial verification is a key step to remove outliers for accurate feature matching in visual place recognition. In this paper, we propose a novel method for outlier detection using a hierarchical spatial verification scheme. Given a set of putative correspondences between a pair of images, we convert the matching problem into a 4D transformation space and identify promising similarity transformations using Hough voting. In the hierarchical scheme, we first use a hypothesize-and-verify technique to identify groups of correspondences according to each similarity transformation. Second, the group with the largest number of correspondences serves as a standard to subsequently remove outliers in other groups by explicit geometric consistency checking. We have compared the proposed method with the state-of-the-art solutions on five popular public datasets to show that our method has better performance in place recognition and loop closure detection.