Incorrect inliers selected by estimateFu​ndamentalM​atrix

I'm using estimateFundamentalMatrix function to geometrically verify the matches detected by a SIFT based matching system (vlfeat). However, it selects matches that match one point in the one of the images to a large number of points in the other image. For example, the following "inlier" matches were returned by the function:

As far as I understand, F constrains the matches of a point to lie on the epipolar line for that point. It should not give so many random matches for a single point. Can someone clarify what's happening here?