Geometric Transform Estimator

Hi

I'm using the geometric transform estimator to remove outliers as in the video stabilization demo, however it is removing perfectly good matches. The estimated transform becomes when approximately 10 points in one image gets matched to only one in the next image, which obviously is wrong. The problem when I make the matches unique and only keep the best ones, then the correct ones get sorted out. What should I do?!