You should multiply by fps, instead of dividing. The time between two frames in seconds is 1/fps.
Also, are you reading the video from a file, or are you processing live video? In the latter case, you would not really know your frame rate, because it would depend on how much processing you are doing. The best thing to do would be to get the timestamp for the frames that you are comparing.