From my understanding you want to read a large video file. Whenever you read a video file the first call to "read" scans the entire file to accurately determine the exact location in the video file to seek to, given a specific frame index. This is what causes the large overhead of reading the first frame.
It is better to use the "readFrame" method of the "VideoReader" class. You can seek to a specific time by setting the "CurrentTime" property of the "VideoReader" object to the time that you wish to seek to. If you want to read frames sequentially from that specific point, you can set the "CurrentTime" once and call "readFrame" repeatedly. In this case, you do not have to pay a penalty for computing the total number of frames.
>> vidObj = VideoReader('filename');
>> vidObj.CurrentTime = timestamp;
>> img = vidObj.readFrame;
When you do this, the video starts decoding at the I-frame closest to the "CurrentTime". The time this takes depends on the number of key-frames in the video: if there are fewer key-frames, it takes more time. For example, if you want to read a frame at time t = 100 seconds, and the nearest key-frame is at 95 seconds, and the video is 30 fps, approximately 5*30 frames have to be decoded before getting to the frame. If you wish to read the video non-sequentially, you can keep setting the "CurrentTime" property, but the performance completely depends on the number of key-frames in the video.
Another thing that can make this process faster is to use AVI format with MJPEG compression for the videos.