Labeling the Same Object in Consecutive Timeframes (MATLAB)

The Problem: I have timelapse images of numerous small moving objects where the motion is relatively small. When I execute a labeling operation on the objects in different timeframes independently, the labels completely change across the consecutive timeframes.

The Objective: I want to execute a labeling operation on the objects in timeframe t = 1, so that the same objects in timeframe t = 2 can have the exact same label as in the previous timeframe.

The Question: I am thinking of using some sort of nearest-object-is-the-same-object criterion to identify the objects in the timeframes, but I am not sure how to go about this and how it can be done.

The Disclaimer: I have read people suggesting the Kalman Filter for this, but I do not think it is necessary and would like to use a simple nearest-object approach to the labeling. Using the Kalman Filter would also require me to implement a Filter on each individual object and I assume that it would be very computationally expensive (there are about 500 small objects in each timeframe). Besides, the relative motion is not very large and tends to move around in its original vicinity.

The Thanks: Would be very grateful of any comments on how to proceed. If any clarification is required, please let me know. Many thanks in advance.