Matching two arrays that have similar values

I have two folders that have time stamped files. The time stamps are in this format: MMDDYY HHMMSS.### where ### is the number of milliseconds.

I need to compare the contents (file names) of the two folders and generate a new array (or structure) that contains all the files from both folders that have time stamps within 12 milliseconds of each other.

I have manipulated the file name strings and generated numeric arrays with the file stamps but then I still have to compare two large numeric arrays to each other with double loops etc.

That just takes up too much time (I have dozens of folders each containing thousands of files).

I was wondering if there is a quicker easier way to do it?

Thank you in advance

[dpb]

Not sure it would be any faster in the end, but a sorted combined vector of all would allow a single processing pass from the beginning to pick the ones sequentially that are no farther than M millisec from present. Then increment to the next beyond that and repeat.

[Cedric]

Sort first as proposed by dpb, and then you can cluster based on time difference (using DIFF and e.g. ACCUMARRAY).

Answer 1

[Arun Mathew Iype]

It seems sorting the data first and then using some matching algorithm is faster by around 65 times on my system.

What I did was to sort the two numeric arrays using “sortrows”. Then I match each entry in the first array with entries which fall in 12 millisec duration in the second array. Since the array is sorted, the search can be stopped if you get a non-match after a match. Additionally, the next element in the first array may start the search from the first matched element in second array for the previous entry in the first array.

The following is the test script based on an input of around 1000 rows in each file array. >> array_grouping Time for generating matches using simple for loops =5.2695 Time for generating matches using sorting =0.082277

Image : Input files

Image : Output matching