Matlab save with -append is quick the first time then slows down dramatically the second time

I'm trying to attach a last used timestamp to some caching in my code so that I can later quickly load in the lastused variable and delete cached entries that haven't been used in the last N days. While loading a single datetime object is quick as I would expect, it seems like save(cacheFile, 'lastused', '-append') is an operation that scales with the size of the file (which can get to GB or larger). Why is this the case?

Also, in my testing, it seems like the save with -append is NOT a problem the first time, but after that it does become file size dependent. Why?

Here is some sample code that I've been running for testing the timing:

% time to save the big file | this is expected to be large
data = rand(3000);
tic; save('test.mat','data'); time.save = toc;

% time to append the first item | this is quick as I would hope all appends are
lastused = datetime;
tic; save('test.mat','lastused','-append'); time.appendFirstTime = toc;

% time to append the second time | this becomes slow
lastused = datetime;
tic; save('test.mat','lastused','-append'); time.appendSecondTime = toc;

% time to append a new variable | still slow, this shows that overwrite isn't causing the issue
othervar = datetime;
tic; save('test.mat','othervar','-append'); time.appendOtherVar = toc;

Answer 1

[Suraj]

Hi Aaron,

I understand that you’re using “save()” function in MATLAB with the “-append” flag to write to an existing “.mat” file.

I suggest that you use MAT-file version “v7.3” which runs with faster save times for the code you’ve provided. You can do use at the time of initially saving your ".mat" file, with the following command.

save('test.mat','data','-v7.3');

You can read more about MAT-file versions from the documentation: https://www.mathworks.com/help/matlab/ref/save.html#btox10b-1-version

I hope this helps!

Best regards,

Suraj.

Answer 2

[Pritesh Parmar]

Hey Aaron,

I had the same issue with MATLAB's "save -append" with updating variables with large dataset. Try the following code: fastSaveUpdate - Efficiently update variables in MAT-file - File Exchange - MATLAB Central

Pritesh Parmar