Change in Data Storage

Hi All,

My basic question is, has there been a change into the way data is saved (the actual structure of it) for structure files in MATLAB within the past year and a half?

Background:

I have a series of scripts which I use to process large amounts of raw data (10+GB) and save into single files (1-3GB). Since multiple users use these, I have it all built into a MATLAB standalone application which multiple users use. Previously this was all done on MATLAB 2017b. Throughout the year as I have used 2019a I have not noticed any differences either. However I recently updated to 2019b and have noticed considerable changes to my write times in the files. Additionally I have noticed even greater read times for 'new' files (all with the same scripts), even when using the same original data.

tic
who('-file','previouslyGeneratedFile.mat');
toc %0.28s

tic
who('-file','newlyGeneratedFile.mat');%Note <25% of above, using same original data set
toc %17.4s

While the top file reads the variable info in 0.28s, the bottom example takes 17.4s, and that coupled with the second file is less than a quarter size (I used the same original fileset, but what previously took ~1 hour to process now could not complete overnight).

There have been other hints that lead to write/read changes as well which I have not included here.

If anyone has noticed this issue and has a solution please let me know, because without changing anything performance has diminished past any usable limits.

[Oren Lee]

Added inofrmation.

• I tried moving the data to my SSD and reprocessing with no effect (meaning load times on the old file are significantly faster than the new one [note the data is identical]).
• I switched back over to my old .exe which was built on 2017b as noted above, and it also takes unreasonably long to "who" the new file, but it is fast on the old file [demonstrating the same behavior]
• I reran the old data from my 2017b .exe and it processed much faster. This final final was also able to be "who"-ed much quicker as well. This tells me the issue is either in my scripts or in the versions of MATLAB used.

Any guidance on this would be much appreciated. I am going to keep experimenting and seeing what can possibly be different...

UPDATE:

I spotted 1 difference, which I would think would be very minor. I pass the application "app" forward in the function in a "new" version, but not the old one. This is used to send notificaitons to the user during processing. Note, that it is referenced exactly 1 time during all of the processing. I also don't see how this would effect a write times during processing AND read later outside of this function.

2nd UPDATE:

I have installed all patches for 2019b (PATCH 2) to no avail. When I get the opportunity I will be reinstalling 2017b. If anyone has any suggestions let me know, I am at a loss of what I can do.

[Guillaume]

Which mat file format are you using in each case? Are you explicitly setting it in the save call, or are you using the default (under Preferences -> General -> Mat-files), and if so is it the same in both versions?

Can you post the files somewhere where we can look at them?

[Oren Lee]

Hi Guillaume,

Thanks for the answer, just checked it, noticed it was NOT set to -7.3, and updated it. After a quick test, things are back to running as they should! You are a lifesaver as I would not have thought that the default preferences would have changed!

Thanks again!

Answer 1

[Oren Lee]

Answering incase anyone else stumbles across this.

The answer is above as Guillaume noted that the default setting for the SAVE command had changed (see comment on how to update) and needed to be reset to save as -7.3 and NOT -7.0 as they were updated to.

Thanks again to Guillaume!