MATLAB Answers

Iteratively save data to mat file

조회 수: 6(최근 30일)
MCM
MCM 10 Oct 2018
댓글: Walter Roberson 11 Oct 2018
I have to read, process and plot data from 700 text files. I want to save the processed data from one text file to a Mat file before moving on to the next text file. This is a precaution in case of error. I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently.
If I use save('myMatFile.m','x') to save the data from each file, then I will need to create 700 variables in the Mat file; not ideal. If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal. Originally, I use dlmwrite to export to a text file but that is slow.
Also, I'd like to know if my approach is wrong and there is a better way to do this. ANYTHING to speed up this incredibly slow IO process.

  댓글 수: 0

댓글을 달려면 로그인하십시오.

답변(2개)

Stephen Cobeldick
Stephen Cobeldick 11 Oct 2018
편집: Stephen Cobeldick 11 Oct 2018
"I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently."
"If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal."
Use matfile, that is exactly what it was designed for.

  댓글 수: 2

MCM
MCM 11 Oct 2018
When I said, "if I append a single variable in the Mat file then Matlab partially loads it each time; not ideal", I was referring to the matfile function.
When I append using matfile, Matlab gives me a warning that the function will partially read the MAT file in order to do the append
Walter Roberson
Walter Roberson 11 Oct 2018
Consider that if you were changing an existing variable, then MATLAB would have to locate the variable in the file, figure out its size, and make a decision about whether to overwrite in place or to delete the variable and write a new one.
When you append a new variable, MATLAB must first search the .mat file to determine that the variable is not already present, after which it makes whatever changes it needs to the .mat file to write the new variable.
The only way that these processes could be done without partially reading the .mat file would be if every change made by matFile were by simply writing new information on the end without looking inside the file to see what was already there. Not impossible, but it would have the consequence that reading the .mat file would require scanning from the beginning to the end for everything, just in case there was an update later on in the file that overwrote the existing information. It could be done that way, but it would make every read inefficient.

댓글을 달려면 로그인하십시오.


gonzalo Mier
gonzalo Mier 10 Oct 2018
First, use the -nocompression option for the save operator if you have a lot of info.
Maybe you could improve your performance writing the info in 700 different mat files as save("mat_"+num2str(i)+".m") so you don't have to read the info each time.

  댓글 수: 1

Walter Roberson
Walter Roberson 11 Oct 2018
I recently posted a timing test using matFile with and without compression. For the rand() that the user was working with, turning off compression made a big difference. It is plausible that in some other cases like a mostly 0 matrix that compression might be more efficient due to reduced disk i/o.
Be sure to save with -v7.3 to use matFile efficiently

댓글을 달려면 로그인하십시오.

태그

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by