Fastest way to save and retrieve floating point vectors

I am writing a code that generates a row vector of size 1x600 (containing floating point numbers) in each iteration. There are about 100,000 such iterations performed in the entire code. Also, I have to read these saved vectors (row wise) in a different code at a later point of time. I have been using 'dlmwrite' so far for appending the vectors row wise to a .csv file, and the function 'load' for retrieving this data. I found that saving and retrieving data in this way is consuming a lot of time. I wanted to know if there is a better way of saving and retrieving these vectors.

I have read some posts in the forum suggesting to save the data in a binary file/.mat file as some alternatives. But I am not sure which would be the best for my case. Could someone please suggest me a faster way to do the above exercise?

Answer 1

[Walter Roberson]

Simplest way, that will take very low overhead for your program (such as might be needed for real-time work):

fid = fopen(OutputFileName, 'w');  %not 'wt'

for ...
    fwrite(fid, YourBufferOfDoubles);
end

fclose(fid)

However, you have about 480 megabytes worth of raw data, and that is enough that unless you are writing to SSD, that the file overhead is going to add up.

Because of that, you might want to consider methods that compress data as you go. You would buffer some data and call compression routines sometimes, and write out the compressed data. One way to do that would be https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPOutputStream.html

A third way... just store all the values in memory as you produce them (having pre-allocated the array), and afterwards save() them to a .mat file. .mat does compression.

[SKP]

Thank you very much for this prompt answer. I shall try out these methods and get back to you.

[SKP]

The suggested methods are very time efficient and occupy lesser memory compared to the ones I was previously using. Thank you.