Importing data from a large data set

I have a ~30 GB worth of data in a .mat file. The .mat file has m*n (m = ~10^9, n = 10).

I need to run a simulation that derives its value from randomly selected row (from a choice of row 1:m).

The column (1:n) has the simulation parameters.

What is the best way (memory and speed) to do it? Is it possible to do it without loading/importing the entire .mat file in Workspace?

Answer 1

[Ameer Hamza]

Yes, you can read data from a mat file without loading it completely. See matfile()https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html

[hal9k]

Here is what worked for me.

tic

importData = matfile(filename);

data_info = whos('-file',filename);

row = data_info.size(1);

rand_row= randi([1 row],1);

output = importData.filename(rand_row,:)

toc

Elapsed time = 0.02 sec.

This is great.

[Ameer Hamza]

Glad to be of Help.