any faster way to load part of the mat file?

hi: I have a large matfile ROP, with the size of 45*156*300000, but I only need part of the data with given i: ROP(i,:,:).

here I have two ways: 1)directly load the file. 2)use matfile commmand.

below is the code I used, and the tic-toc result:

tic
load ROP
toc

tic
m=matfile('ROP.mat');
test=squeeze(m.ROP(1,:,:));
toc

Elapsed time is 39.358792 seconds.
Elapsed time is 32.416581 seconds.

looks it does not make the loading significantly faster than before. and the only positive effect is that it saves the memory.

when I load a matrix same size with the ROP(1,:,:), it only cost about 3 seconds, which is about 10 times shorter.

so my question is: is there anyway to make the loading faster? while keep the same small memory like matfile command?

thanks!

Li

[José-Luis]

How do you create ROP? The fastest way would probably be to only to save what you need from the start.

Answer 1

[Guillaume]

To efficiently use the benefits of matfile it is important to understand how matlab stores matrices in memory (and on file). Matlab stores all the rows of the first column of the first page continuously, then all the rows of the the second column of the first page, etc. until the last column of the first page, then it's the same with the second page. So the storage order is

rop(:, 1, 1), rop(:, 2, 1), ..., rop(:, end, 1), rop(:,1, 2), ..., rop(:, end, end)

With that in mind, using rop(x, :, :) is not efficient at all, since matlab needs to skip size(rop, 1) elements between each element you've requested. However, if you'd requested rop(:, :, x), matlab just needs to seek to the right location and then read the whole lot as one consecutive read. Much faster.

All, this to say, that if you want to speed up matfile read, you ought to change the order of the dimensions when you write that rop variable to file:

rop = permute(rop, [3 2 1]);
save('ROP.mat', 'rop');

m=matfile('ROP.mat');
test = m.ROP(:, :, 1)

[Yu Li]

hi: thanks for your reply, I tried this idea but looks still cost lot of time. below is the code, I load the ROP, re-arrange it and save to ROP1, and tried to load ROP1 with the data interested.

tic
load ROP
toc

tic
ROP1 = permute(ROP, [3 2 1]);
toc

tic
save ROP1 ROP1 -v7.3
toc

tic
m=matfile('ROP1.mat');
test=m.ROP1(:, :, 1);
toc

and here is the result:

Elapsed time is 39.310962 seconds.
Elapsed time is 5.730803 seconds.
Elapsed time is 134.724888 seconds.
Elapsed time is 31.991022 seconds.

however thanks your answer! best! Li

[Urs Hofmann]

For me it decreased the time required to read the matrix from 4.9 s to 2.0 s (on top comes 0.5 s for each loading for permuting and 3.5 s while saving).

% hugeMatrix has dimensions [3000, 1001, 1001] of type 'single')
% size: 11.2 Gb
% cropped version: 1.5 Gb 

% permuted version
hugeMatrix = permute(hugeMatrix, [2, 3, 1]); % 3.5 sec
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all

m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(:, :, 1:400); % 2.0 sec
hugeMatrix = permute(hugeMatrix, [3, 1, 2]); % 0.5 s


% plain version
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all

m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(1:400, :, :); % 4.9 sec

Meaning: I only save once during a measurement and then load it multiple times, so permuted version is the way to go for me.