load file over network vs copyfile

Hi,

So this is not a complex coding problem, more a basic matlab understanding issue.

I have access to huge amounts of data (in the range of 100s of terabytes) in a network folder. When I read this data (using load) on the actual network path i get a speed in the range of 10-30mbit/s (based on information in the task manager). This is quite slow and will not work.

When I instead copy the folders using "copyfile" from the network location to C:\ I get a speed in the range of 150-200 mbit/s.

Then I can continue reading the variables I'm interested in.

What I can't understand is why there is such a big difference in speed using load on the network location and copying the file to C:\ could anyone here please enlighten me? :)

Answer 1

[Walter Roberson]

load() needs to seek() in the file, and potentially read sections of the same file multiple times.

When seeking around is done, the file cannot logically be cached because on a network drive it must be assumed that another process might be writing to the file. Though that is a detail that depends on the exact file system, as on some network file systems, the file could potentially be locked while it is open, even over the network.

[Martin]

Thank you for your answer.

Then my "ugly" solution by copying all files to C: and then analyse them will always be faster.

[Walter Roberson]

I would not say "always", but that would be most common, yes.

[Martin]

Sorry Walter but you seem to know this stuff, can I add one question?

Would it be possible to virtually run Matlab on the network path somehow? (Lets assume X is a mapped network folder, and that I would be permitted by the network admins.)

If the files are located at "X:\My_files\", could I create "X:\tempAnalysis\" and run the workspace buffer and cache from there? Or something similar to get around the "copy stuff to C:" but still be way faster than reading over the network as described in the initial question.

Or how would you have solved a similar problem? If you could just push me in the right direction that would be greatly appreciated!