More efficient way to load data

Hello,

Is there a more efficient way to load in data(array), I have a 1x10^7 which is roughly 4mb but takes almost 70s to load. Am using the

         load('filename','varname')

command

[Matt J]

I have a 1x10^7 which is roughly 4mb

That's not really possible. Even if the data where uint8, it would require 10MB of storage.

[Walter Roberson]

It might end up highly compressed.

[Matt J]

@Natialol: If 4MB is the (compressed) file size, it is the uncompressed size that dictates how long it takes to load.

[Walter Roberson]

Matt, decompression is done at CPU speeds, whereas reading from disk is done at disk speeds, which can be 1000 times slower. The usual grouping of speeds is

CPU registers >> Primary Cache >> Secondary Cache >> RAM >> Disk

where each step to the right is roughly 10 times slower than the one before it.

Therefore if you take an average of (say) 50 decompression instructions per disk byte loaded (and it would normally be far less), then you are still far ahead compared to reading from disk.

[Matt J]

But you still have to move it into RAM. Why wouldn't that transfer limit your speed, since in your hierarchy it's 2nd in slowness only to Disk?

And if not, how does one explain the OP's 70sec load time.

[Walter Roberson]

Loads from disk to RAM generally use DMA, and thus proceed in parallel. The decompression would mostly happen in CPU register and primary cache, with the flushes to secondary cache possibly in parallel (depending on the memory controller and the CPU properties.)