Why can I no longer make an array of this size?

I use Matlab to look at time based signals from a CMOS image sensor. To do that, I take 1500 frames, and stack them together to make one time based signal for each pixel (~5M). I've never had an issue doing this before, but suddenly, I'm not allowed. I understand that 56G of allocated memory is ridiculous, but somehow I was able to do this in the past. Any ideas?

The code shown here is essentially identical to an older file that worked.

 % tic;
stacknum = 1500;
imagewidth=2592;
imageheight=1944;
pixel = uint16(zeros(imagewidth,imageheight,stacknum));
for i=1:stacknum
    data = data.';
    filename=sprintf('frame_%d.mat' ,i);  %name files one at a time
    load(filename); 
    pixel(:,:,i) = data(:,:);
end
fname = sprintf('A2_stack_32C.mat'); %name the data file
save(fname,'-v7.3');
sound(randn(409, 1), 8192)
toc;

I've included a loaded array, created a while ago, to show that I'm not straight up lying.

[Jan]

Please explain, what you observe. Where does the "56G" detail come from? Do not let the readers guess, what you observe.

[Ben Hendrickson]

Not sure where the 56G comes from, it's just what Matlab yelled at me:

I didn't know you could specify datatype like that. I'll give it a try.

Answer 1

[Jan]

zeros(imagewidth, imageheight, stacknum)

creates a 60.5GB array, but casting it afterwards requires an additional 15.1GB array, before the zeros of type double are removed. Better use:

pixel = zeros(imagewidth, imageheight, stacknum, 'uint16');

which creates the 15.1GB array directly without wasting memory.

 I understand that 56G of allocated memory is ridiculous, but somehow I was able to do this
 in the past. Any ideas?

If this has worked in the past, you had a machine with more than 75GB of free RAM. I cannot guess, where or when this memory has been gone.

Note:

save(fname,'-v7.3');

cases all variables to the MAT file, including "i", "fname", the magic "data" (coming from the useless "data = data.'" line. Better define explicitely, what should be saved in this file.

sound(randn(409, 1), 8192)

? Brrr, sounds ugly.

[John D'Errico]

My guess is in the past there was more free disk space available, which allowed Ben to create a large array of that size using virtual memory. Since then, the virtual memory block allocated on the hard disk now has less room, so no 56GB array anymore. Just a guess though.

[Ben Hendrickson]

Thank you! It seems to be running fine now.

I assure you I've never used a machine with 75 GB of RAM. How it ever worked may forever remain a mystery.

Fair point about saving multiple variables, it's never caused an issue, so I've never addressed it. Never hurts to employ good coding practices though.

Haha, ya. I wanted something that would startle me back to work. It's been highly effective.