Out of memory in parfor loop

I am getting an error of being out of memory when running the following code, but only when I use a parfor loop, it works fine with a for loop, albeit much much slower. I am using sparse matrices because of the large size size of the matrix I am calculating.

A=spalloc(m,n,maxElements); %m and n are large, <200,000
parfor index=1:n   
    value=zeros(m,1);
    valIndex=findIndex %findIndex is just a custom function
    val=findArea(n);   %findArea is just a custom function
    value(valIndex)=val;
    A(:,index)=value;
end

I am not sure what the issue is with the memory.

I have also tried indexing the elements of the matrix by storing them in a row, column, and value variable, but then indexing within those variable gets complicated and ends up either not being compatible with the parfor loop or using up more memory.

Can anyone explain to me why I am running out of memory? Is it copying the matrix to each worker before writing to it?

I am trying to figure out a way to save each "value" iteration, and then build the matrix "A" outside the loop, but I haven't come up with a good idea of how to do this?

[Edric Ellis]

How does the out-of-memory error manifest itself?

[John R]

It simply says "Out of memory, run HELP MEMORY for more options" or something along those lines.

Answer 1

[Yoav Livneh]

You are running out of memory because parfor uses a lot more memory.

Let's say for example, that each iteration of the regular for loop reuires 300 MB of RAM. IF you have, for example, 8 workers running the parfor loop, each iteration will require at least 2400 MB of RAM, not including the overhead data used by the parfor loop itself.

Maybe if you decrease the number of parallel workers you could find the optimum point between memory usage and running time.

Another solution is to optimize the contents of your loops. For example you don't need to call zeros(m,1) every iteration, you can use:

value = zeros(m,n);
parfor index=1:n
   valIndex=findIndex;
   val=findArea(n);
   value(valIndex,n) = val
   A(valIndex,index) = val;
end

This will result in much less overhead memory transfers, which will reduce run time and memory usage.

The code above might not be legal in a parfor loop, depending on the custom funtions, but I hope you understand the principle.

[John R]

"valIndex" relies on "index" and therefore the parfor code can't classify "value" if it doesn't start with it being zeros ( I don't understand why making it zeros in the beginning allows for the variable to be classified).

[Yoav Livneh]

Maybe if you could write what findIndex does and what are valIndex's dimensions, we can figure it out together.

[John R]

findIndex produces a vector of indices that correspond to the rows that values from "value" should go in the matrix "A".

I feel like there should be some easy fix for my out of memory problem, I just need to save the column of values for each loop, I don't need the entire matrix A available in memory on each loop, I just need it at the end of the script. Is there some way I can do this? Do I have to resort to using "save" and then reconstruct the matrix after all 200,000 rows are saved on my HD?