parfor: exceed memory

Hi all,

It seems that my problem is an old one and many people have faced it, but I still have not fixed problem yet so hope to get help here.

My problem is that: I ran a program (complex, including many functions) on the lab cluster and used 25gb of memory. If I use "for" loop I did not see any problem, but when I used "parfor" I soon received a message says that the required memory is exceeded than 25gb I arranged.

I made a search on this problem and know some thing that it is likely due to data cloning problem but I did not see a good suggestion how to fix it.

Can anyone suggest me a good solution?

Thank you in advance.

Answer 1

[Walter Roberson]

parfor statically examines all of the variables used in the body of the parfor. Any of the defined variables that are not indexed by the loop variable will generally be completely copied into the process that does the work, even if only a portion of the variable is accessed. If the number of workers times the amount of data read in the loop is greater than available memory, you have an inherent problem. If not all of the data of a variable is needed, sometimes using distributed arrays or co-distributed or sliced helps; other times you should extract subsets of the data before the parfor.

Sometimes the question is one of efficiency rather than amount of storage. In that case see http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper

http://www.mathworks.com/matlabcentral/answers/121831-how-to-make-data-persist-on-workers-between-calls-to-parfor#comment_202694

http://www.mathworks.com/help/distcomp/sliced-variables.html

I know that a recent MATLAB release introduced a replacement for Worker Object Wrapper, but unfortunately I do not remember the name of the feature at the moment.

[Tony Tom]

Hi Roberson,

Thank you for your suggestions. I am learning Worker Object Wrapper, However, I am aware that this function still makes a copy to all workers, thus a decent amount of ram is still required.

As you advised, we should index variables by the loop index, so only data needed for a worker is accessed. That means we slice variables.

But what about the case a heavy data need to be accessed in each loop: for example:

parfor i=1:100
g=a_function(H_D);
end

where H_D is a big array (my case,~9GB). Because the whole data of H_D is accessed in a loop, thus if each worker needs a copy of the whole data of H_D, the total amount of ram required is very big. Can we have some way that H_D stored in a worker and be used for all other workers?

[Walter Roberson]

Have a look in the File Exchange for Shared Array.

[Tony Tom]

Thank you Roberson. I have checked, worker object wrapper does not work well on my case, I still face memory problem. I will try with shared array.