Saving values in parfor loops

I have a large database (1e8,4) and I want to use parfor command to do some calculations. I also want to use matlabpool command. My problem is in saving the output of the calculations. I used the following line to save the outputs:

parsave(sprintf('%d.mat', i), NumCr, NumNi);

where parsave is a function to save variables NumCr and NumNi.

The problem is that since this line of the code creates a huge number of *.mat files, MATLAB crashes. I also know that with matlabpool open command, global variables are not working. Is it possible to mention how I can save the values?

[Edric Ellis]

Why do you want to save data from the workers - is it not sufficient to let the PARFOR loop complete and save the data from the client?

Are you creating one file per 1e8 iterations of your PARFOR loop?

[Ronaldo]

I would appreciate it if you mention how I can modify the following code to save the values of NumCr and NumNi in each iteration?

parfor i=1:size(A,1)

    O=A(i,1:4);
    Om=ones(size(A,1),3);
    Om1=O(1,1)*ones(size(A,1),1);
    Om2=O(1,2)*ones(size(A,1),1);
    Om3=O(1,3)*ones(size(A,1),1);
    d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
    distance=d>=R1m & d<=R2m;
    NumCr=sum(distance.*Cr);
    NumNi=sum(distance.*Ni);
    parsave(sprintf('%d.mat', i), NumCr, NumNi);

end

Answer 1

[Thomas]

I assume it doesn't really crash with an error but it takes close to forever to output that many files...(?)

I don't know about the memory limitations you have .. your database should be around 1.5GB ... but can't you save the results in an matrix or a cell array first? Like

results = zeros(length(database)) %need to define this outside parfor
parfor i=1:length(database)
  result(i) = NumNi;
end;
save...

Maybe you could even do this inplace, so database(i,1) = NumNi to save memory. Alternatively, if you run into memory problems you could divide your loop into smaller chunks and process some 10000 or 100000 elements at a time in parallel, then save and then continue.

If you have a lot of memory you could also try to setup a ramdisk and output your mat files on this disk - which is much faster even compared to an ssd disk.

[Ronaldo]

result(i)=NumNi is not working in parfor loops. Below is the code that I wrote. I would appreciate it if you help me understand what my mistake is.

parfor i=1:size(A,1)

    O=A(i,1:4);
    Om=ones(size(A,1),3);
    Om1=O(1,1)*ones(size(A,1),1);
    Om2=O(1,2)*ones(size(A,1),1);
    Om3=O(1,3)*ones(size(A,1),1);
    d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
    distance=d>=R1m & d<=R2m;
    NumCr=sum(distance.*Cr);
    NumNi=sum(distance.*Ni);
    parsave(sprintf('%d.mat', i), NumCr, NumNi);

end

[Thomas]

There is no reason why result(i)=NumNi should not work with parfor. I just checked it does. However, it is crucial to define the whole result matrix/vector BEFORE the loop as i did above. Otherwise it would grow dynamically, which is not possible with parallel constructs.

Also try if your loop works in general with a small example database.

[Ronaldo]

It works. Thanks a lot for your great help.