Save inside parfor loop at a specific iteration step

Good day, I'm having problems saving inside a parfor loop. What I want to do is to save the results (vector results) from 5000 -> 5000 iterations (parfor loop takes 1 week to complete and I don't want to lose all the temporary result in case that something bad happens with the machine, i.e. reboot, power off..) A simplified version of my code is:

parfor ii = 1:100000
      [output1, output2] = MyFunction(~,~,ii)

% from 5000 to 5000 iterations I want to save the vector results

   if ii == 1:5000:100000
        parsave('results.mat',output1, output2)

% now I want to save the final results

   elseif ii == 100000
      parsave('results.mat', output1, output2)
   end
end

Maybe someone can provide me with an useful parfor.m function, or a way to save it using matfile objects (I don't know how to use them)

[Sean de Wolski]

First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:

if 1 == 1:5,disp('hi'),end

if any(1 == 1:5),disp('hi'),end

[Matt J]

@CalinV

Saving every 5000th iteration will not help you because parfor loop iterations runs in parallel and in arbitrary order. When iteration #5000 gets executed, it does not imply that iterations 1-4999 have also been executed.

Answer 1

[Matt J]

Since MyFunction takes as long as it does (at least 6 seconds), I might just save every iteration within a for-drange loop. It takes no time to save a 1e5x1 vector, and the local part on each worker will be smaller still.

      output1=distributed.NaN(1,1e5); %pre-allocate
      output2=distributed.NaN(1,1e5);

      spmd
       for i=drange(1:1e5)

          [output1(ii), output2(ii)] = MyFunction(~,~,ii);

          parsave(['TEMP', num2str(labindex)],...
                  getLocalPart(output1),getLocalPart(output2));
       end
      end

[CalinV]

This solution is elegant and works great ! I have used it in this form:

      parpool('local',10); % results will be distributed on 10 workers
      output1=distributed.NaN(1,1e5); %pre-allocate
      output2=distributed.NaN(1,1e5);

      spmd
       for i=drange(1:1e5)

          [output1(ii), output2(ii)] = MyFunction(~,~,ii);

          parsave(['output1', num2str(labindex)],...
                  getLocalPart(output1));
          parsave(['output2', num2str(labindex)],...
                  getLocalPart(output2));
       end
      end

I've used 2 times the parsave function so in case that the machine shuts down before the simulation ends, 10 mat files will be saved with temporary results that I'll need to concatenate later (for output1 & output2)

The parsave function is:

      function parsave(fname,data)

      var_name=genvarname(inputname(2));
      eval([var_name '=data'])

      try
      save(fname,var_name,'-append')
      catch
      save(fname,var_name)
      end

% Written by Minjie Xu (<mailto:[e-mail] [e-mail]>)

Answer 2

[Sean de Wolski]

First, the if ii==vector will never return true because all values need to be true. You would need an any() around it; compare:

if 1 == 1:5,disp('hi'),end

if any(1 == 1:5),disp('hi'),end

Second, since two workers might try to write at the same time, you'll need to save to different MAT files and concatenate after:

save(['Results' num2str(ii,'%05i') '.mat'],etc)

Other than that, looks like it should work...

[Matt J]

I can't see how the OP's initial proposal will work, or at least be beneficial. The iterations of a parfor loop are independent, so saving every 5000th iteration will discard everything else. You also don't know in what order the iterations of a parfor loop will be executed. For all one knows, the iterations 1:5000:100000 could be the final 20 that get executed at the end of the week. A power failure mid-week will lose everything.

[Sean de Wolski]

Maybe I didn't read closely enough - I thought he just wanted those iterations saved.

If he cares about order, use parfeval.