Parfor loop increase memory usage

Hello,

I am using parfor to process to read several thousand csv files in a directory, perform some operations on each file independently, and save the output. Each of the csv files is not bigger than a couple of MB and I am using about 15 workers. Yet, I seem to run out of all of my 64GB of RAM memory. I notice that the memory usage per worker gradually increases as the loop progresses, from 500MB to 900MB only after a few dozen files have been read.

The pseudocode looks like this:

file_names = dir( directory_with_files);
L          = length(file_names);

parfor i = 1:L

     %read file
     data   = readtable( file_name );
     result = some_function( data );
     f = figure('visible','off');
     plot( result );
     savefig( f );
     close( f );
end

I am trying to be careful about assigning the read file to the same variable so that it overwrites it, closing all plot figures and so on. The only thing I can think of is that the function I use in the parfor is fmincon and it displays a message of whether the optimization was successful or not, and another function in the parfor also displays some messages. Could this be causing the increasing memory usage? What other things can I do to process these files in parallel? Again, the file sizes are very small and I should have several times enough memory for this task,

thank you,

PS: I am using 2018a

****** EDIT ***** I include below the full code:

list = dir('~/Documents/matlab_files/*.csv');
L    = length(list);

parfor i = 1:L

      data = readtable(strcat('~/Documents/matlab_files/',list(i).name));

      all_para = [0.53,0.3,0.001,0.001];

      try
          [x,fval,exitflag,output,lambda,grad,hessian] = fmincon(@(all_para)func_min(all_para,data),...
              all_para,[],[],[],[],...
              [-5,-5,0.01,0.01],...
              [5,5,5,5]);

          M = func_eval(x, data);
          csvwrite(strcat('~/Documents/output/',list{i},'outM.csv'),M);

          f = figure('visible','off');
          plot(M);
          saveas(f,strcat('~/Documents/output/',list{i},'.png'));
          close(f);

      catch ME
              fprintf('fmincon failed: %s\n', ME.message);
              continue;
      end
  end

[Matt J]

How much memory is allocated by some_function when you run serially? If you're consuming 4 GB in serial mode, it would make sense that around 60 GB would be consumed by 15 workers.

[Diego Leal]

Hello Matt,

thank you for your comment.

I ran a regular for loop and i see a modest increase in the RAM usage of about 1MB per iteration. Each of the files that I read per iteration is less than 1MB. I thought that in the for and parfor loops the temporary variables (as is "data" in the psedocode above) would be rewritten with each iteration, so at most I would have 1MB usage in memory per for loop or 15MB per parfor loop. But it seems as if it was "adding" the size each time. Is this what's going on? What is a way around it?

[OCDER]

The memory bug is not obvious based on the pseudocode. Can you show us the actual code? Is there a custom MEX function involved in the some_function? Also, how are you saving the .fig file to unique file names?

savefig(f)  %without specifying the file name, will write to Untitled.fig by multiple workers.

[Diego Leal]

hello @OCDER,

thank you for your help,

I copy below the full actual code:

list = dir('~/Documents/matlab_files/*.csv');
L    = length(list);

parfor i = 1:L

      data = readtable(strcat('~/Documents/matlab_files/',list(i).name));

      all_para = [0.53,0.3,0.001,0.001];

      try
          [x,fval,exitflag,output,lambda,grad,hessian] = fmincon(@(all_para)func_min(all_para,data),...
              all_para,[],[],[],[],...
              [-5,-5,0.01,0.01],...
              [5,5,5,5]);

          M = func_eval(x, data);
          csvwrite(strcat('~/Documents/output/',list{i},'outM.csv'),M);

          f = figure('visible','off');
          plot(M);
          saveas(f,strcat('~/Documents/output/',list{i},'.png'));
          close(f);

      catch ME
              fprintf('fmincon failed: %s\n', ME.message);
              continue;
      end
  end

where "func_min" is the function I am minimizing in fmincon, and func_eval is another separate function, but note that the input in both is just "data" and x or all_para, which are created inside the parfor.

[Edric Ellis]

A few questions:

1. What release of MATLAB/PCT are you using

2. What OS are you using

3. Are you able to come up with a simplified but executable example that demonstrates the problem you're seeing

I tried to reproduce the problem using R2018a on Linux using 1000 simple numeric CSV files of about 3MB each, where I was calling readtable, plot, and saveas in a parfor loop. I didn't see any increase in memory usage on the workers.