Matlab loop (with load/save) is getting slower and slower

Hi

I'm running a large code that is basically:

    for i = 1:50
      load(i.mat, input);
      output = myfun(input);
      save(i-out.mat, output);
    end

I've run "myfun" hundreds of times and it's long-winded but it works fine and is entirely deterministic. The problem which I'm getting is new, basically since I've added the load/save part, which replaces me just saving the output as one element in a pre-allocated vector. The time taken to execute the loop is getting longer and longer as i increases. The first 4 are completed within hours, the but by the time I reach 20 it's taking a full 24 hours before 3 iterations are completed.

There is no accumulation, each iteration should be totally independent, there is no correlation between the size of i and the input file i.mat. I would test using "for i = 25:1" to make sure order doesn't matter, but I don't have another week.

Does anyone know a simple explanation of why saving/loading (these are large files) might slow this down?

Thanks for any help Mike

[Matt J]

Show an example more faithful to the actual code.

load(i.mat,input)

is not a valid command. Also, describe the contents of the file being loaded.

[Walter Roberson]

Is there any graphics involved? Graphics can build up and slow computation down a lot.

[Michael]

There are no graphics, just a lot of matrix manipulation and fminsolve.

I shortened the code, I'm using num2str with sprintf to load 1.mat when i=1, 2.mat when i=2, etc., the same process is being used to label the saved files

[Michael]

The .mat files contain a handful of variables, all matrices to double precision, a couple are about 50x4 and a couple are about 200x200. The output are two matrices, 200x200.

[Image Analyst]

Let's see the actual line where you call save. The paraphrased/pseudocode line you gave is not helpful enough.

Answer 1

[Image Analyst]

Might not make any difference but I'd delete any existing mat file before saving to it to make sure it's not adding/appending to it. See if that helps. Use tic and toc to find out which is taking longer and longer - the load() or the save().

[Michael]

Actually I am overwriting old files of the same name, I'll give this a try and see if it helps, thanks.

[Image Analyst]

Are you running with any breakpoints set? That can really slow things down.

Answer 2

[Shashank Prasanna]

Michael, try the following: put a memory command in the the loop:

for i = 1:50
      load(i.mat, input);
      output = myfun(input);
      save(i-out.mat, output);
      memory
end

Either paste the output or let us know if the memory is increasing at each iteration, this may be a leak. Or I can think of JIT issues.

Answer 3

[vincent caillet]

Hi,

I may have a solution for your problem. In your loop, try to use close all force.

My work is essentially to read a large number of images, do some filtering, and save them back. When I loop through a large number of images, the computer memory linearly increases. I've tried to clear as much memory as possible as the end of the loop, but was not very successful.

I notice that clearing the variables AND closing worked however. Not sure if that applies to other people, but for me it's works perfectly. See the example below:

 for i = 1:50
      load(i.mat, input);
      output = myfun(input);
      save(i-out.mat, output);
      clearvars -expect i OtherVariableOfInterest
      close all force
end

While the code is running, inspect your computer performances (Alt+Esc) to see whether the memory keeps increases, or also drops at the end of your loop.