check parameter used by function with large memory

Hi:

I ran my code and found that after a certain time the memory is occupied by nearly 50G. I want to find the parameters which used so much memory but there are lots of sub-functions.

is there anyway to find the corresponding parameters, sub-functions, and memory used? so that I can optimize my code?

Thanks! Yu

[OCDER]

https://www.mathworks.com/matlabcentral/answers/97560-how-can-i-monitor-how-much-memory-matlab-is-using

This might be relevant for your case.

[Jan]

How do you determine the amount of "occupied" memory? Maybe all you see is the reserved memory, which does not necessarily mean, that it is in use.

Beside the parsed source code, functions do not "use" any memory at all, except if you store persistent memory. Only data occupy memory.

Without seeing the code, it is impossible to suggest an improvement.

[Yu Li]

Hi: The code is a large ‘for’ loop. At the very beginning the memory is small but after some time I see 50G memory used by Marlab in ‘task manager’. So I think some parameters in the loop are accumulate during the calculation.

Thanks! Yu

Answer 1

[OCDER]

REAL NEW ANSWER

You are generating a ton of invisible figure handles, and clearing the variable name without closing the figures. This will accumulate a ton of figure handles that are not referred to by any variable name (hence whos fail to detect these. Use close and not clear.

for j = 1:10
    f = figure('visible', 'off');
    clear f %Clears the variable, but not the handle! Use "close(f)"!
end
findobj(0, '-class', 'matlab.ui.Figure') %You have tons of figures, hidden.

close all %closes all your figures
findobj(0, '-class', 'matlab.ui.Figure') %All the figures are deleted now.

NEW ANSWER

Use the undocumented memory profiler as shown here: http://undocumentedmatlab.com/blog/undocumented-profiler-options

To summarize, do this in you matlab command window:

profile -memory on
setpref('profiler', 'showJitLines', 1);
profview

%In profview, on the top next to "Run this code:", type in the command to run your code.

OLD ANSWER

Use whos to determine the variable names and size in the for loop. Then print out the largest var and its size. Here's an example.

https://www.mathworks.com/help/matlab/ref/whos.html

 for j = 1:100 %Your large for loop
      ... blah blah blah       

      if mod(j, 10) == 0 %just so  you don't print too many times
          W = whos;
          [MaxByte, MaxIdx] = max([W.bytes]);
          fprintf('Largest Var = %s [%0.3f GB]\n', W(MaxIdx(1)).name, MaxByte/1e9);
      end
 end

[Yu Li]

Hi:

Thanks for your reply. but still could not find the variable.

I re-run the code until it uses 10GB memory, but when I use 'whos' in the main function, it only shows a 100MB variable.

Perhaps the variable which contains a large memory is in some sub-function, but there are about 20 sub-function in my code. except the 'BF' method, i.e. use 'whos' in all the sub-functions, is there any other solution regarding this problem?

Thanks! Yu

[OCDER]

Interesting... just found a undocumented memory profiler. See edited New Answer above.

[Yu Li]

Hi:

Thanks for your reply. The problem was solved.

meanwhile, as f is closed at each loop, the performance of the code looks improved.

Thanks!

Yu

Answer 2

[Jan]

Remember, that the RAM shown in the TaskManager is not necessarily "occupied". If Matlab reserves 100MB frequently in a loop, the released memory is not necessarily given back to the operating system. And if it is, the OS might not find the time currently to overwrite it with zeros, such that it can be delivered to an application again. This means, that the TaskManager does not tell you, how much RAM the code needs. The code might run with 1GB RAM also fluently, although the memory managers of Matlab and the OS reserve more memory, when it is available.

[Yu Li]

Hi: thanks for your reply. assuming I have 96GB memory in my workstation, I think what you mean is: 'even though the task manger shows that 96GB was occupied by Matlab, it does not really be occupied. and Matlab can operated successfully.', do my understanding right?

I will try to re-run the case until the memory usage is full. will get back to you and let you know what I find.

Thanks! Yu

[OCDER]

By any chance, does your program use a custom MEX file?

[Yu Li]

no, all the functions are from Matlab.

[OCDER]

Ok, in that case, not a memory leak from C++... Did the memory profile at least point to the function that's using a lot of memory? You can then narrow things down and use that whos trick to figure out the large variable in that one function.

[Yu Li]

Hi:

My PC has 96GB memory, and after run the code for one night, it is shown that 99% of the memory was occupied in task manager.

I go on run the code again without close the Matlab, the memory usage keeps at 99%, and the performance of my code looks the same like before. I'm not sure why.

Thanks!

Yu

[OCDER]

If you use clear all in your matlab session, does your memory free up, or do you have to restart matlab completely to get your memory back? When you say "performance looks same like before", before what changes?

Try to search for:

• Growing global variables / matrix / cell

X = [X rand(1000, 1)];

• Accumulation of invisible figure handles with a lot of data in each

figure('Visible', 'off')

Can you show us the result of the undocumented profiler result? If all else fails, it'll probably be faster to tediously investigate each function, 1 by 1.

[Yu Li]

Hi:

Thanks for your reply.

1. the clear all does not work, it could not free up the memory, only restart the matlab can complete get my memory back.

2. 'the performance looks same like before', it means the even thought the code runs with 99% memory, performance looks like the same with 10% memory.

3. I've re-run the code and will let you know what shows in the profiler.

4. you are right, I used figure ('visible','off') in my loop, but I have clear the invisible figure at end of each loop, below are my source code:

    for i=1:1:1000000
        f = figure('visible','off');
        ~blabla
        print(fig_name,'-djpeg','-r1200');
        clear f
    end

Thanks!

Yu

[OCDER]

Updated my answer above. That's a tricky bug.

Answer 3

[Guillaume]

I used figure ('visible','off') in my loop, but I have clear the invisible figure at end of each loop

f = figure;
clear f

No you haven't! There's a huge difference between clear and close. clear f gets rid of the variable f but the figure it points to still exist. close(f) closes the figure pointed to by f but keep f as a variable (which is now an invalid figure handle).

So it's no wonder you're low on memory, you've got all these figures open. Replace the

clear f

by

close(f);

In general, you should never use clear. clearing a variable that is created in a loop in particular serves no purpose. close on the other hand is useful.