Out of memory frustration

Matlab is so nice but How I wish it can manage physical memory properly. It does not seem to know how to track the memory it allocates, It says 'out of memory' even when all other variables are cleared. If you restart it and try the same thing again it works but not for very long before it says 'out of memory' once again. It almost seem like the clearing variable has not effect other making the variable invisible.

Its so crazy because pack() which is meant to rescue the situation also gives out of memory error after it has deleted all your variables containing valuable data.

Please if anybody got a solution let me know, I really don't wanna throw this computer through the window if it says 'out of memory' just one more time!!!!!!!!!

Hopefully the computer geniuses at Mathworks will help us clear there footprints

[Yao Li]

try to improve your algorithm. For example, if you want to load data from an Excel or text file, try to load only a few columns or rows at a time instead of loading all the data as a large matrix.

Answer 1

[per isakson]

You should not trough the computer out of the window, even if says "out of memory". You might hurt someone in the street.

Matlab is not perfect, but it mostly recovers memory well. Speculations:

• do you have some persistent data in a function or object?
• there are tools to inspect memory use at the File Exchange and at Undocumented Matlab, e.g. Undocumented profiler options
• Use Task Manager (Windows) to inspect the memory use
• you have found a bug (memory leak). Report to the tech-support with all details.

[bethel o]

I am on the sixth floor and its very windy, hopefully the wind will carry it safely to the bushes when I throw it, because i will certainly need it back when I have calmed down.

I do have a few global variables but they are mostly one element integers. Only one of the global variables is large about [48 x 1536 x 60]. Could this be my problem?

[per isakson]

    >> [48 * 1536 * 60]*8/1e6
    ans =
       35.3894
    >>

35MB is not the problem.

Answer 2

[Jan]

You explained, that even pack fails with an out-of-memory error. pack stores the currently known variables in a MAT file, clears the memory and reloads the data. When this reloading fails, I cannot imagine how the value could fit into the memory before saving.

Exhausting the computer's memory is a general problem and not caused by Matlab itself. The same will happen in any programming language. The only strong solution is installing more RAM and running a 64 bit system, and installing even more RAM. Using smaller data types can help also, but this is a standard procedure and should be applied even if the computer does not crash to save resources.

[Jason Ross]

I was going to come here to say Jan's second paragraph, but he already said it. Some problems just outgrow what a 32-bit OS can provide, and you need to go to a 64-bit OS. Hardware has been capable of 64-bit operation for a long while, and all the OS variants that MATLAB supports have had 64-bit variants for a long time, too.

[bethel o]

More RAM and 64bit, OK. But I am stuck with 32bit because of my realtime simulations which by the way can run for 24-hours without any memory issue.

Well, the failing of 'pack' has to be attributed to matlab because if 'pack' really free the spaces then it should be able to successfully reload the variables at least back to their original space?

If its not matlab's fault then how come when I restart matlab, it starts working again? This means to me that the only way to clear matlabs footprints is to shut it down. I don't mind if have to shut it down every hour or so but for every few minutes=fraustration? Perhaps the command 'pack' should be designed to save the variables, restart matlab and relaod the variables?

Thanks you Jan

[bethel o]

@Jason Ross, I need realtime simulations and this as far as I know does not work well in 64bit matlab and software I have looked at won't work in 64bit.

[Jan]

@bethel: It is not the question, if pack should be designed differently. The builtinin commands works as advertised, and it cannot perform magic. If the memory is "really" free'd depends on the operating system also. Here Windows7 is much smarter than e.g. XP. When your program calls Mex functions or DLLs, which produce memory leaks, restarting Matlab is the only way to free the memory, while pack cannot access the memory (otherwise it would not be a "leak").

Of course a simulation can run for years without exahsuting the memory, when it is carefully designed, or it can fill the memory in milliseconds. Such problems must be solved by the programmer (and the admin, who installs more RAM!), and neither Matlab nor the OS can catch this automagically.

I do not see any reason, why realtime simulation should run worse under 64 bit. When you have troubles with the limited memory, 64 bit (and more RAM) will definitely better than any drawbacks you expect for the actually more powerful system.

Answer 3

[Mike]

You can try feature memstats periodically in your function and it might provide a clue as to what is growing? Maybe use the windows system monitors? cntrl-alt-delete to task manager, then performance tab and use the resource monitor button at the bottom? There are a few views of memory and maybe something else is growing? You went through the "doc Resolving "Out of Memory" Errors"?

A stupid suggestion, but maybe it will spur some "outside of the box" thinking, you might already be doing some of it. Is it possible to save your important data to file, clear everything and reload and restart from where you saved? There are apps out there for windows which will run

If your plot has a lot of data in it and you are using hold, it might be filling up memory? You can try just updating your plot with the latest data. You can just running it without the plot and see how much longer it runs? This is a really helpful discussion: http://undocumentedmatlab.com/blog/plot-performance/

>> feature memstats

    Physical Memory (RAM):
        In Use:                             7142 MB (  1be6bf000)
        Free:                              17430 MB (  4416d7000)
        Total:                             24573 MB (  5ffd96000)
    Page File (Swap space):
        In Use:                             7302 MB (  1c86f1000)
        Free:                              41842 MB (  a3726b000)
        Total:                             49145 MB (  bff95c000)
    Virtual Memory (Address Space):
        In Use:                             8766 MB (  223e7f000)
        Free:                            8379841 MB (7fddc161000)
        Total:                           8388607 MB (7fffffe0000)
    Largest Contiguous Free Blocks:
         1. [at   228010000]             8375214 MB (7fcbae30000)
         2. [at 7feff031000]                4110 MB (  100ec3000)
         3. [at   17c030000]                  63 MB (    3fd0000)
         4. [at 7feebfbb000]                  47 MB (    2fe5000)
         5. [at   13e400000]                  28 MB (    1c00000)
         6. [at 7feeefb7000]                  24 MB (    18e9000)
         7. [at 7fee4fec000]                  24 MB (    1844000)
         8. [at    7bc66000]                  17 MB (    11aa000)
         9. [at 7fef0920000]                  14 MB (     ef0000)
        10. [at 7fee9139000]                  14 MB (     ed7000)
                                            ======= ==========
                                         8379561 MB (7fdca946000)

ans =

             8782048591872

[bethel o]

Mike, thanks for those tips

Answer 4

[bethel o]

So after whinging at everybody about my life and how is all ruined by this 32bit computer and matlab and memory and global warming, yeah, I just got sorted with a 64bit machine runing 64bit matlab. This thing is gold mine. Life is good!!!

Thanks to everyone for all your suggestions, they were all useful!

Bethel

Answer 5

[Walter Roberson]

Are you deleting all the graphics objects as well ?

[bethel o]

I only have one figure window! However I do 'clear all', 'clear functions', 'clear mex' and 'clear java'. I am not sure if the last three have effects? I am not sure how to clear graphics objects although I do not explicitly create many of them?

Thanks

[Jan]

If you have one figure, but create new objects e.g. by image() without deleting the formerly created objects, the memory will be exhausted after a certain number of operations. None of the clearing commands would be helpful then. My impression is, that clearing ios useful under rare conditions only in Matlab.

[bethel o]

OK. Do you know how to properly clear the graphics objects?

Thanks