How come my 64-bit Matlab complied shared lib gets "Out of Memory" error when machine (64bit) still has huge free memory?

My OS is Linux 64 bits and my MCR 2013a is also 64 bits. My shared lib was compiled with Matlab compiler 2013a 64 bits too. My machine has >130GB memory.

I cycled my executable (which calls the Matlab shared lib) using the same input and parameters for a few days to test the robustness of my program. I ran 12 processes of the same executable in parallel: each process runs independently on one core of the machine.

All of the processes ran well for ~1000 first iterations but after that all of them kept showing Matlab "out of memory" from the Matlab compiled shared library.

It showed "out of memory" at Matlab's interp function which created just small matrices (size 1000 x 1000).

When the error "out of memory" happened, the machine still had a ton of free memory:

From free command:

 memSize =132.1375Gb free mem =102.3263 Gb

Using top command for a process ID of my program, it showed:

%MEM: 5.6
%VSZ (i.e.virtual memory size): 18860540 Kbytes (~18.8Gb)
%RSS (i.e. consumed physical memory size): 7469936 Kbytes (~7.4Gb)

I checked limit in Linux it showed unlimited, that means no restriction on memory usage of Matlab.

I am thinking the only reason MCR could show "out of memory" is because it could not get a small contiguous block of memory from the huge free 102.3263 GB. But it does not make sense because I cannot imagine that my memory was that fragmented.

Could you please provide some advice about the reason of this error?

Thank you.

[Walter Roberson]

?? Those figures indicate that the program was consuming 7 1/2 terabytes of physical memory on a system that only has 130 gigabytes ?? Or am I reading 'Mb' incorrectly and it is in megabit, which would still have the process using just under a terabyte on a system that does not have nearly that much memory?

[SpringWater]

Thanks Walter. Sorry for the typos. I just corrected them on my post. The process's memory status is

%VSZ (i.e. virtual memory size): 18860540 Kbytes (~18.8Gb)
%RSS (i.e. consumed physical memory size): 7469936 Kbytes (7.4 Gb)

Answer 1

[Walter Roberson]

Each parallel worker is a different process; you need to total the real memory over all of the workers and add the main process in order to find the total memory consumed.

Also, in general one way to run out of memory is to run out of Java heap, which is typically much more restricted (by configuration file) than main memory. That usually produces a distinct error message, and I don't think it is happening to you, but it is something to keep in mind for future.

Also in general if you are invoking an external process and passing it large objects then you can end up running out of stack memory in the external process -- historically it has been common on Unix type systems for a maximum stack size to be baked in at link time instead of the ideal "the stack grows from one end and the heap grows from the other end and you are fine until they meet in the middle". I don't think that is happening in your case though.