Memory Issues: Almost like clear doesn't fully work

I am running 32b vista, Matlab 2009a. The machine has 4GB of RAM installed, PAE & 3G switch activated. the machine is dedicated to MATLAB and doesn't run anything else.

I have code that reads and writes data from disk. Data is stored in the form of txt files. There are thousands of text files around 200MB each in size. The code loads the text file into RAM processes it and then writes it back to disk.

Each load is done within a loop

for i = 1: numFiles

loadAndWriteData(file(i))

end

function loadAndWriteData(file)

load file

do stuff

save file

clear file (and all other variables I can see lurking around)

end

hence any RAM intensive processes are done in loadAndWriteData.m This returns on each call of the for loop.

I use memory.m to inspect the memomry availble to me. I see that after a few runs around the available memory starts dropping off. Then Matlab crashes with an out of RAM error.

I would have thought that as the function is returning each time around the loop, I would be nicely clearing everything in RAM.

What I am doing wrong? Any help gratefully received. thank you.

Answer 1

[mathworks2011]

please also see http://www.mathworks.com/matlabcentral/answers/5982-batch-files-and-matlab-sequential-execution-and-start-wait

Answer 2

[mathworks2011]

Since MATLAB uses a heap method of memory management, extended MATLAB sessions may cause memory to become fragmented. When memory is fragmented, there may be plenty of free space, but not enough contiguous memory to store a new large variable.

Just to append this. One possible solution is to quit matlab and to start again. This is clearly non-sensical though.

Another, which has the same effect, is to use the pack command. As this cant be used from the command line it is essentially useless too.

This seems to be the most ridiculous problem: essentially it means that matlab can not be used for any processes that require large amounts of data....??!!

[Jason Ross]

Your suggestion of quitting MATLAB and starting again isn't so nonsensical. If this program was amenable to using functions from the parallel computing toolbox, you can set the RestartWorker on the job object to do exactly that.

http://www.mathworks.com/help/toolbox/distcomp/restartworker.html

[mathworks2011]

noted and good point.

However, 1. I dont have the toolbox and (2) I feel such a basic issue shouldnt require specialist treatment.

I dont rememeber much of my C++, but fragmentation isnt an issue there or in C# etc. The problem being matlab doesnt allow you to ever allocate memory according to stack/heap (I dont think...)

http://www.go4expert.com/forums/showthread.php?t=9669

Answer 3

[Matt Fig]

How are you clearing the files? Are you using FCLOSE and checking the return variable?

EDIT

I am not certain that clearing the file identifier will close the file. I don't see it in the documentation...

fid = fopen('myfile')
clear fid  % Is the file in memory or not?
fclose(fid) % Does using this instead of clear help???

[mathworks2011]

using the clear command

[mathworks2011]

I am reading the file as:

M = dlmread(myFullRawPath, ',', 1, 1); %ignore the col headings

fid = fopen(myFullRawPath);

C1 = textscan(fid,'%s%s%s%s%s%s%s%s','Delimiter',',','TreatAsEmpty', {'NA','na'},'CommentStyle', '//');

fclose(fid);

do stuff

clear C1 M;

[Matt Fig]

O.k., so you are already using FCLOSE. That is what I meant in my original question. Why not just use a script instead of functions? That way you can have access to the pack command.

You could inline the contents of loadAndWriteData to a script which has the FOR loop.

That way you could then clear and pack however often you need...

[mathworks2011]

Im not clear on what you mean when you say "Why not just use a script instead of functions? That way you can have access to the pack command".

You do sketch out some code for me?

The reason I never use scripts, is that the architecture of the code is actually very complex (much more so than in MWE I showed here).

[Matt Fig]

A script is an M-file which has no keyword: function in it. For example, this is would be a function M-file:

function B = mysquare(A)

B = A.^2;

and this would be a script M-file:

B = A.^2;

The script will actually execute in the base workspace, and so can call PACK. To change a function to a script, simply copy and paste all of the code into a new M-file. Note that scripts can call function M-files, so just put the bare-bones in a script, then you can call PACK and other command line functions.

Answer 4

[aasim Azooz]

Try this for i = 1: numFiles loadAndWriteData(file(i)) clear all end

if it does not work, you may have a virus in your computer. It happened with me. everything went fine after I scanned my PC for viruses. good Luck Aasim Azooz