Caught std::exception Exception message is: bad allocation

Hi to all!

I am trying to run the code below as suggested by Andrei in this forum to get a data set converted into a 633 by 150005 Matlab matrix for further analysis. The data file itself consists of 633 lines with at most 150005 characters per line plus 1897 headerlines on top of it.

fid = fopen('datafile.txt');
t = textscan(fid,'%150005c','HeaderLines',1897,'BufSize',160000);
fclose(fid);
out = reshape(regexprep(t{1}(:)',{'o',' ','"',','},{'1' '0' 'b' 'e'}),size(t{1}))-'0';

Experiments with 2 lines of 150005 characters yielded a correct 2 by 150005 Matlab variable, at least by changing the BufSize to 160000. However, when I try to run the scripts above using my "real data", it gave the following message and no result at all.

Caught std::exception Exception message is: bad allocation

By typing in memory at the prompt, I get what follows. This may be helpful in resolving the issue.

Maximum possible array:             223 MB (2.340e+008 bytes) *
Memory available for all arrays:    775 MB (8.129e+008 bytes) **
Memory used by MATLAB:             1028 MB (1.078e+009 bytes)
Physical Memory (RAM):             2031 MB (2.129e+009 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.

I attempted pre-allocating by creating zeros(633,150005) but I get "Out of memory. Type HELP MEMORY for your options." Please help.

Thanks,

Bernard

[Andrew Newell]

Could you provide a link to Andrei's answer?

[Walter Roberson]

http://www.mathworks.com/matlabcentral/answers/28742-converting-a-text-file-to-a-matlab-matrix-variable

Answer 1

[Walter Roberson]

Read a line at a time and covert it and store it. The arrangement you have now requires two copies in memory at the same time (original and translated), and that would go through your memory limit even if you were not using 8 bytes per character in the translated output.

As you only have 4 distinct values (it appears from the above), you should store the results as uint8 to save memory.

[Bernard]

Walter, thank you for the advice... Does that mean that I have to back to my old scripts as shown below?

for n=1:50,000;

fid=open('datamap.txt')

for k=1:633;

tline=fgets(fid);

if isletter(tline(n))==1;

Matlab(k,n)=1;

else Matlab(k,n)=0;

end

end

fclose all

end

I find this procedure very slow.... but I'l give it a try though... Do you have any suggestions?

Thanks,

Bernard