out of memory error

I'm trying to join to large arrays (rawS=43914x11 and rawP=23772085x11). My code is

out1=cellfun(@num2str,rawS,'un',0)
out2=cellfun(@num2str,rawP,'un',0)
[idxP,idxS] = ismember(out2(:,1),out1(:,1));
out=rawS(idxS(idxP),[1,2,3,4,5,6,7,8,9,10,11]);
out(:,[12,13,14,15,16,17,18,19,20,21,22]) - rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

This ran just fine the first time I ran it, but runs out of memory now (seemingly at different points each time) on the second line for "out2" . No other programs or processes are running. I Tried restarting the computer of course, using tall arrays, deleting variables after they are not needed and so forth, but nothing seems to works. Short of going next to distributed arrays to parallel workers, I would appreciate greatly any ideas or suggestions. Thanks.

[Jan]

num2str(x) can create char vectors with up to 22 characters (e.g. for -pi*1e16) (maybe more?).

A {23772085 x 11} cell array store 23772085*11 arrays, which have an overhead of about 100 bytes. With 22 chars (a char uses 2 byte) per element, this array requires:

23772085 * 11 * (100 + 20 * 2) Byte = 36.6 GB

How many RAM do you have installed?

Please post the code with the tall approach.

By the way, you can simplify

out(:,[12,13,14,15,16,17,18,19,20,21,22]) - rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

to

out(:, 12:22) - rawP(idxP, 1:11);

[Walter Roberson]

out=rawS(idxS(idxP),[1,2,3,4,5,6,7,8,9,10,11]);

That is 11 columns. All of out is overwritten, so it does not matter if out had more columns before this point.

out(:,[12,13,14,15,16,17,18,19,20,21,22]) - rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

That asks for columns 12 to 22 of out but out only has 11 columns.

If the calculation did work then the result would be discarded because of the semi-colon and lack of assignment statement.

[Mark Bodner]

Sorry, I had accidently mistyped the code--the last line is actually

out(:,[12,13,14,15,16,17,18,19,20,21,22]) = rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

[Mark Bodner]

Sorry, I had accidently mistyped the code--the last line is actually

out(:,[12,13,14,15,16,17,18,19,20,21,22]) = rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

The installed RAm is 64 GB.

The code with the tall approach was simply:

t=tall(rawP);

out1=cellfun(@num2str,rawS,'un',0)

out2=cellfun(@num2str,t,'un',0)

[idxP,idxS] = ismember(out2(:,1),out1(:,1));

out=rawS(idxS(idxP),[1,2,3,4,5,6,7,8,9,10,11]);

out(:,[12,13,14,15,16,17,18,19,20,21,22]) = rawP(idxP,[1,2,3,4,5,6,7,8,9,10,11]);

[Jan]

@Mark Bodner: Again, the code would be better to read, if you replace [12,13,14,15,16,17,18,19,20,21,22] by 12:22 .

Answer 1

[Walter Roberson]

Pre-allocate out to the final size. This will reduce temporary memory use. Also, using : is more memory efficient than indexing at the list of columns.

out = zeros(numel(idxP), 22);
out(:, 1:11) = rawS(idxS(idxP), :);
out(:, 12:24) = rawP(idxP, :);