how to append data from mat file.?

hello. guys... my data is so heavy(4-5 gb data)...it taking so much time for loading ... i have some mat files...they are actuality cell array...files are like brodata.mat <5136 *2>, brodata1.mat <4328*2>,brodata2.mat <4367*2>,brodata3.mat <5328*2>..... i am trying to make only one file (cell array) ..which contain all the variables from these files i.e <19159 *2 >.

[dpb]

What is in each cell in the cell arrays? If it's just a single numerical element per (as might come from having used textscan without 'collectoutput','true') then using cell2mat and resaving as simply numeric arrays will save quite a bit of room and make loading faster besides...

If not, need to know more about what is in them to have any real definitive suggestions.

[pruth]

in each cell array there are two column , like bellow

<15094x3 double>   735965.041666667
<15072x3 double>   735965.083333333
<15096x3 double>   735965.166666667
<15078x3 double>   735965.250000000
<15101x3 double>   735965.291666667
<15070x3 double>   735965.375000000
<15097x3 double>   735965.458333333
.
.
.
<15106x3 double>   736145.625000000
<15137x3 double>   736145.708333333
<15111x3 double>   736145.791666667
<15133x3 double>   736145.833333333
<15132x3 double>   736145.916666667

i want one file which contain all the data from all cell array. i want next cell arrays two column should append here below these two columns. hope u understand. i m little slow in English.

Answer 1

[dpb]

I expect you'll just make your loading time worse but I'd use the matfile object to try this and see how it goes...

Start by making a copy of the first file in a new file so have a way to recover if having a huge file isn't the answer to your problems...

copyfile('brodata.mat','newbro.mat')  % make the initial new file
newObj=matfile('newbro.mat','Writeable','true');  % create matfile object as writeable
d=dir('brodata*.mat');   % get the list of files
for 2=1:length(d)        % process; skip the base (dir returns ordered list)
  data=load d(i).name;   % get the data from the file
  newObj.data=[newObj.data;data];
  clear data             % get rid of extra copy
end

In all likelihood the above will run into memory issues as it does create a copy in memory

"data" above is a placeholder for your cell variable name in your files; use whatever it is.

You can -append to a .mat file, but only additional variables with save; if you use the same variable name it replaces that name, it does not add onto that array internally -- illustration:

>> whos -file new.mat
Name      Size            Bytes  Class    Attributes
c         4x2              6912  cell               
>> c
c = 
  [100x2 double]    [7.3638e+05]
  [100x2 double]    [7.3638e+05]
>> save new.mat c -append
>> whos -file new.mat
Name      Size            Bytes  Class    Attributes
c         2x2              3456  cell               
>>

Net result is the 2x2 from memory replace the already existing 4x2 on file.

Only way I can think of to create larger arrays on file w/o holding in memory would be to create stream files (which don't support cell arrays well). Then you could do simple file system copy but you lose the existing data structure.