MATLAB Answers

Fast access/concatenation of large array structure

조회 수: 12(최근 30일)
D. Plotnick
D. Plotnick 2017년 4월 13일
댓글: Matt J 2017년 4월 19일
I have a large structure array (500K+ items), and I wish to access certain fields of the that array and concatenate the results. Below is a placeholder example.
A{1}.time = 1500
A{1}.data.temp = 70;
A{1}.data.humidity = 20;
A{2}.time = 1501
A{2}.data.temp = 73;
A{2}.data.humidity = 19;
etc. Till we have 500,000 of these. (I have made it a cell array since the actual entries differ in my data, and I have other code that will go through and just grab the cells we want.)
Now, I want to access e.g. all of the 'data' and concatenate it so that I have a simple vector I can plot. Currently this is done using a loop, but that is very slow. Is there a faster way to do this than some version of the below:
fieldNames = fields(A{1}.data);
for ii = 1:length(fieldNames)
out.(fieldNames{ii}) = ...
cat(1,cellfun(@(x) getField(x,'data',fieldNames{ii}), A));
end
where
function out = getField(in, fieldname1,fieldname2)
out = in.(fieldname1).(fieldname2);
end
Again, this certainly works but for extremely large datasets with lots of fields it becomes very very slow. I bet that there is a much more efficient way of gathering all of the data contained in the fields and subfields of a large data set like above. Any help is appreciated.
Thanks, -Dan
An additional discovery: Matlab is somehow storing the field names for each sub-structure individually. In the above example, it has memory allocated for the fieldnames data.temp and data.humidity TWICE (once for each copy). This is why it is so slow. A 50 Mbyte set of data has grown to 3 GB because of this organization scheme. I am going to make a separate post about this (is the memory issue resolved if each entry is a known class? That way the field names aren't stored once for each copy?).
  댓글 수: 5
Walter Roberson
Walter Roberson 2017년 4월 14일
memmapfile is convenient but not mandatory: you can use a bunch of fread() instead.

댓글을 달려면 로그인하십시오.

채택된 답변

Matt J
Matt J 2017년 4월 14일
편집: Matt J 2017년 4월 14일
I don't quite understand why you are using a cell array and not a struct array. Your example will not work unless all A{i} have the same field names so a struct array should have been sufficient.
In any case, you should be able to create a struct array and do faster concatenations with it as follows,
structArray=[A{:}];
dataArray=[structArray.data];
Now you can concatenate individual fields of dataArray in a similar way, e.g.,
out.temp=[dataArray.temp];
You can loop over the field names of dataArray, similar to what you are doing now, to repeat this kind of concatenation for all fields.
I feel obliged to mention, though, that your data organization looks like it is bound for trouble. It is inefficient, in general, to store large amounts of data scattered across structs and cell arrays. Cells and structs do not use contiguous memory, and so are not efficient storage-wise or built for fast access. Nesting them in sub-structs makes the problem worse. You should really be storing all your temp, humidity, etc... data in their own vectors to begin with. Or at least, you should be doing so if the data is going to get large and speed is a priority.
  댓글 수: 6
Matt J
Matt J 2017년 4월 19일
If you really want these results re-inserted back into a struct array ... then you must use a for-loop
My comment here wasn't really very precise. There are alternatives to for-loops, but they will not be faster, e.g,
header1Cell=num2cell( swapbytes([example.header1]) );
[example.header1]= deal(header1Cell{:});

댓글을 달려면 로그인하십시오.

추가 답변(0개)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by