How to Concatenate struct arrays efficiently

I use parallel processing to run roughly 35,000 jobs in parallel. They output a .mat file and I want to concatenate them into one big file.

The process takes too long to run. (I am waiting for it to run while writing this) Is there a more optimal way to concatenate struct arrays so this doesn't take hours to run?

I'm guess the the limiting factor is reallocating "finalData".

Note: outData is a struct array of n elements

inputs = dir([intputDir,filesep,'*.mat']);
  finalData = [];

    if ~exist([outputDir],'dir')
            mkdir([outputDir]);
    end
    l=length(inputs);
    for i = 1:l
        load([inputs(i).folder,filesep,inputs(i).name]);
        finalData = [finalData,outData];
        if (mod(i,100) == 0)
            fprintf('Finished .Mat File # %d of %d',i,l);
        end
    end

    save([outputDir,filesep,'allOutputs'],'finalData');
    struct2xls(finalData,[outputDir,filesep,'Timing_Analysis_Data.xlsx']);

[James Tursa]

Load into variables inside the loop, and then concatenate all of the variables after the loop. Rough outline:

    c = cell(l,1);
    for i = 1:l
        c{i} = load([inputs(i).folder,filesep,inputs(i).name]);
        if (mod(i,100) == 0)
            fprintf('Finished .Mat File # %d of %d',i,l);
        end
    end
    % then concatenate all of the c{:} here

Do all of your individual struct arrays have the same fields?

[Kenneth Yesh]

Yes, they all have the same fields so they concatenate without any issue. I just thought it was taking way to long to run.

Answer 1

[Stephen23]

Your very slow code is due to that your variable finalData changes size on every loop iteration, forcing MATLAB to move it in memory. Read this to know more:

https://www.mathworks.com/help/matlab/matlab_prog/preallocating-arrays.html

It is easy to import that data into one preallocated structure, something like this:

D = dir(fullfile(inputDir,'*.mat'));
N = 1:numel(D);
S = struct('outData',cell(N,1));
for k = 1:N
   F = fullfile(inputDir,D(k).name);
   S(k) = load(F,'outData');
end
Data = [S.outData];
save('newfile.mat','Data')

Note it is recommended to use fullfile rather than fiddling around with file separators and string concatenation.