Save data during matlab loop

Hello community!

Time for another question :)

My current project is a simulation run by matlab. So mainly it is a call of a "SimulateStep" function in which a lot of different things happen.

One of the things is the call of a huge mex function which gives me some internal data. So, during each call I have a relativly big struct that has a lot of data in it.

And my question is now: What is the most efficient way to save this data during each step and save it with the propriate time? Also, it should be easy to access and read it afterwards.

My current attemp is looking like that:

function saveData(this, time)
    % input data    
    namelist = fieldnames(this.structToSave);
    for it = 1: length(namelist)
        name = char(namelist(it));
        if ~isfield(this.timeSeries, name)
          this.timeSeries.(name) = timeseries;
        end
        this.timeSeries.(name) = addsample(this.timeSeries.(name), 'Data', this.structToSave.(name), 'Time', time);
    end
end

It works pretty good. At the end of the simulation I save the "timeSeries" struct to a workspace variable and that's it.

But this attemp is way to slow. If I have a list of ~30-40 signals to save you can heavily feel the simulation getting slower.

Does someone has any better alternative to that? Or is it only my programing that sucks here? ;)

Thanks a lot!

[Nitin Kapgate]

Is all the data in the structure of numeric type?

[Simon Meyer]

Yes, all the data is numeric. Mainly double, some integer.

There might be also boolean, but it's no problem to convert that to an integer.

But! The element of the struct can be a vector containing doubles.

Answer 1

[Walter Roberson]

Performance during simulation is important, but performance after the simulation is not as important. So save quickly and process later

function saveData(this, time)
    if isfield(this, timeSeries)
        this.ntimeSeries = this.ntimeSeries + 1;
    else
        this.ntimeSeries = 1;
        this.timeSeries = cell(10000,2);       %pre-allocate
    end
    this.timeSeries(this.ntimeSeries,:) = {time, this.structToSave};
end

The 10000 pre-allocation is a total guess about how many simulation steps you need. It is best to over-estimate: the overhead for unused rows is only 16 bytes per row, whereas the overhead for growing the cell can be high. (The code can be tweaked for lower overhead for growth.)

After the execution, this.ntimeSeries will record the number of entries actually used, and this.timeSeries will be an N x 2 cell array in which the first column is time information and the second is the struct you wanted to save.

In post-processing you can run though and do the saving.

To do the post-processing saving, I would suggest that you fieldnames() of the second column (the structs) and unique(), and then for each unique field name, run through all of the cell rows that saved that field, accumulate the time information (first cell) in the big vector, accumulate the contents of the vectors (making sure to concatenate along an appropriate dimension) into a 2D array for that variable, and addsample() all of it at one go -- only one addsample() for each different variable.

Of course if you are always saving exactly the same list of variables each time, the logic would be easier.

[Simon Meyer]

Hi, thanks for that answer!

The structure itself doesn't change during simulation, but it will change in between runs (the internal data is extendable pretty easily). I don't know how this will help to make the logic much easier, can you give any hints? Just the preprocessing, correct?

I'm giving myself a try for the postprocessing and will be back later! :)

[Simon Meyer]

Hello Walter,

thanks for your help, it works perfectly!

Here is my post-processing:

function my_timeseries = postProcessData(this)      
  time_vec = this.timeSeries(:,1);
  input_vec = this.timeSeries(:,2);
  internal_vec = this.timeSeries(:,3);
  output_vec = this.timeSeries(:,4);

  idx = cellfun(@isempty, time_vec);
  time_vec = time_vec(~idx);
  input_vec = input_vec(~idx);
  internal_vec = internal_vec(~idx);
  output_vec = output_vec(~idx);

  names = fieldnames(input_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), input_vec, 'UniformOutput',false));
    my_timeseries.input.(name) = timeseries(data, time_vec);
  end
  names = fieldnames(internal_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), internal_vec, 'UniformOutput',false));
    my_timeseries.internal.(name) = timeseries(data, time_vec);
  end
  names = fieldnames(output_vec{1,1});
  for i = 1:numel(names)
    name = char(names(i));
    data = cell2mat(cellfun(@(s) s.(name), output_vec, 'UniformOutput',false));
    my_timeseries.output.(name) = timeseries(data, time_vec);
  end
end

As you can see I saved multiple structs in the same line - I just made an easier example in the question above. Also, I modified the saving a little bit. I already have a counter in the simulation call - so I don't need the own counter. But this counter isn't available in the post processing anymore - that's why I'm filtering all data with the time information.

The output is exactly the same like in my previous attemp, but much faster.

During simulation the saving is not noticable and the post processing is - like you said - no problem because it does not matter. I havent measured, but I guess it is the range of only few seconds.

Thanks a again and have a merry christmas!

Answer 2

[Nitin Kapgate]

As the data is numeric/boolean, I would suggest that you to write the data at every step or periodically to a file (say a CSV file with columns for different signals and rows for every step). That would help in increasing the RAM available for simulation and improve the simulation speed. At the simulation progresses, the growing size of logged data can consume huge amount of RAM and slow down simulations.

[Simon Meyer]

Okay, I'm sorry but forgot something...

Some of the data is still double, but in a vector of up to 10 elements. So saidly the matrix part / file-saving part woudn't work here.

[Simon Meyer]

Also, the problem doesn't seem to be RAM that slows down the process after a specific time - it is already super slow in the first few cycles. The call of "addsample" is the time-consuming call, according to the profiler.

[Simon Meyer]

I wanted to give you an answer as well.

We tried to do the csv-file attemp too - but it was also way to slow. It doesn't matter how the data is given (raw, serialized, etc.) and which method is used to write (dlmwrite, fprint, etc.), it took way to long to do it every cycle.

But thanks anyway for that idea! :)

[Nitin Kapgate]

I am glad that the issue is resolved.