speeding up the data analysis

I am working on a project where I have to analyse high frequency wind data (50 Hz) coming from a wind turbine data measurement system. this amount of data must be converted from .dat binary files to .mat files which I can use in matlab. The data must then be filtered and then averaged over 10 minutes to be compared to the data from another measurement system. Doing all this requires the analysis of thousands of data and it's very time consuming (right now it is about 105 seconds just for the data of 2 days). How can I speed the process up?

for ii = 3:length(filedir)
    filename = filedir(ii).name;
    newdata = ReadFamosDataIntoTimeTable(filename);
    % filter
    IsConsidered    = newdata.avbladeangleGRe<40 ... % normal operation
        & newdata.RAWS9>0.5 ... % good availability 
        & ~isnan(newdata.iv10mswindspeed2GRe) & newdata.av100msabswinddirectionGRe>180 & newdata.av100msabswinddirectionGRe<250;
    TurbineData(ii-2).date = filename;
    TurbineData(ii-2).iv10mswindspeed2GRe = mean(newdata.iv10mswindspeed2GRe(IsConsidered));
    TurbineData(ii-2).ivactivepowerGRe = mean(newdata.ivactivepowerGRe(IsConsidered));
    TurbineData(ii-2).CalculatedAirdensity_GRe = mean(newdata.CalculatedAirdensity_GRe(IsConsidered));
    TurbineData(ii-2).HWShub1 = mean(newdata.HWShub1(IsConsidered));
    TurbineData(ii-2).av100msabswinddirectionGRe = mean(newdata.av100msabswinddirectionGRe(IsConsidered));
end

The function ReadFamosDataIntoTimeTable is to convert the .dat binary data into .mat data

Answer 1

[Steven Lord]

What does the Profiler indicate is the section of code that takes the most time?

Does the Code Analyzer app or the code analyzer information included in the Editor when you edit these files make any recommendations that suggest improved performance?

I suspect that ReadFamosDataIntoTimeTable is the bottleneck (simply because it's doing file I/O) but you haven't showed us the code of that function so we can't offer any specific suggestions about it.

One potential (likely small) optimization would be to index into the timetable array once using IsConsidered rather than indexing into each variable separately. Compute something like:

newdataFiltered = newdata(IsConsidered, :);
TurbineData(ii-2).iv10mswindspeed2GRe = mean(newdataFiltered.iv10mswindspeed2GRe);
TurbineData(ii-2).ivactivepowerGRe = mean(newdataFiltered.ivactivepowerGRe);
% etc

[Federico]

Ok I just checked and for some reason it is faster with the filtering of the single variables (timefine = 119.28) than with the filtering of the whole matrix (249.9). I will now share the code for ReadFamosDataIntoTimeTable:

function TT = ReadFamosDataIntoTimeTable(FileName)

% read in data and get time

Channel = importfamos(FileName);

nt = min(cat(1,Channel.length)); % lowest number

Idx = find(cat(1,Channel.length)==nt,1,'first');

dt = str2double(Channel(Idx).dt(1:end-1));

time = [0:nt-1]'*dt;

% create a time table

TT = timetable();

% loop over channels

nChannel = length(Channel);

for iChannel = 1:nChannel

% data

ThisLength = Channel(iChannel).length;

if ThisLength == nt

Data = Channel(iChannel).data;

else % interpolation on lowest resolution

ThisDt = str2double(Channel(iChannel).dt(1:end-1));

ThisTime = [0:ThisLength-1]'*ThisDt;

ThisData = Channel(iChannel).data;

Data = interp1(ThisTime,ThisData,time);

end

% add to time table object

TT = addvars(TT,Data);

% add units and name

TT.Properties.VariableNames{iChannel} = Channel(iChannel).name;

TT.Properties.VariableUnits{iChannel} = Channel(iChannel).yUnit;

end

% add more time information at the end to avoid to add them for each data

TT.Time = seconds(time);

TT.Properties.StartTime = datetime(Channel(Idx).t0);

end

[Federico]

also I checked the profiler and the most time consuming part of the code is the importfamos function so if you want I can send that too