improving data averaging in a loop (is too slow)

Good Morning, I have a series of data in a matrix and i want to take the average data every

SECONDS_SCAN = (number of seconds),

"remembering" the deltaT of time between the data.I tried with this double loop, where I used variables like the ones given in the examples, but much larger. The script is:

     deltaT=zeros(size(TIME(:,1)));
      D=zeros(size(ATOT));
      for k=1:length(ATOT(1,:))
          i=1;
      for j=1:(length(TIME(:,1)))
          if etime(TIME(j,:),TIME(i,:))>=SECONDS_SCAN
              deltaT(j,1)=etime(TIME(j,:),TIME(i,:));
              %R(j,1)=1;
              D(j,k)=nanmean(ATOT(i:j,k));
              i=j+1;
          end
      end
      end

The variables are:

SECONDS_SCAN = (number of seconds)

TIME= 
        2007           4           6          13          50          33
        2007           4           6          13          50          36
        2007           4           6          13          50          38
        2007           4           6          13          50          40
        2007           4           6          13          50          42

ATOT=    
   30.2000   30.2000   29.9000   29.9000   29.9000   29.9000   29.4000
   30.2000   29.9000   29.9000   29.7000   29.7000   29.4000   29.9000
   29.9000   29.9000   29.9000   29.9000   29.7000   29.7000   29.4000
   30.4000   30.2000   29.9000   29.7000   29.4000   29.9000   29.7000
   30.2000   29.9000   29.9000   29.7000   29.7000   29.4000   29.9000

I also would like to reduce the matrix D, because in the final one i have plenty of "0": I'm just able to do it AFTER having done the averaging.

I'm quite new to Matlab, but I need this data processing program, and I don't know what to do to make it faster, because with my large data files, that implicates hours of time to be done.

Thanks

[Sean de Wolski]

I'm not clear on exactly what yuou want. What is SECONDS_SCAN what are the columns of TIME what does ATOT have to do with anything?

There is a likely a very fast and elegant way to do this with histcounts/histc and accumarray, but I'm not clear on what the inputs would be.

[ludvikjahn]

ATOT is the series of data I want the average (by column) every SECONDS_SCAN seconds (it's me who decides it, every 3 seconds, 100 seconds, 10000 seconds, etc.). TIME is a matrix with

   yyyy mm dd HH MM SS

eEvery raw of TIME is referred to the corrisponding ATOT raw. Data of the same type are in the same column.

Is it clear to you now? I want to have the average for every series of data making this average EVERY ("SECONDS_SCAN") SECONDS. Thanks

Answer 1

[Stephen23]

Here is an outline of how you could do this without loops:

1. Convert the date vectors to date numbers using datenum. Note datenum accepts a matrix of date vectors. Call this dtn.
2. Create a vector of date numbers that represent the bin edges of the times that you want to group together. This is where you define how many seconds the intervals have. Call this edg.
3. Use histcounts or histc to count the dtn within the bins defined by edg. Call this cnt.
4. Use mat2cell to split the data into those blocks defined by cnt. Only split along the first dimension, the columns can stay together. Call this spl.
5. Use cellfun and mean to calculate the mean of the data in spl. Note that you must specify mean's second argument, in case there is only one row of data in some cell, which means you will need an anonymous function .
6. Concatenate it all back together into a numeric array. You could use vertcat and cell array expansion to generate a comma separated list.

You should now have a matrix of the mean values, without any loops at all. This should be much faster than calculating the mean in nested loops. It could be implemented like this:

SECONDS_SCAN = 4;
TIME = [...
    2007, 4, 6, 13, 50, 33;...
    2007, 4, 6, 13, 50, 36;...
    2007, 4, 6, 13, 50, 38;...
    2007, 4, 6, 13, 50, 40;...
    2007, 4, 6, 13, 50, 42];
ATOT = [...   
    30.2000, 30.2000, 29.9000, 29.9000, 29.9000, 29.9000, 29.4000;...
    30.2000, 29.9000, 29.9000, 29.7000, 29.7000, 29.4000, 29.9000;...
    29.9000, 29.9000, 29.9000, 29.9000, 29.7000, 29.7000, 29.4000;...
    30.4000, 30.2000, 29.9000, 29.7000, 29.4000, 29.9000, 29.7000;...
    30.2000, 29.9000, 29.9000, 29.7000, 29.7000, 29.4000, 29.9000];

dtn = datenum(TIME)*(24*60*60);
edg = min(dtn):SECONDS_SCAN:SECONDS_SCAN+1+max(dtn);
cnt = histc(dtn,edg);
spl = mat2cell(ATOT,cnt(1:end-1),size(ATOT,2));
out = cellfun(@(m)mean(m,1),spl, 'UniformOutput',false);
out = vertcat(out{:});

out =
       30.2    30.05     29.9     29.8     29.8    29.65    29.65
       30.15   30.05     29.9     29.8     29.55   29.8     29.55
       30.2    29.9      29.9     29.7     29.7    29.4     29.9