How to calculate 8 days mean from ten years data?

I have a code which calculates the 8days mean from single year. It computes 45 eight days from 45X8 = 360 days and last 8days mean from remaining 5 or 6 days of the year. I wish to use this code over ten years data with the same method as mentioned in previous line. It should not jump to next year to complete last 8 days. It should start from the 1st January and finish at 31 December of each year and then keep on doing similar way so till the end of all the data. I request you all to kindly suggest me some checks in this code to run this code over a length of continuous ten years data without overlapping between years. The code is as follows

data_matrix=readmatrix('c:/data/test.csv');
start = 1:8:height(data_matrix);
stop = unique([start(2:end)-1, height(data_matrix)]);
nbins = numel(stop);
dataAverages = nan(nbins, width(data_matrix));
for i = 1:nbins
    rows = start(i) : stop(i);
    dataAverages(i,:) = mean(data_matrix(rows, :), 1, 'omitnan');
end
disp(dataAverages)
size(dataAverages)
% Write output as matrix in csv format
filename = 'c:/data/test_out.csv'
writematrix(dataAverages,filename);

Devendra

[Mathieu NOE]

it would help if uu share some csv files

one csv file per year or you have all data in one single csv file ?

[Cris LaPierre]

Context: https://www.mathworks.com/matlabcentral/answers/1992983-how-to-compute-8-days-mean-from-one-year-data

[Devendra]

I have all the data in one csv file. Right now I am attaching the three year data as one Test.csv file. Thanks a lot for your kind cooperation.

Devendra

[Devendra]

Please ignore the first row and first two columns.

Devendra

[Cris LaPierre]

All the code you have shared has been written by @Adam Danz. Please share what you have tried.

[Mathieu NOE]

the code writen by @Adam Danz still works on the multi years csv file you posted

there is no time limit that would prevent to use that code in its original state

data_matrix=readmatrix('Test.csv');
size(data_matrix) % ans =  1096          11
windowSize = 8;  % 8x1
start = 1:windowSize:height(data_matrix); 
stop = [start(2:end)-1, height(data_matrix)]; 
nbins = numel(stop); 
dataAverages = nan(nbins, width(data_matrix)); 
for i = 1:nbins
    rows = start(i) : stop(i);
    dataAverages(i,:) = mean(data_matrix(rows, :), 1, 'omitnan');
end
size(dataAverages)  % 137    11

what is your issue ?

[Devendra]

Thanks a lot for your kind support. Yes, it is working for all three years data.

But what I wish to do is it must compute 46 eight days mean of nine variables ( column 3 to 11) each year. Therefore, for three years data it must have 46x3=138 data points. But it is generating 137 data points.

Beacuse after 45th eight days mean (45x8=360) only 5 or 6 days are left in each year. So the last 46th eight days mean must be computed from these remaining 5 or 6 days only depending upon normal year or leap year.

Similarly it can keep on doing for all the three years data.

I appreciate your kind cooperation.

Devendra

[Devendra]

I am highly obliged to you and thank you so much for your kind help. I have kept the name of this matlab code as Mathieu_NOE.m because this is from your kind courtesy.

The code is working very fine. But it is writing only one year data (46,9) in csv file.

Once again thanks a lot for your generous help.

Devendra

[Devendra]

I am Sorry for my mistake. It has written all the data(138,9). Please ignore my prevoius post.

Devendra

[Mathieu NOE]

my pleasure !

glad it works

[Mathieu NOE]

I am just seing that I actually didn't post my answer in the answer section

so I am gonna move it , so you could accept it if you want !

[Devendra]

Certainly I will accept it. Devendra

Answer 1

[Mathieu NOE]

ok so we have to do the processing for each year and pay attention to the remaining (variable length) days

NB that as we now have access to each year results , you can decide wheteher you want to store them in a separate file

for the time being it's doing a vertical concatenation of all results

to load the data I used this fex submission

readlog - File Exchange - MATLAB Central (mathworks.com)

which gave me a fast loading , but of course there are plenty of alternatives if you have another preference

i also tested the case where the recording would stop randomly before a 31 december

for the provided file Test.csv, I have 138 data points as expected

%% read data file
[lines, ~] = readlog('Test2.csv'); % fex : https://fr.mathworks.com/matlabcentral/fileexchange/96364-readlog?s_tid=srchtitle
data_matrix = split(lines,',');
[m,n] = size(data_matrix);
ind_start_year = find(startsWith(data_matrix(:,1),'1-Jan-')); % find row corresponding to first day of the year
ind_stop_year = find(startsWith(data_matrix(:,1),'31-Dec-')); % find row corresponding to last day of the year
toc

%% main loop
windowSize = 8;  % define buffer length for averaging
cols = 3:n; % define cols to be processed
dataAverages_total = [];

% if the data file stops before a 31 Dec , we must add this : 
if numel(ind_stop_year) < numel(ind_start_year)
    ind_stop_year = [ind_stop_year; m];
end

for ck = 1:numel(ind_start_year)% loop over year number
    ind_start_thisyear = ind_start_year(ck);
    ind_stop_thisyear = ind_stop_year(ck);
    start = ind_start_thisyear:windowSize:ind_stop_thisyear; 
    stop = [start(2:end)-1, ind_stop_thisyear]; 
    dur = stop-start+1;
    nbins = numel(start); 
    dataAverages = nan(nbins, numel(cols)); 
    for ci = 1:nbins
        rows = start(ci) : stop(ci);
        tmp = data_matrix(rows, :);
        dataAverages(ci,:) = mean(str2double(tmp(:,3:n)), 1, 'omitnan'); % results of one year
    end
    sum(dur)
    dataAverages_total = [dataAverages_total ; dataAverages]; % concatenate every year's result (dataAverages) 
end

Answer 2

[Mathieu NOE]

you can use the existing standard readlines function instead of readlog as shown above

lines = readlines('Test.csv','EmptyLineRule','skip'); 

but I had to add 'EmptyLineRule','skip' to avoid empty (last) line being loaded

[Devendra]

I am using one Random Forest Regression matlab code. This code is giving table variables related errors. I am attaching the code and input file. I request you to kind have a look on it and suggest me how to fix it.

I would appreciat your kind help.

Devendra