Averaging each .STD files

Greetings. I have been pondering and working hard on this issue since I came here to ask for solutions. I have over 100 .std files in which each of them has 10 headers and each header has 24 hours data of which each hour has 60 values. I want to loop through the hundred data files and get the average of the 24hrs data with graphs. I was only able to do for each data at a time and it's very tedious.

[Steve Eddins]

If you can provide more information about the content and structure of your files, it might help people come up with answers for you. It would be ideal if you could attach a couple of sample data files.

Based on your description so far, I would probably be looking to create a timetable containing all the data and then use the retime function to get 24-hour averages.

[Moses Joseph]

I have sent the attached files. I would be grateful if I can get it done.

Thank you.

Answer 1

[Steve Eddins]

[Update: Based on comments from Moses, I submitted another answer to this question. See below.]

Try this:

files = dir("*.Std");
N = numel(files);
Filename = strings(N,1);
Date = NaT(N,1);
Mean1 = zeros(N,1);
Mean2 = zeros(N,1);
Mean3 = zeros(N,1);
Mean4 = zeros(N,1);

for k = 1:numel(files)
    name = string(files(k).name);
    A = readmatrix(name,"FileType","text");
    
    [~,base_name,~] = fileparts(name);
    base_name_parts = split(base_name,"-");
    
    Filename(k) = name;
    Date(k) = datetime(join(base_name_parts(2:4),"-"));
    means = mean(A,1);
    Mean1(k) = means(1);
    Mean2(k) = means(2);
    Mean3(k) = means(3);
    Mean4(k) = means(4);
end

T = timetable(Date,Filename,Mean1,Mean2,Mean3,Mean4);

When I run this with your data files, T is this:

  8×5 timetable

       Date                Filename            Mean1     Mean2     Mean3     Mean4
    ___________    ________________________    ______    ______    ______    _____

    01-Jan-2015    "bjco001-2015-01-01.Std"    11.992     24.82    2.6294    6.38 
    02-Jan-2015    "bjco002-2015-01-02.Std"    11.992     22.25    2.1003    6.38 
    03-Jan-2015    "bjco003-2015-01-03.Std"    11.992    25.834    1.9662    6.38 
    04-Jan-2015    "bjco004-2015-01-04.Std"    11.992    32.088    1.6014    6.38 
    05-Jan-2015    "bjco005-2015-01-05.Std"    11.992     25.03    1.8935    6.38 
    06-Jan-2015    "bjco006-2015-01-06.Std"    11.992    31.423    1.5822    6.38 
    07-Jan-2015    "bjco007-2015-01-07.Std"    11.992    31.528    1.8784    6.38 
    08-Jan-2015    "bjco008-2015-01-08.Std"    11.992     31.08    1.6912    6.38 

    ...

Plot sample:

    plot(T.Date,T.Mean2)

[Moses Joseph]

In short, I am short of words for this kind of response from you Sir. I am really grateful Sir for your valuable help only that it hasn't answered my question. If you noticed let's say for instance, in bjco001-2015-01-01.std, we have 4 columns and each column has 0-23 of which 0 has almost 60 values ie, 0.00-0.93 which represent a day value. Another day value starts with 1.00 till we no longer see 1 then continue from 2 again till we get to 23.

So for the four columns, I need the average value of 0.00-0.93, 1.000-1.983, 2.000-2.983 till 23.000-23.983.

Which means, for the first file, I should have 4 columns each column having 24 rows values. So I want to do this for the whole files instantaneously without having doing them one file at a time.

D = load('bjco001-2015-01-01.txt'); M = ceil(D(:,1)+1E-4); HrMeans = accumarray(M, (1:numel(M)).', [], @(x){mean(D(x,2:4))}); Result = [(0:23).' cell2mat(HrMeans)]

figure plot(Result(:,1), Result(:,2)) grid xlabel('Time') ylabel('VTEC')

The above was the code I got from here but it only worked for one file at a time and I have to convert the STD file to txt file before I could use it.

It's so stressful using it but I still appreciated it.

So I need the one that can loop through the who STD files I sent to you and give give me 4 columns with 24 rows data for each file with a graph for each. I have tried many possibilities before coming to ask online here but no result.

Please to use the above code I sent, it can only work after converting the file to txt which I believe you know it sir.

I would be indebted to you if this is achieved here. Nothing is impossible and that's why I am here.

Thank you in anticipation for a positive result.

Answer 2

[Moses Joseph]

Attached are the files but compressed

Answer 3

[Moses Joseph]

Attached are the files in zip format

Answer 4

[Steve Eddins]

Try this:

files = dir("*.Std");
N = numel(files);
Date = NaT(0,1);
X = zeros(0,3);

for k = 1:numel(files)
    name = string(files(k).name);
    A = readmatrix(name,"FileType","text");
    
    [~,base_name,~] = fileparts(name);
    base_name_parts = split(base_name,"-");
    
    Date = [Date ; datetime(join(base_name_parts(2:4),"-")) + (A(:,1)/24)];
    X = [X ; A(:,2:4)];
end

T = timetable(Date,X(:,1),X(:,2),X(:,3));
T = sortrows(T);
head(T)

ans =

  8×3 timetable

            Date            Var1     Var2    Var3
    ____________________    _____    ____    ____

    01-Jan-2015 00:00:00    12.72    2.69    6.38
    01-Jan-2015 00:01:01    12.69    2.47    6.38
    01-Jan-2015 00:01:58    12.66    2.54    6.38
    01-Jan-2015 00:03:00    12.63    2.86    6.38
    01-Jan-2015 00:04:01     12.6    2.39    6.38
    01-Jan-2015 00:04:58    12.57    2.52    6.38
    01-Jan-2015 00:06:00    12.53    2.81    6.38
    01-Jan-2015 00:07:01    12.49    2.65    6.38
    

T2 = retime(T,'hourly','mean');
head(T2)

ans =

  8×3 timetable

            Date             Var1      Var2      Var3
    ____________________    ______    _______    ____

    01-Jan-2015 00:00:00     10.85     2.0272    6.38
    01-Jan-2015 01:00:00    8.4937     1.8062    6.38
    01-Jan-2015 02:00:00     7.681     1.1152    6.38
    01-Jan-2015 03:00:00    6.0948      1.016    6.38
    01-Jan-2015 04:00:00    3.7198     1.0692    6.38
    01-Jan-2015 05:00:00     3.066     1.1472    6.38
    01-Jan-2015 06:00:00    8.8583    0.97183    6.38
    01-Jan-2015 07:00:00    18.533     1.1425    6.38

[Moses Joseph]

Can't say how grateful I am Sir. Thank you a whole lot.

Please I am getting this error "Index exceeds array bounds"

[Moses Joseph]

What should I have adjusted to make it work?

[Steve Eddins]

Did you copy-and-paste the code here directly into a code file in your MATLAB editor, or did you retype it manually? If you retyped it, then I suspect a typing mistake.

Include the complete error message here, including the line of code where the error appeared.

For troubleshooting a program, I highly suggest using the MATLAB Debugger. To get started, set a breakpoint on the first line of the file, run the code, and then single-step through each code line. Look at the values that get assigned at each line and make sure they are what you expect.

[Moses Joseph]

Yes, I copy pasted the code.

Let me try what you asked me to do then I revert

[Moses Joseph]

I tried using the code one after the other, I did not get any error until I added

Date = [Date ; datetime(join(base_name_parts(2:4),"-")) + (A(:,1)/24)]; Then got index exceeds array bounds

[Steve Eddins]

To troubleshoot this, you need to examine the sizes and values of each variable at each step to determine where something is going wrong. Here, the variables base_name_parts and A being indexed. Do they have the expected size here? If not, why not? For example, if some of your filenames have a different form than the set you attached, then maybe base_name_parts isn't always a 4-element string vector.

Or maybe you'll find that A is empty just before the error occurs, which would suggest that the call to readmatrix failed for some reason, which in turn might suggest that one or more files in your dataset is malformed.

[Moses Joseph]

Before adding this line Date = [Date ; datetime(join(base_name_parts(2:4),"-")) + (A(:,1)/24)];

I have A=1440bx4 double base_name= "" base_name_parts="" Date=0x1 datetime files=313 N=313 name="bjco0-2015_12_31 X={}

And I didn't receive any error

But the moment I added it, I got index exceeds array bounds. I am still wondering why it worked there and won't work here.

I am using R2019b.

Even I tried the previous code you sent, I got same error message. Could it be system date or what? Still racking my little brain sir.

I must confess, you have really tried for me and I know many folks like me will get this result helpful if after aLL we find solution to it.

[Moses Joseph]

I am still a baby in MATLAB may be that is why I am finding it a bit difficult to solve the issue cause I can't see reason why it's working well for you and not me.

You just have to help me completely sir. I value you. If there is anything you can put through to do, I don't mind sir.

Thank you.

[Moses Joseph]

Index exceeds array bounds.

Error in Untitled2 (line 61) Date = [Date ; datetime(join(base_name_parts(2:4),"-")) + (A(:,1)/24)];

Explanation The size of the indicated variable or array appears to be changing with each loop iteration. Commonly, this message appears because an array is growing by assignment or concatenation. Growing an array by assignment or concatenation can be expensive. For large arrays, MATLAB must allocate a new block of memory and copy the older array contents to the new array as it makes each assignment. Programs that change a variable's size in this way can spend most of their run time in this inefficient activity. For the same reasons, there is significant overhead in shrinking an array or in changing the size of a variable on each iteration. Suggested Action Consider preallocating a variable or array before entering the loop by using zeros, ones, cell, or a similar function. Preallocating avoids the need for MATLAB to copy the data from one array to another inside the loop. For examples of code that do and do not preallocate an array, see “Preallocating Arrays”. When an array grows by assigning past the end of the array or by using concatenation, preallocation alone does not improve the performance. The code must also use explicit indices. If you do not know the size of an array before the loop begins, preallocate it, and then if necessary, shrink it after the loop completes. If any of the following conditions are true, it might be appropriate to suppress this message, as described in Adjust Code Analyzer Message Indicators and Messages: The loop code contains a conditional block where the array grows. (In this case, it can be reasonable to grow the array only as the loop finds such conditions.) For each iteration in the loop, the length of the concatenation varies (such as character vectors with variable length). The total size of the array is not calculable before entering the loop. The array is small and will remain small; therefore, the impact of recopying is also small. The code preallocates the array, but in a way that Code Analyzer does not recognize.

I got the above.

I tried taking shredding the files by taking the first 10 files to get where the error is from but could not figure it out

[Moses Joseph]

Very sorry sir. I have gotten the values you got earlier on. I have 4 columns with each having 8 rows that's for 1 file out of 313 files.

But I should have 4 columns with each having 24 rows for each file.

That means, I should have 313 files having 4 columns with 24 rows each.

The code I sent earlier on did the job but it's for one file and I need it to loop for the whole files and give me tables like the one you will get if you use the code I sent to you.

This one you did worked but gave me 8 rows for only 1 file instead of 4 columns by 24 rows each for 313 files so I should have 313 tables of 4columns by 24 rows.

Thank you sir. Still waiting

[Moses Joseph]

Sir, you are genius.

Thank you a whole lot.

I have been able to figure it out. It worked fine for me at least.

I love you Sir and I salute MATLAB for this great platform.

Thank you very much.

Hope I can still come here and ask more questions to get answers just like today. Please?

I am deeply grateful Sir.

I value you and GOD bless you real good for me. If I have issue on this again or any other ones, hope you will answer me?

[Steve Eddins]

I'm glad you were able to figure it out!

[Moses Joseph]

Kudos sir 👍

[Moses Joseph]

Greetings Sir.

Once again thank you.

If you wouldn't mind, I have 2 questions to ask from you as regards this work.

1. Can you please explain each line of this codes as to a lame man cause is of no use of I can't explain well to people or understand its functionalities. Differentiate which is MATLAB inbuilt functions/variables and personal functions/variables.

2. The table of results gotten from this computation, can you help in splitting it into days and hours for each month which means 31days(31 rows) and 24 hrs(24 columns) for each month and calculating the mean, media, standard deviation, variation for each of the hour or column?

If this can be automated using MATLAB, I will so much appreciate. I have computed the one you did for me manually for one year using Excel spreadsheet but man, it's too much work for me going for 5 years data. Copying and pasting which is error prone.

Hope I am not asking for too much sir? I believe the solution will be of help to many people doing this kind of project.

Happily waiting your positive response sir.

Thank you