Working with grouped data - data access and analysis (standard deviation)

I have a relatively large dataset in which I need to group data based on the cycle of reading, and calculate the median for these cycles. This worked so far. Now I need to compare a value from each member in each cycle with the median for that cycle within some standard deviation range. This should be recorded for some sort of timeseries analysis.

However, I am not able to find a proper solution with the matlab so far. It would be great if someone could help me with this.

so, to recap, - group by cycle - calculate median per group - compare each member valu of the cycle (same members in every cycle) to the group median - save results for each cycle/ member pair in timeseries?

Thanks

Answer 1

[Tom Lane]

I can get you partway there. Here is code to generate some cycle values, generate random data, compute the medians for each cycle, and subtract the cycle medians from the data:

cycle = sort(randi(5,30,1));
a = cycle+randn(size(cycle))/5;
meds = grpstats(a,cycle,@median)
a-meds(cycle)

I don't understand how you intend to convert this to a timeseries, but perhaps you can figure that out. Also, if your cycle values are not consecutive integers starting from 1, you may find it helpful to use the grp2idx function to generate group numbers.

[Valentina]

Thanks, Tom. This helps with the grouping part.

I'll see about the time series thing.

Answer 2

[Lola Davidson]

For those stumbling on this more recently, you can keep all your data together in a timetable and compute the grouped calculations using grouptransform, introduced in R2018b.

% generate some random timestamped data and collect it in a timetable
cycle = sort(randi(5,30,1));
a = cycle+randn(size(cycle))/5;
t = timetable(hours(1:30)',cycle,value);
% use grouptransform to add a column for the desired calculation. Here we
% subtract off the median of the group from each group member
t = grouptransform(t,"cycle",@(x)x-median(x),"value",ReplaceValues=false)

It might also be helpful to note that grouptransform has a handful of built-in methods for common calculations. For example you can normalize the data in each cycle to have mean 0 and std 1 using z-score:

t = grouptransform(t,"cycle","zscore","value",ReplaceValues=false)