how to fill a time series from burst sampled data?

I have data recorded in ~3 minute bursts at a frequency of 4hz and taken every ~13 minutes. On plotting, therefore, a line is created connecting each of these bursts [see image attached]. To remove this and also to make a complete timeseries with time intervals running complete from the start to end [rather than a table with only the times at which data was recorded], I figured I could create a new table with a complete timeseries with appropriate time interval, merge it with the 'burst time series' and then pack the missing data with NaNs.

I have created the new timeseries

dtn = datenum(['9/11/2016 08:29:11:000';'9/11/2016 17:02:09:000'],'dd/mm/yyyy HH:MM:SS.FFF'); 
dts = datestr(t(1):1/24/60/60/4:t(2),'dd/mm/yyyy HH:MM:SS'); %create [char] time series 08:29:11-17:02:09 for 9/11/2016 
dts=datetime(dts,'InputFormat','dd/MM/yyyy HH:mm:ss');

and with a table of both sets of data, thought I could then use a 'join' to merge them inserting data at appropraite times into 'dts' with gaps to be subsequently filled by NaNs at other times, but haven't had any success in this.

A=table(vpdtime,vppres,...
    'VariableNames',{'Time' 'Pressure'});
B=table(dts,...
    'VariableNames',{'Time'});

[Peter Perkins]

I think the three lines you have to construct a datetime vector would be more succintly written as

dts = datetime('9/11/2016 08:29:11'):seconds(.25):datetime('9/11/2016 17:02:09');

Generally speaking, for both performance and clarity, it's best to avoid mixing datenum/datestr and datetime.

[Dom Smith]

Thanks, I do manage to get endlessly confused on converting times!

Answer 1

[Peter Perkins]

Dom, since it appears you have R2016b, here's what I came up with using a timetable:

>> % t0 = datetime(2016,9,11,8,29,11,'Format','yyyy-MM-dd HH:mm:ss.SSS');

Could have done it with datetimes, like the above, but since it's all in one day, maybe a duration makes more sense?

>> t0 = duration(8,29,11,'Format','hh:mm:ss.SSS');
>> burst = (0:seconds(.25):minutes(3))';
>> time = [t0 + burst; t0 + minutes(13) + burst; t0 + minutes(26) + burst];
>> signal = seconds(time - t0) + 100*randn(size(time)); % fake data
>> t1 = time(end);
>> tt = timetable(time,signal);

>> tt(1:1000,:)
tt =
        time         signal 
    ____________    ________
    08:29:11.000      57.136
    08:29:11.250     -50.832
    08:29:11.500     -33.331
[snip]
    08:32:10.500      545.13
    08:32:10.750      5.3862
    08:32:11.000      352.54
    08:42:11.000      764.23
    08:42:11.250      657.03
    08:42:11.500      740.39
    08:42:11.750      888.41
[snip]

and so on. This

plot(tt.time,tt.signal)

creates the same plot you already made, with lines that span the empty spaces. As you say, NaNs will prevent that in the plot. I'm not sure what you would then do with all those NaNs, but I think this is what you are asking for:

tt2 = retime(tt,[t0:seconds(.25):t1]);

So now this

plot(tt2.time,tt2.signal)

will show only the measured data. It seems like maybe you'd then do some kind of smoothing/interpolation to fill in the gaps that are now filled with NaNs.

Hope this helps.

[Dom Smith]

Cheers! With a bit of tweaking to fit, I think this is what I was looking for.

Answer 2

[Walter Roberson]

Have you considered using a timetable() ?

[Dom Smith]

I have yeh, I can do it using synchronise making A and B a timetable and then synchronising to new 'secondly' timestep... but then I loose some of the data's resolution.

C=synchronize(Att,Btt,'secondly');

Is there any way of synchronising to timesteps smaller than a second with the new timetable function?

[Walter Roberson]

Instead of 'secondly' pass in a vector of the times to synchronize against.