How can I detect gaps in a time series matrix and insert NaN's

Hi, I have a time series matrix with the date and time in separate columns and I need to find the gaps and fill them in with NaN's. By looking at other suggestions through the File Exchange, I figured out how to find the gaps using:

sp=1; %sampling period is every hour

t=continuouswind3(:,3);

idx=find(diff(t)>(2*sp))'+1; %to detect gaps greater than twice the sampling period

Since my date and time stamps are in individual columns [YYYY MM DD hh mm] I have to first find the missing minutes (10 min time stamp), then hour (1 hour), then days (1 day). From running the script several times changing the columns, it appears that I am not missing any months. I now need to fill in the missing data in columns 6 through 10 with NaN's and the corresponding missing time stamps in columns 1 through 5. I am working with wind data measured once every 10 min from January 1 2006 through December 31 2014. Thank you!!!!

[Image Analyst]

Attach a .mat file with continuouswind in it, and a screenshot of your plot.

[Janet Reimer]

Here is the .mat file

Answer 1

[Image Analyst]

To replace the indexes specified by idx with NANs, do this;

t(idx) = nan;

[Janet Reimer]

The index only gives me where the gap starts. In most cases the gap is much longer. Would you recommend a For cycle?

Answer 2

[David Young]

I think this will do what you want:

% test data
data = [2015 01 20 01 10   1 2 3 4 5; ...
        2015 01 20 01 20   1 2 3 4 5; ...  20 min gap
        2015 01 20 01 40   1 2 3 4 5; ...  
        2015 01 20 01 50   1 2 3 4 5; ...  1hr 10 min gap
        2015 01 20 03 00   1 2 3 4 5; ...  30 min gap
        2015 01 20 03 30   1 2 3 4 5];

% parameter in days
timestep = 10 / (24 * 60);     % 10 minute increment

% convert time array to simple vector of datenumbers
timevecs = [data(:, 1:5) zeros(size(data,1), 1)];
timestamps = datenum(timevecs);

% Round to get index into output array. This assumes that all the times in
% the data are close to multiples of the timestep after the first one. It
% would be easy to check the assumption at this point if there is any
% doubt.
indexes = 1 + round((timestamps - timestamps(1))/timestep);

% Create output array correct size
nfull = indexes(end);
fulldata = NaN(nfull, size(data,2));

% populate the first 5 columns with the new dates
fulltimestamps = timestamps(1) + timestep * (0:nfull-1);
fulltimevecs = datevec(fulltimestamps);
fulldata(:, 1:5) = fulltimevecs(:, 1:5);

% populate the last columns with the original data
fulldata(indexes, 6:end) = data(:, 6:end);

[Janet Reimer]

Thank you! That did it. Reading the code it all seems so simple now.

[Alexandre Canitano]

Great code, thanks for sharing.