How to detect whether 72 hours have passed after the occurrence of a certain event? - datenum gives error for longer time series.

I am working on a time series which consists of sensor outputs at various date and time. I need to find out whether the sensor output is 72 hours apart from a certain event. I can save the date and time of event occurrence for comparison purpose. The time series I am working upon has time given as - 8/19/2016 9:41:49 AM , or, 8/13/2016 12:00:51 PM - format.

If I use datenum function, it fails as the length of time series is more then 600000 readings. So, I would also like to know the maximum no. of dates and time datenum can handle.

[dpb]

"... datenum ... fails as the length of time series is more then 600000 readings."

Need amplification here...fails how, precisely?

>> d=datenum(2016,8,13,12,0,51+[0:600000].');
>> datestr(d(end))
ans =
20-Aug-2016 10:40:51
>> size(d)
ans =
    600001           1
>> 600000/86400
ans =
  6.9444
>> d(end)-d(1)
ans =
  6.9444
>> dt=datenum(0,0,0,72,0,0)
dt =
   3
>>

datenum has no problem with 600,000 values here; now if you've got an array of some large number of columns besides the 600K rows, you may have a memory problem essentially unrelated to datenum

As the last shows, 72 hr is just 3 days so any difference of 3 in the date number is the interval. Now, if the data are sampled randomly, you may not have many that are precisely 3 days apart to the second is another issue.

In short, we need more info.

NB: In recent releases, there's a new datetime class that may make the differential time stuff a little easier than with "classic" datenums, but I don't have it here to illustrate.

[na ja]

I used the datenum function on a dataset consisting on 456681 timing values to plot those values, and I get following error.

Error using datenum (line 178)
DATENUM failed.

Error in work_tag2 (line 18)
dates = datenum (z1.tagtime,'mm/dd/yyyy HH:MM');

Caused by:
    Error using dtstr2dtnummx
    Failed on converting date string to date number.

Now if I use the same function on smaller dataset, say having range of 10000 timing values, it gives output correctly.

If I get precision up to the level of hours and minutes, that would be fine! I am trying to implement something where, if, current time is 72 hours more than previously noted time, certain action should be taken.

[dpb]

The error isn't the length of the input, it's that there's a value in the larger array that's throwing an error; probably by not being in the correct format. Either winnow down the dataset in question by something like binary search to isolate the problem record or set a breakpoint and use the debugger to check what the input is when the error occurs.

I'm virtually positive it isn't the number of elements, however; as shown, datenum worked just fine on a much larger array here. BUT, I can reproduce the error easily here as well in one of several ways--each having to do with the format of the date string vis a vis the input datestring variable--

>> datenum(d0,'mm.dd.yyyy HH:MM:SS AM')  % periods between m/d/y
Error using datenum (line 179)
DATENUM failed.
Caused by:
  Error using dtstr2dtnummx
  Failed on converting date string to date number. 
>> d0 = '8/19/2016 9:41:49';  % no trailing AM/PM string
>> datenum(d0,'mm/dd/yyyy HH:MM:SS AM')
Error using datenum (line 179)
DATENUM failed.
Caused by:
  Error using dtstr2dtnummx
  Failed on converting date string to date number. 
>>

Any such mismatch between the input for any element in the array will cause the error before the function call completes.

[na ja]

Thanks dbp, I rechecked the timing array. There was format change in few of the incoming values. When I was taking smaller array for troubleshooting, they were not the part of data, so I didn't get error. You are correct. So, I modified accordingly.

I think I can use datevec function to verify whether three days have passed or not.

[dpb]

OK, that's what I expected. datenum isn't particularly user-friendly in reporting it's errors; it could be a reasonable enhancement request to ask that it be improved to at least return the string it was trying to convert when it croaked and better would be also the location in the input array.

As for the followup comment, datevec doesn't seem the simplest way to tell about elapsed time periods, no. As I noted in first response simply looking for the difference of >=3 in the two datenums is more than likely the simplest technique. You've not given enough background on the actual requirements to be able to suggest a specific syntax or technique; as noted the question of whether this is for exact time match is one issue; we don't know the purpose nor the manner in which the timestamps were taken such as to know what are the issues needing addressed.

[na ja]

Hi dpb, the date format used in the series I am working upon is 'mm/dd/yyyy HH:MM:SS AM'. So, I am planning to use datevec in following manner.

1. Store past date and time

2. apply datevec on it and get time in vector [y, m, d, h, mn, s].

3. get the same vector for future time values.

4. if future date datevec(3) - past date datevec(3) == 3, consider 3 days have passed till the past date.

5. now future date becomes past date and is get compared to newer timing values.

I don't want precision in terms of hour, just want to get an idea whether three days have passed.

Now for datemun function, suppose I have a date and time '8/13/2016 12:00:51 PM'. How can I infer whether three days have passed by comparing its datenum output with following time values?

Answer 1

[dpb]

"if future date datevec(3) - past date datevec(3) == 3, consider 3 days have passed till the past date."

Except what if futureDate is 11/01 and pastDate is 10/30? Then

>> d2='11/01/2016'; d1='10/30/2016';
>> dv2=datevec(d2,'mm/dd/yyyy');
>> dv1=datevec(d1,'mm/dd/yyyy');
>> is3Day=(dv2(3)-dv1(3))>3
is3Day =
   0
>>

Looks ok, but now if the new date is 11/10 that is clearly over 3 days, we get...

>> d2='11/10/2016';
>> dv2=datevec(d2,'mm/dd/yyyy');
>> is3Day=(dv2(3)-dv1(3))>3
is3Day =
     0
>> >> d2='11/10/2016';
>> dv2=datevec(d2,'mm/dd/yyyy');
>> is3Day=(dv2(3)-dv1(3))>3
is3Day =
     0
>>

Houston, we have a problem! And, can't solve it by abs as in

   >> is3Day=abs(dv2(3)-dv1(3))>3
is3Day =
     1
>>

because then the first case would then be wrong. Using datevec the problem can only be solved by a lot more logic to keep track of month rollover, which then entails keeping track of days per month which then entails leap year and on and on and on...

OTOH,

>> d2='11/01/2016';   % refer back to original case again...
>> dn2=datenum(d2,'mm/dd/yyyy');
>> is3Day=(dn2-dn1)>3
is3Day =
   0
>> d2='11/10/2016';   % and back to second case...
>> dn2=datenum(d2,'mm/dd/yyyy');
>> is3Day=(dn2-dn1)>3
is3Day =
   1
>>

Now both work "out of the box" and all that other stuff is taken care of for you automagically.

It still isn't totally clear to me about what your intent is regarding partial days; is it required that at least a full three, 24-hour days have passed since the start time or is it that it's any time in the third or greater day that is ok? The latter is, I presume, the reason for the question regarding the example start time given above. If that is the case, the answer is also essentially trivial, just take integer portion of the start date datenum; as I think I noted before, datenums are expressed in days and fractions thereof; whole numbers are days and the fraction is the partial portion of a day within the current one. So, your dn1 for the above date would become

>> d1= '8/13/2016 12:00:51 PM';
>> dn1=fix(datenum(d1,'mm/dd/yyyy HH:MM:SS AM'));  % start at midnight for counting
>> datestr(dn1)                                    % show is what think it is...
ans =
13-Aug-2016
>>

And, of course, doing it this way you can write the function isNday to return a logical vector of those dates which are >=N (or > exclusively, or ==(*) if that's desired) all at once simply by looking at

function is=isNday(dates,N)
% returns True for locations >N days apart for datenum vector

    is=[false diff(dates)>N];

irrespective of any particular start date. You can make it specific by passing in the dates vector beginning with the specific date. is=[false diff(dn-dn0)>=N];

(*) NB: depending upon the sampling rate that while one presumes there's at least one sample per day, this will make for as many instances of there being a TRUE value as there are samples in a day as samples within a day are then indistinguishable.

Answer 2

[Peter Perkins]

Unless you're using a version of MATLAB older than R2014b, you'll be much better off using datetimes:

>> times = datetime(2016,10,1:5)'
times = 
  5×1 datetime array
   01-Oct-2016
   02-Oct-2016
   03-Oct-2016
   04-Oct-2016
   05-Oct-2016
>> eventTime = datetime(2016,10,1)
eventTime = 
  datetime
   01-Oct-2016
>> times - eventTime
ans = 
  5×1 duration array
   00:00:00
   24:00:00
   48:00:00
   72:00:00
   96:00:00
>> times - eventTime == hours(72)
ans =
  5×1 logical array
   0
   0
   0
   1
   0

Hope this helps.

[dpb]

Thx for jumping in Peter; I'd've suggested the datetime excepting don't have a release here that incorporates it and figured as soon did then OP'd ask for specific code...