fitlm, how to use two datasets with identical size and timestamps

Hi

I have two data sets. Each dataset has the sensor values and a timestamp.

I would like to use fitlm to determin a linear function between both these sets of data, but the data used in this function should only be data that has timestamp in both two data sets, to that when data is being compared it at the same timestamp.

The issue is that a certain periods the dataset might have some data samples missing a certain time stamps.

This is my current code.

            ref_timetable_time=[ref_data_timestamp]';
            ref_timetable_ug=[ref_data_sensordata]';           
            ref_timetable = timetable(ref_timetable_time,ref_timetable_ug);
            timerperiod_Ref=ref_timetable(timerperiod,:);

            timetable_time=[live_t]';
            timetable_ug=[no2_ppb];           
            timetable_ug = timetable(timetable_time,timetable_ug);
            timerperiod_ug=timetable_ug(timerperiod,:);
            hourlyaveraged_no2_data = retime(timerperiod_ug,'hourly','mean');                       
            %model sensor data to ref data
            inModels{k} = fitlm( timerperiod_Ref.ref_timetable_ug, hourlyaveraged_no2_data.timetable_ug, "linear", "VarNames", ["Ref Data", "SensorValue"] );

What can be done so that dataset are equal in both size and timestamps?

Thank You

[Image Analyst]

You forgot to attach the data files.

See retime or interp1

If you have any more questions, then attach your data and code to read it in with the paperclip icon after you read this:

TUTORIAL: How to ask a question (on Answers) and get a fast answer

[Dharmesh Joshi]

Hi

The datafiles, are very large, so i did not attach it.

I will read the two links provided.

[Dharmesh Joshi]

I have tried retime, but it does not add missing data points.

Please see my plot below.

The blue is ref data, and orange is sensor data.The sensor data, has some missing points, which need to be filled before i can compare data.ata and the organge is sensor data

[dpb]

retime will certainly "fill in" and synchronize two timeseries(*), but I see no reasonable way to create missing data prior to the beginning of the sensor time trace back to the time of the origin of the other trace--there isn't anything to go on to even guess what that might have been with the erratic nature of what data are there. I'd venture the only sensible option here is to begin both observations at the innitial time of the sensor trace and go on from there.

But, that done, I've no idea of what you must be thinking with the stated objective of "use fitlm to determine a linear function between both these sets of data" since there really appears to be almost no correlation whatever between the two traces. What model are you propoosing and what purpose would this fit be supposed to serve?

(*) There appears to be/may be a problem in the sensor data around Aug 21; the plot above shows a break in the line. That would indicate a NaN or Inf in either the data or the date vector -- plot() will draw a straight line between points if there were just a gap in time, but either variable being ~isfinite() will result in those points not being drawn; hence the apparent break. That would also produce NaN in the output of retime for those locations if don't use one of the options to handle missing data, explicitly.

[Dharmesh Joshi]

Thanks

The data is correlated, but requires some correction due to tempeture, so expirmenting on a temperture correction factor.

Yes, the sensor was switched off during certain points, therefore that data is not avilable.

[dpb]

"The data [are] correlated, but requires some correction due to tempe[ra]ture, ..."

Well, good luck with that from the appearance that's going to take some mighty serious "some" and need to be a very complex temperature gradient to make those two traces have any semblance of even an approximate resemblance to each other.

As for the other missing data in the sensor; you really can only interpolate to fill those missing positions -- at that location in the recovering rise, at least, it doesn't appear that just a linear approximation there would be that far out of line with what the previous recoveries appear to be...

Be most interesting to see what it would look like if you were to post the same figure as above but add the temperature as another trace -- this would appear a good place for yyaxis right

[the cyclist]

If you are just trying to identify points in the two datasets that share exact timestamps, then you may be able to use the ismember function or the intersect function to trim the data to the shared timestamps.

[KSSV]

To find the missing values, you can use either interp1 or fillmissing.

[Dharmesh Joshi]

Thanks