Data Source for downloaded data from an HTTP API advice

Hi

I am downloading data from any HTTP API. As shown in code below:

for i = 1:datelistsize(2)
disp([datelist.start(i),datelist.stop(i)]);
live_data_downloaded_data_temp =device_data(deviceid,datelist.start(i),''+datelist.stop(i));                
live_data_downloaded_data =[live_data_downloaded_data live_data_downloaded_data_temp'];                
end    
 save("deviceid_"+deviceid,'live_data_downloaded_data','ref_data_timestamp','ref_data_sensordata')

As time progresses i do not want to be re downloading data that i alread have.

If i am correct i could use datasource.

I would need to store the last times stamp of my data, then what i aim to do is on every instance i call my code, i would save "live_data_downloaded_data" into a new .mat file. Then when i need to read the data, i would have multiple files.

Does anyone have example how i would need to modify my code, so a data source can acklowedge it?

Answer 1

[Adithya]

To implement a data source that acknowledges previously downloaded data and avoids re-downloading it, you can use the concept of "checkpoints". It allows you to save the state of your data retrieval process, so you can resume from where you left off instead of starting from the beginning. Here's an example of how you can modify your code to incorporate checkpoints:

deviceid = 'your_device_id';

checkpointFile = 'checkpoint.mat'; % File to store the checkpoint data

% Load previous checkpoint if it exists

if exist(checkpointFile, 'file')

load(checkpointFile);

else

checkpoint = 1; % Starting checkpoint (first data point)

end

for i = checkpoint:datelistsize(2)

disp([datelist.start(i), datelist.stop(i)]);

% Download data only if it hasn't been previously downloaded

if i >= checkpoint

live_data_downloaded_data_temp = device_data(deviceid, datelist.start(i), ''+datelist.stop(i));

live_data_downloaded_data = [live_data_downloaded_data live_data_downloaded_data_temp'];

end

checkpoint = i + 1; % Update the checkpoint to the next iteration

% Save the checkpoint data

save(checkpointFile, 'checkpoint');

end

% Save the final downloaded data

save("deviceid_" + deviceid, 'live_data_downloaded_data', 'ref_data_timestamp', 'ref_data_sensordata');

In this modified code, a "checkpoint" is introduced to keep track of the last successfully downloaded data point. The checkpoint value is stored in a "checkpoint" file (checkpoint.mat) to persist the state between different instances of running the code.

When you run the code, it checks if a "checkpoint" file exists. If it does, it loads the checkpoint value, indicating the last successful iteration. The code then continues from that checkpoint, downloading the remaining data points.

The downloaded data is only fetched if the current iteration is at or beyond the checkpoint value. This ensures that previously downloaded data is not re-downloaded.

After each iteration, the "checkpoint" value is updated to the next iteration, and the checkpoint file is saved to store the progress.

Finally, the final downloaded data is saved along with any other required data.

By using this approach, you can resume data retrieval from the last successful iteration and avoid redundant downloads in subsequent runs of the code.