Dealing with large timetables

I generated two timetable [283580428x5] and [21317257x2] and have also written them to a file using writetimetable().

Now I would like to synchronize them using synchronize [table1, table 2] but I get 'out of memory' very soon after executing the command.

Is there a way to process these table blockwise or any advice how I can handle these kind of large timetable efficiently w/o running out of memory? I appreciate any help or guidance

Answer 1

[Star Strider]

The only option I can suggest to you is in the timetable documentation section on Extended Capabilities under Tall Arrays. (I have no personal experience with huge data sets.)

[Lutetium]

Thanks, thats actually what I need.

However, it seems that I keep running into problems.

I used the write to writetimetable to write my as *csv. I then used datastore to read it in

ds1 = tabularTextDatastore('file.csv')

convert it to a tall array

tt = tall(ds1)

Thats how the data looks like in the preview which seems fine.

Then I try to get a timetable again but timevector is not recognized. Any idea how the syntax needs to be in this case? I have googled/tried a lot but wasnt able to recover my timetable.

TT = timetable(rowTimes, tt.Var1, tt.Var2, ...)

The conversion from the original variable (in-memory timetable) works perfectly fine

TT = tall(tt).

[Star Strider]

As always, my pleasure!

I am not certain what the problem could be. If you are having problems importing the time vector, it could be a problem with the format, and I am not certain how to solve that. It might be better to store it as a more conventional datetime or duration object that MATLAB expects, rather than an unusual format that may require tweaks to the detectImportOptions options structure. That is a useful function, however it can be frustrating (for me at least) to get to work correctly.

For the time being, unless it needs to be read by software other than MATLAB, it might be best to save it as a .mat file, since that eliminates the format problem. Be sure to back it up to another computer or the MathWorks Cloud (see Use Files on the Cloud for more information). I have no experience with MATLAB Drive either, however with a sufficiently large file, it is always best to back it up, and that is a convenient option.

[Lutetium]

Thanks, I got it to work now with your input. It may be the lack of experience but I agree, figuring out the correct import options can be quite frustrating.

In the end, I converted to a tall array and performed the synchronization. With write the synchronization is executed.

Using the datastore, I can recall the timetable again without any importing options. In temp, the data is stored a *.mat files.

ds1 = tall(variable)
ds2 = tall(variable)

ds3 = synchronize(ds1,ds2)
location = 'E:\Temp'
write(location,ds3)
location = 'E:\Temp'
ds = datastore(location)

[Star Strider]

As always, my pleasure!

I have obnly used the datastore function once, and forgot to mention it as an option.