MATLAB Answers

Dealing with large timetables

조회 수: 2(최근 30일)
Newbie 28 Oct 2020
댓글: Star Strider 29 Oct 2020
I generated two timetable [283580428x5] and [21317257x2] and have also written them to a file using writetimetable().
Now I would like to synchronize them using synchronize [table1, table 2] but I get 'out of memory' very soon after executing the command.
Is there a way to process these table blockwise or any advice how I can handle these kind of large timetable efficiently w/o running out of memory? I appreciate any help or guidance

  댓글 수: 0

댓글을 달려면 로그인하십시오.

채택된 답변

Star Strider
Star Strider 28 Oct 2020
The only option I can suggest to you is in the timetable documentation section on Extended Capabilities under Tall Arrays. (I have no personal experience with huge data sets.)

  댓글 수: 4

표시 이전 댓글 수: 1
Star Strider
Star Strider 29 Oct 2020
As always, my pleasure!
I am not certain what the problem could be. If you are having problems importing the time vector, it could be a problem with the format, and I am not certain how to solve that. It might be better to store it as a more conventional datetime or duration object that MATLAB expects, rather than an unusual format that may require tweaks to the detectImportOptions options structure. That is a useful function, however it can be frustrating (for me at least) to get to work correctly.
For the time being, unless it needs to be read by software other than MATLAB, it might be best to save it as a .mat file, since that eliminates the format problem. Be sure to back it up to another computer or the MathWorks Cloud (see Use Files on the Cloud for more information). I have no experience with MATLAB Drive either, however with a sufficiently large file, it is always best to back it up, and that is a convenient option.
Newbie 29 Oct 2020
Thanks, I got it to work now with your input. It may be the lack of experience but I agree, figuring out the correct import options can be quite frustrating.
In the end, I converted to a tall array and performed the synchronization. With write the synchronization is executed.
Using the datastore, I can recall the timetable again without any importing options. In temp, the data is stored a *.mat files.
ds1 = tall(variable)
ds2 = tall(variable)
ds3 = synchronize(ds1,ds2)
location = 'E:\Temp'
location = 'E:\Temp'
ds = datastore(location)
Star Strider
Star Strider 29 Oct 2020
As always, my pleasure!
I have obnly used the datastore function once, and forgot to mention it as an option.

댓글을 달려면 로그인하십시오.

추가 답변(0개)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by