How read extremely large HDF5 format data and resolve out of memory issue?

I need to read several datasets from a 2TB HDF5 file which are needed for further computation.

If I simply coding as following,

varible1=h5read('path to .h5 file', 'path to dataset')

It would require ~500 GB array memory.

Is there any good way to solve this problem?

Thanks!

[Jonas]

does this help? https://de.mathworks.com/matlabcentral/answers/423693-read-and-divide-hdf5-data-into-chunks

maybe you could also spit data using oython beforehand, i saw multiple those scripts when googling

Answer 1

[ROSEMARIE MURRAY]

You could use a fileDatastore with the read function h5read, which would allow you to specify a certain amount to read at a time.

Or you could try this datastore: https://www.mathworks.com/matlabcentral/fileexchange/64919-hdf5-custom-file-datastore-for-timeseries-in-matlab