how to convert the database file into hdf5 format.

My database file can have large file sometime due to which I can't read and write data from it, as it crashes the MATLAB. So i am thinking to convert my database into hdf5 format, how can I convert the db file to hdf5 format because a sql query from the db can crash MATLAB.

[KSSV]

REad about h5create, h5write

[MakM]

Hi KSSV,

The thing is, database is too large to read, for the h5create I need to first read the db file then write it into hdf5 format. Is there any way I can directly shift the data from db to hdf5?

[KSSV]

To my knowledge. You need to read and then write.

[MakM]

Any other way do u sugeest, I can try to prevent MATLAB crash ?

Answer 1

[Divyam]

Hi @MakM,

As a first step, create a dataset in the HDF5 file with unlimited dimensions in accordance with your data. Refer to the following documentation links as a guide:

• https://www.mathworks.com/help/matlab/ref/h5create.html?s_tid=doc_ta#:~:text=sizeRatio%20%3D%20%0A0.9432-,Create%20Dataset%20with%20Unlimited%20Dimension,-Input%20Arguments
• https://www.mathworks.com/help/matlab/ref/h5write.html#mw_7a31bb1f-d7f5-4140-bb0c-9df02664d138:~:text=14%5D%2C%5B5%207%5D)-,Write%20Data%20to%20Unlimited%20Dataset,-Input%20Arguments

To ensure that the SQL query from the database does not crash MATLAB, use a loop, and import a chunk with appropriate chunk size from the data such that the chunk size doesn't exceed MATLAB memory limits.

% Pseudocode

% Define the connection to your database
conn = database(<>);

% Define the chunk size
chunkSize = 1000;

% Initialize the HDF5 file
h5FileName = 'my_data.h5';

% Example: Create a dataset in the HDF5 file with unlimited dimensions
h5create(h5FileName, '/my_dataset', [Inf, numColumns], 'ChunkSize', [chunkSize, numColumns]);

% Keep track of the number of rows written
rowsWritten = 0;

while true
    % Retrieve a chunk of data from the database
    sqlQuery = sprintf('SELECT * FROM my_table LIMIT %d OFFSET %d', chunkSize, rowsWritten);
    
    % conn is your database connection
    dataChunk = fetch(conn, sqlQuery);
    
    % Break the loop if no more data is returned
    if isempty(dataChunk)
        break;
    end
    
    % Convert the data chunk to a format suitable for HDF5
    dataMatrix = table2array(dataChunk);
    
    % Write the data chunk to the HDF5 file
    h5write(h5FileName, '/my_dataset', dataMatrix, [rowsWritten + 1, 1], size(dataMatrix));
    
    % Update the number of rows written
    rowsWritten = rowsWritten + size(dataMatrix, 1);
end

% Close the database connection
close(conn);

For more information regarding the "h5create" and "h5write" functions, refer to the following documentation links:

• https://www.mathworks.com/help/matlab/ref/h5create.html
• https://www.mathworks.com/help/matlab/ref/h5write.html