How to store and deal with very very large matrices?

% input variables
T = 2274;
N = 58;
B = 10000;
H = 250;
K = 3;

% I simulate matrices of similar size to give the idea of the problem

rng('default') % even if replicability is not necessary in this case

% first set (based on dimension)
P0 = lognrnd(0, 1, T, N);
R0 = rand(T, N) - rand(T, N);

% second set (based on dimension)
R = rand(T, N, K) - rand(T, N, K);
V0 = rand(T, N, K);
Z0 = randn(T, N, K);

% third and foruth set (based on dimension)
Z1 = randn(H, B, N, K);
R1 = rand(H, B, N, K) - rand(H, B, N, K);
V1 = rand(H, B, N, K);
P1 = lognrnd(0, 1, H, B, N, K);

Z2 = randn(H, B, N, K);
R2 = rand(H, B, N, K) - rand(H, B, N, K);
V2 = rand(H, B, N, K);
P2 = lognrnd(0, 1, H, B, N, K);

The above code is not executable due to the large size of the matrices involved. While MATLAB can handle the computations, saving and loading these big matrices is extremely slow, and loading the '.mat' files often results in crashes.

I have attempted to use the matfile function to access the variables without loading them, but it doesn't seem to provide a satisfactory solution in terms of speed.

I would greatly appreciate any suggestions or improvements on how to efficiently deal with such big matrices in MATLAB. Is there a more optimized way to handle these large datasets, particularly for saving and loading operations?

Answer 1

[Animesh]

Hello Barbab,

I understand that you are encountering difficulty while trying to load and store large-sized matrices.

In this case, I would recommend you to use ‘Tall Arrays’ in MATLAB. Tall Arrays are used to work with out-of-memory data that is backed by a datastore. Datastores enable you to work with large data sets in small blocks that individually fit in memory, instead of loading the entire data set into memory at once.

Please refer the following MathWorks Documentation link on ‘Tall Arrays’ for better understanding on how to use them for large matrices: https://www.mathworks.com/help/matlab/tall-arrays.html

Furthermore, for additional information on ‘Tall Arrays’ in MATLAB, consider going through the reference links below:

1. Documentation for Tall Arrays: https://in.mathworks.com/help/matlab/import_export/tall-arrays.html
2. Using Tall Arrays: https://in.mathworks.com/help/matlab/tall-arrays.html
3. Deferred Evaluation of Tall Arrays: https://in.mathworks.com/help/matlab/import_export/deferred-evaluation-of-tall-arrays.

I hope this helps!