3D sparse matrix

I have 1500 images almost each have 5000 rows and 7000 columns. so I need to load one by one. make a large matrix. The matrix size will be 5000x7000x1500. Image are Greyscale Image.

A portion of code is below. In the images variable preallocaiton will be done. To save the memory I need to do the 3D sparse matrix.

Any of your help will be appreicated. Thank you so much.

filename = sprintf(basefilename,start_idx);
full_path = fullfile(image_dir, filename);
f1 = imread(full_path);
nframes = end_idx-start_idx+1;


        % preallocating the matrix

        sz0 = [size(f1,1:2) nframes];
        images = zeros(sz0,class(f1));
        
        fidx = int8(start_idx:end_idx);

        tic

        parfor k = 1:length(fidx) % these are output frame indexes
            
            % reading one by one frame operation.........
            images(:,:,k) = tmp_img(:,:,1);
        end
        
        

Answer 1

[Walter Roberson]

MATLAB sparse matrices are strictly vectors or 2D, with no capacity for 3D.

The File Exchange Contribution https://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays can emulate multi-dimensional sparse matrices.

Note that performance of sparse matrices can be poor if you add more entries than there are available "unused slots".

A basic sparse matrix has one 8-byte pointer per column . If the column is empty, the pointer is NULL (binary 0). If the column is not empty, then the pointer points to a list of row indices and corresponding values.

Making assignments into a sparse matrix can require copying the entire structure of the existing sparse matrix into a new structure that has more space, doing the assignment in the new structure, and throwing away the old structure.

However, when you create a sparse matrix using spalloc you can designate the number of unused slots to allocate. Each unused slot takes 16 bytes. When you assign into a sparse matrix that has an unused slot, then the sparse matrix can record the row and column and corresponding value without having to copy the matrix.

As you plan to eventually write in entire layers of image data, if you do not pre-allocate lots of memory, you would end up with a notable amount of copying the sparse matrix to make room for the new data. And if you do pre-allocate then you need 16 bytes per pre-allocated entry, which is going to add up enough that it is going to be worse than having a full matrix that is twice as large.

Sparse matrices that are densely occupied are not efficient.

[Walter Roberson]

        parfor k = 1:length(fidx) % these are output frame indexes
            
            % reading one by one frame operation.........
            images(:,:,k) = tmp_img(:,:,1);
        end

have you considered

        parfor k = 1:length(fidx) % these are output frame indexes
            
            % reading one by one frame operation.........
            images_cell{k} = tmp_img(:,:,1);
        end

If later you have operations that need to work across multiple images, then you can cat(3,images_cell{:}) to create a grand array. Or extract portions of each cell and cat(3) those portions together if you only need a segment of the cuboid.