Workaround for dealing with heavy figure file without slowing or freezing or crashing matlab?

I have a large dataset represented as a matrix with dimensions 740864x8192, containing double precision values. The matrix is essentially an image with 740864 rows and 8192 columns, and I intend to visualize it using the pcolor function in MATLAB. However, when I attempt to plot this data, MATLAB slows down and crashes due to the sheer size of the dataset.

As a workaround, I tried to delete unnecessary variables during the process to free up memory and ensure that the matrix size doesn't overwhelm the system. Despite these efforts, MATLAB still crashes when it attempts to display the final figure, which is when the graphical representation of the data appears on the screen.

I used following command:

pcolor(X, Y, Z)

Is it possible to deal with this heavy figure file without slowing or freezing or crashing matlab?

[Les Beckham]

How do you even load that matrix? A matrix that size using doubles is 45.2 gigabytes.

Even if you have enough RAM to load it, you will run into problems trying to do anything with it.

740864*8192*8 / (1024*1024*1024)

ans = 45.2188

Also, where do X and Y come from in your call to pcolor (assuming that your big matrix is what you called Z and the pcolor documentation calls C)? They are going to use up memory as well.

[dpb]

740864*8192*8/1024/1024/1024

ans = 45.2188

is 45 GB; how much physical real memory does your machine actually have for starters?

If you used single instead of double it would be 25 GB which you might be able to deal with computationally.

But, even a fancy 8K monitor would have only something approaching 8K pixels in resolution in the one direction if every pixel were used for the image;

740864/8192

ans = 90.4375

is 100X more points than pixels to display the image on -- and it's more likely you've got only 1920 which is a factor of 4X less than that.

IOW, "decimate, decimate!" -- keep only every 4-500 points in each direction to even have a chance.

But, it's not at all clear how the 2D array relates to the plot3 X,Y,Z arrangement...

[megha]

Absolutely! this is 45 GB datafile - chunked into 15 netcdf files.

Let me check how do I take 4-500 points in each direction..

Just a small clarification - It is Pcolor plot not plot3!

Meanwhile, any suggestions welcomed!

Answer 1

[Anurag]

Hi Megha,

I understand you want to visualise a matrix with large dimensions, please follow the steps below to do the same:

• Use down sampling, it is the process of reducing the number of samples in a signal or dataset, often to decrease data size or processing complexity.

downsample_factor = 10;
X_downsampled = X(1:downsample_factor:end, :);
Y_downsampled = Y(1:downsample_factor:end, :);
Z_downsampled = Z(1:downsample_factor:end, :);
pcolor(X_downsampled, Y_downsampled, Z_downsampled);

• Plot smaller sections iteratively to avoid overwhelming MATLAB.

section_size = 1000;
for i = 1:section_size:size(Z, 1)
    end_idx = min(i + section_size - 1, size(Z, 1));
    pcolor(X(i:end_idx, :), Y(i:end_idx, :), Z(i:end_idx, :));
    pause(0.1);
end

• Consider exporting data for visualization in external tools like ParaView or exporting images.

imwrite(Z, 'large_data.png');

Hope this helps!

[Walter Roberson]

You might want to consider using imresize instead of downsampling by indexing: imresize() offers a choice of resizing methods, some of which would allow the intermediate data to contribute to the final pixel.

img = rgb2gray(imread('flamingos.jpg'));

for K = 1 : 50
    img(K:100:end, 1:100:end) = 255;
    img(1:100:end, K:100:end) = 255;
end
L = numel(img(1:71:end));
img(1:71:end) = 255-mod(1:L, 256);

imshow(img)

methods = ["nearest", "bilinear", "bicubic", "box", "lanczos2", "lanczos3"];
for K = 1 : numel(methods)
    figure();
    newimg = imresize(img, [100 100], methods(K));
    imshow(newimg); title(methods(K));
end

Answer 2

[Shubham]

Hey Megha,

I understand that you want to visualize an image but the MATLAB crashes while attempting to display the result. As pointed out by Les and dpb in the comments, dealing with a 45 GB file computationally is difficult.

You can try out the following:

1. Reduce the resolution of the image by down sampling the dataset. You can reduce the dataset size by taking averages within specified intervals. For your case, a sampling frequency of the order of 1000 (or at least 100) should be used to reduce the dataset size significantly. You can try out Anurag's and Walter's suggestion as well. You can also try breaking your dataset into chunks if you only need to visualise a certain part of the image.
2. Refer to the documentation of Visualizing Tall arrays: https://www.mathworks.com/help/matlab/import_export/tall-data-visualization.html “Tall arrays” allow you to work with large datasets. The above documentation provides some of the functions which can be used with “Tall arrays". The key idea used here is to create bins such that the data can be aggregated into small intervals.
3. Have a look at this previous MATLAB answer: https://ms-intl.mathworks.com/matlabcentral/answers/1915155-how-to-plot-2d-density-plot-with-very-large-files In the above case, the solution describes to divide the data into intervals on both the axis to create a rectangular grid and then averages out the values that fall into a grid point.
4. You can use function “reduce_plot” which reduces the plot according to the number of pixels available and updates if the user tries to zoom in. It could be useful for exploring large datasets. You can find the function “reduce_plot” available on file exchange: https://www.mathworks.com/matlabcentral/fileexchange/40790-plot-big

Hope this helps!