Highly Challenging Problem : Delete a matrix from a binary file after reading it.

Hello Everyone,

I want to store matrices into a binary file, the in a second time, for another process, I wan to read the matrices one-by-one from the binary file.

But, since when I read a matrix I can store it into a variable, I don't need it anymore holding memory in my binary file.

clear; clc; close; warning('off','all'); fclose('all');
delete 'All_K.bin'

% Write Part
N   =  5;
n   =  3;
K1  =  3*ones(N * n);
K2  =  5*ones(N * n);
K3  =  9*ones(N * n);

fileID  =  fopen('All_K.bin', 'a');
fwrite(fileID, K1, 'double');
fwrite(fileID, K2, 'double');
fwrite(fileID, K3, 'double');
fclose(fileID);

% Read Part 
fid  =  fopen('All_K.bin', 'r');

% Read K1 from the file
A1  =  fread(fid, [N*n N*n], 'double');

% Erase K1 from the file
% ******************************* %

% Read K2 from the file
A2  =  fread(fid, [N*n N*n], 'double');

% Erase K2 from the file
% ******************************* %

% Read K3 from the file
A3  =  fread(fid, [N*n N*n], 'double');

% Erase K3 from the file
% ******************************* %

fclose(fid);

So, please can Anyone help me solve this difficult problem.

Thanks in advance!

[Adam]

Is it necessary to store binary files rather than .mat files where deletion of variables is trivial? Also you appear to be deleting each matrix in turn anyway so why not just read the whole file then delete the file itself?

[AMEHRI WALID]

a Mat file is limited to matlab only, that's why I am using binary files.

It's a good idea, but, this is just an example, In reality I'm storing large matrices (500 r, 500 c), and sometimes I need to store thousands of them

[Adam]

I'm no expert in file structures, but binary files really don't seem like they are setup for this type of operation, just arbitrarily deleting parts of them.

[AMEHRI WALID]

Okay, no problem

Thank you !

[Stephen23]

"a Mat file is limited to matlab only..."

I regularly use Octave, R, and numpy to read/write .mat files.

[AMEHRI WALID]

If I may ask you, since you have told me about mat files.

How can I read matrix by matrix, instead of reading all matrices ?

% Use MAT Files

clear; clc; close; warning('off','all'); fclose('all');

delete 'All_K.bin'

N  =  10;
n  =  3;

K  =  3*ones(N * n);
save('All_K', 'K')

K  =  5*ones(N * n);
save('All_K', 'K', '-append')

K  =  9*ones(N * n);
save('All_K', 'K', '-append')


clear;

load 'All_K'

[Adam]

Same syntax as you do for saving:

loadedData = load( 'All_K', 'K1' );
K1 = loadedData.K1;

would generally be safest.

[AMEHRI WALID]

Sorry, I just changed my code

I will have only one changing Matrix K to store.

The problem is that it is saving just the last one.

So how can I do it ?

[Stephen23]

"So how can I do it ? "

You would be better off accumulating your data into one (container) array, and then saving it all at once. That would be simpler and more efficient.

If that does not suit your process, then consider using matfile to add data to an existing (container) array.

If that does not suit, then generate a scalar structure with the fieldname/s that you require, save using the '-struct' and '-append' options (untested, something like this)

S.('nameOfField') = [...]
save(...,'-append','-struct','S')

Then when loading always load into an output variable (otherwise you will force yourself into writing slow, complex, buggy code):

S = load(...)

Note that the -append option is badly named:

https://www.mathworks.com/matlabcentral/answers/322891-how-to-append-new-structure-array-element-in-existing-mat-file#comment_425176

[James Tursa]

"Deleting" a section of a binary file amounts to rewriting the file, and reading/writing files to disk is one of the slowest things a computer does. Better to follow the suggestions made to avoid this. If you really need it in a raw binary form, try to read them sequentially.

[AMEHRI WALID]

Okay, thank you everyone one for your help and your ideas.