append bug maybe.

In the .mat file, append the variable with the append command, which has the same name as the original variable in the file. Although it is overwritten, the file size is the sum of the two variables, which results in the file getting larger and larger. In fact, multiple variables in the file After saving from the workspace is very small. We think this is a bug and there is no real coverage. wonder if it is or not

Answer 1

[Walter Roberson]

This is by design. When you -append, then the .mat file as a whole is not rewritten: instead the place in the .mat file that stored the variable is marked as unused and the new information is put at the end of the file.

If you were to later save a smaller version of the same variable, then the now-empty block in the file might be re-used, but this is not promised.

-append makes no attempt to "compact" the file to remove the hole. It was designed for the situation where small update time was more important than smallest file.

[hangjun ji]

thank you, I guess that through practical using save function with -append.

but I must say, the following text lead to confusion about this issue:

example

save(filename,variables,'-append') adds new variables to an existing file. If a variable already exists in a MAT-file, then save overwrites it with the value in the workspace.

[Walter Roberson]

save overwrites the variable. That does not tell you anything about how it represents that inside the file.

[hangjun ji]

overwrite? let me say, I mean the same name variable will not overwrite. it just add the new same name variable to the end of file when you save, thus the file will get bigger and bigger. we update the logfile each hour. It is 3.2M when the APP start, yesterday we find the logfile 57.4M, but the actual change only just 30k.

that shows if you want use save(filename,,'variable name','-append'), it will not relplace the old version of the same name variable, but really append to the end of the file.

[Walter Roberson]

.mat files are data structures. As far as the data structure is concerned, the old value is no longer present and has been overwritten with the new value. Mathworks choose to implement that in a way that is efficient in time, but not in disk space.

[hangjun ji]

I see, but the file will be too large when be loaded into working area again.

[hangjun ji]

I mean that the word overwrite is inappropriate here and will be misunderstood.

[Walter Roberson]

The variable is overwritten, the storage is (usually) not.

update the logfile each hour.

Perhaps using a .mat file is not the best choice for your situation.

If you are consistently appending additional data (without removing any of the old data), then a binary or text file and appending data would perhaps be better.

If you are going through a cycle of filtering and consolidation and adding new material, so that what is to be written replaces what is there but does not include all the detail of what was there before, then a database might make more sense.

[hangjun ji]

ok, I got it.

[Walter Roberson]

Often files such as .mat files are designed so that if something goes wrong during an update, that you do not lose the old information, or at least designed to reduce the risk of losing the old information. If you were to re-use the disk space immediately (in the case where the new version of the variable can fit into the old location) then if you stop writing for some reason (reboot, kill the process) then the .mat file would end up with the old variable version being gone and the new variable version not being all there yet. It is more reliable to write the new material at the end of the file (or at least in an empty block that is big enough and is after the original variable in the file) and then once it is in place, to invalidate the old variable that is earlier in the file. If you do this, then at any time, the valid version of the variable is the first one you can find in the file.

If you were to deliberately look for holes in the file that are large enough to fit the new version of the variable (and you would have to know its compressed size to know if it would fit) then as you write out the new version, until it is completely written out, scanning the file would find the new not-ready version first. There are ways to work with that: you could write out a version marked as not being in use yet, then afterwards go back and mark it as active, then go find the old version later in the file and remove it, But the logic is simpler to just put the new version at the end until it is done and then to go and mark the old earlier version as invalid.

[hangjun ji]

thanks a lot, I got it.