How to quote variables whose names are in order in a for loop?

Hi! I want to store a 6D tensor A which is so large that I meet the 'out of memory' error. So I want to store it by parts. i.e. store A by sub-tensors A1,A2,...An. I know I can create variables whose name are in order by eval(). For example, I can create x1,...,x10 by:

VarNum = 10; 
vars = []; 
for i = 1:VarNum
    eval([['x',num2str(i-1)],'=','i',';']); 
end

But I don't know how to quote x1,...x10 in a for-loop when I don't store x1,...x10 into an array or cell which contains too large space and I will encounter 'out of memory' error. Can anyone help me? Thank you!

[Walter Roberson]

http://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

[weikang zhao]

There is no difference between storing x1,...x10 into an array or cell and directly storing 6D tensor A in memory. Because the data is very large, you can only get xi from the data file one by one, instead of loading them all into the memory at once.

[Stephen23]

The content of a cell array is not stored in contiguous memory, so there is no reason why you cannot use a cell array (assuming that all of the data can fit into memory simultaneously).

If the data cannot all fit into memory at once, then definining variable names will not magically increase the amount of memory your PC has installed: in that case you should use the appropriate tools, e.g.:

https://www.mathworks.com/help/matlab/tall-arrays.html

[Jason Yang]

@Stephen CobeldickThanks! Could you clarify what 'contiguous memory' means? So arrays are stored in contiguous memory, right?

[Walter Roberson]

When MATLAB asks the operating system for memory, the operating system looks around for unused physical "pages". In theory there are a few different possible physical page sizes, but in practice for Windows it will almost always be 4 kB. That is, the physical memory is broken up into chunks of 4096 bytes and those are allocated as chunks instead of trying to keep track of every byte individually.

So, the operating system will figure out how many 4 kilobyte chunks of memory are needed, and will find them somewhere (or make them available.) Then it will put the addresses of all of those physical pages together into a translation table that matches between consecutive logical addresses and corresponding physical addresses. The program (MATLAB) uses consecutive numeric addresses for the chunk of memory, and the hardware works to convert the numeric addresses into physical addresses.

As far as the program is concerned, it just sees all the memory in one array as if it were adjacent physical chips. But for a different array, it might have been told a quite different numeric base address. Which might be a kind of fiction too -- the second array might perhaps have ended up in physical memory in chips immediately beside the ones for the first array. Or not. MATLAB asked for memory, the operating system says "Here is a base address, use this base address plus a linear offset and the hardware will take care of accessing the right scattered physical memory pages". MATLAB asked for more memory, the base address might be quite "far" from the first array, no matter where the physical memory used is.

It is meaningful to talk about chunks of contiguous physical memory -- physical memory pages whose chips are right beside each other. For most purposes, the operating system does not need contiguous physical memory... but if it can happen to get it, then that can reduce overhead a little. (Contiguous physical memory is sometimes required for talking to peripherals.) In some cases of a system being fairly loaded with programs and chunks of memory, that reduction of overhead can turn out to be important; for example the hardware might only be able to track 65536 different chunks of memory at the same time, so it might become important to use consecutive physical blocks so that the hardware can say "this page and the next three" instead of listing the pages individually. There is some use to it... but until the system gets overloaded in that way, the system doesn't care how scattered the physical memory pages are for a program.

For the purposes of user-level programs, they only care that they are given a base address and can access any location in the chunk by using the basic address and a linear offset -- if byte offset 4097 is associated with a physical memory page that is not the "next" physical memory page after byte offset 4095, then that is taken care of in hardware. Any one array of the low-level data types (uint8, uint16, uint32, uint64, int8, int16, int32, int64, single, double, logical, char) is taken care of at the MATLAB level in terms of start address and offset, with the hardware taking care of the physical details. The addresses for any one numeric array are all contiguous at the "virtual" memory level.

When you use cell arrays or structs or any other composite data type, then the different entries are not specifically allocated as being "beside" or even "close to" the other entries at the composite level. The structure entry abc(1).def might be allocated some distance away from abc(2).def or abc(1).ghi -- but all of the numeric array abc(1).def is contiguous virtual memory within itself.

Why does this all matter in the situation?

Well, when you have limited amounts of available memory, then depending on what has happened before in the program, you might encounter a situation where there is not enough available consecutive virtual memory to hold (say) a 6 gigabyte block, but there might be enough consecutive virtual memory to hold six 1 gigabyte blocks. The total amount of available virtual memory can, in some cases, be less important than how fragmented it is. There are definitely situations where if you had allocated that 6 gigabyte block "first thing" in a program that it would have given it to you and then arranged everything else to fit, but if you ask for it later in the program it might tell you it cannot do it because there just isn't enough in one piece at the moment.

Thus, Stephen was suggesting that as long as you have enough available memory to hold the total amount of data, that using a cell array might be a way to split the requirements into smaller pieces that are more likely to individually fit where one big chunk did not fit.

Using a cell array will not help in the situation where the total amount of data to be loaded is larger than can fit in your system, In those kinds of situations, using tall() arrays or using datastore object can sometimes help.

[Jason Yang]

@Walter Roberson That helps a lot! My question is solved. Thank you very much!

[Steven Lord]

An analogy that might make this clear is that of a hardcover version of the combined Lord of the Rings series of books. That volume requires a certain width of shelf space to store, and all that space has to be on the same shelf.

If you were to split the series into three volumes they would take up (basically) the same total width, but you could put the first volume on one shelf and the other two on the next shelf down. Though if you misplace volume 2 it could take you a while to find it when you want to read from it.

The combined hardcover is a double array, the three volumes are a cell array each cell of which contains part of the larger double array.

Answer 1

[Steven Lord]

See if some of the tools in MATLAB for working with data that is too large to fit in memory on one machine all at once will satisfy your needs.