How to efficiently integrate big data without using memory / (How to create big data)

• in a study i will produce large arrays.
• Each array will have at least 500 MB size.
• Each array will have the same number of rows.
• the total size of dataset will be approximately 20 GB or over.
• Somehow I have to create a single variable/array which includes all data and size of 20 GB.

matfile seems a good solution. However when the size of file increases, it gets slower. How can i handle this problem?

[blaat]

How you store and access big data is strongly dependent on what you need to do with it. Do you need all arrays at the same time? Do you need specific values from these arrays at the same time? Can you somehow partition your calculations to a subset of the data?

Without more information it is very difficult to give advice on your problem.

[Mehmet OZC]

I do not need all arrays at the same time. I think I can easily pick a few colums with matfile command. The issue is to bring together all variables to a single file. I must write all variables to a single file. this file will be a large file. I don't know how to combine/integrate separate files.

Any help is appreciated.

[Mehmet OZC]

For example;

A1 = 3918x1330 (21 MB on disk)

A2 = 3918x10000

A3 = 3918x20000

...

A99 = 3918x100000

All variables will have same number of rows. What i want to do is to write the following to a single file

[A1 A2 A3 ... A99];

[blaat]

If can process the arrays separately, perhaps it would be more convenient to keep them as separated files. Or is there another reason you want a single, large file?

If a single file is required, I would advise against storing everything in a single variable. As far as I know, there is no way of reading only part of an array from a .mat-file, so the file will require 20 GB of memory to load.

[Steven Lord]

Why do you need to write them to a single file? Why not put each in its own file; that way if something were to happen to one of the files you wouldn't lose all of your data?

[Mehmet OZC]

there can be more than one way to solve a problem. Saving all separate files to a single file on hard disk will ease my other calculations.

Matlab can easily load part of variables

http://www.mathworks.com/help/matlab/import_export/load-parts-of-variables-from-mat-files.html

The issue is to write such that big file.

[blaat]

So, if I understand correctly, your problem is this: you want to write 20 GB of data to a single variable in a .mat-file, but it's getting unworkably slow? Or doesn't it work at all?

[Mehmet OZC]

It works to a degree. When i try to append a 2 GB file to a 4GB file it gets slower. MATLAB does wonderful things. I believe it can handle this or is it impossible to create a really large file with using ordinary computers?

[Walter Roberson]

I wonder if compression is leading to slowdowns? I do not know whether -v7.3 with matfile uses compression; see discussion http://www.mathworks.com/matlabcentral/answers/15521-matlab-function-save-and-v7-3 and http://www.mathworks.com/matlabcentral/answers/137592-compress-only-selected-variables-when-saving-to-mat

Answer 1

[JMP Phillips]

Here are some things you could try:

Use the matfile function, which allows you to access and change variables directly in MAT-files, without loading into memory: http://au.mathworks.com/help/matlab/large-mat-files.html http://au.mathworks.com/help/matlab/ref/matfile.html

Structure your data differently: - if you are representing the data as doubles, maybe you can afford less accuracy e.g. use int32. For example, you can use scaling of 1e4 to represent a double value such as 100.3425 as an integer 1003425.

With MATLAB:

• use 64 bit matlab version
• try disabling compression when saving the files, with the -v6 option

Optimize your PC for your task:

• in task manager, close any unnecessary processes running at the same time, including taskbar junk (adobe update, java update etc)
• disable your anti-virus which might be trying to scan the file and slowing it down
• under task manager, give higher priority to the MATLAB process (see http://www.sevenforums.com/tutorials/83361-priority-level-set-applications-processes.html)
• increase your virtual memory or page file size http://windows.microsoft.com/en-au/windows/change-virtual-memory-size#1TC=windows-7
• defragment your hard drive
• run MATLAB from your local hard drive and not a network drive or external harddrive
• save the .mat file to your local hard drive where it has plenty of space, not a network drive or external harddrive.
• For faster hard drive access, use a Solid State Drive (SSD)

[Walter Roberson]

The -v6 option is incompatible with matfile and with objects over 2 Gb.

[Mehmet OZC]

In one of the links provided above I have run across following code

example = matfile('example.mat','Writable',true);

[nrowsB,ncolsB] = size(example,'B');
for row = 1:nrowsB
  example.B(row,:) = row * example.B(row,:);
end

And that solved my problem. Thanks