How to read large data file in Matlab?

I have a GPS signal data values stored in a .dat file of 200 GB. The data is in the form of uint8. I want to read 16 GB of data from the start and downsample it by a factor of 10 before storing the new signal in a separate file. I have used fread but I cam unable to read large data file. Is there any other way which I can use to read data in chunks and put them together afterwards? Anyone please help me out. Its very important that I read this data.

The specifications of software, platform & PC are:

Matlab R2012a. Windows 8 64-bit, Core i5 3rd Gen., 4 GB RAM, 500 GB of hard Disk Space

Answer 1

[Simon]

Hi!

Is it a binary file or a normal text file? For text files you may use "fopen" and "fgetl" to read line by line. If you know your row format you may use "fscanf" to read a given number of values. Repeated calls to "fscanf" will read your whole file.

Can you give some (!) example lines? It is easier to help.

[Umair Nadeem]

Its a binary data, stored at two amplitude levels and it is in the form of a column vector. From your answer I get the idea of reading small chuncks of data in a loop and writing the downsampled data in another file. Then maybe I will be able be read 16GB by using chuncks of 100MB in every loop. Do you think it will work or the result would be same i.e. extremely slow response...?

[Simon]

Hi!

fid = fopen(FielName);
A = fread(fid, [100, 3]);
fclose(fid);

This reads 100 lines from a file with 3 columns and stores it in a matrix A of size [100, 3] (as double). you can put a loop around fread to read the file in chunks. There should be no problem.

[Sania Zahan]

Hi My data shape is (39412, 3, 300, 25, 2), its numpy file with just numbers. Therefore above method changes the shape. I need to maintain the last 4 indexes. could you please suggest how may I do that. Thanks

[Walter Roberson]

A = fread(fid, [39412, 3, 300, 25, 2]);

However, you really need to test first, as it would be common to find the first two dimensions transposed between different programming languages. You might need to use

A = permute(fread(fid, [3, 39412, 300, 25, 2]), [2 1 3 4 5]);

Answer 2

[Sheikh Alsheikh]

Same problem I had o face! I first break the data into chunks

[Hammad Intersoft]

how to open a large file by breaking it into chunks. i am facing a problem while opening a large file that has a size of 48*42*2414 and bytes 4866624. when i load this file in matla it says "cannot display summaries of variables with more than 524288 elements."