I got a large data matrix (3Gb) of signal data which I saved to a .mat file. Since the data size is large I figured out it will be wiser to access the file and read each time a block of data, process it and move to the next block instead of loading the complete file. I'm wondering should I use the matfile method, fopen and fread or should I use a datastore and read command. What is faster from the processing point of view and easier to work with? Or any other considerations? Thanks

댓글 수: 3

per isakson
per isakson 2016년 11월 11일
"data set" &nbsp is that one numerical array?
"fairly large data set" &nbsp how large compared to the physical memory?
Talmon Alexandri
Talmon Alexandri 2016년 11월 12일
data set - plain numerical array 10 million lines by 48 columns Total RAM size - 3G
per isakson
per isakson 2016년 11월 14일
Given
  1. "large data matrix (3Gb) of signal data which I saved to a .mat file"
  2. "Total RAM size - 3G[B]"
  3. "[one] plain numerical array 10 million lines by 48 columns"
I assume the mat-file is version 7.3, because that's needed to store an array of that size.
It's the easiest to work with and I don't think there is any other way that is significantly faster.

댓글을 달려면 로그인하십시오.

답변 (1개)

Walter Roberson
Walter Roberson 2016년 11월 11일

0 개 추천

I think maybe using memmapfile and a binary file might be easiest for access.

카테고리

도움말 센터File Exchange에서 Large Files and Big Data에 대해 자세히 알아보기

질문:

2016년 11월 11일

댓글:

2016년 11월 14일

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by