Long time to load 40 gig file
조회 수: 5 (최근 30일)
이전 댓글 표시
Hi, I have a gaming laptop which I recently bought for 4000€. Massive CPU and lots of ram. However when I want to load a 30 gig matlab file from my hard drive which consists of variables with huge cell array tables, I still need a lot of time. How can I make this process faster? Is data store useful?
댓글 수: 4
Jan
2017년 10월 20일
Please mention the details: How long is "a lot of time"? What is the acceptable speed for you? Which MAT format is used?
Steven Lord
2017년 10월 20일
There's some additional information that would be useful in trying to determine what's consuming most of the time and whether there's a better approach you can use.
- Is this a text file, a binary file, a MAT-file, etc?
- Do you need all the variables in the file at once or could you do what you need accessing each variable in turn, keeping only one at a time in memory?
- Can you post a small sample (not all 30-40 GB, but 5-10 lines) of what the data looks like in the file, and whether the file has a consistent format throughout or if the format changes periodically
- Copy and paste the command that you're using to try to read in the file, or describe the interactive process you're using (if you're using the Import Tool, for example.)
- Can you clarify what "a lot of time" means: a couple minutes, half an hour, an hour, etc.?
- Can you clarify what "lots of ram" means: how many GB?
답변 (2개)
Edric Ellis
2017년 10월 20일
You don't mention how you're loading the data right now. If you have Parallel Computing Toolbox, you can read the data in parallel using parfor together with datastore. Or, better still, use tall arrays, which can automatically take advantage of Parallel Computing Toolbox.
댓글 수: 0
Jeremy Hughes
2017년 10월 20일
"cell array tables"
A good bet is that the cell arrays are the issue. Each cell of the array takes a 114 bytes of overhead. Without knowing anything about your data, except that you described them as tables, I suggest looking at the MATLAB datatype, table.
Data can be efficiently stored as tables for many common usages. I wish I could give a better answer, but I'd need to see what's in the file.
Cheers,
Jeremy
댓글 수: 0
참고 항목
카테고리
Help Center 및 File Exchange에서 File Operations에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!