What is the fastest way to load many large files and then reuse that data

조회 수: 54 (최근 30일)
Daniel . 2020년 8월 20일
댓글: Daniel . 2020년 8월 28일
I have upwards of 200 .csv files that are around 500 MB each. Each file contains a one line text header and 10 columns of numeric data with many, many rows. I only need to load columns 2-4 once from any one of the files as that information is identical in all files. From all of the files, I need columns 5-8 only. The files are all in one folder with a systematic naming convention if that helps at all. What is the fastest way to do this the first time? I've tried importdata, textscan, and readmatrix and have either not been able to do what I want above or have found it still too slow. Once it's loaded, I'll do some manipulation and save it as a .mat to work on later. Am I right that saving as .mat will produce the fastest load times in the future?
  댓글 수: 3
Daniel 2020년 8월 28일
One whole file is 301,409,168 bytes once loaded. I had been loading all the data and plotting some things as a sort of quality check. I don't suppose I'd know how to just load it and save it to a .mat in a piecewise fashion, though that might suffice for what I want.

댓글을 달려면 로그인하십시오.

답변 (1개)

Jalaj Gambhir
Jalaj Gambhir 2020년 8월 24일
Have a look here. You can probably use textscan.
  댓글 수: 1
Daniel 2020년 8월 24일
There are several answers in that first link. Which are you suggesting?

댓글을 달려면 로그인하십시오.


Help CenterFile Exchange에서 Large Files and Big Data에 대해 자세히 알아보기




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by