Importing large CSVs and consolidating in one matrix.
조회 수: 1 (최근 30일)
이전 댓글 표시
Hello I need to consolidate around 800 CSVs(total size ~ 6GB) into 1 matrix. I need to process(mainly filtering) the data and plot the graphs. What should be the approach of the task so that the processing time in loading the CSVs and plotting takes minimum time.
TIA
댓글 수: 0
채택된 답변
Titus Edelhofer
2015년 1월 12일
Hi Pranjal,
if you have R2014b available, then datastore + mapreduce would be the choice of tools.
Titus
댓글 수: 2
Titus Edelhofer
2015년 1월 12일
In this case use textscan for reading the data. Each file is probably not too large (assuming the 800 csv are somewhat similar in size), i.e., about 7-10 MByte. Therefore reading each file in one sweep should be no problem.
You might use the Parallel Computing Toolbox (parfor) to read the files in parallel. Either process the data right away or save to .mat so that you can read them later faster in a loop.
Titus
추가 답변 (0개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Large Files and Big Data에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!