Use datastores and tall arrays.
Python-like generator for reading large CSV file?
조회 수: 3 (최근 30일)
이전 댓글 표시
I have a large csv file, typically I just read everything into a array like this
num =csvread(filename,2,2)
if csv file is big, the num array will be big as well and sometime it blowout my memory.
In python, there is a way to read the file but without holding the data in the memory. the methods use generator object. For example,
Base case(w/o using generator object):
value=[len(x) for x in open('filename.tx')] this case will hold the data in the memory
another case(use generator object:
it=(len(x) for x in open('filename.tx'))
the it is a generator object and it only hold the memory size for the generator expression without holding memory for all the data, later, we just extract the data by using next(it).
I am wondering if Matlab has similar generator object as Python when reading csv file to save the memory? Thanks
댓글 수: 0
채택된 답변
추가 답변 (1개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Standard File Formats에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!