Read one column only of a large file, line by line?

조회 수: 2 (최근 30일)
Louise Wilson
Louise Wilson 2020년 10월 6일
답변: Seth Furman 2020년 10월 30일
I have a large table e.g. 7836x72001. I am interested in extracting the first column from the table which is a vector of datenumbers. Would there be a way of doing this quickly rather than waiting to load in the entire file?
I have considered using fopen and fgetl but this is taking a long time.
folder=('Y:\SoundTrap\Boats\PSD Output\Duty cycle data\');
files = dir(fullfile(folder,'*.csv'));
nf = length(files);
i=1;
a=1;
for i = 1:nf
filename = files(i).name;
disp(filename);
try
fid=fopen(fullfile(folder,filename));
tline=fgetl(fid); %read first line (column headers)
a=1; %relevant row of new table we are adding data to
while ~feof(fid) %read every row until last row reached
tline=fgetl(fid); %read next line
t=tline(1);
output(a)=t;
a=a+1;
end
writetable(a,fullfile('L:\PSD output\Boats\Duty cycle data\TVEC',strcat('TVEC_', filename)));
end

채택된 답변

Walter Roberson
Walter Roberson 2020년 10월 6일
I would suggest textscan with a format that is a %T (or %D as appropriate), followed by %*[\n] to consume the rest of the line.
textscan is much faster than looping yourself. Although you know that you could "do less work" by not processing the rest of the line, your knowledge would be implemented at the m-script level, with the threaded interpreter and Just In Time engine, whereas textscan is compiled into machine code, much lower overhead. And you need to parse the characters for end of line anyways, so textscan in this form is more efficient than one might think.
  댓글 수: 3
Louise Wilson
Louise Wilson 2020년 10월 22일
(it can take up to 20 mins to read a file using readtable)
Walter Roberson
Walter Roberson 2020년 10월 22일
My understanding is that readtable on text files calls textscan, except possibly for fixed-width text... I don't know how fixed-width text is handled.

댓글을 달려면 로그인하십시오.

추가 답변 (1개)

Seth Furman
Seth Furman 2020년 10월 30일
You might also want to consider using a tall table.

카테고리

Help CenterFile Exchange에서 Text Data Preparation에 대해 자세히 알아보기

제품


릴리스

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by