Import data tool speed

조회 수: 3 (최근 30일)
Brad_EE
Brad_EE 2020년 12월 11일
댓글: Yair Altman 2020년 12월 13일
Platform: W10-64bit, 4 core, 500GB SSD, 32GB Ram
File to import description: 80MB, contains comma delimited numbers and short character arrays. Overall format 27 columns x 400K lines.
Clicking the import data tool icon prompts for a file name, shows a message "Opening a large text file ..." and then displays the file contents in a table format on the GUI after ~5 seconds. The table displayed matches the file contents.
On the same GUI I set the output type to cell array and range to A2:AA395201 (entire file minus header line). Clicking the Import Selection button displays message "Importing Data..." and a status bar that stays gray for 35 minutes before it suddenly disappears. At that point the import is complete and the variable name appears in the workspace.
Why does the initial opening large text file finish in 5 seconds but the import take 35 minutes? It seems for the opening large text file step to complete and display the data in the GUI table, it has essentially imported the data but 500X faster!

채택된 답변

Yair Altman
Yair Altman 2020년 12월 13일
The import tool GUI only shows you a preview of the data, based on the top N lines in the file, it does not read and process the entire file. Only when you click the <Import Selection> button is the entire file processed based on the selected range that you specified and the file format detected by the preview. This naturally takes much longer than the preview processing.
  댓글 수: 4
Brad_EE
Brad_EE 2020년 12월 13일
Perhaps it is reading 35 lines at a time based on the relative position of the scroll bar. So for instance if the scroll bar is pulled down 3/4 of its length, only 35 lines from that location in the file is actually read and displayed.
Yair Altman
Yair Altman 2020년 12월 13일
Yes, this is exactly what I meant

댓글을 달려면 로그인하십시오.

추가 답변 (0개)

카테고리

Help CenterFile Exchange에서 Whos에 대해 자세히 알아보기

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by