Efficient way to read in large dataset

Hi all.

I am to read in a set of data to Matlab which has 6 columns and more than 2 million rows. I tried reading the data as a tab delimited .txt file with the dataset function as such:

   pricedata = ...
  dataset('File',file,'Delimiter','tab','ReadObsNames',true)          ;

This has gone on for more than 3 hours. Can someone suggest a better way to do it?

Answer 1

[Peter Perkins]

Mads, you will most likely find that this:

>> help dataset/dataset
dataset Create a dataset array.
[snip]
DS = dataset('File',FILENAME,'Format',FORMAT, ...) creates
a dataset array using the TEXTSCAN function to read column-
oriented data in a text file.  Specifying a format can
improve speed significantly for large files.

will read the file much faster. In your case, FORMAT might be something like '%f%f%f%f%f%f' if all the columns are numeric. Hope it helps.

[Mads]

Thx, I will give that a shot.