Use of textscan instead of load in a for loop

Hi,

I'm at the moment loading data using the following code:

 data = cell(1,n_sim);
for m_a = 1:n_sim

    for m_b = 1:n_seed

     data{1,m_a}{m_b} = load(fileList{1,m_a}{m_b}, 'A1:end');

    end

end

This takes a long time since there are a lot of files and they are quite big. The resulting data cell is a 1xn_sim cell, and each cell has 1xn_seed cells inside. Each of those i.e. data{1,1} contains a 11000x25 double.

I would like to use textscan to get the same format on my results. The files that I'm reading have 25 columns and 11000 rows. Can I get some help with the scripting of texscan for this case?

Thanks

[Alexander Jensen]

Can I ask, why would you like to use textscan? and I think it would be helpful if you added a screenshot of the file (opened in some text-editor), at least I think that I need to see how it looks, in order to make the textscan code :)

[Isma_gp]

I tried running just one case with textscan and it is much faster. I have attached a screenshot of the file. Thanks

[Walter Roberson]

That code is not valid. A1:end is not a valid option (those start with a minus sign) and is not a valid variable name.

[Isma_gp]

I don't have any issue running this code, except the one I presented....that it is very slow.

[Walter Roberson]

It appears that in practice loading from a text file ignores any parameters that do not start with '-' . It would be better to remove the 'A1:end' from the load() in order to reduce confusion. In particular, all of the file will be read in, not just column 1.

[Walter Roberson]

Is the data space delimited or comma delimited ?

Answer 1

[Walter Roberson]

ncol = 25;
fmt = repmat('%f', 1, ncol);

data = cell(1, n_sim);
for m_a = 1:n_sim
    data_mb = cell(1, n_seed);
    for m_b = 1:n_seed
      fid = fopen('fileList{1,m_a}{m_b}', 'rt');
      data_mb(m_b) = textscan(fid, fmt, 'CollectOutput', 1);   %notice () not {} on output
      fclose(fid);
    end
    data{1,m_a} = data_mb;
end

Answer 2

[Alexander Jensen]

I imagine that the reason why it takes a while is due to the lack of preallocation of memory, I would however do like this (I know it is not using textscan, but why would you? :D )

dims = [11000 25];
data = nan(n_sim, n_seed, dims(1), dims(2));
for m_a = 1:n_sim
    for m_b = 1:n_seed
       data(m_a,m_b,:,:) = load(fileList{1,m_a}{m_b}, 'A1:end');
    end
end

[Isma_gp]

Hi, thanks for this. It is still taking the same amount of time though.