Read files in a specific order

Hi, I have a series of files named with a date, for every third hour i.e. "NUM_1985_6_4_0_0.dat", "NUM_1985_6_4_3_0.dat", "NUM_1985_6_4_6_0.dat" ,....., "NUM_1985_6_4_21_0.dat",......, "NUM_1985_6_10_21_0.dat", "NUM_1985_6_11_0_0.dat". I'm using the dir command before using the fopen command. Is there any option when using the dir command to be able to list the files on a specific order. I would like to read the files by dates, however, the dir command gives me a list of files where the first one is "NUM_1985_6_10_0_0.dat" following a different logic, where the number 1 goes first in the list.

Thanks

Answer 1

[Stephen23]

Download my FEX submission natsorfiles, which was designed to fix the exact problem that you are having. Use it to simply sort the output from DIR:

D = '.'; % folder path
S = dir(fullfile(D,'*.dat'));
S = natsortfiles(S); % alphanumeric sort by filename
for k = 1:numel(S)
    F = fullfile(D,S(k).name)
    ... import/process file F
end

Here is natsortfiles used on your example filenames:

>> C = {'NUM_1985_6_4_0_0.dat';'NUM_1985_6_4_3_0.dat';'NUM_1985_6_4_6_0.dat';'NUM_1985_6_4_21_0.dat';'NUM_1985_6_10_21_0.dat';'NUM_1985_6_11_0_0.dat'};
>> C = C(randperm(numel(C))) % "random" order
C =
  NUM_1985_6_11_0_0.dat
  NUM_1985_6_4_0_0.dat
  NUM_1985_6_4_21_0.dat
  NUM_1985_6_4_3_0.dat
  NUM_1985_6_4_6_0.dat
  NUM_1985_6_10_21_0.dat
>> D = natsortfiles(C) % alphanumeric sort
D =
  NUM_1985_6_4_0_0.dat
  NUM_1985_6_4_3_0.dat
  NUM_1985_6_4_6_0.dat
  NUM_1985_6_4_21_0.dat
  NUM_1985_6_10_21_0.dat
  NUM_1985_6_11_0_0.dat

[Isma_gp]

Thanks, that's great!

[Stephen23]

@Isma_gp: another solution is to always use ISO 8601 date formats (e.g. yyyy-mm-dd, and which always include leading zeros), then you can use a simple character sort (e.g. MATLAB's inbuilt sort).

[Luke Riddell]

Hi Stephen,

I've tried this but I get an error returning.

When using:

S = dir(fullfile(FinalFolderloc,'*.ASC'))
S = natsortfiles(S)

It returns the error in line with the natsortfiles function:

" Error using bsxfun

Non-singleton dimensions of the two input arrays must match each other.

Error in natsortfiles (line 231)

mat(logical(bsxfun(@eq,vec,nmn).')) = fnm; % TRANSPOSE bug loses type (R2013b)"

Do you have any advice?

[Stephen23]

@Luke Riddell: you seem to have found a bug in NATSORTFILES. I just uploaded a new version to FEX, please download it, try it, and let me know if it works as expected.

Answer 2

[Rik]

You can sort the output or dir any way you like. See below for an example.

filelist=dir('NUM_*.dat');
dates=[filelist.datenum];
[~,order]=sort(dates);
filelist=filelist(order);

[Isma_gp]

Can we sort the name field from dir?

[Stephen23]

This is a very fragile solution, because it assumes that the files themselves were written in the same order as the data was recorded: this will break easily, e.g. any time a file is edited, or if the files are copied from one folder to another.

"Can we sort the name field from dir?"

Yes. See my answer to know how.

[Bin Qi]

Thank you!