Import range of files based on file name

I have a folder containing around 300,000 files. I don't need to import all the files.

Problem: How can I import a range of files based on specific file name?

This is an extension on the problem I asked in another post, link: https://www.mathworks.com/matlabcentral/answers/587738-import-files-based-on-file-name#answer_488336

Problem example:

In the picture below I have a section of the files, which are all in the same folder. I only want to import the .data files. But I don't need all the .data files to be import only the last 5 of every serie.

Up until now I have the following code (thanks to Stephen Cobeldick):

% Select only .data file from the last time step of each simulation
Folder = uigetdir;
files = dir(fullfile(Folder,'*.data*'));
spl = regexp({files.name},'\.data\.','split','once');
spl = vertcat(spl{:});
vec = str2double(spl(:,2));
[~,idx] = sort(vec);
[~,idy,idz] = unique(spl(idx,1),'last');
out = {files(idx(idy)).name};
expData = cell(length(out),1);

for i = 1:length(out)
    fid = fopen(fullfile(Folder,out{i}),'r');
    %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
end

This code only extracts the last .data file of every serie, but as I explained I want to have the last 5 files of every serie.

Answer 1

[Cris LaPierre]

Acknowledging there are a lot of ways to do this, here's how I thought to do it. It borrows from the code you've shared.

% Select only .data file from the last time step of each simulation
Folder = "datafiles";
files = dir(fullfile(Folder,'*.data*'));
spl = regexp({files.name},'\.data\.','split','once')';

% Convert the cell to a table so that you can sort and group two different data types
tbl = splitvars(cell2table(spl),1,"NewVariableNames",["Name","Ext"]);
tbl.Ext = str2double(tbl.Ext);
% this gives you the unique experiments and the last extension in the series
exp = groupsummary(tbl,"Name","max","Ext"); 

for n = 1:height(exp)
    for i = 4:-1:0
        % Use the Name and Ext values to rebuild the file names 
        fid = fopen(fullfile(Folder,exp.Name(n)+".data." + string(exp.max_Ext(n)-i)))
        
        % Read all the data from the file
        dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    end
end

[Tessa Kol]

Thank you, that really helped.

[Cris LaPierre]

Of course, this assumes your files are sequential with no omissions, and that each series has at least 5 files.

[Tessa Kol]

I added lines of code:

clear
clc
rhoPart = 2500;
% Select only .data file from the last time step of each simulation
Folder = uigetdir;
files = dir(fullfile(Folder,'*.data*'));
spl = regexp({files.name},'\.data\.','split','once')';
% Convert the cell to a table so that you can sort and group two different data types
tbl = splitvars(cell2table(spl),1,"NewVariableNames",["Name","Ext"]);
tbl.Ext = str2double(tbl.Ext);
% this gives you the unique experiments and the last extension in the series
exp = groupsummary(tbl,"Name","max","Ext"); 
for n = 1:height(exp)
    for j = 4:-1:0
        % Use the Name and Ext values to rebuild the file names 
        fid = fopen(fullfile(Folder,exp.Name(n)+".data." + string(exp.max_Ext(n)-j)));
       %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    end
end
%% Reading the data
rhoPart = 2500;
dim1 = 9;
dim2 = 9;
Nstudy = dim1*dim2;
Ndatafiles = 5;
Ntot = Nstudy*Ndatafiles;

for i = 1:Ntot
    % Write only the x, y, and z components of the particles, particle radius,
    % z component+ particle radius and volume of the particle
    expData{i} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    % Write only the vx,vy,vz of the particles and magnitude
    velData = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
end

How do I now in what order the .data files are being imported to expData?

It seems to me that matlab imports the same file 405 times. I think it is because expData is in another loop then dataRead.

[Cris LaPierre]

Look at the line of code that creates the fid.

[Tessa Kol]

In that line you open from every serie the last 5 files. I tried to put the expData inside the loop, like below, but without succes.

for n = 1:height(exp)
    for j = 4:-1:0
        % Use the Name and Ext values to rebuild the file names 
        fid = fopen(fullfile(Folder,exp.Name(n)+".data." + string(exp.max_Ext(n)-j)));
       %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    expData{j,n} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    % Write only the vx,vy,vz of the particles and magnitude
    end
end

[Cris LaPierre]

You need to think through your code execution. For example, you can't access dataRead{3} if it hasn't yet been created.

[Tessa Kol]

I don't really understand what you are saying. Should I put something like this:

fid{j,n}

[Cris LaPierre]

I mean you are just trying things now without thinking what is happening. I also don't know what question you are wanting an answer to any more.

[Tessa Kol]

Yes you are right. I am used to read and write all the files instead of only a range of files. Below is my code how I read and write all the .data files and put every data file into a seperate cell.

files = dir(fullfile(uigetdir,'*.data*'));
[~,Index] = natsort({files.name});
files = files(Index);
expData = cell(length(files),1);
for i = 1:length(files)
    fid = fopen(fullfile(files(i).folder,files(i).name),'r');
    %% Reading the data
    % Read all the data from the file
    dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
    frewind(fid);
    % Write headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
    runData{i} = strsplit(fgetl(fid), ' ');
    % Write only the x, y, and z components of the particles, particle radius,
    % z component+ particle radius and volume of the particle
    expData{i} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
    % Write only the vx,vy,vz of the particles and magnitude
    velData{i} = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
    fclose('all');
end

That same procedure I want to accomplish with the last 5 data files of every serie. And that is what I am struggling with.

[Cris LaPierre]

I would expect you to be getting an error when you run that code.

[Tessa Kol]

I ran that whole code so many times and I didn't got an error. But how can I perform the same procedure for the last 5 data files?

[Cris LaPierre]

Ah, I misunderstood how the code ran.

How to modify it? Replace the code that opens every file with the code I shared already that opens the last 5 in every series.

rhoPart = 2500;
files = dir(fullfile(uigetdir,'*.data*'));
spl = regexp({files.name},'\.data\.','split','once')';

% Convert the cell to a table so that you can sort and group two different data types
tbl = splitvars(cell2table(spl),1,"NewVariableNames",["Name","Ext"]);
tbl.Ext = str2double(tbl.Ext);
% this gives you the unique experiments and the last extension in the series
exp = groupsummary(tbl,"Name","max","Ext"); 

c=1;
for n = 1:height(exp)
    for i = 4:-1:0
        % Use the Name and Ext values to rebuild the file names 
        fid = fopen(fullfile(Folder,exp.Name(n)+".data." + string(exp.max_Ext(n)-i)))
    
        %% Reading the data
        dataRead = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f %f','HeaderLines',1);
        frewind(fid);
        % Write headerline N, time, xmin, ymin, zmin, xmax, ymax, zmax
        runData{c} = strsplit(fgetl(fid), {' ','\t'});
        % Write only the x, y, and z components of the particles, particle radius,
        % z component+ particle radius and volume of the particle
        expData{c} = [dataRead{1}(:,1) dataRead{2}(:,1) dataRead{3}(:,1) dataRead{7}(:,1) dataRead{3}(:,1)+dataRead{7}(:,1) rhoPart*(4/3)*pi*(dataRead{7}(:,1).^3)];
        % Write only the vx,vy,vz of the particles and magnitude
        velData{c} = [dataRead{4}(:,1) dataRead{5}(:,1) dataRead{6}(:,1) sqrt(dataRead{4}(:,1).^2 + dataRead{5}(:,1).^2 + dataRead{6}(:,1).^2)];
        fclose(fid);
        c=c+1;
    end
end

[Tessa Kol]

Yes. Thank you so much for all the help!