How to load all data files of all subfolders ?

I save my data files in multiple subfolders of one folder. The one data folder (containing the subfolders) and the Matlab script to analyze the data are stored in the same directory. I'm stuggeling with reading the data from the subfolders.

First I tried to read only data from one folder. It worked fine:

path = 'C:\Users\Marcus\Documents\ETH\Matlab\Calculation_of_Z2S_and_Standard_Deviation\data\'; % path to one subfolder where all data is stored
liste = dir(strcat(path,'*.txt')); % creates a string with path and adds '*.txt'
files = {liste.name};
for k = 1:numel(files);
  z{k} = load(fullfile(path,files{k})); 
  p(k) = STDfun(z{k}); % function STDfun analysis my data
end

Then I tried to modify this code to load also the data in the subfolders. As it was recommended in other posts I used genpath, but it did not work:

originalpath = 'C:\Users\Marcus\Documents\ETH\Matlab\Calculation_of_Z2S_and_Standard_Deviation\data';
foldername = fullfile(originalpath);
pth = genpath(foldername);
path = regexp([pth ';'],'(.*?);','tokens');
liste = cellfun(@(c)[c '*.txt'],path,'uni',false);
files = {liste.name};
for k = 1:numel(files);
  z{k} = load(fullfile(path,files{k})); 
  p(k) = STDfun(z{k});
end

pth shows all subfolders just separated by ';'. I think there is some problem with the format afterwards. The error I get is:

Struct contents reference from a non-struct array object. Error in main_20160810_Minimalbeispiel (line 7) files = {liste.name};

Thanks for any advice!

Answer 1

[Adithya Addanki]

Hi Holger,

I think the variable "path" in the script above has some issues as the last entry is empty.

I was thinking about the usecase and edited the script like below:

originalpath = 'C:\Trials\testfilepath';
foldername = fullfile(originalpath);
pth = genpath(foldername);
pathTest = regexp([pth ';'],'(.*?);','tokens');

for k = 1:size(pathTest,2)-1
   list{k} = dir([pathTest{1,k}{:} '\*.txt']);
end

for i = 1:numel(list)
    if(~isempty(list{i}))
        temp = list{i};
        for j = 1: numel(temp)
            disp(fullfile(temp(j).folder,temp(j).name))
        end
    end
end

PS: It is not suggested to overload in-built functions. The script above is just an example to give a rough idea, instead you may even create a separate function that can be called as required iteratively when the list is found to be an array of file names etc.

The general workflow is given here:

http://www.mathworks.com/matlabcentral/answers/92761-how-can-i-recursively-process-files-in-subdirectories-using-matlab

I hope this helps.

Thank you,

Adithya

Answer 2

[Image Analyst]

Holger, do NOT use path as the name of your variable - you'll overwrite the prior, existing path and that could really mess things up. path is a built-in variable that you should only mess with via the "set path" button on the home tab of the toolribbon, unless you really know what you're doing.

To split your path into separate paths (get rid of semicolons), simply use strsplit:

pth = genpath(foldername);
pth = strsplit(pth, ';') % Separate into individual folders.

pth is now a cell array. To get the kth folder, use braces:

thisFolder = pth{k}; % Extract kth folder from a cell into a regular string character array.