Problem with "change directory" in a for loop

I'm having a problem with "cd" when looping through folders. Below is the problematic part of my script:

datadir='wherethedatais';

cd(datadir);

dirlist=dir('sub*');   % list of different subjects

for h=1:length(dirlist)
cd(dirlist(h).name)

Everything is fine until the 7th iteration of the loop when I get this error:

Error using cd
Cannot CD to sub0010 (Name is nonexistent or not a directory).

All the folders are named subXXXX where XXXX is 0001 to 0114.

Any ideas what might be causing this?

[Stephen23]

@Ahmed: to read/ write/access files or directories you do not need to use cd at all. In fact it slow way to access directories, and it makes debugging a pain because you can get stuck in unexpected directories when the code throws an error. It is prone to lots of error, like the one you have now.

You can make your code much simpler by using full file paths, which are supported by all relevant MATLAB functions, like dir, fopen, and all file reading/writing functions:

mypath = 'C:\MyFolder\MyExperiment'; % Absolute path
mypath = 'Subfolder'; % subfolder of the current directory
myfile = fullfile(mypath,'match*.txt');
dir(myfile)

Answer 1

[Thorsten]

Using cd just to read data is error prone. Because you have to ensure that after cd to change back to your original dir, otherwise the next dir will not be found. So if you first change to sub001, then the next change to sub002 will work from within sub001, i.e., tries to change to sub001/sub002 with respect to your original dir.

Instead of cd, use something like

filename = fullfile(dirlist(h).name, 'data.mat')); %assuming 'data.mat' is the name of your mat-file in the various subdirectories
load(filename, 'X'); % assuming 'X' is the variable you need from file data.mat

[Ahmed]

At the end of the for loop I just added the line:

cd ..

Because all I'm doing is going from the main directory "datadir" to a subdirectory then back, then to the next subdirectory, etc.

[Thorsten]

ok, then your code should work. Have to tried to change to sub0010 from the command line? At what position does sub0010 appear in the list of your directories? Does the error always occurs for the same directory? Does the error persist if you add a pause(1) after the cd ..? Have you considered following my advice to get rid of cd?

[Ahmed]

OK I figured it out. I considered following your advice but I'm not so experienced with MATLAB so I wanted to keep it fairly simple for me.

The problem was, there's an "if" clause in the for loop that ends with "continue" - it skipped the "cd .." line in iterations where the "if" clause was not true. So you were right, it ended up looking for "sub0010" in the "sub0009" directory. Thanks so much for your help!

[Guillaume]

Hence why both Thorsten and I recommend not using cd. Using cd in an interactive environment makes sense. For code, there's no reason to use interactive tools. Using full paths makes a lot more sense.

Answer 2

[Guillaume]

Possibly, sub0010 is not a directory but a file. You never actually check that you're cd'ing to a directory

for subdir = dir('sub*')'  %directly iterate over the list of subdirectories without using an index
    if subdir.isdir
       cd(subdir);
       %...
    end
end

Why do you need to cd into all these directories anyway? Why not just use read/write/list data from these directories using their full path?

[Ahmed]

sub0010 is definitely a directory. I need to cd into all these directories because they all have files with the same names and I want to loop through them. If I understood your question correctly?

[Guillaume]

You don't need to cd into any directory to read/write/list files, regardless of their name:

function filelist = getsubdirlist(root, filter)
    %getsubdirlist returns all files that match filter in root and all its subdirectories however deep
    %root: full path of root directory where to start search
    %filter: filter pattern to pass to dir command
    %filelist: a column cell array of file names (full path)
    matchedfiles = dir(fullfile(root, filter));
    filelist = fullfile(root, {matchedfiles.name})';
    allcontent = dir(root)';
    for subdir = allcontent([allcontent.isdir] & ~ismember({allcontent.name}, {'.', '..'}))
         filelist = [filelist; getsubdirlist(fullfile(root, subdir.name), filter)]; %#ok<AGROW> 
    end
end

usage example:

getsubdirlist(pwd, '*.m')