Create FOR loop to process and create datasets from all files found

Forgetting the need to append all data (which will be my ulimate goal), I need to process all files found to create x number of individual datasets. How do I run the following script with a FOR loop to create these datasets??

% sets base directory and lists all files stored within its subfolders

base_dir = 'Farm34Maize/';
[status, list] = system('dir /B /S TOA5_2436.flux*.dat');
result = textscan(list, '%s', 'delimiter', '\n');
filelist = result{1};

%%Read files
% Run FOR loop to read all .dat files
for i = 1:numel(filelist);
fid = fopen(filelist{i}, 'r');

%%Extract column headers
row = fgetl(fid{i}); % Skip header row
row = fgetl(fid{i}); % Extract second row that represents columns headers
cols = textscan(row, '%q', 'Delimiter', ','); % Identify column headers
cols = strtrim(cols{1}).';

%%Extract data and timestamp
row = fgetl(fid{i}); % Skip third row
row = fgetl(fid{i}); % Skip fourth row 
c = length(cols);
format = ['%q', repmat('%f', 1, c-1)];
data = textscan(fid, format, 'Delimiter', ',', 'TreatAsEmpty', {'"NAN"', '"INF"'}, 'CollectOutput', 1, 'headerlines', 1);

%%Split data and timestamp
data = [datenum(data{1}) data{2}]; % In order to work correctly, input file date formats must be 'yyyy-mm-dd HH:MM:SS' or 'yyyy/mm/dd HH:MM:SS PM' or 'y

fclose(fid{i});
end

Thanks in advance.

[Wieger Duursema]

Hi Bugguts99, I am wondering how you did manage to save the data within the FOR LOOP so that they do not overwrite. So, how did you do that? Thanks, Wieger

[bugguts99]

1 year later....I ended up writing and appending to a delimited text file :

dlmwrite('20110420.txt', data, 'delimiter', ',', '-append');

Answer 1

[Walter Roberson]

Don't use system() to get the file names, use dir()

fileinfo = dir('TOA5_2436.flux*.dat');
for K = 1 : numel(fileinfo);
  thisfile = fileinfo(K).name;
  fid = fopen(thisfile,'rt');
  % Extract column headers
  row = fgetl(fid); % Skip header row
  row = fgetl(fid); % Extract second row that represents columns headers
  %and so on
  fclose(fid)
 end

Your code was trying to index fid, which was not a cell array in your code. That would have crashed your code. The change to dir() is just better programming.

[bugguts99]

Thanks Walter, but before I accept your answer I have a couple of other queries related to your suggestions:

1. The reason that I use system() is to set a base directory with a number of levels of subfolders that contain the files that I need. This negates the need to reset the working folder within the matlab environment each time. Can I do the same thing by using dir()?

Also, I have been reading up on saving out variables within the FOR loop so that they do not overwrite, but I have had no satisfaction thus far. For instance I have 107 files from which I would like to create 107 datasets. Any suggestions??

[bugguts99]

Hi Walter, I have solved my 2nd query by writing each iteration to txt file (or the like)....so thanks anyhow. But if do have some suggestions re: dir() vs. system(dir), I would be very interested to hear these?

[Walter Roberson]

Your code never uses base_dir . The system() call is *not* going to make use of that variable.

%find what is in the base directory

basedirinfo = dir(base_dir);

%throw away the info on everything that is not a directory

basedirinfo = basedirinfo([basedirinfo.isdir]);

%throw away directories "." and ".."

basedirinfo = basedirinfo(~ismember({basedirinfo.name},{'.','..'}));

%everything left is a subdirectory. Loop over all of them

data = {}; %to hold the data

for didx = 1 : length(basedirinfo)

thisdir = basedirinfo(didx).name;

subspec = sprintf('%s\%s', thisdir, 'TOA5_2436.flux*.dat');

fileinfo = dir(subspec);

numfiles = length(fileinfo);

%extend the data cell array to hold more entries

oldlen = size(data,1);

data(oldlen+numfiles) = {};

%now process the files

for fidx = 1 : numfiles

thisfile = sprintf('%s\%s', thisdir, fileinfo(fidx).name);

fid = fopen(thisfile,'rt');

% extract column headers

row = fgetl(fid); %Skip header row

..... %copy appropriate code here

thisdata = textscan(fid, format, 'Delimiter', ',', 'TreatAsEmpty', {'"NAN"', '"INF"'}, 'CollectOutput', 1, 'headerlines', 1);

thisdata = [datenum(thisdata{1}) thisdata{2}]; % In order to work correctly, input file date formats must be 'yyyy-mm-dd HH:MM:SS' or 'yyyy/mm/dd HH:MM:SS PM' or 'y

data{oldlen+fidx} = thisdata;

fclose(fid);

end %of file loop within directory

end %of directory loop

%now data{1}, data{2}, and so on are the accumulated data sets

Answer 2

[Jan]

To learn more about the DIR command:

doc dir

There you find an example of how to scan a specified folder:

dir(fullfile(matlabroot, 'toolbox/matlab/audio/*.m')

Although this does not work recursively, it is much safer.