[Beginer] How can I extract column from several .txt and add them to another .txt ?

Hello,

I have several .txt datas but I am only interested by one column in each, so I would like to take this column from each and create a .txt file which contains each columns (the number of columns of this new .txt is the number of files treated).

The name of each file is a number (from 1 to the number of file into the folder).

I had the following idea: to do a loop that create an empty .txt, and add the column specified column form each file scanned until it covered all files, then save it. There are probably more igenious ways to do that.

I am struggling to create a loop that takes into account the order of files.

If you have any advices, I thank you a lot.

Greetings

Answer 1

[dpb]

If you do that (create the names with

fname=sprintf('BaseName%03d.txt',i);

that is), then it's a piece o' cake because dir() returns the list in sorted order...

ColumnToKeep=8;                                                 % what it says, use variables
d=dir(fullfile('yourWorkingDirectory','BaseName*.txt'));        % use appropriate wild card name
fnameNew=fullfile('yourWorkingDirectory','OutputNameOfChoice.txt');
nFiles=numel(d);                                                % how many are there?
for i=1:nFiles
  data=readmatrix(fullfile(d(i).folder,d(i).name));             % read the file in succession
  if i==1, OutData=zeros(size(data,1),nFiles);end               % preallocate the output data array
  OutData(:,i)=data(:,ColumnToKeep);                            % store ith file Nth column
end
writematrix(OutData,fnameNew)                                   % and write it out when done

See doc for the various functions used for all the details on each. In particular, writematrix is comma-delimited by default, you can change as desired.

Above assumes the data are all numeric and no header; there are other options to deal with that or if mix of data types, then read/writecell are probably more apropos. Attaching an example data file would let write specific code.

[Ilan Boulet]

Thanks a lot !!

I am trying to run your code but it write some errors because of the function OutData which is not define. I guess it is because there is a problem with 'data', I'm working on this, I will try to use 'textscan' or 'writecell' as you suggested over 'readmatrix' because of the data I have I guess (linked example of the datas I use).

[Ilan Boulet]

close all

ColumnToKeep=8; % what it says, use variables
D='path';
d=dir(fullfile(D,'0*.txt'));        % use appropriate wild card name
fnameNew=fullfile('D');
nFiles=numel(d);                                                % how many are there?
for i=1:nFiles
  data= importdata(fullfile(D,d(i).name))%,'%*s %*s %*s %*s %*s %*s %*s %s' ,'delimiter','','HeaderLines',1);             % read the file in succession
  if i==1, OutData=zeros(size(data,1),nFiles);
  end               % preallocate the output data array
  Cell2mat(:,ColumnToKeep)=data;                                 % store ith file Nth column
end
writecell(OutData,fnameNew)
disp(data)

Unrecognized function or variable 'OutData'.

This is where I stand with the modification to make to fit ma datas but it still doesn't work. I guess I don't understand yet what the 'fullfile(d(i).folder,d(i).name)' into the 'readmatrix' means : you tell it to read every file into the .folder ? How does it knows he has to read the .txt only for example ? It is because you wrote d to read only the .txt ?

As it didn't work I tried the 'importdata' which as far as I understood is more "polyvalent" to read files.

This still doesn't work but I feel like I am moving forward :)

I am sorry I am really trying to understand how its work..

[dpb]

Several issues in what you changed from example...

ColumnToKeep=8; % what it says, use variables

D='path';

Is your directory name REALLY the string "path" ??? That's pretty unusual naming convention; I've don't believe I've ever seen anybody with a directory named 'path' before.

This should be something like

D='C:\UserName\Working\Project Fred';

or whatever is shown by using

>> cd

C:\Users\Duane\Documents\MATLAB\Work

at the command prompt if that is where these files are located. It's the directory to your files string.

d=dir(fullfile(D,'0*.txt'));        % use appropriate wild card name

The above wild card string, even if the directory were correct will only return files whos names begin with the digit '0' and end with '.txt'. If your files were named "001.txt", etc., then that part is ok, but the path needs to be correct as well.

Test what you get at the command line with the various steps along the way -- or set a breakpoint in the debugger.

fnameNew=fullfile('D');

The above just creates a string "D" -- that's not a very good filename...and it doesn't use any path...that's also permissible, but is cleaner to ensure you know where you write stuff...

>> fnameNew=fullfile('D')
fnameNew =
    'D'
>> 

See??? That's not likely to be what you meant...and certainly not what I suggested -- which would be to use something like

D='yourWorkingDirectory';
d=dir(fullfile(D,'BaseName*.txt'));        % use appropriate wild card name
fnameNew=fullfile(D,'OutputNameOfChoice.txt');

If you fix up the names to match the actual working directory structure you have and create a reasonable file name for the output, then the rest of the code I wrote would likely work as is.

Read the doc for fullfile -- it is simply a helper function to build file names from the component pieces ensuring correct placing of the delimeters between levels of subdirectories--nothing more, nothing less. Just saves some manual string catenation code and perhaps adding an extra "\" (or leaving one out).

Again, if you would attach one of these files for actual illustration of content, could know for sure about reading it with whatever would be appropriate i/o function given the content type -- we're flying blind on that end.

[Ilan Boulet]

Hello,

Thank you for your answer (and your patience ..)

ColumnToKeep=8; % what it says, use variables
D='path';

I am sorry it wasn't clear, for sure I put 'path' as an example and it wasn't the real path I used, because the name of the path is a bit long.

I inserted to this comment two files which contains my datas, please tell me if you can't read it. As you can see, their names are 001 and 0070, which is the form 00i.

"If you fix up the names to match the actual working directory structure you have and create a reasonable file name for the output, then the rest of the code I wrote would likely work as is."

The current directory has to be the same than the one called by D ?

This is where I am:

close all
clear all

ColumnToKeep=7; % what it says, use variables
D='C:\Users\boulet\Documents\Recherche\2022\Mesures\Transport\Transistors série 3\Graphène Série 3\In Situ (3)\Z1-2\S3\Après deuxième recuit 12H57 P=5.4e-6 mbar 70 cycles';
d=dir(fullfile(D,'*.txt'));        % use appropriate wild card name
nFiles=numel(d);
fnameNew=fullfile(D,'OutputNameOfChoice.txt');% how many are there?

for i=1:nFiles
  data= readmatrix(fullfile(D,d(i).name),'delimiter','','HeaderLines',1);             % read the file in succession
  if i==1, OutData=zeros(size(data,1),nFiles);
  end               % preallocate the output data array
  OutData(:,ColumnToKeep)=data;                                 % store ith file Nth column

end
writecell(OutData,fnameNew);

I have this message when I'm trying to run it :

'??? Undefined function or method 'readmatrix' for input arguments of type 'char'.'

I tried to solve this by changing the 'readmatrix' function by 'textscan' because I read the 'readmatrix' function isn't working on "old" Matlab versions (I am using the R2007b) and I don't know if it is true.

Again thanks a lot you're helping me way more than I deserve, I'll be extremely grateful if we figure this out ..

[dpb]

"(I am using the R2007b)"

Wow...that IS a relic from the past, indeed! There are a lot of additions since that are of great ease of use improvements; particularly in the i/o arena.

That definitely changes things because none of the readXXXX routines were introduced until about R2016x and the table/readtable only preceded them by R2013b.

I don't have anything old installed so will have to poke at the files some and revert to long since put away toolsets that rarely use any more...

"Please stand by..."

[Ilan Boulet]

I can invest in the last version, I think I started to like spend some time on Matlab so it will be worth for sure :)

[dpb]

OutData(:,ColumnToKeep)=data;                                 % store ith file Nth column

There's an error in the above -- I put the keeping column in the wrong place...it's the data in the read in array to keep and write to the next column in the output array...

OutData(:,i)=data(:,ColumnToKeep);

is what was intended, of course.

Answer 2

[dpb]

See the MATLAB FAQ @FAQ#How_can_I_process_a_sequence_of_files? for a fairly lengthy exposition of several alternatives. I'm partial to the dir solution outlined there, but if you didn't name your files with leading zeros, then they won't sort in numeric order -- in which case the FEX submittal natsort will be helpful.

There's also a section in the MATLAB doc itself that's a shortened version under the title "Import or Export a Sequence of Files"

As common as is, it isn't all that easy to find the example and it isn't excessively long, for sure.

Neither of the above have been updated to reflect the new Strings class which allows one to build the numeric file name with the overloaded plus "+" operator in lieu of the explicit format string building route --

>> numfiles=4;
>> for k = 1:numfiles
    myfilename = "file" + k +".txt"
end
myfilename = 
"file1.txt"
myfilename = 
"file2.txt"
myfilename = 
"file3.txt"
myfilename = 
"file4.txt"
>> 

which is handy if you didn't use the leading zeros -- if did, then still need the '%02d' formatting string.

[Ilan Boulet]

Thanks a lot for your answer dpb.

I actually can name my files "file00i" directly when they are exporte from my experiment.

So if I understood well, since my files are ordered with the 00i at the end, I can create an already sorted list?

It would be amazing because what I want to do is a for-loop that extracts the eighth column of my files and put them into an empty table,but by keeping the good order of the files! Still struggling on that lol.

Thanks again

Answer 3

[dpb]

OK, the other may be of some use to others; given the revelation of old version of ML in use here I'll just post a different Answer entirely...same ideas hold, just use old

ColumnToKeep=8;                                                 % what it says, use variables
d=dir(fullfile('yourWorkingDirectory','BaseName*.txt'));        % use appropriate wild card name
fnameNew=fullfile('yourWorkingDirectory','OutputNameOfChoice.txt');
nFiles=numel(d);                                                % how many are there?
for i=1:nFiles
  data=dlmread(fullfile(d(i).folder,d(i).name),' ',1,ColumnToKeep-1);  % space-delimited, skip first row, read one column
  if i==1, OutData=zeros(size(data,1),nFiles);end               % preallocate the output data array
  OutData(:,i)=data;
end
dlmwrite(fnameNew,OutData,' ','precision','%.6E')               % and write it out when done

The above only reads the specified column; I note that in the given files column 8 is all zero which isn't that interesting.

Also NB: the row and column offsets in dlmread are zero-based--hence the "-1" in from the ColumntToKeep and the "1" for the row offset.

I'll also note you complained before that "function OutData" wasn't found -- it is NOT a function but the output array you're creating -- it is preallocated in the if..end construction the first pass thru the loop -- it's inside the loop so have the first file read from which to be able to determine the needed size from the length of the first file. This does presume all files are the same length as must be to go into a single double array -- otherwise would have to use a cell array instead that gets a lot more painful to write with older release.

Answer 4

[Ilan Boulet]

I bought the 2022 version and your previous code works perfectly !!

I'm so grateful, thanks a lot !!

[dpb]

No problem, teaching is part (most?) of the fun of participation here...

NB: The "ColumnToKeep" value can be a vector of columns, not just a single value...although the column addressing in the assignment to OutData would have to be updated to handle more than the one column it does at present.