How to filter and arrange text in a file using textscan

Hello, I have a Log.txt file with data;

ITEM: TIMESTEP
1000
ITEM: NUMBER OF ATOMS
113
ITEM: BOX BOUNDS mm mm mm
-1.1 1.1
-1.00019 1.9
-0.75007 0.75007
id  type  x  y  z
1  1  0.450082  1.76687  -0.101338
2  1  -0.0949301  1.76202  -0.0205438
3  1  0.0403266  1.78975  0.0742209
4  1  0.195614  1.75958  0.0106906
5  1  0.227547  1.84632  -0.098714
1  1  0.144155  1.76642  0.108974
2  1  0.288044  1.85155  0.0387214
3  1  0.370195  1.81785  -0.10084
4  1  0.383223  1.78207  0.0387085
5  1  -0.285847  1.72996  -0.00240017
1  1  0.450142  1.77285  0.117999
2  1  -0.318942  1.74455  -0.0651567
3  1  -0.198368  1.76376  0.0788363
4  1  0.255091  1.74895  0.0369646
5  1  -0.128421  1.72431  -0.113898
1  1  0.486869  1.83805  0.0239884
2  1  0.0405222  1.80871  -0.10944
3  1  -0.359026  1.75984  -0.00066415
4  1  -0.436089  1.73765  -0.0688759
5  1  -0.15772  1.81532  -0.0857257

I wrote a code as given below to skip the first 8 lines and subsequently read the entire text and in the process leaving non-numbers.

fid = fopen('Log.txt','r');
%Advance 8 lines:
linesToSkip = 8;
for ii = 1:linesToSkip-1
    fgetl(fid);
end
%Process all remaining lines
tline =  fgetl(fid);
To_Excel_Data = []; %You should allocate if you know how large your data is
while ~feof(fid)
    tline = fgetl(fid);
    %Getting rid of non-numbers
    tline = regexprep(tline,'[^0-9\s+-.eE]','');
    To_Excel_Data = [To_Excel_Data; str2num(tline)];
end
fclose(fid);

The Output of the code is;

1  1  0.450082  1.76687  -0.101338
2  1  -0.0949301  1.76202  -0.0205438
3  1  0.0403266  1.78975  0.0742209
4  1  0.195614  1.75958  0.0106906
5  1  0.227547  1.84632  -0.098714
1  1  0.144155  1.76642  0.108974
2  1  0.288044  1.85155  0.0387214
3  1  0.370195  1.81785  -0.10084
4  1  0.383223  1.78207  0.0387085
5  1  -0.285847  1.72996  -0.00240017
1  1  0.450142  1.77285  0.117999
2  1  -0.318942  1.74455  -0.0651567
3  1  -0.198368  1.76376  0.0788363
4  1  0.255091  1.74895  0.0369646
5  1  -0.128421  1.72431  -0.113898
1  1  0.486869  1.83805  0.0239884
2  1  0.0405222  1.80871  -0.10944
3  1  -0.359026  1.75984  -0.00066415
4  1  -0.436089  1.73765  -0.0688759
5  1  -0.15772  1.81532  -0.0857257

Which is well and good. Now I am looking for help to extent the code with textscan (or an alternative) so that I can filter and group the data according to the identity of the first element in column 1 so that the output is;

1  1  0.450082  1.76687  -0.101338
1  1  0.144155  1.76642  0.108974
1  1  0.450142  1.77285  0.117999
1  1  0.486869  1.83805  0.0239884
        
        
2  1  -0.0949301  1.76202  -0.0205438
2  1  0.288044  1.85155  0.0387214
2  1  -0.318942  1.74455  -0.0651567
2  1  0.0405222  1.80871  -0.10944
        
        
3  1  0.0403266  1.78975  0.0742209
3  1  0.370195  1.81785  -0.10084
3  1  -0.198368  1.76376  0.0788363
3  1  -0.359026  1.75984  -0.00066415
        
        
4  1  0.195614  1.75958  0.0106906
4  1  0.383223  1.78207  0.0387085
4  1  0.255091  1.74895  0.0369646
4  1  -0.436089  1.73765  -0.0688759
        
        
5  1  0.227547  1.84632  -0.098714
5  1  -0.285847  1.72996  -0.00240017
5  1  -0.128421  1.72431  -0.113898
5  1  -0.15772  1.81532  -0.0857257

I look forward to your help. Thank you.

Answer 1

[KSSV]

fid = fopen('LOG.txt','r') ;
S = textscan(fid,'%f','delimiter','\n','HeaderLines',9) ;
fclose(fid) ;
data = reshape(S{1},5,[])' ;
% seperate the data 
[c,ia,ib] = unique(data(:,1)) ;
for i = 1:length(c)
    idx = data(:,1)==c(i) ;
    iwant(:,:,i) = data(idx,:) ;
end

[Theo Score]

Thank you so much KSSV, You saved the day a lot.

Answer 2

[Jan]

To sort the array using the first column only:

[dummy, Index] = sort(To_Excel_Data(:, 1));
To_Excel_Data  = To_Excel_Data(Index, :);

[Theo Score]

Thank you so much Jan. The data is now arranged as I want. Just a small glitch. This is my overall code;

files = dir ('*.conveyor');
fileout = 'Merged.txt';
fout = fopen (fileout, 'w');
for cntfiles = 1:length(files)
    fin = fopen (files(cntfiles).name);
    linecount =0;
    while ~feof(fin)
        linecount = linecount + 1;
        linetext = fgetl(fin);
        if linecount >= 10 % Number of header lines being skipped
            fprintf (fout, '%s %d\n', linetext, cntfiles);
        end
    end
    fclose(fin);
end
fid = fopen('Merged.txt','r') ;
S = textscan(fid,'%f','delimiter','\n') ;
fclose(fid) ;
data = reshape(S{1},19,[])' ;
[dummy, Index] = sort (data(:,1));
data = data (Index,:);
% seperate the data 
output = 'Results.xlsx';
xlswrite (output, data);

The small problem I have is that the data files are being read randomly and merged randomly. These are time series results so I want the data to be read sequentially and merged in the order. This is to say I need 1000.conveyor, 2000.conveyor, 3000.conveyor,..., (n).conveyor and the data appended in that sequence.

Thank you again.

Answer 3

[John BG]

Deside

the command sort sorts your data the way you are asking for

format short
sort(To_Excel_Data,1)
  ans =
      1.0000    1.0000   -0.4361    1.7243   -0.1139
      1.0000    1.0000   -0.3590    1.7300   -0.1094
      1.0000    1.0000   -0.3189    1.7376   -0.1013
      1.0000    1.0000   -0.2858    1.7446   -0.1008
      2.0000    1.0000   -0.1984    1.7490   -0.0987
      2.0000    1.0000   -0.1577    1.7596   -0.0857
      2.0000    1.0000   -0.1284    1.7598   -0.0689
      2.0000    1.0000   -0.0949    1.7620   -0.0652
      3.0000    1.0000    0.0403    1.7638   -0.0205
      3.0000    1.0000    0.0405    1.7664   -0.0024
      3.0000    1.0000    0.1442    1.7669   -0.0007
      3.0000    1.0000    0.1956    1.7729    0.0107
      4.0000    1.0000    0.2275    1.7821    0.0240
      4.0000    1.0000    0.2551    1.7898    0.0370
      4.0000    1.0000    0.2880    1.8087    0.0387
      4.0000    1.0000    0.3702    1.8153    0.0387
      5.0000    1.0000    0.3832    1.8179    0.0742
      5.0000    1.0000    0.4501    1.8381    0.0788
      5.0000    1.0000    0.4501    1.8463    0.1090
      5.0000    1.0000    0.4869    1.8516    0.1180

do you really need the rows grouped in fours?

if you find these lines useful would you please mark my answer as Accepted Answer?

To any other reader, if you find this answer of any help please click on the thumbs-up vote link,

thanks in advance for time and attention

John BG

<mailto:[e-mail] [e-mail]>

[Theo Score]

John, thank you so much, however this seems to rearrange the data per column of which this data has to be treated per row. So the data will be mixed up.