Speed Up the for loop

 for j=1:120000
     disp(j);
     for i=1:7000000
         disp(i);
          if (strcmpi(x(i),y(j)))
           if (T1(i)==W1(j) && DOY2(i)==W2(j))
               if ((T6(i)-0.125<=W3(j))&&(W3(j)<=T6(i)+0.125))
             
             
   fprintf(fileID,'%s \t %6.2f \t %6.2f \t %6.2f \t %6.2f \t %6.4f \t %6.4f\t %6.2f \t %6.2f \t %6.4f \t %6.4f\t %6.2f \t %6.2f \t %6.2f \t %6.2f',cell2mat(y(j)),T1(i),DOY2(i),T6(i),T7(i),T8(i),T9(i),T10(i),W1(j),W2(j),W3(j),W4(j),W5(j),W6(j),W7(j));
 
  fprintf(fileID,'\n');   
  
                end  
           end        
          end
     end
 end

[Arun Kumar Singh]

This loop has to be run for i=1:7000000 into another loop for j=1:120000 times, How to Speed up this loop ?

Answer 1

[Jan]

Fmt1 = '%6.2f \t %6.4f \t %6.4f\t %6.2f \t %6.2f \t %6.2f \t %6.2f';
Fmt2 = '%s \t %6.2f \t %6.2f \t %6.2f \t %6.2f \t %6.4f \t %6.4f\t %6.2f \t %s\n';
fileID = fopen(FileName, 'W');  % Uppercase!
for j = 1:120000
   W3u  = W3(j) + 0.125;
   W3l  = W3(j) - 0.125;
   % [EDITED] the {j} belongs to y, not to x:
   m    = strcmpi(x, y{j}) & T1 == W1(j) & DOY2 == W2(j) & ...
            W3l <= T6 & T6 <= W3u;
   WStr = sprintf(Fmt1, W1(j), W2(j), W3(j), W4(j), W5(j), W6(j), W7(j));
   for i = 1:7000000
      if m(i)
         fprintf(fileID, Fmt2, ...
             y{j}, T1(i), DOY2(i), T6(i), T7(i), T8(i), T9(i), T10(i), WStr);
      end
   end
end
fclose(fileID);

AS far as I understand, y and x are cell strings. Then cell2mat(y(j)) is a slow version of y{j} .

[Arun Kumar Singh]

Matrix dimensions must agree.

Error in DID_BASE_Files_Faster (line 23)

m = strcmpi(x{j}, y) & T1 == W1(j) & DOY2 == W2(j) & W3l <= T6 & T6 <= W3u;

[Jan]

Because I do not have the input data, I cannot debug this for you. But you canuse debugger to find out, which dimensions are not matching:

dbstop if error   % let Matlab stop, when the error occurs

% Then:
size(y)   % same size as: strcmpi(x{j}, y)
size(T1)
size(DOY2)
size(W3l)
size(W3u)
size(T6)

What do you see? Which Matlab version are you using?

If you are using Matlab < R2016b, it might be the problem, that one of the arrays is transposed. Then insert a (:) or .' accordingly.

[Arun Kumar Singh]

Kindly find the attached data

[Arun Kumar Singh]

Here is full code

[Jan]

These are 2 text files, but what is their relation to the question? Please post the sizes of the input arguments, then it is veyr easy to solve the problem.

Maybe it would be useful to post the code you are using for the import of the files.

[Arun Kumar Singh]

Data i have sent you are just the samples of actual data, File_i is the data for (T1,T2...) and File_j is for (w1,w2..)

[Arun Kumar Singh]

File_i is as sample of All_Combined_Previous_1.txt and File_j is a sample of A_Swarm.txt and see the code then you will have better idea.

[Arun Kumar Singh]

I have attched above the code.

[Jan]

There was a typo in my code. Why do you avoid to answer my question for the sizes?

I've inserted the code in a function and cleaned it up. For the first 1000 elements my code needs 0.4 seconds instead of 24 seconds of the orignal version:

function DOY_Date_Jan(InFile1, InFile2, OutFile)

fid = fopen(InFile1, 'r');
M   = textscan(fid, '%s%f%f%f%f%f%f%f%f%f%f', 'collectoutput', true);
fclose(fid);

x    = M{1, 1};
M1   = M{1, 2};
T1   = M1(:, 1);
T2   = M1(:, 2);
T3   = M1(:, 3);
% T4   = M1(:, 4);
% T5   = M1(:, 5);
T6   = M1(:, 6);
T7   = M1(:, 7);
T8   = M1(:, 8);
T9   = M1(:, 9);
T10  = M1(:, 10);
date = datetime(T1, T2, T3);
DOY2 = day(date, 'dayofyear');

fid1 = fopen(InFile2, 'r');
datacell1 = textscan(fid1,'%s%f%f%f%f%f%f%f', 'collectoutput', true);
fclose(fid);

M2 = datacell1{1,1};
M3 = datacell1{1,2};
y  = M2(:, 1);
W1 = M3(:, 1);
W2 = M3(:, 2);
W3 = M3(:, 3);
W4 = M3(:, 4);
W5 = M3(:, 5);
W6 = M3(:, 6);
W7 = M3(:, 7);

Fmt1 = '%6.2f \t %6.4f \t %6.4f\t %6.2f \t %6.2f \t %6.2f \t %6.2f';
Fmt2 = '%s \t %6.2f \t %6.2f \t %6.2f \t %6.2f \t %6.4f \t %6.4f\t %6.2f \t %s\n';

fid = fopen(OutFile, 'W');  % Uppercase!

tic
yOld = '';
for j = 1:numel(y)
   % Compare all strings only, if y{j} differs from the previous one:
   if ~strcmp(y{j}, yOld)
      yOld = y{j};
      x_yj = strcmpi(x, y{j});
   end
   
   W3u  = W3(j) + 0.125;
   W3l  = W3(j) - 0.125;
   m    = x_yj & T1 == W1(j) & DOY2 == W2(j) & W3l <= T6 & T6 <= W3u;
   WStr = sprintf(Fmt1, W1(j), W2(j), W3(j), W4(j), W5(j), W6(j), W7(j));
   for i = find(m)
      fprintf(fid, Fmt2, ...
         y{j}, T1(i), DOY2(i), T6(i), T7(i), T8(i), T9(i), T10(i), WStr);
   end
end
toc

fclose(fid);
end

I call it with the provided files as:

DOY_Date_Jan('File_i.txt', 'File_j.txt', 'All_Previous_Swarm_A_2.xls')

Some hints:

• Do not forget to close files after opening them.
• Write one command per line, because this is easier to read.
• Hiding numbers in the names of variables is not useful. "T8" is not better than "M1(:, 8)".
• Avoid hard-coded limits of loops, but adjust them to the size of the data.
• Instead of inserting the file names in the code, provided them as inputs to a function.

[Arun Kumar Singh]

Thank you

[Arun Kumar Singh]

Just a simple question, why does it produce output like this ? Line number 17 and 18 in excel file

[Jan]

Sorry, the interface of the forum does not let my type, what I want to. Code formatting should be easy and intuitive, but the editor in this forum is a mess.

Change the line " for i = find(m) " to " for i = find(m).' "

You know, that you can use the debugger to fix such problems also?

[Arun Kumar Singh]

Thanks

Answer 2

[Bjorn Gustavsson]

A couple of suggestions:

0, check the time spent on different lines by running this script with smaller limits on j and i with the profiler running.

1, cut out the disp(i) and disp(j) entirely. In your version you'll have matlab printing some 8.4*10^11 numbers to the command-window. There's no way that your screen-pixels will survive that. (perhaps put something like:

if mod(j,1000)==0
  fprintf('%d:  %s\n',j,datestr(now,'HH:MM:SS'))
end

in to see that it turns over. You can modify the divisor to get more or less frequent updates

)

2, put the '\n' into the first fprintf call - should/might speed things up.

3, combine all conditions into one if-test, they are supposed to shortcut as soon as the TRUE/FALSE is determined - this might speed up or slow down, but ought to be worth a test.

4, instead of calling cell2mat in every fprintf-call do that separately first for all cells in y to create an array of strings, this might speed things up a bit.

HTH

[Arun Kumar Singh]

Thanks