Index exceeds matrix dimensions but only in parfor and not for

hey I am new to parallel computing. I have been trying to get my simulation run using parfor, however when i am using parfor i am getting the following error:

I have got this weird error which i don't know how to handle. I don't get any error when using for instead of parfor. any idea? Thanks! Naty

Error using parallel_function (line 598)

Index exceeds matrix dimensions.

Error stack:
run_ode.m at 62
run_sim.m at 9

[Matt J]

Try rebooting.

[Naty S]

Nope, that didn't help :/ I am attaching the loop, maybe it would help.

 for k=1:1:20
       engctrl.w=0.37+k*0.1;
  for i=1:1:n;
       xd0=x(i);
       parfor j=1:1:n
           y0=y(j);
           ret=run_sim(slip, 'const_terrain', ctrl,engctrl, y0, xd0, 100, 30);

      valid(j)=ret.valid;
      xd_st(j)=xd0;
      y_st(j)=y0;
      xd_f(j)=ret.x(length(ret.x(:,1)),3);
      y_f(j)=ret.x(length(ret.x(:,1)),2);

% end
     end
     valid_f(i,:)=valid';
     xd_stable(i,:)=xd_st';
    y_stable(i,:)=y_st';
    xd_final(i,:)=xd_f';
    y_final(i,:)=y_f';
 end

[Jan]

Most of all it would be helpful, if you explain us, what the line 62 or run_ode.m is.

[Naty S]

amm... The weird thing is that it's all working when i am using plain "for" loops . only when I am using parfor it claims that there is an error in the function. The error occurs at the line with xout, just before the last line.

background: It's solves ODE with event's functions. so it has finished solving a previous ODE and now finished solving another ODE ,the problem with the "for" loop combining the two vectors into a longer vector, the "stance to flight" is required to switch from polar to Cartesian axis. Thanks!

      options=odeset('Events', @liftoff_event, 'AbsTol', 1e-14, 'MaxStep', 0.01, 'Refine', 3);
      [t,x,te,ye,ie]=ode45(@stance_dynamics,[tstart,t_f],x0,options);
      nt=length(t);
      tout=[tout;t(2:nt)];
              for it1 = 2 : nt %Changing for Polar to Cartisen  Cord.
              tmp_x = stance_to_flight(x(it1, :));
              xout = [xout; tmp_x(1:4), x(it1, 1:2)];
              end

[Matthew Phillips]

I'm having this exact same problem. Pretty hard to debug given that you can't put breakpoints within a parfor loop.

Running

>> dbstop if error

halts on this block in parallel_function.m, line 579ff:

    try
        R = distributed_execution(...
            P, base, limit, W, k, NN, NN0, consume, supply, ...
            reduce, R, concatenator, ...
            consume_supply_reduce_concat, ...
            consume_bit, supply_bit, reduce_bit, concat_bit);
        break;
    catch err

...

              [err, possibleSourceFiles] = iMaybeTransformMissingSourceException(err, F);
                if isempty(possibleSourceFiles) || ~attachFilesAndRetryOnError
                    throwAsCaller(err);
                end

possibleSourceFiles is empty. parallel_function.m is called directly from the file containing my parfor loop so it is not possible for me to track down its inputs (in particular, the function handle 'consume') and the possible source of this error.

Answer 1

[F Poelwijk]

One common reason for this error to occur is when you try to insert entries to an array that is not pre-defined outside the parfor loop. A for loop has no problems with this (though the editor will suggest pre-allocation for speed), but a parfor loop does. Try pre-defining the variable, e.g. x = zeros(n,m).

[Matt J]

Not sure why parfor would care about pre-allocation more than for. In the following test script, I also get no errors

parfor i=1:10000

      x(i)=i;

end

[Neill Weiss]

This worked for me. I needed to preallocate an array of objects. Thanks F Poelwijk.

Answer 2

[Khalid]

Hi, I ran into the same problem with some code, and thought I'd explain the issue and fix in case someone else gets the same error for the same reason. Below is a minimal reproducible example of the problem. You will see the for and parfor loops below are identical. The for loop runs ok, but the parfor loop faults.

This is likely to come up if you have different running options and some variables are set in some of them but not in others. It appears the parfor error checking is thrown, it is probably checking that all possible execution paths through the parfor loop are acceptable, whereas in the example below they are not. Note that if you set doAssignment to true below both types of loop will run. The fix in my case is to do per-allocation for all possible running options through the parfor loop.

- Khalid.

doAssignment = false;
if doAssignment 
  a = zeros(2,1);
end

for i = 1:2
  if ~doAssignment
      disp('run here');
  else
      disp('dont run here');
      disp(a(i)); %#ok<*SAGROW,*UNRCH>
  end
end

parfor i = 1:2
  if ~doAssignment
      disp('run here');
  else
      disp('dont run here');
      disp(a(i));
  end
end

[Thomas Dolivo]

Thank you very much for your comment. I had a similar situation that I identified thanks to your hint.

foo = zeros(4,1);
parfor i = 1:8
    if i <= 4
        bar = foo(i); % this causes an error in parfor, but not in for
    else
        bar = foo(i-4);
    end
end

By unifying the access to the array, the error was fixed:

foo = zeros(4,1);
parfor i = 1:8
    if i <= 4
        j = i;
    else
        j = i-4;
    end
    bar = foo(j);
end

@Mathworks: The error message should be more clear. Why can’t it at least give a line number of where the “Index exceeds matrix dimensions” occured?

Answer 3

[Matt J]

There's too little information about the code being run to say much. My guess is that the loop iterations might not really be order-independent for some subtle reason. For example, maybe some of the functions you are calling within the parfor loop contain persistent or global variables -- something which is supposed to remember the outcome of previous iterations in a for loop. That would be illegal in a parfor, of course, since the sequence of iterations can't be predicted.

[Matt J]

One way to test this would be to replace the parfor loop with a normal for loop, but run the iterations in a crazy random order

 for i=randperm(1:Nsteps)
   ...
 end

and see if the result is affected.