Run-Time of Parfor becomes unbelievably lengthy sometimes

I have run my program multiple times using Parfor with profiling in Matlab-R2018a and there is no run-time error of the program. In most of times, the run-time is around 3000 seconds with the majority of run-time spent in "java.util.concurrent.LinkedBlockingQueue(Java method)".

But sometimes, the run-time became unbelievably lengthy up to 44370 seconds while the majority of which is also spent in "java.util.concurrent.LinkedBlockingQueue(Java method)".

As I have read a relevant MATLAB Answer in https://uk.mathworks.com/matlabcentral/answers/41353-parfor-takes-a-lot-of-time-to-run, the item "java.util.concurrent.LinkedBlockingQueue(Java method)" in profiler does not give any clue about where the workers are spending time, hence I guess there might be some unexpected internal run-time behaviours of Parfor (parallel computing toolbox) in Matlab-R2018a that could happen sometimes depending on the state of device.

I got such guess based on the thread posted by Meng-Yuan Huang in https://groups.google.com/forum/#!topic/comp.soft-sys.matlab/HYqR91wiWyk to the above MATLAB Answer in the first hyperlink, where it said a specific strange behaviour of Parfor it had encountered in Matlab-R2016b was solved by restarting Matlab.

[Andrea Stevanato]

I have this problem too, more than I expected. Are you solved this issue?

[OCDER]

Is it faster to run this in a regular for loop? Try to isolate if this is a parfor issue or a data size issue or a slow code issue. Remove any parfor and repeat the iteration that takes 44000 seconds (or use a smaller dataset so you don't have to wait that long).

Here's @Andrea's thread that's related to this issue, but no resolution yet.

https://www.mathworks.com/matlabcentral/answers/410248-parfor-hangs-during-execution

[Yijun Li]

@Andrea I have not solved this problem yet.

@OCDER Actually, using normal for loop is much longer than 3000 seconds (normal parfor behavior) but is not as crazy as 44370 seconds...

[OCDER]

Is your code accessing or writing a common file in the hard drive? The common issue that is unique to parallel computing is deadlock https://en.wikipedia.org/wiki/Deadlock. Or do you have some sort of random number generator, in which a worst-case scenario is triggered leading to a very long computing time?

[Andrea Stevanato]

Maybe, and i tell maybe, I have solved the problem. My code was something like this

for i = 1:10000
  parfor i = 1:N
    % something
  end
end

I change my code in something like this:

i = 1;
while i < 10000
  [result, i] = function(i, result);
end

function [result, i] = (i, result)

parfor i = 1:N
  %something
end  

if ~mod(i,3)
    return;
end
[result, i] = function(i+1, result);
end

I'd observed that the hangs was after ~5 iterations, so every each 3 iterations I'll return back and restart all like the first time but remembering previous result for start again where i left. This due I'd observed that the first ~5 iterations works fine and the hangs was coming after that.

[Andrea Stevanato]

I was wrong, why matlab have this behaviour???

[OCDER]

To resolve yours or Yijun's slow parfor issue, someone has to provide EVERY code required to recreate the stalling behavior. So if someone in the forum downloads the .m files (or zip file) and runs your main function, it should repeat exactly what you do right before the program stalls.

>> myMainFcn %your main function that'll replicate EXACTLY what you do to get it stall

We need something to work with other than pseudocodes. Even better is if you can provide a simple example script that still replicates the stalling problem without all the extra lines of codes.