What happens when a parallel worker finishes early?

Hi everyone,

I have a benchmark code running where I test the time it takes from start to finish. I noticed that the workers are not synchronized. What will happen when one worker finishes early? This is a parfor loop of 20 workers on a 20-core server.

Good day!

Answer 1

[Edric Ellis]

When MATLAB runs a parfor loop, it attempts to split the iterations of the loop across the workers to keep them all busy simultaneously. However, this is not always possible (especially if you have only a very small number of loop iterations, and the iterations take different amounts of time). So, towards the end of the parfor loop, some of the workers will be idle waiting for the last workers to finish.

[Walter Roberson]

To expand slightly: in situations where the number of tasks exceeds the number of workers, then as one worker completes a task it will be allocated new tasks if any are remaining.

[Tyler Warner]

So what you are saying is that MATLAB will continue to work the current task for each worker individually, even while some stand idle? I understand that this may be a very complicated situation for Mathworks to implement, but why not provide a solution to get them to continue to work?

[Edric Ellis]

To expand on my answer: we don't simply divide up the parfor loop iterations across the workers - we split the iterations in differing sized groups such that there are (generally) around 3 "intervals" (groups of loop iterations) per worker. Subsequent intervals get dispatched to workers as they complete operating on their intervals - this is a relatively simple but effective means of load-balancing. We don't currently provide any way for you to tailor this scheduling, and you're right we don't try and subdivide late-finishing intervals to see if we can get other workers to operate on them.

The current scheme provides a trade-off between communication overhead and late-finishing. There are always going to be pathological cases where no scheduling system can work well. Consider the following silly example:

parpool('local', 3);
parfor idx = 1:3
    if idx == 3
        pause(100);
    end
    out(idx) = rand();
end

There is no possible scheduling that we could use for this case to keep all the workers busy at the same time.

If you want full control over the parallel scheduling, you can use parfeval.

[Tyler Warner]

Thank you for the assistance!