Computing on multiple notes with multiple cores

Hello,

I have a code with the following structure:

for ... % Split between multiple nodes
        parfor ... % Split betwenn mutliple cores
            ...
        end
end

Is it possible to tell MATLAB, that it has to split the for-Loop between the nodes? Or does that MATLAB automatically if I use parfor 2 times? I.e.

parfor ... % Split between multiple nodes
        parfor ... % Split betwenn mutliple cores
            ...
        end
end

Answer 1

[Matt J]

The problem is, that the outer loop has 10 repetitions and the inner loop 50 repetitions

Combine the loops into one, i.e., instead of

for i=1:10% Split between multiple nodes
        parfor j=1:50 % Split betwenn mutliple cores
            ...
        end
end

do instead,

parfor k=1:10*50
    
    [i,j]=ind2sub([10,50],k);
    
    ...
    
end

[Paul Muster]

I have to check, if I have to change my used data structure for that. But thank you, that's an option.

[Matt J]

You're welcome, but if you find that it works (it should), please Accept-click the answer.

Answer 2

[Harald]

Hi Paul,

you can only have one level of parallelization, thus no nested parfor statements within a single script / function.

If the parallel pool that runs the parfor has cores on multiple nodes, they will automatically be used. While there might be reasons to parallelize the inner loop, I typically first try to parallelize the outer loop.

Best wishes,

Harald

[Paul Muster]

Hello Harald,

thank you for your answer.

The problem is, that the outer loop has 10 repetitions and the inner loop 50 repetitions. If I use parfor for the outer loop, I can only use 10 workers, no?

Paul

[Harald]

Hi Paul,

that's correct. With a reasonably long computation time, the additional overhead of parallelizing the inner loop should become negligible anyway.

I hope that the answer has overall addressed your question. If not, please let me know.

Best wishes,

Harald

[Paul Muster]

Hello Harald,

I don't know if I understood your answer correctly.

You meant above that I can parallelize either the outer or the inner for-loop.

for i=1:10
    for j=1:50
        ...
    end
end

Let's assume I have 50 workers available (calculation on a computing cluster) and the runs of the inner for loop all take about the same amount of time. Then it makes sense to use parfor for the inner loop. Because then all 50 workers can be used. If I use parfor for the outer loop, only 10 workers can be used, right?

So the only solution would be to combine the two for loops into one, as described by Matt J?

[Harald]

Hi Paul,

in many cases, there may not be a problem of parallelizing the inner loop as you initially suggested.

Matt J has offered a nice alternative to thinking about which loop to parallelize.

My understanding is that your main question was if your code was going to use multiple cores on multiple nodes, and the answer to that was: If the parallel pool that runs the parfor has cores on multiple nodes, they will automatically be used.

Please let me know if I can be of further help.

Best wishes,

Harald

[Paul Muster]

Ok, thank you very much!