Parfor and multithreading

Hi Konrad,

I writing regarding the answer you offered about distributed computing and multithreading in: http://www.mathworks.com/matlabcentral/answers/20060-parallel-matrix-multiplication-on-a-distributed-computing-system

I work with Matlab R2011b on a cluster with 16 nodes with each 2 quad core CPU. The MDCS is set up with 32 license: 2 workers per node. I would like to set up the each worker to use multithreading on each CPU.

I tried you solution. But when I change the maxNumCompThreads to 4, this does not change the number of core that work on the cluster.

Here an example of the code I am running.

matlabpool open N
pctRunOnAll maxNumCompThreads(4)
parfor a=1:M
    FunctionUsingMultithreading;
end
matlabpool close

Am I doing something wrong ? I there something to set up on the MDCS side to be able to take advantage of the multithreading ? The are setting up like:

bpsh 12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27 /opt/matlab/toolbox/distcomp/bin/startworker -clean -name worker1 -jobmanager p41matlab -jobmanagerhost p41cluster
bpsh 12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27 /opt/matlab/toolbox/distcomp/bin/startworker -clean -name worker2 -jobmanager p41matlab -jobmanagerhost p41cluster

Thanks a lot

Answer 1

[Jason Ross]

Are the CPUs really quad core, or are they dual-core with hyper-threading enabled? You'll likely need to look up the CPU specs to know.

If you have hyper-threading, it's likely this:

http://www.mathworks.com/matlabcentral/answers/34673-didn-t-recognize-full-workers-when-using-matlabpool

[Nicolas]

Thanks for the response.

I checked the CPUs. Each nodes have 2 CPU units each Quad-core AMD Opteron Processor 2378. These are real quad-core cpu for my understanding.

I redefined the cluster configuration with a single worker on each node and then changed the maxNumCompThreads ot 4 or 8 opening various size of matlapool.But, when monitoring, I still see only one single core on each node working.

When I am opening matlab directly on the node (not using MDCS) and I am running the same code without any pool/parfor. I can cleary monitor 8 cores working when executing the function I'm testing.

Any clues on what I am doing wrong ?

Does the name of the workers has to be different on each nodes ?

Thank you very much for you insight.

[Jason Ross]

Great. If you look in Konrad's post, there is some code to test that the setting has been applied. Do you get the results you expect when you run that code? (it's the spmd block where you open the matlabpool and run maxNumCompThreads).

[Nicolas]

Yes. The maxNumCompThread(N) return N when ran one more time.

The simple code I am using to test:

matlabpool open 2

spmd

maxNumCompThreads(4)

end

parfor a=1:2

NumThread(a) = maxNumCompThreads;

A = randn([256 256 256]);

for b=1:10

A = fftn(A);

end

end

matlabpool close

which return: NumThread = [ 4 4 ];

When monitoring the CPU, I still see only one core working.

I tried also to run the test with only one worker setup on the MDCS side and by opening one single matlabpool. No more success :(

[Jason Ross]

If you add the "Threads" counter to the performance monitor, do you see a change?

[Nicolas]

No. I also tried to use the createJob/createTask/submit/etc...

code with running the same function whichwith the maxNumCompTreads command at the beginning. No more success.

Could it be that the MDCS side on the cluster is wrongly settup ? Is there something to modify in the startworker script to enable the multithreading ?

[Jason Ross]

It sounds like you have it all done correctly. You might want to call support, perhaps there is something else in your environment that's preventing this from initializing as you would expect.

[Nicolas]

Thanks Jason. This is getting very confusing. Actually, I figured out that the 'test' code I was using (with fftn) does not support multithreading when using PCT/MDCS. If I use linear algebra operation instead, It turns to work perfectly with multi threading.

I figured out that, when in PCT/MDCS, FFT/FFTn on 1dim array is also multihtreaded but not with array with dim > 1.

Any insights on what happening with this FFT/FFTn functions ?