Difference between simple matlabpool parfor and batch parfor.

I see a huge difference between apparently similar approaches and would like to understand what's happening and why the discrepancy.

I just created this M-file

    % code
mywave.m
parfor i=1:10024
  A(i) = sin(i*2*pi/1024);
end

And tried to run it using 2 different approaches:

      %code
matlabpool open 100 
%connected to 100 labs.
tic
mywave 
toc

Elapsed time is 0.652927 seconds.

     %code
2. batch('mywave','matlabpool',100)

MaximumNumberOfWorkers : 101 MinimumNumberOfWorkers : 101 Elapsed time is 166.722475 seconds

My configuration is with 132 processors (workers) passing all the validatiions. connected to 100 labs.

With resource list parameter as : -l walltime=12:00:00 -l nodes=11:ppn=12

(i have a linux cluster with 12 nodes & 12 ppn , with shared file system)

Thanks in advance.

Answer 1

[Edric Ellis]

The amount of computation that your PARFOR loop contains is far too small to benefit from launching such a large matlabpool. Launching 100 workers and having them all set up communications etc. takes a long time - much longer than it takes for MATLAB to run the body of your FOR loop. You can only expect PARFOR to speed up your program when the amount of work to be done exceeds the overheads of setting up the workers and launching the parallel loop.

[Walter Roberson]

See http://www.mathworks.co.uk/matlabcentral/answers/44734-there-is-aproblem-in-parfor for a small mathematical analysis of why parfor can be slower.

[nah]

Thanks Edric and Walter for your answers. Yes, I understand the overhead could be more than the gain one could get from parfor (it's parallelization 101). But, my problem is more with distributing the calculation. I just have some 1000s of short calculations to do & been wondering how I could achieve it. Parfor is the easiet & obvious solution, but it does behave oddly. My question was more about the differences between the same matlabpool job (similar overheads) but having different execution times when one is submitted as a batch job. And the answers here do not yet address those differences.