Maximizing the number of workers used in a parfor loop using the "batch" command

I have four cores, and would like to use all four of them in parallel. To accomplish this, I have attempted to use the "batch" command, combined with a parfor loop as follows. (The example is completely silly, I've tried to strip it down to the most basic components for clarity.)

I have a program called myTest.m as follows:

parfor ii=1:3;
    sillyProgram(ii);
end;
sillyProgram;

where sillyProgram is defined as follows (it's silly):

function sillyProgram(ii);
ctr = 0;
fid = fopen(['Silly' num2str(ii)],'a');
while 1 > 0;
    ctr = ctr + 1;
    fprintf(fid,'%i\n', ctr);
    pause(10);
end;

I then try to run myTest with the following command:

batch('myTest','matlabpool',3)

(I'm running R2013a, hence matlabpool instead of pool.)

The idea was: use three workers in the parfor loop, then exit out of the loop and use the fourth worker to run sillyProgram one more time. But for some baffling reason, the fourth job does not get run. And consequently I'm unable to get all 4 workers running simultaneously.

Am I making a stupid mistake? If not, and this is intended behavior. is there a more intelligent way to use batch, but not waste a a precious worker on the wrapper program, that really does nothing but spawn the real jobs?

Answer 1

[Thomas Ibbotson]

I think it might help to explain how batch with the 'matlabpool' argument works. This causes 1 worker to run 'myTest' as the client, and the other 3 workers are used to form the pool with that client worker. The client worker then runs the code in 'myTest' just as it would if you had run the script on your local machine.

This means that on the client worker the code containing the parfor runs and calls 'sillyProgram' within the parfor on each of the 3 other workers. Because 'sillyProgram' has an infinite loop it will never complete, and your client worker will wait forever for the parfor to finish, never reaching the second call to 'sillyProgram' outside the loop.

To answer your second question, you might want to look at the jobs and tasks API.

[Leo Simon]

Thanks very much, Thomas. The link to http://www.mathworks.com/help/distcomp/create-simple-independent-jobs.html#f3-10810, i.e., jobs and tasks API (I don't know how to work the buttons) looks like exactly what I need. I'll check and clarify.

[Leo Simon]

Thanks again, Thomas

The approach you recommended works very well, in case anybody else is interested the code below generates numCores independent jobs

numCores = 4;
parallel.defaultClusterProfile('local');
c   = parcluster();
j   = createJob(c);
for ii=1:numCores
    createTask(j,@<myFunct>,<myFunctOutputs>,{<myFunctInputs>});
end

The one problem I'm having is getting some kind of output feedback. Is there an option as there is in batch, to capture the diary? If so it's not well documented! If not, it seems like a glaring omission.