Parallel computing: how best to retrieve outputs as jobs complete?

I am doing the following: run a set of jobs on a cluster, collect their outputs, process, [downstream code].
My question is how to improve my implementation of the "collect their outputs" step. The only way I've found to do this involves waiting and periodically checking for completed jobs. Is there a better way? Can jobs signal their completion and trigger processing of their output?
My simplified code is below. Note that it uses an LSF scheduler instead of the Matlab Job Scheduler (MJS), which means that some of the method names are different (.destroy instead of .delete, .getAllOutputArguments instead of .fetchOutputs).
MATLAB code
%construct & submit jobs
jobArray{i} = cell(1,numJobs)
for i=1:numJobs
jobArray{i} = createJob(schedulerObject);
createTask(jobArray{i},@somFunct,numOutputs,someInputs);
submit(jobArray{i});
end
%collect outputs, delete jobs
outputs = cell(size(jobArray));
jobsDeleted = 0;
numJobs = length(jobArray);
jobsNotDone = ~cellfun('isempty',jobArray);
pause on
while jobsDeleted < numJobs
%wait to check for completed jobs
pause(someTime);
%check for completed jobs
for i = find(jobsNotDone(:))'
if strcmp(jobArray{i}.State,'finished')
%get output
thisOutput = jobArray{i}.getAllOutputArguments;
%process output
% [...]
%record output
output{i} = thisOutputProcessed;
%clean up
jobsNotDone(i) = false;
jobArray{i}.destroy;
jobsDeleted = jobsDeleted + 1;
elseif strcmp(jobArray{i}.State,'failed')
%print to error file
% [...]
jobsNotDone(i) = false;
jobsDeleted = jobsDeleted + 1;
end
end
end
pause off
Thank you.

답변 (0개)

카테고리

도움말 센터File Exchange에서 MATLAB Parallel Server에 대해 자세히 알아보기

질문:

2012년 10월 7일

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by