- Suppress the output arguments you don't need
- Save to a buffer and save that to disk when it becomes too large.
- Have each worker save its own output, not aggregating. You can put all the files together yourself afterwards
- Use sparse arrays
Info
이 질문은 마감되었습니다. 편집하거나 답변을 올리려면 질문을 다시 여십시오.
parfor loops with large outputs
조회 수: 2 (최근 30일)
이전 댓글 표시
Hi,
I'm using MATLAB's parallel toolbox on a shiny server (32 physical cores, 64 logical, 512GB RAM). All my parallelization needs can be classified as 'embarrassingly parallel', e.g.:
parfor i_par = 1:n_tot
[CalcOutputs_array(:,:,i_par), AdvancedCalcOutputs_array(i_par)] = CPU_DemandingFunction(InputsStruct_array(i_par));
end
In order to minimize overheads, i've already split the input such that each worker receives only what it needs.
The output, however, can become quite large (1GB-10GB), and data collection from the workers is vastly over-weighting the calculation itself.
I'm positive that this is the case since by lowering the amount of data outputted from the function (i have a flag that controls it) - the runtime improvement factor approaches the # of cores i'm using.
Can that be improved? Is there any remedy for data-collection overheads?
Thanks a lot!
Yanir
댓글 수: 1
José-Luis
2016년 6월 30일
This is a very general question.
You could:
etc...
답변 (0개)
이 질문은 마감되었습니다.
참고 항목
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!