'parfor' Memory Usage Greater Than Expected

조회 수: 7 (최근 30일)
Jay
Jay 2014년 9월 5일
댓글: Jay 2014년 9월 12일
Just before the execution of a particular loop in my code the memory usage will be ~6GB.
If I use 'for' then the memory usage will increase by ~600MB during execution of a single loop and drop back down.
If I use 'parfor' with just 2 workers the memory usage will exceed my 32GB of RAM and 32GB of swap.
Here is the loop:
parfor n=1:length( loop_Unique_Handle_Object_Array )
result{n} = someFunction( shared_Data,...
loop_Unique_Handle_Object_Array(n) );
end
'shared_Data' is 100MB
Each instance of loop_Unique_Handle_Object_Array is 200MB
I dont understand why the amount of RAM required by 'parfor' is greater than the amount required by 'for' * numWorkers + Overhead. Is this a result of using handle objects?

채택된 답변

Matt J
Matt J 2014년 9월 5일
편집: Matt J 2014년 9월 5일
Is this a result of using handle objects?
Possibly. All data that a handle object points to will be cloned and transmitted to the workers. It's not clear how you would have measured the memory footprint of everything your handle object points to and concluded that it is 200MB. MATLAB doesn't have any direct tools that would let you measure this footprint (whos() will not be reliable, in particular).
An easy test would be to just do a simplified parfor which only broadcasts the handle objects
parfor n=1:length( loop_Unique_Handle_Object_Array )
result{n} = loop_Unique_Handle_Object_Array(n);
end
  댓글 수: 8
Matt J
Matt J 2014년 9월 9일
That sounds like a good idea. You could also try trimming the memory footprint of the anonymous functions using this FEX file,
and see if that's of any help.
Jay
Jay 2014년 9월 12일
Excellent, that looks real handy. I just ran into a different problem involving parfor and function handles that I think this may fix.

댓글을 달려면 로그인하십시오.

추가 답변 (0개)

카테고리

Help CenterFile Exchange에서 Parallel for-Loops (parfor)에 대해 자세히 알아보기

제품

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by