optimize cluster configuration for virtual machine
조회 수: 2 (최근 30일)
이전 댓글 표시
I'm testing a parfor loop where each parallel iteration has to do some pretty heavy processing (~70 GB peak mem usage, ~3 h runtime without parfor). Eventually I need to run this on hundreds of datasets, so I want to ensure a reasonable cluster configuration. I'm running Matlab 2020a (Win 10) interactively on a configurable virtual machine (256 GB RAM with 32 virtual processors).
Here's what I do:
myClus=parcluster('local'); %from virtual machine perspective, all hardware is local
myClus.NumWorkers=3; %limited by RAM: 3*70 GB < 256 GB capacity
myClus.NumThreads=10; %10 threads (=virtual processors?) per worker: 3*10 < 32 available processors
parpool(myClus);
%parfor loop...
Is this a reasonable (or at least not terribly inefficient) setup in this situation? Specifically:
- is setting NumThreads = #virtual processors/NumWorkers (as I'm essentially doing) an acceptable strategy? I'm assuming a thread is equivalent to a virtual processor in my case, but I'm not entirely sure
- would increasing NumThreads be expected to lead to further improvements? Since I have the luxury of configuring my hardware, I could conceivably go for e.g. 64 processors and set NumThreads=20
- bonus points for other optimization suggestions
댓글 수: 0
답변 (1개)
Raymond Norris
2022년 1월 31일
When you run the code serially, do you see high core activity? If not, you might not see any appreciable difference increasing NumThreads. Depends on the work the parfor loops is running. Might be worth testing it by throttling NumThreads and seeing what performance improvements you get.
Have you consider at some point the need to scale your code to a cluster/cloud?
댓글 수: 2
Raymond Norris
2022년 1월 31일
I don't have a rule of thumb, but keep in mind, these are threads, not processes. By default, MATLAB has already been mapping the computation thread count to your physical core count all this time. If you haven't seen a degregation in your system, but fine to leave it as is. But as you say, testing might be the best confirmation.
참고 항목
카테고리
Help Center 및 File Exchange에서 Cluster Configuration에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!