Parpool worker distribution on HPC cluster in Windows Server 2012. How to choose/constraint the number of cores in each compute node?

조회 수: 1 (최근 30일)
In my University HPC cluster there are 32 computes nodes x 16 cores each, for a total of 512 cores, and each node has 64 GB RAM. So using the cluster at full capacity, my parallel (SPMD) code running in each worker should not use much more than 4 GB RAM, or the computations will be slowed down due to system page swapping, or pool will crash with "out of memory" error.
The cluster runs on Windows Server 2012, and I noticed that MATLAB allocate workers consecutively. That is, that the cores on the compute node N+1 are only assigned after all the cores in nodes 1, 2, ..., N, have been assigned.
My issue is that I am developing a program that will probably demand 8 GB per core, and I need to use as many cores as possible, so I was planning to call a parpool in a way that I will get only 8 cores from node 1, 8 cores from node 2, ..., and so, but I don't know how to do it without wasting resources.
Is there is any way of solving this in Windows?
  댓글 수: 1
Alvaro
Alvaro 2022년 12월 21일
This might depend on how your university cluster is setup, but you could try this at first:
Maybe elaborate a bit more onto how your university cluster is setup, are you sending batch jobs through some custom or specific scheduler? Are you using Matlab Parallel Server by any chance?

댓글을 달려면 로그인하십시오.

답변 (0개)

카테고리

Help CenterFile Exchange에서 Parallel Computing Fundamentals에 대해 자세히 알아보기

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by