Is there a utility like nvidia-smi within matlab to determine which gpus are in current use?

10 GPUs, 4 users. Want to avoid usage conflict.

Answer 1

[Walter Roberson]

No, there is no utility like that built into MATLAB.

My understanding is that the order of GPUs in the gpu device list depends upon the order in which the GPUs answered the probes at boot time, and so is potentially variable. However, if I understand correctly it is consistent during any particular boot session.

Therefore, if you can get the cooperation of all users, you could use simple resource management techniques such as creating a binary file with 10 entries in it that record whether a program is requesting to use the given GPU. If you were to write process IDs into the file, then you could have a management layer that checked whether the recorded process still existed and if not then freed up the entry.

[Sam Marshalik]

In combination with Walter's response, here is an idea.

Big disclaimer: Keep in mind this is not a perfect solution, as the information you get from nvidia-smi or gpuDevice will be stale after you retrieve it and may not be utilized in that momment in time but is being used by someone.

You can use the gpuDevice command in MATLAB to view GPU resources. You can use view memory usage to see if a GPU is being used in that momment (snippit from gpuDevice)

TotalMemory: 2147287040 (2.15 GB)

AvailableMemory: 1615209678 (1.62 GB)

You can also use the system command to call nvidia-smi via MATLAB. You can then use MATLAB text parsing to programmatically look at usage, get a handle to that GPU (have not played with this, but assuming it is possible), and use that GPU.

[RLeighton]

Using the second approach, based on temperature. Didn't think of first approach. Thanks

Answer 2

[Joss Knight]

gpuDeviceTable is the utility for listing your devices and their properties. You can also see all your devices in the Parallel menu on the Home toolstrip.

The menu entry will also tell you when the device was last used by your MATLAB, so you can check whether your code is successfully accessing the correct GPU.

The simplest way to control GPU access to multiple users is to put the devices into Exclusive Process compute mode, which means that only one user will be able to select each device at a time. You can use a command like "nvidia-smi -i 0 -c 3" to set the compute mode.

For a more sophisticated automated management of which user has access to which GPU, you will need a more sophisticated scheduling or resource management system. This can use environment variables like CUDA_VISIBLE_DEVICES to ensure that each user starting a process on a resource accesses a particular GPU only.