Using a system with multiple gpus and multiple users, how can we share resources?

I am trying use a system with a Tesla K80 which has multiple GPU devices (8). How can I effectively share these resources without affecting other peoples work? I am currently selecting a GPU device according to available memory. Unfortunately, this is not foolproof. Several devices show that memory available is 'NaN'. Any advice on how to implement it properly would be appreciated!

Extra Info: Windows Multipoint Server Accessed via remote desktop

Answer 1

[Joss Knight]

This is difficult to answer fully without a lot more information about your system and environment. Probably the best way to deal with multiple users is to manage them via an MDCS cluster which people can connect to open pools or send batch jobs. The administrator can then make sure there is one worker per GPU and each worker has selected a specific GPU on startup. Another way would be to use nvidia-smi to put the devices in EXCLUSIVE_PROCESS mode. You can find some answers to similar questions here:

https://uk.mathworks.com/matlabcentral/answers/351824-selecting-specific-gpus-for-parpool https://uk.mathworks.com/matlabcentral/answers/346023-limit-total-gpu-memory-usage