GPU memory accessed even when code runs on CPU only (See attached code)

I am using 10 workers to submit a job. 1 worker is assigned to the GPU while the rest is assigned to the CPU cores. createTask() calls a function that checks for the current worker. Based on the current worker assigned to the task, that function will branch into the GPU or the CPU code.

The problem is that even when it branched into the CPU code, GPU memory is still pre-allocated. I found that initializing the GPU code with gpuArray.zeros() causes MATLAB to pre-allocate GPU memory even if the program never executes that line of code. This is a major issue because while the one GPU worker is running, the other 9 workers are grabbing GPU memory causing it to run out of memrory.

A quick fix was to initialize gpu arrays with gpuArray(zeros()) but this significantly slows down the GPU worker since data has to be pushed to the GPU each time. I have attached some simple code to reproduce the issue. I used nvidia-smi to look for GPU memory usage.

Does anyone know of a workaround the issue that I just described. Thank you. Geremy

[Joseph Cheng]

*cough cough* you forgot to attach the code.

Answer 1

[Edric Ellis]

I'm still investigating this problem, but as a workaround you can also use the 'like' syntax which avoids the problem:

z = zeros(10000, 'like', gpuArray)

[Geremy Michel]

Hi Edric - so I am going to give it a try but I am not sure I understand the difference with z = gpuArray(zeros(10000)).

[Sean de Wolski]

Look at gpuArray(zeros(10000)) as the interpreter sees it:

10000 % a number
zeros(10000) % build a double precision 10000x10000 zeros array
gpuArray(zeros(10000)) % move that 10000x10000 array to gpu

With gpuArray.zeros or zeros(...,'like') the intermediate double array never needs to be created on the cpu

[Edric Ellis]

Sean is correct that both gpuArray.zeros and zeros(..., 'like', gpuArray) are built directly on the GPU - the twist is that zeros(..., 'like', gpuArray) avoids the particular problem that you're seeing which is related to the existence of static gpuArray method calls (i.e. things which look like gpuArray.zeros or gpuArray.rand) inside your function (even if you're not actually calling them).

[Geremy Michel]

Edric - you nailed it! I understand the subtle difference between gpuArray.zeros() and zeros(...,'like') now. This did solve my issue. I don't see any gpu memory usage unless it is explicitly called by the function. And this works faster than gpuArray(zeros()) since the data is not created on the matlab workspace before being pushed the the GPU.

Sean - thanks for clarifying the interpreter, it all makes sense now.

Answer 2

[Geremy Michel]

Sorry, see attached!