"Maximum variable size allowed on the device is exceeded." when using predict on GPU

Hello there,

I have a neural net and want to do a prediction using predict. When I do that I get the error

Error using dlnetwork/predict
Execution failed during layer(s) 'fc1'.

Caused by:
Error using parallel.internal.gpu.mtimesAdd
Maximum variable size allowed on the device is exceeded.

Afaik it's because on GPU Matlab only supports 2^32 elements in one array. Whats the best way to circumvent this?

I know I can disable the GPU completely by setting CUDA_VISIBLE_DEVICES to -1, but I still want to use it in other functions.

The data I pass into predict is a normal array and not a gpuarray.

Is there a way to force predict to use CPU?

Cheers

Answer 1

[Joss Knight]

Is there a way to force predict to use CPU?

Yes. Pass data in as an in-memory array. If your data is on the device, you may need to call gather.

z = predict(net, gather(x));

If this does not work it will be because some of your network weights are on the GPU. So you may need to call gather on the network as well:

net = dlupdate(@gather, net);
net.State = dlupdate(@gather, net.State);

It's much more likely, however, that you're just passing too much data into your network. If you are trying to call predict on a large dataset, consider using minibatchpredict instead, to process the data in batches.

[hanspeter]

Thanks, that's it. I already used gather(x), but didn't think of also gathering the weights

Answer 2

[Shivam Lahoti]

Hi hanspeter,

I understand you are encountering a “Maximum variable size allowed on the device is exceeded" error when using predict on a GPU.

As you mentioned, the current maximum number of elements allowed in a gpuArray is intmax('int32'), or 2^31. This is because CUBLAS passes array lengths as int (int32_T). This is a known limitation and is mentioned in the documentation link below:

https://www.mathworks.com/help/parallel-computing/gpuarray.html#mw_6d599a79-d350-4ffd-8b2c-0b4b353f858b

Moreover, none of the following can exceed intmax('int32'):

• The number of elements of a dense array.
• The number of nonzero elements of a sparse array.
• The size in any given dimension. For example, zeros(0,3e9,'gpuArray') is not allowed.

I hope this helps you understand the issue.

Regards,

Shivam