Matlab does not see CUDA GPU

Colleagues, hello!

After updating of the Nvidia driver 390.65 to 419.69 Matlab (R2018b) ceased to see video cards (Quadro K6000 + 3xTesla K40c).

The CUDA Toolkit updating from v.9.1 to v.10.1, NVIDIA_Nsight_Visual_Studio, FrameWork 4.7, Visual Studio Tools, VS community & msys2 installations, and Matlab replacement to R2019a did not help.

CPU: 2x Xeon E5-2697v2

Operating System: Windows 8.1

>> gpuDeviceCount

ans =

0

[Joss Knight]

Can nvidia-smi see your card?

(At Windows Command Prompt)

cd "C:\Program Files\NVIDIA Corporation\NVSMI"
nvidia-smi

[Alexander Sonin]

Joss, thank you very much for responding to my problem. I tried to enter your magic commands. Before their first entering, driver 419.69 (optimal for Tesla k40c) and CUDA Toolkit 10.1 was installed.

And after the entering of your commands, Windows saw three Tesla k40c video cards:

Matlab also saw these video cards:

>> gpuDeviceCount

ans =

3

But the very first Quadro K6000 video card (which is connected to monitors) was not visible neither in Command Prompt nor in Matlab.

Therefore, I installed the driver 419.67 (optimal for the Quadro K6000) on top.

As a result, Windows in the Command Prompt saw all 4 video cards:

but Matlab did not see one:

>> gpuDeviceCount

ans =

0

Removing drivers 419.67, 419.69 and a clean installation 419.67 did not change the result: in the Command Prompt video cards are visible, but not in Matlab.

[Alexander Sonin]

And finally, I again installed the newest recommended driver 419.69 from above (in the hope that at least 3 video cards will be visible). But as a result, Matlab did not see a no one video cards, while all 4 video cards were visible in Command Prompt.

[Joss Knight]

"infoROM is corrupted"? This does not sound good. Was MATLAB running when you did this? If so, see if you get this error without MATLAB running. If you do, this is definitely nothing to do with MATLAB. I suggest contacting NVIDIA support.

Otherwise the next step is to contact MathWorks support. Something is broken.

[Alexander Sonin]

What is "infoROM"? Is it ROM on Quadro K6000 GPU?

I get the warning " infoROM is corrupted" without MATLAB running.

NVIDIA Support bounce me to MathWorks Support

[Walter Roberson]

https://devtalk.nvidia.com/default/topic/850602/cuda-setup-and-installation/warning-message-of-nvidia-smi/

https://github.com/thomas-krenn/check_gpu_sensor_v1/blob/master/check_gpu_sensor.POD

https://bitcointalk.org/index.php?topic=1854250.1580

[Alexander Sonin]

Walter, thanks for the links.

At the

https://bitcointalk.org/index.php?topic=1854250.1580

recommended reflashing (reinstall? update?) the GPUs ROM (BIOS?) with NVFlash.

I am download a latest BIOS for Quadro K6000:

https://www.techpowerup.com/vgabios/164411/nvidia-quadrok6000-12288-131024

and NVIDIA NVFlash:

https://www.techpowerup.com/download/nvidia-nvflash/

but don’t know, how apply NVFlash to reflashing BIOS into GPU.

The NVFlash folder include nvflash.exe- and nvflash64.exe- files:

If to run any of them, command window will appear, similar to the Windows command prompt:

However, the entering of any command in this window is not possible.

Where can I get detailed instructions on the use of NVFlash?

[Walter Roberson]

Sorry that is outside my experience.

[Alexander Sonin]

I am reflashed BIOS by means of NVFlash.

Instructions for NVFlash usage:

https://forums.overclockers.ru/viewtopic.php?f=3&t=62747&start=7380

http://nastroyvse.ru/devices/comp/proshit-pereproshit-bios-videokarty-nvidia.html

Now GPU-Z for Quadro K6000 indicate a new latest BIOS (80.80.35.00.01) and nv-smi is without any WARNINGs:

But MATLAB still does not see the any video cards:

>> gpuDeviceCount

ans =

0

[Walter Roberson]

I notice in the GPU-Z that CUDA is not checked ??

[Alexander Sonin]

Yes, in GPU-Z the tick opposite CUDA is not present in both Quadro K6000 and Tesla K40c.

[Walter Roberson]

If it were my system I would probably try activating the tick and then rebooting and see if anything worked.

[Alexander Sonin]

I will try to do it again

[Alexander Sonin]

These boxes opposite CUDA and DirectCompute (for Tesla K40c) are not clickable and the checkboxes cannot be set:

On another computer (with Win10Pro x64, Titan (also GK110) and GTX780 (also GK110)) there are ticks in all the squares (and GPUs is visible in MATALAB):

As I understand it, this is the reason for the invisibility of video cards in MATLAB. Is it possible to activate CUDA in another way, for example from the command prompt or nv-smi?

[Walter Roberson]

Sorry, at present I have no ideas as to why you cannot enable CUDA on those devices. (Well, other than vague ideas of "firmware problems".)

Would you be able to experiment by swapping one of the cards between systems, to see whether the problem stays with the system or stays with the card ?

[Alexander Sonin]

Yes, I be able to experiment by swapping one of the cards between systems, but first I will try to install the old driver, like 39x.xx (optimal for CUDA Toolkit 9.1 or 9.2), which present earlier, at which GPU was visible. If CUDA returns and the GPUs become visible, does it make sense to change the GPUs?

Walter, how do you suggest changing GPUs?

1. Replace one of the GPUs in a faulty system (Quadro K6000 and Tesla K40c) with another type of GPU from a working system (for example, GTX780)?

2. Remove all GPUs from a faulty system (Quadro K6000 and Tesla K40c) and put one of the same (Tesla K40c) from the available stock?

3. Take video cards one at a time out of the faulty system and see what happens?

4. In any other way?

[Walter Roberson]

If CUDA returns when you downgrade drivers, I do not think swapping cards would help.

I would suggest exchanging one faulty K40c with a working system, so that both systems have a mix of suspected faulty and known good K40c.

[Alexander Sonin]

Hello, Walter!

The devil pulled my leg and I did the opposite: I started by rearranging the videocards.

1. Remove one bottom Tesla K40c

In PC are present: Quadro K6000 + Tesla K40c + Tesla K40c

NVIDIA Driver 419.67

CUDA in GPU-Z not visible (both for Quadro K6000 and Tesla K40c)

In MATLAB gpuDeviceCount=0

2. GTX 780 inserted in bottom slot instead removed Tesla K40c

In PC are present: Quadro K6000 + Tesla K40c + Tesla K40c + GTX 780

Unknown where did driver 388.19 come from!!!

I did not install it!

In GPU-Z CUDA became visible for all GPUs (Quadro K6000, 2xTesla K40c, GTX 780)

And in MATLAB all GPUs became visible, but not usable:

>> gpuDeviceCount

ans =

4

>> gpuDevice(1)

Error using gpuDevice (line 26)

There is a problem with the graphics driver or with this

GPU device. Be sure that you have a supported GPU and

that the latest driver is installed.

Caused by:

The graphics driver was found, but it is too old. The

graphics driver on your system supports CUDA version

9.1. The required CUDA version is 10 or greater.

[Alexander Sonin]

3. Video cards from all slots except the first one are extracted.

In PC are present: only Quadro K6000

388.19 NVIDIA driver stayed

In MATLAB Quadro GPU is also visible, but not usable:

>> gpuDevice()

Error using gpuDevice (line 26)

There is a problem with the graphics driver or with this GPU

device. Be sure that you have a supported GPU and that the

latest driver is installed.

Caused by:

The graphics driver was found, but it is too old. The

graphics driver on your system supports CUDA version

9.1. The required CUDA version is 10 or greater.

4. Removed 388.19 driver.

nv-smi does not work

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

5. 419.67 driver is installed (optimal for Quadro K6000 and CUDA Toolkit 10.1)

The nv-smi sees the Quadro K6000

In GPU-Z is not present CUDA.

In MATLAB Quadro GPU is not visible:

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

6. CUDA Toolkit 10.1 has been installed.

In GPU-Z PhysX added, but is not present CUDA.

In MATLAB Quadro GPU is not visible:

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

7. The Quadro K6000 was removed and the GTX 780 was installed instead.

419.67 driver disappeared.

NVSMI sees the 382.05 driver, but in Windows Programs & Components ony 3D Vision 390.41 driver is present.

In GPU-Z CUDA and everything else is present!

In MATLAB GTX 780 is visible, but not usable:

>> gpuDeviceCount

ans =

1

>> A=gpuArray([1 2 3])

There is a problem with the graphics driver or with this GPU

device. Be sure that you have a supported GPU and that the

latest driver is installed.

Caused by:

The graphics driver was found, but it is too old. The

graphics driver on your system supports CUDA version 8.

The required CUDA version is 10 or greater.

8. Driver 425.31 (optimal for Win8.1 and GTX 780) is installed on top of the existing one.

GTX 780 in MATLAB is visible and usable, with newest driver:

>> gpuDeviceCount

ans =

1

>> A=gpuArray([1 2 3])

A =

1 2 3

>> B=gather(A.^2)

B =

1 4 9

>> whos

Name Size Bytes Class Attributes

A 1x3 4 gpuArray

B 1x3 24 double

ans 1x1 8 double

>> gpuDevice

ans =

CUDADevice with properties:

Name: 'GeForce GTX 780'

Index: 1

ComputeCapability: '3.5'

SupportsDouble: 1

DriverVersion: 10.1000

ToolkitVersion: 10

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 6.4425e+09

AvailableMemory: 6.1924e+09

MultiprocessorCount: 12

ClockRateKHz: 941000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 1

CanMapHostMemory: 1

DeviceSupported: 1

DeviceSelected: 1

All is well!

But, Quadro and Tesla video cards must be installed in the system.

9. Therefore, the GTX 780 was extracted, and the Quadro K6000 was installed again in the system. Also 398.75 driver is installed (optimal for CUDA ToolKit 9.2).

CUDA is not visible in GPU-Z.

In MATLAB:

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

10. CUDA ToolKit 9.2 with patches was installed.PhysX added in GPU-Z, but CUDA is not visible.

In MATLAB:

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

11. Reinstal NVIDIA driver to 391.29 (optimal for CUDA ToolKit 9.1)

CUDA is not visible in GPU-Z.

In MATLAB:

>> gpuDeviceCount

ans =

0

[Alexander Sonin]

12. CUDA Toolkit 9.1 was installed instead CUDA Toolkit 9.2

And CUDA appeared in GPU-Z!

But where did the 391.29 driver go?

Instead, there is a 388.19 driver!

Apparently, this is an internal driver from the CUDA Toolkit 9.1?

And thanks to him CUDA is appeared?

In MATLAB:

>> gpuDeviceCount

ans =

1

13. Then CUDA Toolkit 9.1 patches were installed.

The situation has not changed

[Alexander Sonin]

14 Then I began to return the Teslas into the system.

Two Tesla K40c were installed into the system.

398.75 driver is resurrected, and CUDA has disappeared in GPU-Z.

In the MATLAB video cards are not visible:

>> gpuDeviceCount

ans =

0

15. Installation of an additional fourth TeslaK40x did not change the situation.

[Alexander Sonin]

16. Then I installed the 390.65 driver (suitable for CUDA Toolkit 9.1)

17. And reinstalled CUDA Toolkit 9.1 with patches.

388.19 driver activated and CUDA returned.

In MATLAB R2019a all GPUs are visible, but not usable:

>> gpuDeviceCount

ans =

4

>> gpuDevice(1)

Error using gpuDevice (line 26)

There is a problem with the graphics driver or with this GPU device. Be sure that you

have a supported GPU and that the latest driver is installed.

Caused by:

The graphics driver was found, but it is too old. The graphics driver on your system

supports CUDA version 9.1. The required CUDA version is 10 or greater.

>> A=gpuArray([1 2 3])

There is a problem with the graphics driver or with this GPU device. Be sure that

you have a supported GPU and that the latest driver is installed.

Caused by:

The graphics driver was found, but it is too old. The graphics driver on your

system supports CUDA version 9.1. The required CUDA version is 10 or greater.

[Alexander Sonin]

18. Then I try enter same commands in МАТLAB R2018b, and after that (oh, wonders!) the GPU devices was earned:

>> gpuDeviceCount

ans =

4

>> gpuDevice(1)

ans =

CUDADevice with properties:

Name: 'Tesla K40c'

Index: 1

ComputeCapability: '3.5'

SupportsDouble: 1

DriverVersion: 9.1000

ToolkitVersion: 9.1000

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 1.1976e+10

AvailableMemory: 5.3635e+09

MultiprocessorCount: 15

ClockRateKHz: 745000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 0

CanMapHostMemory: 1

DeviceSupported: 1

DeviceSelected: 1

>> gpuDevice(2)

ans =

CUDADevice with properties:

Name: 'Tesla K40c'

Index: 2

ComputeCapability: '3.5'

SupportsDouble: 1

DriverVersion: 9.1000

ToolkitVersion: 9.1000

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 1.1976e+10

AvailableMemory: 5.3635e+09

MultiprocessorCount: 15

ClockRateKHz: 745000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 0

CanMapHostMemory: 1

DeviceSupported: 1

DeviceSelected: 1

>> gpuDevice(3)

ans =

CUDADevice with properties:

Name: 'Quadro K6000'

Index: 3

ComputeCapability: '3.5'

SupportsDouble: 1

DriverVersion: 9.1000

ToolkitVersion: 9.1000

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 1.2885e+10

AvailableMemory: 5.7037e+09

MultiprocessorCount: 15

ClockRateKHz: 901500

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 1

CanMapHostMemory: 1

DeviceSupported: 1

DeviceSelected: 1

>> gpuDevice(4)

ans =

CUDADevice with properties:

Name: 'Tesla K40c'

Index: 4

ComputeCapability: '3.5'

SupportsDouble: 1

DriverVersion: 9.1000

ToolkitVersion: 9.1000

MaxThreadsPerBlock: 1024

MaxShmemPerBlock: 49152

MaxThreadBlockSize: [1024 1024 64]

MaxGridSize: [2.1475e+09 65535 65535]

SIMDWidth: 32

TotalMemory: 1.1976e+10

AvailableMemory: 5.3802e+09

MultiprocessorCount: 15

ClockRateKHz: 745000

ComputeMode: 'Default'

GPUOverlapsTransfers: 1

KernelExecutionTimeout: 0

CanMapHostMemory: 1

DeviceSupported: 1

DeviceSelected: 1

>> A=gpuArray([1 2 3])

A =

1 2 3

At link

https://www.mathworks.com/help/parallel-computing/gpu-support-by-release.html#responsive_offcanvas

I was disappointed to find that the R2019a only works with a CUDA version no lower than 10.

[Alexander Sonin]

Thus, by experience, I found:

1. Installing CUDA Toolkit on top of the installed driver can solve the GPU visibility problem (which differs from the statements made on mathworks.com).

2. In order for MATLAB to see GPUs, CUDA must be visible in GPU-Z.

3. In order for MATLAB to use GPUs, the version of CUDA Toolkit, installed on a PC, must be at least specified in the “GPU Support by Release” section.

But, I did not understand, why GPU-Z never saw CUDA when installing CUDA Toolkit 10.x.

[Walter Roberson]

Whew! What a bunch of bother!

[Hina Ashraf]

I am having same problem? How did you fix it?

[Hina Ashraf]

Sorry, I restart the MATLAB R2020b and the problem is fixed! :)

Answer 1

[Walter Roberson]

Those are Kepler architecture. The 10 series drivers dropped support for Kepler if I recall correctly.

[Walter Roberson]

No, this turned out to be wrong. The Fermi architecture was dropped last year.

End of support for the Kepler mobile gpus was announced last month; https://www.techspot.com/news/79115-nvidia-end-support-mobile-kepler-gpus-starting-april.html but they should still have drivers until next year. Support for the desktop Kepler will continue after that.

[Alexander Sonin]

Walter, thank you very much for reviewing my problem!

The Titan and GeForce 780 (Kepler GK110) GPUs are installed on another personal computer. OS Win7, driver 419.67. MATLAB2018b has seen these video cards.