CUSOLVER_S​TATUS_INTE​RNAL_ERROR with RTX 4080 Super in MATLAB 2022a C MEX Function

Hello,

I'm encountering a problem when running a CUDA-based C MEX function in MATLAB 2022a on a machine equipped with an RTX 4080 Super GPU. The specific error occurs when executing the line:

cusolverStatus_t status = cusolverDnCreate(&handle);

Here are the details:

• The MEX function compiles successfully using the mexcuda command.
• The same code operates correctly on this machine when using an RTX 2080Ti GPU with exactly the same software setup.
• When the RTX 4080 Super is used, this error appears, even though the function runs without issues in Visual Studio, indicating that CUDA (Version 12.5) is properly configured on my system.
• Upgrading to MATLAB 2024a resolves the issue, suggesting a compatibility problem with MATLAB 2022a. However, I am restricted to using MATLAB 2022a due to dependencies on other external software.

Additional context:

• NVIDIA Driver Version: 555.99
• CUDA Toolkit Version: 12.5

Has anyone else experienced similar issues with newer NVIDIA GPUs in MATLAB 2022a? Any insights or suggestions on how to resolve or circumvent this error with the RTX 4080 Super would be greatly appreciated.

Thank you!

[Umar]

Hi Stefanos, It seems that the compatibility issue lies between MATLAB 2022a and the RTX 4080 Super GPU. Upgrading to MATLAB 2024a resolves the problem, indicating a version-specific conflict. Since you are constrained to MATLAB 2022a, consider checking for any MATLAB updates or patches that address GPU compatibility issues. Additionally, ensure that all CUDA libraries and drivers are up to date. You may also try adjusting compiler options or CUDA configurations specific to the RTX 4080 Super GPU. Seeking support from MathWorks or NVIDIA forums could provide further insights or workarounds tailored to your setup.

Answer 1

[Joss Knight]

This is a known issue with CUDA 11.2, Ada cards, and cusolver. The solution is to upgrade MATLAB to R2023a or above, because it requires a new version of CUDA which cannot be provided in an update release.

It is possible that you can be walked through dropping a fixed version of cusparse and cusolver from CUDA 11.3, that would be compatible with MATLAB R2022a and not broken. However, it would require you to edit your MATLAB installation. (SPOILER: This turned out to be the only solution that worked.) Upgrading is definitely preferable.

Since this is a mex function, you have the option to compile using your own CUDA toolkit rather than the one shipped with MATLAB. To do this you will need to make some edits to mex options in order to allow mexcuda to use a version of the toolkit that doesn't match the version of MATLAB. I can walk you through this if this is your preferred route. At the moment mexcuda is not using the 12.5 tookit you installed because the version is wrong.

EDIT: Actually, I think I know how to force mexcuda to compile with your toolkit without any options file changes:

mexcuda -v CUDA_LIB_PATH='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64' CUDA_ROOT='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5' LIBDEVICE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\nvvm\libdevice' cusolver_handle_mex.cu

The -v will help you work out whether the correct version of nvcc was used. What I can't be sure of is that using CUDA 12.5 alongside CUDA 11.2 won't have unforeseen problems. If it does you might be better off installing CUDA 11.3 instead, which will be compatible with R2022a but won't have the cusolver bug.

[Stefanos]

I adapted it a bit to work with my case (not sure if it is correct) but i still get the same cuSOLVER internal error even with this:

mexcuda -v ...
    CUDA_LIB_PATH='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/lib/x64' ...
    CUDA_ROOT='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5' ...
    LIBDEVICE='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/nvvm/libdevice' ...
    'NVCCFLAGS=-gencode=arch=compute_89,code=sm_89' ...
    cusolver_handle_mex.cu ...
    -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64" ...
    -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\include" ...
    -lcusolver

Let me know if I should also share the compilation log. Thanks a lot for your help!

[Joss Knight]

I think share the verbose log - we need to be sure you're using the right version of nvcc and the right libraries.

Also run version -modules after running your function so we can see which libraries are loaded.

[Stefanos]

I attached the mexcuda log and the output of the version modules since they are a bit too long.

[Joss Knight]

It does not look as though you have called your mex function before running version -modules because none of the cuda libraries are loaded.

Also in the verbose output I see

1. Your path entries in the call to mexcuda are not in quotes, and
2. Your path separators are the wrong way round ('/' instead of '\').

Can you double-check these?

[Stefanos]

Can you please elaborate on which path entries require quotes? this is my current command

mexcuda -v ...
    CUDA_LIB_PATH='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/lib/x64' ...
    CUDA_ROOT='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5' ...
    LIBDEVICE='C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/nvvm/libdevice' ...
    'NVCCFLAGS=-gencode=arch=compute_89,code=sm_89' ...
    cusolver_handle_mex.cu ...
    -L"C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/lib/x64" ...
    -I"C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/include" ...
    -lcusolver

Also, attached iss the updated version -modules output.

[Joss Knight]

Quotes are required around the file paths that contain spaces. Quotes look correct there, but the path separators are still the wrong way round ('C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.5/lib/x64' -> 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64').

Also, do not specify -L or -I.

[Joss Knight]

Your updated version output shows that you definitely have the CUDA 11.2 version of cusolver loaded and not the CUDA 12.5 version.

I'm wondering if this is impossible on Windows. There are two versions of the DLL, cusolver64_11.dll and cusolver64_12.dll. However, the LIB itself, which is used at link time, just has the one name cusolver.lib. I'm not sure if, at runtime, the OS knows you want to load the other cusolver.

Anyway, let's keep going, see if we can just tidy up these other issues.

[Stefanos]

Now it should be fine and I also removed the -L and -I. I have also added the new output of mexcuda

mexcuda -v ...
    CUDA_LIB_PATH='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64' ...
    CUDA_ROOT='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5' ...
    LIBDEVICE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\nvvm\libdevice' ...
    'NVCCFLAGS=-gencode=arch=compute_89,code=sm_89' ...
    cusolver_handle_mex.cu ...
    -lcusolver

[Joss Knight]

And the new output of version -modules?

[Joss Knight]

I wonder if we need to force the path as well. So try adding C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\bin to the beginning of your PATH environment variable. You might have to restart MATLAB. I didn't think this was necessary since the path is edited automatically when you run the mex function, but it's worth a try.

[Joss Knight]

Also, try specifying the full path to the lib just in case, like

-l'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64\cusolver.lib'

[Stefanos]

I already had C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\bin at the beginning of my PATH environment variable, I believe CUDA also adds it automatically during Windows installation.

I tried specifyingthe full path to the lib but it comes back with an error: Error using mex

$$LIB not detected; check that you are in the correct current folder, and check the

spelling of '$$LIB'.

Error in mexcuda (line 168)

[varargout{1:nargout}] = mex(mexArguments{:});

Error in mex_compile (line 14)

mexcuda -v ...

I found that adding this line the following way worked but maybe you can confirm whether it is the same thing:

-L'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64\cusolver.lib' ...

Also attrached you will find the new mexcuda output and modules -version, I apologize for not including them in my last message.

[Joss Knight]

Well, it's still definitely loading the older version of cusolver and not the newer one. You can see for yourself in the output of version -modules:

C:\Program Files\MATLAB\R2022a\bin\win64\cusolver64_11.dll Version <unknown>     

I can't tell whether it's because the OS doesn't know not to load that version, or whether it's something to do with the fact that the link line in your output doesn't look quite right, it says

link C:\Users\SFLORE~1\AppData\Local\Temp\mex_349698245540363_18908\cusolver_handle_mex.obj C:\Users\SFLORE~1\AppData\Local\Temp\mex_349698245540363_18908\c_mexapi_version.obj  cusolver.lib   "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64\cusolver.lib"

This looks okay except the /LIBPATH line looks suspect, because it specifies the file directly instead of just the path to the lib directory (I was expecting /LIBPATH:"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64"). I think perhaps this is just because of you trying what you said above as -L. But on a positive note, this is the first time your output link line has actually contained the path to the lib folder with all the path separators in the right direction.

So let's try one more time, but this time put the -L back, and then let's look at the mexcuda verbose output. You want

mexcuda -v ...
    CUDA_LIB_PATH='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64' ...
    CUDA_ROOT='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5' ...
    LIBDEVICE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\nvvm\libdevice' ...
    'NVCCFLAGS=-gencode=arch=compute_89,code=sm_89' ...
    cusolver_handle_mex.cu ...
    -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64" ...
    -lcusolver

Failing this I feel our only choices are to go with one of the other two options:

1. Upgrade to at least R2023a
2. Edit your MATLAB installation to add unbroken versions of cusolver and cusparse DLLs from CUDA 11.3

[Stefanos]

Here are the new verbose output and also the version -modules output.

I already tried with Matlab 2023a and the CUDA code works. The issue we have with this is that we also depend on a piece of hardware that is locked to work with only Matlab 2022a. As a result, solution 2 might be a good hack to get it to work until we can use our hardware with a newer Matlab release.

[Joss Knight]

Okay, that didn't work. The link line looks correct, to my eyes.

Okay, so download CUDA 11.3, extract the cusolver64_11.dll and cusparse64_11.dll from the bin directory (...\CUDA\v11.3\bin), and copy them over those files in your matlab installation (C:\Program Files\MATLAB\R2022a\bin\win64\cusolver64_11.dll). Ideally, you might save the original somewhere just in case.

[Stefanos]

This worked! Thank you very much for all your help and taking so much time looking over the logs and everything!

[Joss Knight]

Good to hear. Pity we had to resort to that though.

[Stefanos]

Yes, it would have been nice to not have to hack it this way but since it works it is a good solution for the time being until we can use a newer Matlab release. Thank you again!

Answer 2

[Joss Knight]

It's probably necessary for you to provide your code, since any number of things could have gone wrong prior to intializing cusolver which could prevent it from intializing successfully. For instance, did you start your mex function by calling mxGPUInit?

[Stefanos]

I am not sure what mxGPUInit as I cannot seem to find any documentation on it. Could you point me to a page or reference? Also, it seems weird that the same code works on the 2022a version with another GPU or with the actual GPU I have the issue with the 2024a version. In addition, I have never had to use this previously mxGPUInit and have been working with MEX GPU code for quite a while.

Here is a simplified version of my MEX code that produces the error in question:

#include "mex.h"
#include <cusolverDn.h>
#include <cuda_runtime.h>
#include <cstdio>

// Function to convert cuSOLVER status to a string
const char *cusolverGetErrorString(cusolverStatus_t status)
{
    switch (status)
    {
    case CUSOLVER_STATUS_SUCCESS:
        return "CUSOLVER_STATUS_SUCCESS";
    case CUSOLVER_STATUS_NOT_INITIALIZED:
        return "CUSOLVER_STATUS_NOT_INITIALIZED";
    case CUSOLVER_STATUS_ALLOC_FAILED:
        return "CUSOLVER_STATUS_ALLOC_FAILED";
    case CUSOLVER_STATUS_INVALID_VALUE:
        return "CUSOLVER_STATUS_INVALID_VALUE";
    case CUSOLVER_STATUS_ARCH_MISMATCH:
        return "CUSOLVER_STATUS_ARCH_MISMATCH";
    case CUSOLVER_STATUS_MAPPING_ERROR:
        return "CUSOLVER_STATUS_MAPPING_ERROR";
    case CUSOLVER_STATUS_EXECUTION_FAILED:
        return "CUSOLVER_STATUS_EXECUTION_FAILED";
    case CUSOLVER_STATUS_INTERNAL_ERROR:
        return "CUSOLVER_STATUS_INTERNAL_ERROR";
    case CUSOLVER_STATUS_MATRIX_TYPE_NOT_SUPPORTED:
        return "CUSOLVER_STATUS_MATRIX_TYPE_NOT_SUPPORTED";
    default:
        return "Unknown cuSOLVER error";
    }
}

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
    // Print initial diagnostic information
    mexPrintf("Starting cuSOLVER handle initialization.\n");

    // Declare a cuSOLVER handle
    cusolverDnHandle_t handle;

    // Initialize the cuSOLVER handle
    cusolverStatus_t status = cusolverDnCreate(&handle);
    if (status != CUSOLVER_STATUS_SUCCESS)
    {
        mexErrMsgIdAndTxt("cusolver:initializationFailed",
                          "CUSOLVER initialization failed: %s", cusolverGetErrorString(status));
    }

    mexPrintf("cuSOLVER handle initialized successfully.\n");

    // Destroy the cuSOLVER handle
    status = cusolverDnDestroy(handle);
    if (status != CUSOLVER_STATUS_SUCCESS)
    {
        mexErrMsgIdAndTxt("cusolver:destructionFailed",
                          "CUSOLVER destruction failed: %s", cusolverGetErrorString(status));
    }

    mexPrintf("cuSOLVER handle destroyed successfully.\n");
}

In addition here is the compilation command used:

% Set CUDA_PATH to the appropriate path
% setenv('MW_CUDA_FORWARD_COMPATIBILITY', '1');
setenv('CUDA_PATH', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5');

% Add CUDA binaries to the system PATH
setenv('PATH', [getenv('CUDA_PATH') '\bin;' getenv('PATH')]);

% Compile the MEX function
mexcuda -v cusolver_handle_mex.cu ...
   -L"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\lib\x64" ...
   -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5\include" ...
   -lcusolver -lcudart -lcublas

[Joss Knight]

mxInitGPU

(sorry, misspelt it)