clear mex causes segmentation fault when using cuda RT API

Hi all,

I have a mex function with CUDA code in it so that I can take advantage of my Quadro 4000 gpu (Driver version 4.2 / Runtime version 4.2) on my 64-bit linux machine.

Any time I make any call to a CUDA Runtime API function (cudaMalloc, cudaMemset, cudaMemcpy, etc.), I get a segmentation fault if I clear mex (or clear all).

Here is my simple .cu code:

#include "mex.h"
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
    // Allocate and free one float on GPU memory
    float *a = NULL;
    cudaMalloc((void **)a, sizeof(float));
    cudaFree(a);
    // cudaDeviceReset();
}

After compiling the .cu to .cpp using nvcc and then making myMexFunction.mexa64 with mex, I call my function in MATLAB2012a:

>> myMexFunction;
>> clear all
>> Segmentation fault (core dumped)

One workaround I've found is calling "cudaDeviceReset()" at the end of my mex function. It doesn't seem like resetting the device every time I call my function should be necessary though.

Interestingly, calls to the Driver API have no such issues.

Has anyone encountered this before?

Answer 1

[Jill Reese]

Do you have the Parallel Computing Toolbox installed as well? If so, conflicts can arise between the version of the CUDA libraries that the PCT provides and the version used when compiling Mex code. In the past the workaround has been to call some GPU function, like gpuDevice() , in MATLAB before executing the mex routine.

[dhyun]

That was exactly the problem. Running gpuDevice prior to running my function prevents seg faults when I clear my mex function.

Thanks!

[L]

While running gpuDevice prior to running the mex code prevented segfaults after doing a clear all/clear mex, for me it just was a bandaid to the problem -- memory was not being cleared when the MEX file was run repeatedly this way.

I fixed the issue by following A.Goude's advice here. Even though the poster mentions that there are still some slight memory leaks (and I have noticed this as well), adding the cleanup function called with:

 mexAtExit(cleanup)

where the cleanup function calls

 cudaThreadExit()

did solve the segfaults upon doing a clear all/clear mex. For that matter, I am running an Linux AMD64 (X86_64) system. I also noticed that even the R2013a version of MATLAB does not come with a libstdc++.so.6 library that is compatible with CUDA 5.0 -- I got errors referencing GLIBCXX 3.14 or 3.15 were not supported.

To fix that issue, I had to delete the libstdc++.so.6 symlink in MATLAB's /sys/os/glnxa64/ directory and recreate it pointing to my OS' native version and that let me run my CUDA 5.0 compiled code on Ubuntu 12.10/13.04 under MATLAB R2012a and R2013a.

@Jan Simon - I did not ask a question, and the answer was relevant to prevent the behavior. That being said, cudaThreadExit is now deprecated, and is functionally equivalent to cudaDeviceReset. This segfault issue is still present in MATLAB 2017a, however either calling cudaDeviceReset or gpuDevice will avoid a segfault, as was the case years ago.

[Jan]

@L: Please do not highjack an existing thread. Appending a new question as comment to an answer to an existing question increases the confusion level, because it is not clear anymore, if answers concern your or the original question. So please delete this comment and open a new question. Thanks.