C++ Mex file crashes matlab on 64bit linux, but not 32 bit windows, but program runs fine outside Matlab

warning: the code I provide in this question may crash your machine

I have written a mex gateway function to a C++ class. If I compile this mex function on 32Bit windows using R2008a I have no problems. If I compile and run on Matlab R2011a running 64bit Scientific Linux (a version of Red Hat Enterprize Linux) matlab exits with a segfault when the mexfunction is called, although it appears to run about halfway through the program. The C++ class can be compiled and run (with a main function) outside of Matlab on both platforms with no errors. I am using Microsoft Visual C++ Express Edition on the windows machine, and gcc 4.4.5 on the Linux machine.

What is the cause of this and can I fix it?

A zip file containing the code and data files necessary to reproduce the problem can be downloaded from http://www.see.ed.ac.uk/~s0237326/downloads/mexcrash.zip. This zip file contains the .m and .cpp source code, and a text file for testing (Temp.fem). The file fmehsersetup.m shows the commands I am using to compile. The file Test_mexfmesher.m runs the mexfunction with an appropriate input for testing. The mex gateway function is mexfmesher.cpp, it calls the fmesher class which is made up of the files in the fmesher directory.

Below is a backtrace from the segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff721be36 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwservices.so
(gdb) backtrace
#0  0x00007ffff721be36 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwservices.so
#1  0x00000034daa914f2 in std::basic_ostream<char, std::char_traits<char> >::flush() () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libstdc++.so.6
#2  0x00007ffff72302b0 in ioFlush () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwservices.so
#3  0x00007ffff61303f5 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#4  0x00007ffff61330ec in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#5  0x00007ffff6130c7a in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#6  0x00007ffff6131741 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#7  0x00007ffff618a7d9 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#8  0x00007ffff686d7ef in Mfh_file::dispatch_fh(int, mxArray_tag**, int, mxArray_tag**) () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_dispatcher.so
#9  0x00007ffff61731f0 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#10 0x00007ffff6114975 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#11 0x00007ffff612e96e in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#12 0x00007ffff61330ec in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#13 0x00007ffff6130c7a in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#14 0x00007ffff6131741 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#15 0x00007ffff618a7d9 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#16 0x00007ffff686d7ef in Mfh_file::dispatch_fh(int, mxArray_tag**, int, mxArray_tag**) () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_dispatcher.so
#17 0x00007ffff61669b2 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#18 0x00007ffff6128e13 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#19 0x00007ffff6127eb7 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#20 0x00007ffff6128397 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwm_interpreter.so
#21 0x00007ffff6d378fe in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwbridge.so
#22 0x00007ffff6d384ae in mnParser () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwbridge.so
#23 0x00007ffff6ae0d39 in mcrInstance::mnParser_on_interpreter_thread() () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#24 0x00007ffff6ac3db2 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#25 0x00007ffff6ac3ec0 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#26 0x00007fffee8badb6 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/libmwuix.so
#27 0x00007fffee8c413d in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/libmwuix.so
#28 0x00007fffef3110bd in sysq::during_F<boost::weak_ptr<sysq::ws_ppeHook>, boost::shared_ptr<sysq::ws_ppeHook> > std::for_each<__gnu_cxx::__normal_iterator<boost::weak_ptr<sysq::ws_ppeHook>*, std::vector<boost::weak_ptr<sysq::ws_ppeHook>, std::allocator<boost::weak_ptr<sysq::ws_ppeHook> > > >, sysq::during_F<boost::weak_ptr<sysq::ws_ppeHook>, boost::shared_ptr<sysq::ws_ppeHook> > >(__gnu_cxx::__normal_iterator<boost::weak_ptr<sysq::ws_ppeHook>*, std::vector<boost::weak_ptr<sysq::ws_ppeHook>, std::allocator<boost::weak_ptr<sysq::ws_ppeHook> > > >, __gnu_cxx::__normal_iterator<boost::weak_ptr<sysq::ws_ppeHook>*, std::vector<boost::weak_ptr<sysq::ws_ppeHook>, std::allocator<boost::weak_ptr<sysq::ws_ppeHook> > > >, sysq::during_F<boost::weak_ptr<sysq::ws_ppeHook>, boost::shared_ptr<sysq::ws_ppeHook> >) ()
   from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/libuij.so
#29 0x00007fffef312989 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/libuij.so
#30 0x00007fffef30f4ae in svWS_ProcessPendingEvents(int, int, bool) () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/../../bin/glnxa64/libuij.so
#31 0x00007ffff6ac21c7 in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#32 0x00007ffff6ac260a in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#33 0x00007ffff6ac2d6f in ?? () from /opt/matlab-2011a/bin/glnxa64/../../bin/glnxa64/libmwmcr.so
#34 0x00000034d7a077e1 in start_thread () from /lib64/libpthread.so.0
#35 0x00000034d72e573d in clone () from /lib64/libc.so.6
(gdb) quit

Answer 1

[Jan]

I try to run your fmeshersetup with MSVC 2008. I get these errors and warnings from the compiler:

 FMESHER\fmesher.cpp(103) : warning C4800: 'int' : forcing value to bool 'true' or 'false' (performance warning) 
FMESHER\fmesher.cpp(105) : warning C4800: 'int' : forcing value to bool 'true' or 'false' (performance warning) 
FMESHER\fmesher.cpp(439) : error C2589: '(' : illegal token on right side of '::' 
FMESHER\fmesher.cpp(439) : error C2059: syntax error : '::' 
FMESHER\fmesher.cpp(626) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(646) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(678) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(883) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(895) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(935) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(941) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(998) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1004) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1110) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1116) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1251) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1257) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data 
FMESHER\fmesher.cpp(1337) : warning C4800: 'int' : forcing value to bool 'true' or 'false' (performance warning) 
FMESHER\fmesher.cpp(1371) : warning C4800: 'int' : forcing value to bool 'true' or 'false' (performance warning) 
FMESHER\fmesher.cpp(1447) : warning C4800: 'int' : forcing value to bool 'true' or 'false' (performance warning)

The error in line 439 concerns "std::min" and can be fixed easily. Afterwards the next run of the setup reveals, that there are a lot of size_t to int conversions in writepoly.cpp also.

I currently have the impression, that your function fails in writepoly. I do some further tests, but please do not wait for me.

EDITED: The function crashs in writepoly.cpp->Line 1025. You try to free memory which points to NULL, e.g. in.pointattributelist, out.pointattributelist, out.neighborlist, out.segmentlist, out.segmentmarkerlist.

On one hand this let my Matlab crash such, that it cannot be restarted without restarting the OS. On the other hand it is not smart to call free() without testing for NULL. If I see, that the success of malloc() is not checked also, I would not trust this software.

[Richard Crozier]

Thanks, I appreciate this. I may also have made an error and not uploaded the very latest version (I'm confusing myself working on several different computers). I have now replaced this. I do not see these warnings with this version from the mex command on Windows XP 32 Bit, and the Visual Basic Express Edition 2008 C++ compiler. I get no messages at all in fact from the mex command, unless I set -v obviously. I have also made the Test_fmesher.m slightly easier to use provided the directory structure is preserved.

The problem is not urgent but I would like to get to the bottom of it eventually.

[Richard Crozier]

Re: your edit, apologies if this has caused you any problems, having to restart your whole system! I have not experienced this on either windows or Linux, maybe I've just been lucky. I will look into the correct way to deal with this memory allocation.

[Jan]

@Richard: I do not test foreign C-code on a productive machine, therefore the crashs are ok.

Of course it would be smart, if free() checks for NULL itself. But a careful C-programmer would not rely of the library functions to catch this. Most likely other StdC libs include such a check.

The warnings do not appear under 32 bit systems, because the size_t to int causes troubles in 64 bit systems only.

[Richard Crozier]

@Jan, thanks again for the help, I have also added a warning to others who might download the code to the question.

Answer 2

[Jan]

You've posted 1.7MB source code. You cannot expect, that somebody debugs this large code for you. The conversion fom 32 to 64 bit systems is critical, if the data model does not consider this and assumes that int and pointers are signed INT32 values.

If you have the impression, that the program appears to run about halfway, please explain, why you think so. Either use a debugger or insert some output until you find the line causing the crash.

[Richard Crozier]

The source is not 1.7 MB, much of this is made up of the test data file Temp.fem (896 kB). I had also accidentally included some svn files, and other unnecessary files. I have also provided two mfiles which provide exact commands to compile and run with appropriate input. I have now stripped all unnecessary files from the source. There are now a total of 13 C++ source files. Unfortunately I do not have access to a graphical debugger on 64bit linux that can interface with Matlab, and was hoping someone would immediately see the problem, or it would be a known problem. As I stated, when run not as a mex file, the program runs fine, so I cannot locate the error in another tool, and I'm not very familiar with gdb. I believe the problem might be related to a calls to the C++ function 'free'. I will try and come back with some further information.

[Richard Crozier]

To explain why I think the program runs halfway, the program is a meshing program which creates a finite element mesh. For meshes with a periodic boundary, it takes two passes at the mesh creation. In the first pass it creates several files, (.edge, .ele, .poly) in the second pass it overwrites these files and creates a .pbc files. The first set of files are being created, but not the final .pbc file. Between the first pass and second pass is several calls to 'free' which is where I believe the problem might be. I will try to investigate further though.

[Richard Crozier]

Actually, to correct myself the data file was not this big, but the source does make up just under a MB.

[Jan]

If I unzip the file, I get 1'796'481 bytes in C files inside \temp\fmesher, while the file "Temp.fem" has only 10kB.

[Richard Crozier]

There were a few other unnessessary c and cpp files now removed. On another forum some has suggested the problem may be due to the fact that matlab use its own libstdc++ (stored in /opt/matlab-2011a/bin/glnxa64/libstdc++.so.6) instead of the main one (stored in /usr/lib/x86_64-linux-gnu/). Unfortunately I am not a sudoer and cannot force matlab to use the main libstdc++ on my system.