Error importing matlab.engine into Python

I tried the solution from a previous question. I'm on OSX 10.9.5 using MatlabR2015a and Python 3.4.3 (miniconda).

>>> import os
>>> os.environ['DYLD_LIBRARY_PATH'] = '/Users/Evan/miniconda/lib/'
>>> import matlab.engine
Traceback (most recent call last):
  File "/Users/Evan/miniconda/lib/python3.4/site-packages/matlab/engine/__init__.py", line 41, in <module>
    pythonengine = importlib.import_module("matlabengineforpython"+_PYTHONVERSION)
  File "/Users/Evan/miniconda/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'matlabengineforpython3_4'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/Evan/miniconda/lib/python3.4/site-packages/matlab/engine/__init__.py", line 57, in <module>
    pythonengine = importlib.import_module("matlabengineforpython"+_PYTHONVERSION)
  File "/Users/Evan/miniconda/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "<frozen importlib._bootstrap>", line 539, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1715, in load_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
ImportError: dlopen(/Applications/MATLAB_R2015a.app/extern/engines/python/dist/matlab/engine/maci64/matlabengineforpython3_4.so, 2): Library not loaded: @rpath/libpython3.4m.dylib
  Referenced from: /Applications/MATLAB_R2015a.app/extern/engines/python/dist/matlab/engine/maci64/matlabengineforpython3_4.so
  Reason: image not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Evan/miniconda/lib/python3.4/site-packages/matlab/engine/__init__.py", line 59, in <module>
    raise EnvironmentError('The installation of MATLAB Engine for Python is '
OSError: The installation of MATLAB Engine for Python is corrupted.  Please reinstall it or contact MathWorks Technical Support for assistance.

Answer 1

[Bo Li]

This error message means that the "libpython3.4m.dylib" is not found or not compatible:

 "ImportError: dlopen(/Applications/MATLAB_R2015a.app/extern/engines/python/dist/matlab/engine/maci64/matlabengineforpython3_4.so, 2): Library not loaded: @rpath/libpython3.4m.dylib"

Is there a file called "libpython3.4m.dylib" located under "/Users/Evan/miniconda/lib/"?

[Evan]

Yes.

[Bo Li]

Would you set the environment variable DYLD_LIBRARY_PATH before launching Python and try Python Engine again? Changing the environment variable using os.environ does not effect current Python session.

[Evan]

Using

export DYLD_LIBRARY_PATH=/Users/Evan/miniconda/lib

before starting the Python session works.

However, I've read that it is not advisable (also here and here) make changes to DYLD_LIBRARY_PATH on the system in general, and that if this value requires modification then you're doing something wrong (also here). In fact `$man dyld` states that the intent of this value is that "It allows you to test new versions of existing libraries."

export DYLD_FALLBACK_LIBRARY_PATH=/Users/Evan/miniconda/lib

Also works and might avoid interaction with other software, but apparently not in all cases.

Do you have any insight as to why this step is necessary? Is it because my Python installation is not where the Matlab module is expecting it to be? What is @rpath, and how is it set if $DYLD_LIBRARY_PATH does not exist? Is this problem fixable?

Finally, is there a way to do this from within the current Python session? It seems onerous to have to set this value manually each time I want to use the Python engine.

[Bo Li]

Python Engine links against the libpython library, that's why it does not work when libpython3.4m.dylib is not found.

The default installation of Python on Mac creates a soft link of this library in /usr/lib, which is the searched by the loader by default. You probably can create the soft link for your Python version manually so there is no need to set the environment variable.

Another way to avoid setting the environment variable DYLD_LIBRARY_PATH each time is to set it when shell is launched. For example, add the setting to the file ~/.bash_profile or ~/.bashrc.

Changing the system environment variable within Python does not work for current Python session. You may consider changing os.environ in Python and launch another Python process from current Python, but I am not sure how this works better than other ideas.

[Evan]

Thanks for the alternate solution. I tried

sudo ln -s /Users/Evan/miniconda/lib/libpython3.4m.dylib /usr/lib

And that seems to have fixed things. However, I'm not quite comfortable modifying these types of directories manually, especially those which are root-protected, so I'm not sure if I'll leave it there.

Just out of curiosity, would it be feasible for this to be fixed from The Mathworks' end in a future release of the Matlab engine for Python? I don't know enough about this sort of thing, but it seems like it should be possible to detect that there is a non-default Python install on the system and find the appropriate library directory.