How to call python h5py from windows MATLAB? (Works in OS-X and Linux)

I'm developing a cross platform application which calls Python from Matlab. It works on the Mac (OSX) with no configuration required, and on Linux (after some configuration commands), but I've not been able to get it working on Windows. In Windows, the Python fails to load "h5py", e.g. executing "py.importlib.import_module('h5py')" fails. For all three environments, Python and h5py are installed using Anaconda (and also for Windows, Enthought). Output from the three environments (OSX, Linux, Windows) are given below. Again, the problem is with Windows. Any suggestion for how to get the Windows version working?

----

OSX - this works.

Versions: OSX 10.11.6., Matlab R2017a, (Python 2.7.12 Anaconda 2.4.1) or Python 3.5.2 |Continuum Analytics, Inc. (default, Jul 2 2016, 17:52:12).

Matlab dialog:

>> computer

ans =

      'MACI64'

>> version

ans =

      '9.2.0.556344 (R2017a)'

>> pyversion

         version: '3.5'
      executable: '/Users/jt/anaconda/envs/python3/bin/python'
         library: '/Users/jt/anaconda/envs/python3/lib/libpython3.5m.dylib'
            home: '/Users/jt/anaconda/envs/python3'
        isloaded: 0

>> py.importlib.import_module('h5py')  # This works on OSX

ans = 

    Python module with properties:

                      get_enum: [1×1 py.builtin_function_or_method]
                      get_vlen: [1×1 py.builtin_function_or_method]
                           h5d: [1×1 py.module]
                           h5o: [1×1 py.module]
                          h5ac: [1×1 py.module]
                       is_hdf5: [1×1 py.function]
                       version: [1×1 py.module]
                           h5r: [1×1 py.module]
                           h5t: [1×1 py.module]
                           h5l: [1×1 py.module]
                         Group: [1×1 py.abc.ABCMeta]
                           h5i: [1×1 py.module]
                           h5s: [1×1 py.module]
                      new_enum: [1×1 py.builtin_function_or_method]
                          h5ds: [1×1 py.module]
                   check_dtype: [1×1 py.cython_function_or_method]
                           h5p: [1×1 py.module]
                      HLObject: [1×1 py.type]
                         tests: [1×1 py.module]
                     run_tests: [1×1 py.function]
                     Reference: [1×1 py.type]
                      HardLink: [1×1 py.type]
                     highlevel: [1×1 py.module]
                          File: [1×1 py.abc.ABCMeta]
                           h5z: [1×1 py.module]
                 special_dtype: [1×1 py.cython_function_or_method]
                    get_config: [1×1 py.builtin_function_or_method]
                      new_vlen: [1×1 py.builtin_function_or_method]
      enable_ipython_completer: [1×1 py.function]
               absolute_import: [1×1 py.__future__._Feature]
                  ExternalLink: [1×1 py.type]
                           h5a: [1×1 py.module]
                           h5f: [1×1 py.module]
              AttributeManager: [1×1 py.abc.ABCMeta]
               RegionReference: [1×1 py.type]
                           h5g: [1×1 py.module]
                       Dataset: [1×1 py.type]
                      SoftLink: [1×1 py.type]
                          h5fd: [1×1 py.module]
                         utils: [1×1 py.module]
                            h5: [1×1 py.module]
                          defs: [1×1 py.module]
                       filters: [1×1 py.module]
                      Datatype: [1×1 py.type]

      <module 'h5py' from '/Users/jt/anaconda/envs/python3/lib/python3.5/site-packages/h5py/__init__.py'>

>>

Linux - this works, if configuration statements done first

Versions: Ubuntu 16.04.02 LTS, Matlab R2017a, Python 3.5.3 Continuum Analytics, Inc. (default, Mar 6 2017, 11:58:13)

Configuration commands (must be executed first; for example, stored in "startup.m"):

% matlab startup for linux use of h5py
setenv('HDF5_DISABLE_VERSION_CHECK', '1')
% RTLD_NOW = 2;
% RTLD_DEEPBIND = 8;
% flag = bitor(RTLD_NOW, RTLD_DEEPBIND); % RTLD_NOW | RTLD_DEEPBIND
% py.sys.setdlopenflags(int32(flag));
py.sys.setdlopenflags(int32(10));  % this is equivalent to the above

Matlab dialog:

>> computer

ans =

      'GLNXA64'

>> version

ans =

      '9.2.0.556344 (R2017a)'

>> pyversion

       version: '3.5'
    executable: '/home/jt/anaconda3/envs/py35/bin/python'
       library: '/home/jt/anaconda3/envs/py35/lib/libpython3.5m.so'
          home: '/home/jt/anaconda3/envs/py35'
      isloaded: 1

>> py.importlib.import_module('h5py')  # This works on Linux

ans = 

    Python module with properties:

                    get_config: [1×1 py.builtin_function_or_method]
                          defs: [1×1 py.module]
                           h5o: [1×1 py.module]
                          h5ds: [1×1 py.module]
                      get_enum: [1×1 py.builtin_function_or_method]
                          File: [1×1 py.abc.ABCMeta]
              AttributeManager: [1×1 py.abc.ABCMeta]
                       is_hdf5: [1×1 py.function]
                      new_vlen: [1×1 py.builtin_function_or_method]
                       version: [1×1 py.module]
                      HLObject: [1×1 py.type]
                     highlevel: [1×1 py.module]
                      SoftLink: [1×1 py.type]
                           h5a: [1×1 py.module]
                         Group: [1×1 py.abc.ABCMeta]
                 special_dtype: [1×1 py.cython_function_or_method]
                         utils: [1×1 py.module]
                           h5f: [1×1 py.module]
                   check_dtype: [1×1 py.cython_function_or_method]
                      HardLink: [1×1 py.type]
                     run_tests: [1×1 py.function]
                           h5g: [1×1 py.module]
                       Dataset: [1×1 py.type]
                     Reference: [1×1 py.type]
                         tests: [1×1 py.module]
      enable_ipython_completer: [1×1 py.function]
                      new_enum: [1×1 py.builtin_function_or_method]
               absolute_import: [1×1 py.__future__._Feature]
                           h5t: [1×1 py.module]
                           h5d: [1×1 py.module]
                          h5ac: [1×1 py.module]
               RegionReference: [1×1 py.type]
                  ExternalLink: [1×1 py.type]
                      Datatype: [1×1 py.type]
                      get_vlen: [1×1 py.builtin_function_or_method]
                           h5p: [1×1 py.module]
                          h5fd: [1×1 py.module]
                           h5r: [1×1 py.module]
                         Empty: [1×1 py.type]
                       filters: [1×1 py.module]
                           h5i: [1×1 py.module]
                           h5l: [1×1 py.module]
                           h5s: [1×1 py.module]
                            h5: [1×1 py.module]
                           h5z: [1×1 py.module]

      <module 'h5py' from '/home/jt/anaconda3/envs/py35/lib/python3.5/site-packages/h5py/__init__.py'>

>>

Windows. This does not work. My question is how to get it working?

Versions: Microsoft Windows [Version 10.0.15063]. Anaconda conda 4.3.21. Python 3.5.3 Continuum Analytics, Inc. (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)] on win32. Also tried: Python 2.7.13 Continuum Analytics, Inc. (default, May 11 2017, 13:17:26) [MSC v.1500 64 bit (AMD64)] on win32; and Enthought Python 2.7).

Matlab dialog with Python 3.5:

>> computer

ans =

      'PCWIN64'

>> version

ans =

      '9.2.0.556344 (R2017a)'

>> pyversion

         version: '3.5'
      executable: 'c:\Users\jt\Anaconda3\envs\py35\python.exe'
         library: 'c:\Users\jt\Anaconda3\envs\py35\python35.dll'
            home: 'c:\Users\jt\Anaconda3\envs\py35'
        isloaded: 0

>> setenv('HDF5_DISABLE_VERSION_CHECK', '1') # this does not seem to affect the behavior on Windows
>> py.sys.setdlopenflags(int32(10));  % Windows Python does not support this
Undefined variable "py" or class "py.sys.setdlopenflags".

>> py.importlib.import_module('h5py')  # This is what I need to get working on Windows
Error using _objects>init h5py._objects
(D:\Build\h5py\h5py-2.7.0\h5py\_objects.c:7748) (line 1)
Python Error: ImportError: DLL load failed: The specified procedure could not be
found.

Error in _objects>init h5py.h5r (D:\Build\h5py\h5py-2.7.0\h5py\h5r.c:3222) (line
12)

Error in h5r>init h5py._conv (D:\Build\h5py\h5py-2.7.0\h5py\_conv.c:7536) (line
21)

Error in __init__><module> (line 34)
from ._conv import register_converters as _register_converters

Error in <frozen importlib>_call_with_frames_removed (line 222)

Error in <frozen importlib>exec_module (line 678)

Error in <frozen importlib>_load_unlocked (line 673)

Error in <frozen importlib>_find_and_load_unlocked (line 958)

Error in <frozen importlib>_find_and_load (line 969)

Error in <frozen importlib>_gcd_import (line 986)

Error in __init__>import_module (line 126)
    return _bootstrap._gcd_import(name[level:], package, level)

>>

Dialog with Anaconda Python 2.7:

>> pyversion

         version: '2.7'
      executable: 'C:\Users\jt\Anaconda3\envs\py27\python.exe'
         library: 'C:\Users\jt\Anaconda3\envs\py27\python27.dll'
            home: 'C:\Users\jt\Anaconda3\envs\py27'
        isloaded: 0

>> py.importlib.import_module('h5py')  # This is what I need to get working on Windows
Error using _objects>init h5py._objects
(C:\Minonda\conda-bld\h5py_1496869654914\work\h5py\_objects.c:7762) (line 1)
Python Error: ImportError: DLL load failed: The specified procedure could not be
found.

Error in _objects>init h5py.h5r
(C:\Minonda\conda-bld\h5py_1496869654914\work\h5py\h5r.c:3236) (line 12)

Error in h5r>init h5py._conv
(C:\Minonda\conda-bld\h5py_1496869654914\work\h5py\_conv.c:7550) (line 21)

Error in __init__><module> (line 34)
from ._conv import register_converters as _register_converters

Error in __init__>import_module (line 37)
    __import__(name)

>>

Dialog with Enthought Python 2.7.

    >> computer

    ans =

        'PCWIN64'

    >> version

    ans =

        '9.2.0.556344 (R2017a)'

    >> pyversion

           version: '2.7'
        executable: 'C:\Users\jt\.edm\envs\edmp27\python.exe'
           library: 'C:\Users\jt\.edm\envs\edmp27\python27.dll'
              home: 'C:\Users\jt\.edm\envs\edmp27'
          isloaded: 0

    >> setenv('HDF5_DISABLE_VERSION_CHECK', '1') # this does not seem to affect the behavior on Windows
    >> py.sys.setdlopenflags(int32(10));   % Windows Python does not support this
    Undefined variable "py" or class "py.sys.setdlopenflags".

    >> py.importlib.import_module('h5py')  # This is what I need to get working on Windows
    Error using _objects>init h5py._objects
    (C:\pisi\tmp\h5py-2.7.0-1\work\h5py-2.7.0\h5py\_objects.c:7754) (line 1)
    Python Error: ImportError: DLL load failed: The specified procedure could not be
    found.

    Error in _objects>init h5py.h5r
    (C:\pisi\tmp\h5py-2.7.0-1\work\h5py-2.7.0\h5py\h5r.c:3228) (line 12)

    Error in h5r>init h5py._conv
    (C:\pisi\tmp\h5py-2.7.0-1\work\h5py-2.7.0\h5py\_conv.c:7542) (line 21)

    Error in __init__><module> (line 34)
    from ._conv import register_converters as _register_converters

    Error in __init__>import_module (line 37)
        __import__(name)

This issues seems to affect others. Some related posts I found are:

https://www.mathworks.com/matlabcentral/answers/327193-calling-python-module-from-matlab-causes-segmentation-fault-in-h5py

https://stackoverflow.com/questions/30605179/why-cant-matlab-import-this-python-library

[Peter Cook]

Perhaps not what you're looking for, but native MATLAB can I/O hdf5 files via h5info(), h5read(), h5write() etc. I work with a lot of large hdf5 files ( O(TB) ) and prefer MATLAB to anaconda/h5py for faster reads. E.G. You could read your data into MATLAB with h5read then pass it py as py.numpy.ndarray

[Jonathan Cooper]

I agree that's best for fast reading. The issue for Jeff (and me) is that we have a substantial Python library that automatically adds & validates metadata & structure within the HDF5 file on write. Jeff has written a Matlab wrapper around this library, but alas failing to import h5py stops it working on Windows.

Another benefit of h5py, of course, is that it makes it much easier to find out what datasets are in the file without having to resort to the low-level HDF5 API...

[Asad Hayat]

Have you find any solution to this yet? As i am facing the same thing now while importing tensorflow

[Johannes Ackermann]

I'm having the same issue for other matlab modules. Has anybodz found a solution?

[Matthias Walle]

I am facing the same Issue now using h5py=2.8.0 did not help.

[Sterling Olson]

I have a Python librbary with a MATLAB wrapper as well with a large database of hdf5 files we need to read. I need MATLAB to be able to work with h5py and I am looking for an answer to this issue.

Answer 1

[Alberto Escalante]

I would suggest to try the following:

1) Instead of using anaconda's python, use a global python installation compiled for 64 bit (as explained in matlab's documentation)

2) Install tensorflow in your gloabal python environment

3) uninstall package h5py (tensorflow still works!)

4) Save and retrieve your trained networks in tensorflow using a format that does not use h5py (hdf5)

Answer 2

[Sean Roginski]

I was having issues importing Tensorflow in MATLAB because of h5py. Changing pyenv (I know you used pyversion, but pyenv is more recommended at this point) to run OutOfProcess is a solution, but this configuration can cause significant overhead. I just got it to work when pyenv is InProcess while running tensorflow==2.5.0 and h5py==3.1.0. Hope this helps!