How to build h5py package

joshua79.kim · January 23, 2019, 8:33am

Hi all. I'm trying to build h5py package in imx6 environment.
Basically openwrt does not support some python packages like h5py, cython, pkgconfig, numpy.
So, I created new openwrt makefiles for them.
Most makefiles are working well, build process was succeded except h5py.

I share you a error log and makefile for h5py like below.
How can I fix it?

[Makefile]

include $(TOPDIR)/rules.mk

PKG_NAME:=python-h5py
PKG_VERSION:=2.9.0
PKG_RELEASE:=1

PKG_SOURCE:=h5py-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://files.pythonhosted.org/packages/43/27/a6e7dcb8ae20a4dbf3725321058923fec262b6f7835179d78ccc8d98deec
PKG_HASH:=9d41ca62daf36d6b6515ab8765e4c8c4388ee18e2a665701fef2b41563821002

PKG_BUILD_DIR:=$(BUILD_DIR)/$(BUILD_VARIANT)-h5py-$(PKG_VERSION)

include $(INCLUDE_DIR)/package.mk
include ../python-package.mk
include ../python3-package.mk

PKG_UNPACK:=$(HOST_TAR) -C $(PKG_BUILD_DIR) --strip-components=1 -xzf $(DL_DIR)/$(PKG_SOURCE)
HOST_UNPACK:=$(HOST_TAR) -C $(HOST_BUILD_DIR) --strip-components=1 -xzf $(DL_DIR)/$(PKG_SOURCE)

define Package/python-h5py/Default
  SECTION:=lang
  CATEGORY:=Languages
  SUBMENU:=Python
endef

define Package/python-h5py
$(call Package/python-h5py/Default)
  TITLE:=python-h5py
  DEPENDS:=+libstdcpp +python-light +python-pkgconfig +python-six +python-numpy +python-cython
  VARIANT:=python
endef

define Package/python3-h5py
$(call Package/python-h5py/Default)
  TITLE:=python3-h5py
  DEPENDS:=+libstdcpp +python3-light +python3-pkgconfig +python3-six +python3-numpy +python3-cython
  VARIANT:=python3
endef

define Package/python-h5py/description
  h5py is a C-coded Python extension module
endef

define Package/python3-h5py/description
$(call Package/python-h5py/description)
.
(Variant for Python3)
endef

define PyBuild/Compile
	$(call Build/Compile/PyMod,,install --prefix="/usr" --root="$(PKG_INSTALL_DIR)")
endef

define Py3Build/Compile
	$(call Build/Compile/Py3Mod,,install --prefix="/usr" --root="$(PKG_INSTALL_DIR)")
endef

define Host/Compile
	$(call Build/Compile/HostPyMod,,install --prefix="" --root="$(STAGING_DIR_HOSTPKG)")
endef

Host/Install:=

$(eval $(call HostBuild))

$(eval $(call PyPackage,python-h5py))
$(eval $(call BuildPackage,python-h5py))

[Build Error Log]

running install
running build
running build_py
creating build
creating build/lib.linux2-2.7
creating build/lib.linux2-2.7/h5py
copying h5py/highlevel.py -> build/lib.linux2-2.7/h5py
copying h5py/h5py_warnings.py -> build/lib.linux2-2.7/h5py
copying h5py/__init__.py -> build/lib.linux2-2.7/h5py
copying h5py/ipy_completer.py -> build/lib.linux2-2.7/h5py
copying h5py/version.py -> build/lib.linux2-2.7/h5py
creating build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/dataset.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/group.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/base.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/compat.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/datatype.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/dims.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/vds.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/attrs.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/selections2.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/filters.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/files.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/__init__.py -> build/lib.linux2-2.7/h5py/_hl
copying h5py/_hl/selections.py -> build/lib.linux2-2.7/h5py/_hl
creating build/lib.linux2-2.7/h5py/tests
copying h5py/tests/__init__.py -> build/lib.linux2-2.7/h5py/tests
copying h5py/tests/common.py -> build/lib.linux2-2.7/h5py/tests
creating build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_slicing.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_selections.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_dataset.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_h5p.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_datatype.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_objects.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_attrs_data.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_base.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_file.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_h5f.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_attrs.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_group.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_h5t.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/__init__.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_file_image.py -> build/lib.linux2-2.7/h5py/tests/old
copying h5py/tests/old/test_h5.py -> build/lib.linux2-2.7/h5py/tests/old
creating build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_filters.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_deprecation.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_datatype.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_file.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/__init__.py -> build/lib.linux2-2.7/h5py/tests/hl
copying h5py/tests/hl/test_threads.py -> build/lib.linux2-2.7/h5py/tests/hl
creating build/lib.linux2-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux2-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux2-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux2-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux2-2.7/h5py/tests/hl/test_vds
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
Traceback (most recent call last):
  File "./setup.py", line 168, in <module>
    cmdclass = CMDCLASS,
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/command/install.py", line 563, in run
    self.run_command('build')
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/home/user/prj/openwrt/openwrt/staging_dir/target-arm_cortex-a9+neon_musl_eabi/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/setup_build.py", line 156, in run
    from Cython.Build import cythonize
  File "/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.eggs/Cython-0.29.3-py2.7-linux2.egg/Cython/Build/__init__.py", line 1, in <module>
    from .Dependencies import cythonize
  File "/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.eggs/Cython-0.29.3-py2.7-linux2.egg/Cython/Build/Dependencies.py", line 45, in <module>
    from ..Compiler.Main import Context, CompilationOptions, default_options
  File "/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.eggs/Cython-0.29.3-py2.7-linux2.egg/Cython/Compiler/Main.py", line 28, in <module>
    from .Scanning import PyrexScanner, FileSourceDescriptor
ImportError: /home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.eggs/Cython-0.29.3-py2.7-linux2.egg/Cython/Compiler/Scanning.so: wrong ELF class: ELFCLASS32
Makefile:74: recipe for target '/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.built' failed
make[2]: *** [/home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.built] Error 1
make[2]: Leaving directory '/home/user/prj/openwrt/openwrt/feeds/packages/lang/python/python-h5py'
time: package/feeds/packages/python-h5py/python/compile#206.37#7.58#113.83
package/Makefile:107: recipe for target 'package/feeds/packages/python-h5py/compile' failed
make[1]: *** [package/feeds/packages/python-h5py/compile] Error 2
make[1]: Leaving directory '/home/user/prj/openwrt/openwrt'
/home/user/prj/openwrt/openwrt/include/toplevel.mk:216: 'package/feeds/packages/python-h5py/compile' 타겟에 대한 명령이 실패했습니다
make: *** [package/feeds/packages/python-h5py/compile] 오류 2

jeff · January 23, 2019, 6:09pm

joshua79.kim:

ImportError: /home/user/prj/openwrt/openwrt/build_dir/target-arm_cortex-a9+neon_musl_eabi/python-h5py-2.9.0/.eggs/Cython-0.29.3-py2.7-linux2.egg/Cython/Compiler/Scanning.so: wrong ELF class: ELFCLASS32

Looks like you may be trying to run the tests of a module that contains machine-specific code for your target on your build machine. Two options would be skipping the test target of the module, or somehow spinning up a virtualized replica of your target on your build machine. The first seems a reasonably sane approach, given the complexity of the second!

joshua79.kim · January 24, 2019, 2:51am

Thank you.
But, I can't understand your comment. Could you explain it more detail?

jeff · January 24, 2019, 4:33am

The module looks like it has C or other code that is compiled for the target. As the target isn't the same as the build system, it is cross compiled and the result will not run on the build host. When the Python build scripts try to test the module, it calls python on the host system. It can't load the module as is for a different architecture.

The easiest solution is to modify the Python build instructions to omit the test phase.