Error building Python3 on openSUSE

For Developers
1

Building Python3 OpenWRT 18.06.1 on openSUSE LEAP 15.0:

gcc -shared -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib64 -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib64 -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib64 -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -O2 -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/include -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/include -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/include -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/include -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/include -I/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/include build/temp.linux-x86_64-3.6/local/jan/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.6/local/jan/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.6/local/jan/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.6/local/jan/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/stgdict.o build/temp.linux-x86_64-3.6/local/jan/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/cfield.o -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/local/jan/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -lffi -ldl -o build/lib.linux-x86_64-3.6/_ctypes.cpython-36.so
*** WARNING: renaming "_ctypes" since importing it failed: libffi.so.6: cannot open shared object file: No such file or directory

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_tkinter
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
atexit pwd time
zlib

Failed to build these modules:
_lzma _ssl

Following modules built successfully but were removed because they could not be imported:
_ctypes

Later on the lack of the _ctypes module is fatal. My guess is the import test uses the wrong path, and looks in the host system where there is no libffi.so.6, the host has libffi.so.7. The same config and OpenWRT version builds on Ubuntu (I guess it's Ubuntu 18.04):

gcc -pthread -shared -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -lssl -lresolv -lcrypto -lpthread -lcrypto -lresolv -lpthread -Wl,--no-as-needed -lrt -O2 -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/include -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/include -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/include -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/include -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/include -I/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/target-arm_cortex-a8+vfpv3_musl_eabi/host/include build/temp.linux-x86_64-3.6/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.6/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.6/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.6/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/stgdict.o build/temp.linux-x86_64-3.6/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/build_dir/hostpkg/Python-3.6.5/Modules/_ctypes/cfield.o -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/host/lib -L/home/oen/work/gtu/mcg/openwrt-h1164-mockup-25/staging_dir/hostpkg/lib -lffi -ldl -o build/lib.linux-x86_64-3.6/_ctypes.cpython-36.so
*** WARNING: renaming "_crypt" since importing it failed: build/lib.linux-x86_64-3.6/_crypt.cpython-36.so: undefined symbol: crypt
*** WARNING: renaming "nis" since importing it failed: build/lib.linux-x86_64-3.6/nis.cpython-36.so: undefined symbol: yp_master

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_curses _curses_panel _dbm
_gdbm _sqlite3 _tkinter
readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
atexit pwd time
zlib

Failed to build these modules:
_lzma _ssl

Following modules built successfully but were removed because they could not be imported:
_crypt nis

Maybee Ubuntu uses the same version of libffi, I don't currently have access to that machine so I cannot check.

By symlinking /usr/lib64/libffi.so.6 to /usr/lib64/libffi.so.7 the build gets passed this, indicating that my assumption that the _ctypes module is linked against staging, but the import test is against the host is correct. But making such a symlink is definitely the wrong solution.

Now I get the following error (now I have also changed packages from 18.06 to master):

Package python3-light is missing dependencies for the following libraries:
libuuid.so.1

But that's another story I suppose.

If there are any experts on the python build I appreciate some help with this. I guess this bug will come and bite people later on when Ubuntu upgrades libffi.

2

You are hitting that issue because 18.06 uses Python 3.6 and PYTHON3_SO_SUFFIX:=cpython-$(PYTHON3_VERSION_MAJOR)$(PYTHON3_VERSION_MINOR).so

So since the host system has a newer libffi it does not match the defined one.
Master should build, but I have no idea how to build 18.06 on systems that use new libffi.

3

No, packages master fails to build python 3.7.0

4

Hm, then I hit the same issue on Fedora28 and now on 29.
Tried googling around,but cant find anything similar with any answers

5

I'm guessing there is something wrong in the makefiles, as they specify where the system should look for to find libraries during the build. But I don't know enough about python nor the makefiles to find the error myself.

The makefiles to compile python are here in the source https://github.com/openwrt/packages/tree/master/lang/python (and in the /feeds/packages/lang/python in the local folder you use to compile OpenWrt), in case you want to try to look into this yourself.

6

Moved the -rpath linker option in the package makefiles. The change is in https://github.com/jan-kardell/openwrt-packages/tree/TQ_fix_python3, tried to make a pull request. Master now compiles on LEAP 15.

The build system is a bit complicated, so I am not sure it is the correct fix, but it seems to work.

7

Will give it a go on Fedora 29.
Did you open a PR?

8

Yes, as I wrote, I did a pull request.