Python3 host build fails to install

Mazilo · June 15, 2017, 11:49am

I don't know if this only happens to me or not. However, for the past week, LEDE buildroot has run into building python3 host with the following error messages during installation. Any idea how to fix this?

rm /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/_sysconfigdata*.py rm -r /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/__pycache__ rm: cannot remove '/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/__pycache__': No such file or directory Makefile:1413: recipe for target 'sharedinstall' failed make[3]: [sharedinstall] Error 1 (ignored) /usr/bin/install -c -m 644 ./Misc/python.man \ /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1/python3.6.1 if test ! -d /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; then \ echo "Creating directory /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig"; \ /usr/bin/install -c -d -m 755 /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; \ fi if test -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3 -o -h /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3; \ then rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3; \ else true; \ fi (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6 python3) if test "3.6" != "3.6"; then \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3.6-config; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-config python3.6-config); \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig/python-3.6.pc; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; ln -s python-3.6.pc python-3.6.pc); \ fi rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3-config (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-config python3-config) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig/python3.pc (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; ln -s python-3.6.pc python3.pc) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/idle3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s idle3.6 idle3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/pydoc3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s pydoc3.6 pydoc3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/2to3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s 2to3-3.6 2to3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/pyvenv (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s pyvenv-3.6 pyvenv) if test "x" != "x" ; then \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3-32; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-32 python3-32) \ fi rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1/python3.1 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1; ln -s python3.6.1 python3.1) if test "xinstall" != "xno" ; then \ case install in \ upgrade) ensurepip="--upgrade" ;; \ install|*) ensurepip="" ;; \ esac; \ ./python -E -m ensurepip \ $ensurepip --root=/ ; \ fi Traceback (most recent call last): File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__main__.py", line 4, in <module> ensurepip._main() File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 189, in _main default_pip=args.default_pip, File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 102, in bootstrap _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 27, in _run_pip import pip File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 28, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/vcs/mercurial.py", line 9, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 36, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/utils/glibc.py", line 4, in <module> File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ModuleNotFoundError: No module named '_ctypes' Makefile:1064: recipe for target 'install' failed make[3]: *** [install] Error 1 make[3]: Leaving directory '/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1' Makefile:287: recipe for target '/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/stamp/.python3_installed' failed make[2]: *** [/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/stamp/.python3_installed] Error 2 make[2]: Leaving directory '/mnt/exports/devel/LEDE-git/feeds/packages/lang/python/python3' package/Makefile:109: recipe for target 'package/feeds/packages/python3/host/compile' failed make[1]: *** [package/feeds/packages/python3/host/compile] Error 2 make[1]: Leaving directory '/mnt/exports/devel/LEDE-git' /mnt/exports/devel/LEDE-git/include/toplevel.mk:207: recipe for target 'package/python3/host/compile' failed make: *** [package/python3/host/compile] Error 2

commodo · June 16, 2017, 4:36pm

Looks like one of those voodoo things that's very Python-esque.
The host build of Python is influenced by stuff you may have installed on your system.
Because Python detects stuff you have on your host, and could try to use them.
[ Well, a lot of other packages try to do the same, but Python takes it to another level ]

Can you tell me what distro you're using ?
I could try to reproduce this and get an idea of what & how to fix

Thanks

Mazilo · June 17, 2017, 12:24am

I am using OpenSuSE 42.2 on an old AMD64 Phenom II X3 with a 4GB RAM. After the commit (68312f49c0b4456e665ed438a86a44738caa765a) to move python packages, the compilation still worked. Then, a few days later on, it no longer compiles with such an error message.

commodo · June 19, 2017, 7:39pm

Sorry for taking so long to get back (weekend and other stuff I'm working on).

I'll have to spin up a VM with OpenSUSE 42.2.
I haven't used SUSE in a while.

Curious: did you try to build this with a clean checkout ?

This issue sort of reminds me of this one.
https://github.com/openwrt/packages/issues/4020
It's not identical, but similar.

The fix was here:
https://github.com/commodo/packages/commit/519df664a5a578070c68f4b8046603af868e97c1

If you have time, could you try to add that line to your Python3 Makefile ?
I'll try to reproduce it locally anyway

Mazilo · June 21, 2017, 12:18pm

I followed your suggestion to create a patch file as shown below and ended up with the same error messages. So, I did a make dirclean and then rebuild/recompile from scratch to end up with another compilation on libiconv-full package as shown here before it reaches to build python3 host package. So, until I can get libiconv-full package properly built, it looks like I can't test to rebuild python3 host package.

[code]--- a/feeds/packages/lang/python/python3/Makefile
+++ b/feeds/packages/lang/python/python3/Makefile
@@ -258,6 +258,7 @@ define Py3Package/python3/filespec
endef

HOST_LDFLAGS += \

  -Wl,--no-as-needed -lrt \
  $$$$(pkg-config --static --libs libcrypto libssl)

HOST_CONFIGURE_ARGS+= [/code]

commodo · June 23, 2017, 6:18am

Hey,

On my openSUSE 42.2 freshly installed VM it worked fine.
I used the latest trunk repo [from sometime yesterday].
And the latest packages feeds/repo.

I admit I did not install the full openSUSE stuff ; so just to get a console working.
Maybe there are some other packages in openSUSE that interfere with the Python3 host build ; this sometimes happens on some distros.

Any other info here/now would help about your setup.

Mazilo · June 23, 2017, 9:43pm

I have both my OpenSuSE v42.2 and LEDE updated to the latest. I even tried to re-download LEDE source from git repo into an empty directory and ended up with the same error message for libiconv-full. The only thing I can think of is I configure my LEDE to use binutils-2.28, gcc-7.x, and musl. Also as I mentioned before, python3 used to build OK even right after I updated with commit 68312f49c0b4456e665ed438a86a44738caa765a (moving all pythonX packages under python)

commodo · July 2, 2017, 12:29pm

So, the move itself should not be a problem.

Will retry with your settings.
I'm testing with all LEDE defaults.

commodo · July 3, 2017, 1:46pm

Actually, if it's not too much trouble.

Can you send me your .config file ? ; if you have certain stuff that's specific to your setup (i.e. package configuration, gcc, binutils, etc), I'm hoping it would be easier for me to reproduce.

zhangguanzhang · May 19, 2022, 5:32pm

github.com/openwrt/packages

Python3: Fix host build on OpenSUSE

committed 09:28AM - 20 May 20 UTC

jan-kardell

+1 -1

The linker option -rpath is required to find libs in staging_dir. Now it is incl…uded when building host modules. Without it the import test of the _ctypes and _uuid modules would fail. The _ctypes module uses libffi.so.6 from staging, but OpenSUSE LEAP 15 has libffi.so.7. It will also fail on LEAP 42.x, Fedora28 and 29 and future or old versions of Ubuntu. Fix needed in master and 18.06 branches. Signed-off-by: Jan Kardell <jan.kardell@telliq.com> (cherry picked from commit 6ade5a1e3adb189d062b707e2b8f731c9faab844)

add them:

 -Wl$(comma)-rpath=$(STAGING_DIR_HOSTPKG)/lib

Mazilo · May 20, 2022, 10:42am

Thank you.