Python3 host build fails to install

I don't know if this only happens to me or not. However, for the past week, LEDE buildroot has run into building python3 host with the following error messages during installation. Any idea how to fix this?

rm /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/_sysconfigdata*.py rm -r /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/__pycache__ rm: cannot remove '/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/python3.6/lib-dynload/__pycache__': No such file or directory Makefile:1413: recipe for target 'sharedinstall' failed make[3]: [sharedinstall] Error 1 (ignored) /usr/bin/install -c -m 644 ./Misc/python.man \ /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1/python3.6.1 if test ! -d /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; then \ echo "Creating directory /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig"; \ /usr/bin/install -c -d -m 755 /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; \ fi if test -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3 -o -h /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3; \ then rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3; \ else true; \ fi (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6 python3) if test "3.6" != "3.6"; then \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3.6-config; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-config python3.6-config); \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig/python-3.6.pc; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; ln -s python-3.6.pc python-3.6.pc); \ fi rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3-config (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-config python3-config) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig/python3.pc (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/lib/pkgconfig; ln -s python-3.6.pc python3.pc) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/idle3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s idle3.6 idle3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/pydoc3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s pydoc3.6 pydoc3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/2to3 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s 2to3-3.6 2to3) rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/pyvenv (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s pyvenv-3.6 pyvenv) if test "x" != "x" ; then \ rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin/python3-32; \ (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/bin; ln -s python3.6-32 python3-32) \ fi rm -f /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1/python3.1 (cd /mnt/exports/devel/LEDE-git/staging_dir/hostpkg/share/man/man1; ln -s python3.6.1 python3.1) if test "xinstall" != "xno" ; then \ case install in \ upgrade) ensurepip="--upgrade" ;; \ install|*) ensurepip="" ;; \ esac; \ ./python -E -m ensurepip \ $ensurepip --root=/ ; \ fi Traceback (most recent call last): File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__main__.py", line 4, in <module> ensurepip._main() File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 189, in _main default_pip=args.default_pip, File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 102, in bootstrap _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ensurepip/__init__.py", line 27, in _run_pip import pip File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 28, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/vcs/mercurial.py", line 9, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/download.py", line 36, in <module> File "/tmp/tmpu3_5yog3/pip-9.0.1-py2.py3-none-any.whl/pip/utils/glibc.py", line 4, in <module> File "/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1/Lib/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ModuleNotFoundError: No module named '_ctypes' Makefile:1064: recipe for target 'install' failed make[3]: *** [install] Error 1 make[3]: Leaving directory '/mnt/exports/devel/LEDE-git/build_dir/hostpkg/Python-3.6.1' Makefile:287: recipe for target '/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/stamp/.python3_installed' failed make[2]: *** [/mnt/exports/devel/LEDE-git/staging_dir/hostpkg/stamp/.python3_installed] Error 2 make[2]: Leaving directory '/mnt/exports/devel/LEDE-git/feeds/packages/lang/python/python3' package/Makefile:109: recipe for target 'package/feeds/packages/python3/host/compile' failed make[1]: *** [package/feeds/packages/python3/host/compile] Error 2 make[1]: Leaving directory '/mnt/exports/devel/LEDE-git' /mnt/exports/devel/LEDE-git/include/toplevel.mk:207: recipe for target 'package/python3/host/compile' failed make: *** [package/python3/host/compile] Error 2

Looks like one of those voodoo things that's very Python-esque.
The host build of Python is influenced by stuff you may have installed on your system.
Because Python detects stuff you have on your host, and could try to use them.
[ Well, a lot of other packages try to do the same, but Python takes it to another level ]

Can you tell me what distro you're using ?
I could try to reproduce this and get an idea of what & how to fix

Thanks

I am using OpenSuSE 42.2 on an old AMD64 Phenom II X3 with a 4GB RAM. After the commit (68312f49c0b4456e665ed438a86a44738caa765a) to move python packages, the compilation still worked. Then, a few days later on, it no longer compiles with such an error message.

Sorry for taking so long to get back (weekend and other stuff I'm working on).

I'll have to spin up a VM with OpenSUSE 42.2.
I haven't used SUSE in a while.

Curious: did you try to build this with a clean checkout ?

This issue sort of reminds me of this one.
https://github.com/openwrt/packages/issues/4020
It's not identical, but similar.

The fix was here:
https://github.com/commodo/packages/commit/519df664a5a578070c68f4b8046603af868e97c1

If you have time, could you try to add that line to your Python3 Makefile ?
I'll try to reproduce it locally anyway :slight_smile:

I followed your suggestion to create a patch file as shown below and ended up with the same error messages. So, I did a make dirclean and then rebuild/recompile from scratch to end up with another compilation on libiconv-full package as shown here before it reaches to build python3 host package. So, until I can get libiconv-full package properly built, it looks like I can't test to rebuild python3 host package. :frowning:

[code]--- a/feeds/packages/lang/python/python3/Makefile
+++ b/feeds/packages/lang/python/python3/Makefile
@@ -258,6 +258,7 @@ define Py3Package/python3/filespec
endef

HOST_LDFLAGS += \

  •   -Wl,--no-as-needed -lrt \
      $$$$(pkg-config --static --libs libcrypto libssl)
    

HOST_CONFIGURE_ARGS+= [/code]

Hey,

On my openSUSE 42.2 freshly installed VM it worked fine.
I used the latest trunk repo [from sometime yesterday].
And the latest packages feeds/repo.

I admit I did not install the full openSUSE stuff ; so just to get a console working.
Maybe there are some other packages in openSUSE that interfere with the Python3 host build ; this sometimes happens on some distros.

Any other info here/now would help about your setup.

I have both my OpenSuSE v42.2 and LEDE updated to the latest. I even tried to re-download LEDE source from git repo into an empty directory and ended up with the same error message for libiconv-full. The only thing I can think of is I configure my LEDE to use binutils-2.28, gcc-7.x, and musl. Also as I mentioned before, python3 used to build OK even right after I updated with commit 68312f49c0b4456e665ed438a86a44738caa765a (moving all pythonX packages under python)

So, the move itself should not be a problem.

Will retry with your settings.
I'm testing with all LEDE defaults.

Actually, if it's not too much trouble.

Can you send me your .config file ? ; if you have certain stuff that's specific to your setup (i.e. package configuration, gcc, binutils, etc), I'm hoping it would be easier for me to reproduce.