What is the correct way to configure an external, cross-compiling toolchain, where the host architecture's executable are in a different sysroot than the target's runtime libs?
I'm trying to cross compile OpenWrt v22 on a x86_64 build machine targeting a powerPC64 NXP t1024 based board.
I'm able to build the Linux kernel through OpenWRT's build using my external toolchain, but as soon as it gets to the OpenWRT libraries built with CMake (specifically libnl-tiny), it seems to not be using my external toolchain correctly anymore.
I see another forum thread [Help] [External toolchain] How to properly set external toolchain that I anticipate is having the same issue, but does not help me resolve my difficultly.
I have a cross-compiling toolchain distributed by NXP installed at /path/to/.../sysroots outside the OpenWRT build tree.
The toolchain has two sysroots:
x86_64-linux, which contains the x86 toolchain executable at
x86_64-linux/usr/bin/powerpc64-fsl-linux/powerpc64-fsl-linux-{gcc,g++,ld, ...},
it does not contain a c-runtime libs likecrt1.o/crti.o/-lc/-lgcct1024rdb-64bwhich contains no x86 executables,
it does contain ppc64 c-runtime libs liket1024rdb-64b/usr/lib64/crt1.oglibc, libgcc etc
The similar thread I mentioned earlier [Help] [External toolchain] How to properly set external toolchain had a solutiion posed that the toolchain must just not contain libc.so.6, but I think the setup might have been similar where, since it's a cross compiling toolchain, the host/build arch's executable gcc, g++, ld, etc are in a different sysroot than the target arch's runtime libs
I have in my OpenWRT .config
CONFIG_TOOLCHAIN_PREFIX="powerpc64-fsl-linux-"
CONFIG_TOOLCHAIN_ROOT="/path/to/.../sysroots/t1024rdb-64b/"
CONFIG_EXTERNAL_TOOLCHAIN_LIBC_USE_GLIBC=y
CONFIG_TOOLCHAIN_LIBC="glibc"
CONFIG_TOOLCHAIN_BIN_PATH="../x86_64-linux/usr/bin ../x86_64-linux/bin"
CONFIG_TOOLCHAIN_INC_PATH="./usr/include"
CONFIG_TOOLCHAIN_LIB_PATH="./usr/lib64 ./lib ./lib64"
To hopefully reflect that the BIN path is in the x86_64 sysroot while the INC and LIB path
is in the ppc sysroot
(I've tried other variations tpp, with the root at sysroots and BIN at ./x86_64-linux/... and INC/LIB paths at ./t1024rdb-64b/usr/...)
The most progress I've gotten is using a CMake environment variables and setting
# expose the x86 toolchain executables in the PATH
export PATH="/path/to/.../sysroots/x86_64-linux/usr/bin/powerpc64-fsl-linux:$PATH"
export ARCH=powerpc64
export CROSS_COMPILE=powerpc64-fsl-linux-
# Add the ppc64 sysroot to the invocation of CC and LD
export TARGET_CFLAGS='--sysroot=/path/to/.../sysroots/t1024rdb-64b'
export TARGET_LDFLAGS='--sysroot=/path/to/.../sysroots/t1024rdb-64b'
Again, this works for builing the kernel build with Kbuild, but as soon as the CMake based build starts, it flounders.
At this moment, it fails when trying to install libnl-tiny citing runtime library libcc.so.6 is missing:
install -d -m0755 /path/to/.../openwrt/build_dir/target-powerpc64-fsl-linux_glibc/libnl-tiny-2021-11-21-8e0555fb/ipkg-powerpc64_e5500/libnl-tiny/usr/lib
install -m0644 /path/to/.../openwrt/build_dir/target-powerpc64-fsl-linux_glibc/libnl-tiny-2021-11-21-8e0555fb/ipkg-install/usr/lib/libnl-tiny.so /path/to/.../openwrt/build_dir/target-powerpc64-fsl-linux_glibc/libnl-tiny-2021-11-21-8e0555fb/ipkg-powerpc64_e5500/libnl-tiny/usr/lib/
find /path/to/.../openwrt/build_dir/target-powerpc64-fsl-linux_glibc/libnl-tiny-2021-11-21-8e0555fb/ipkg-powerpc64_e5500/libnl-tiny -name 'CVS' -o -name '.svn' -o -name '.#*' -o -name '*~'| xargs -r rm -rf
Package libnl-tiny is missing dependencies for the following libraries:
libc.so.6
I can confirm I do I have libc.so.6,, but only in the t1024rdb-64b sysroot
❯ find -name libc.so.6
./t1024rdb-64b/lib64/libc.so.6
./t1024rdb-64b/t1024rdb-64b/lib64/libc.so.6
./t1024rdb-64b/t1024rdb-64b/usr/include/glibc-locale-internal-ppc64e5500-fsl-linux/lib64/libc.so.6
./t1024rdb-64b/usr/include/glibc-locale-internal-ppc64e5500-fsl-linux/lib64/libc.so.6
Other tidbits:
I saw the cross compiling page on the wiki but did not find my answer https://openwrt.org/docs/guide-developer/toolchain/crosscompile
I also saw a script in the OpenWRT build at openwrt/scripts/ext-tool.sh for manipulating toolchain/info.mk and I investigated using the script to create a wrapper that adds the sysroot paramater to CC and LD, but couldn't work it out:
./scripts/ext-toolchain.sh \
--toolchain /path/to/.../sysroots/x86_64-linux/usr/bin/powerpc64-fsl-linux \
--wrap /path/to/.../openwrt/staging_dir/toolchain-wrappers