Distributed building

[532910]

Is it possible to use distcc for openwrt compilation?
I've tried 60-second instructions from https://distcc.github.io/ without success.

Are there some other tools for this?

[user31894691]

I think we can look into staging_dir/host/bin/.
After make download, gcc and g++ inside staging_dir/host/bin/ are pointing to /usr/bin/cc and /usr/bin/g++.

Apparently, we can try changing these two symbolic links so that they point to:

• /usr/lib/distcc/bin/gcc and /usr/lib/distcc/bin/g++ for distcc
• /usr/lib/icecream/libexec/icecc/bin/gcc and /usr/lib/icecream/libexec/icecc/bin/g++ for icecc (icecream)

I tried icecream with this method and it seemed OK. However, I think this may only accelerate the "tools" stage and not the "toolchain" and "package" stages.

[532910]

Indeed icecc works!
But in addition to the links replacement, PATH should be prepended with an icecc's bin (which is /usr/lib/icecc/bin in debian): PATH=/usr/lib/icecc/bin:$PATH

[user31894691]

I think there may be a way to use distcc for OpenWrt image building, but I can't guarantee that this method will work for everybody.
Also, I haven't tested distcc + ccache.

This method involves using an external toolchain (prebuilt OpenWrt toolchain).
Currently, some steps are needed to make external toolchain work - please refer to this comment.
In addition, someone reported that libgpg-error fails to build when using external toolchain, so YMMV.

Steps (Replace mipsel with your device's architecture):

1. Configure external toolchain by following this.
2. In staging_dir/host/bin/, create 4 wrappers (distcc_host{cc,cxx} distcc_{cc,cxx}) pointing to the corresponding compilers and chmod +x.
   Examples for distcc_hostcc and distcc_cc:

#!/bin/sh
exec distcc gcc "$@"

#!/bin/sh
exec distcc mipsel-openwrt-linux-musl-gcc "$@"

3. Edit rules.mk so that the wrappers are used:

HOSTCC:=distcc_hostcc
HOSTCXX:=distcc_hostcxx
HOSTCC_NOCACHE:=distcc_hostcc
HOSTCXX_NOCACHE:=distcc_hostcxx
TARGET_CC:=distcc_cc
TARGET_CXX:=distcc_cxx
TARGET_CC_NOCACHE:=distcc_cc
TARGET_CXX_NOCACHE:=distcc_cxx
export TARGET_CC_NOCACHE
export TARGET_CXX_NOCACHE
export HOSTCC_NOCACHE
export HOSTCXX_NOCACHE
export TARGET_CC
export TARGET_CXX
export HOSTCC
export HOSTCXX

4. Repack the external toolchain and install it on other machines
5. Setup distcc on the machines:

• Create symlinks mipsel-openwrt-linux-musl-{c++,cc,cpp,g++,gcc} in /usr/lib/distcc pointing to /usr/bin/distcc.
• Add/edit PATH= in /etc/conf.d/distccd so that distcc can find out where the compilers are located.

6. Tweak DISTCC_HOSTS and try compiling.