OpenWrt Forum Archive

Topic: [Howto] Build with ccache and distcc

The content of this topic has been archived on 25 Mar 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

Build with ccache and distcc

What is ccache? --> http://en.wikipedia.org/wiki/Ccache
What is distcc? --> http://en.wikipedia.org/wiki/Distcc

Requirements:
- At least two computers with the same architecture connected via LAN (if you like to compile on a mixed farm (with 32bit and 64bit hosts) you've to take a look at the crosstool and build extra toolchains. E.g. Canadian cross)
- The same libc and compiler version on all computers in the compiling farm
- In this example I used a 32bit Ubuntu Feisty Fawn OS on all computers (on the distcc servers a LiveDVD)
- The OpenWrt Kamikaze build-system on the distcc client

distcc client:
- Intel Core Duo (with 32bit Ubuntu Feisty Fawn on external USB harddisk)
- Two cores
- 2 GB RAM
- IP: 192.168.178.31

distcc server:
- AMD Turion64 (with 32-Bit Ubuntu Feisty Fawn LiveDVD)
- Two cores
- 1 GB RAM
- IP: 192.168.178.50



Prepare the distcc client:

Install some packages
- distccmon-gnome
- distcc
- nfs-kernel-server

Configure the NFS server
Create a new directory for the share

mkdir -p /home/ubuntu/openwrt

Configure a new NFS share in /etc/exports

/home/ubuntu/openwrt   192.168.178.0/255.255.255.0(ro,sync,subtree_check)

Restart the NFS server

sudo /etc/init.d/nfs-kernel-server restart

Configure distcc
Change /etc/default/distcc

STARTDISTCC="true"
ALLOWEDNETS="127.0.0.1 192.168.178.0/24"
LISTENER=""

Restart distcc

sudo /etc/init.d/distcc restart

Prepare the distcc server:

Install some packages:
- distcc
- nfs-common

Configure the NFS client
Access the share. It's important that the path for the shares are the same on the distcc client and server.

mkdir /home/ubuntu/openwrt
sudo mount -o ro 192.168.178.31:/home/ubuntu/openwrt /home/ubuntu/openwrt

Configure distcc
Change /etc/default/distcc

STARTDISTCC="true"
ALLOWEDNETS="127.0.0.1 192.168.178.0/24"
LISTENER=""

Restart distcc

sudo /etc/init.d/distcc restart

Prepare and start building on the distcc client

Get the OpenWrt Kamikaze build-system

cd ~/openwrt/
svn -q checkout https://svn.openwrt.org/openwrt/trunk/ trunk/

Change $(TOPDIR)/rules.mk

# APPLICATIONS #
HOSTCC:=distcc gcc
TARGET_CC:=distcc $(STAGING_DIR)/bin/$(TARGET_CROSS)gcc

Run 'make menuconfig' and select some additinal thing
- [*] Advanced configuration options (for developers)
  - Build Options
  - (8) Number of jobs to run simultaneously
  - [*] Use ccache

Now tell distcc which distcc servers it should use for compiling. The order of the hosts in DISTCC_HOSTS is important. Use the fastet machine first. You can also add localhost. You can add more than two distcc servers.

export DISTCC_HOSTS="192.168.178.50 localhost"

Start the build normally. The tools and toolchain isn't distributed over distcc. The first time where you see distcc avtivities is compiling the target kernel and then of course the packages.

time make world

With distccmon-gnome you can watch distcc's activities. To get some statistics for ccache run:

CCACHE_DIR=staging_dir_i386/ccache staging_dir_i386/usr/bin/ccache -s

(Last edited by forum2006 on 3 Aug 2007, 10:19)

Interesting.

Correct me if I'm wrong, but doing so, you only cache calls to native (typically x86) gcc, but not to mipsel-linux-uclibc-gcc.  Caching those would surely lead to another boost.

Any hints on how to do this ?

Deliciously interesting stuff. Thanks for sharing : )

jmranger wrote:

Correct me if I'm wrong, but doing so, you only cache calls to native (typically x86) gcc, but not to mipsel-linux-uclibc-gcc.  Caching those would surely lead to another boost.

I think this line from his howto does exactly that:

TARGET_CC:=distcc $(STAGING_DIR)/bin/$(TARGET_CROSS)gcc

(Last edited by DerAgo on 4 Apr 2008, 23:51)

Hi, does this still work for compiling out of TRUNK?

I followed the steps and it does not seem to work. The only problem I had with it was that I could not find where to configure number of jobs:
---
Run 'make menuconfig' and select some additinal thing
- [*] Advanced configuration options (for developers)
  - Build Options
  - (8) Number of jobs to run simultaneously
  - [*] Use ccache

?

The discussion might have continued from here.