How to use ccache effectively

For Developers
1

Hi all:

I have written a script to automate building OpenWrt from scratch (a kind of private build bot). I am currently building both tag "v19.07.2" and "master".

Because I want to build repeatedly, and it takes quite long, I am investigating using ccache. Can someone please shed some light on this subject? The build system is complex and I do not understand what is going on.

First of all, I wonder if the date inside file "version.date" is used as a compilation flag. I hope not, because that would invalidate a lot of caching.

Passing CC="ccache gcc" and "CXX=ccache g++" to "make" seems to have no effect.

Setting CONFIG_CCACHE=y in the configuration has interesting results:

a) During "make download", you get these warnings:

make[2]: Entering directory '/home/rdiez/rdiez/Freifunk/OpenWrt/WorkingOpenWrtGitRepository/toolchain/musl'
bash: ccache_cc: command not found
bash: ccache_cc: command not found
bash: ccache_cc: command not found

b) The first compilations do not use ccache, for example, when building tools/xz .

c) Later on ccache does get used, for example, when building tools/lzma .

d) And it finally fails to build the Linux kernel:

make -f ./scripts/Makefile.build obj=scripts/basic
rm -f .tmp_quiet_recordmcount
Cannot use CONFIG_CC_STACKPROTECTOR_REGULAR: -fstack-protector not supported by compiler
make[5]: *** [Makefile:1134: prepare-compiler-check] Error 1

I guess there are 3 stages about ccache:

(1) When building the first tools that run on the host, like tar or xz, the OpenWrt build system should probably use the host's ccache. Like, for example, when building the sandox's private ccache, see the next stage.

(2) OpenWrt builds then its own, slightly outdated ccache. I wonder why. Ubuntu's ccache version has always been good enough for me.

I do not know yet whether the ccache version in the build sandbox is only used when cross-compiling for the target, or when building other host tools too.

(3) Hopefully all cross-compilation with the sandbox's toolchain will use the sandbox's ccache. The trouble is, the sandbox ccache's cache is hard to reuse.

If you rebuild inside the sandbox, the sandbox cache will help. But if you rebuild the entire sandbox, the private cache will be gone.

I could try to reuse that private cache, but even if the cross-compiling toolchain ends up being exactly the same, it is hard to check or guarantee that this is actually the case. You may build exactly the same toolchain version for the same target, but with different build flags. So it is hard to say that a particular object in the cache can be safely reused with the cross-compilation toolchain just built.

Has anybody got any hints about how to optimise the ccache usage?

Thanks in advance,
rdiez

2

some hints

3

Thanks for the quick answer. I had already seen that post, but unfortunately, it does not reall cover the questions I raised above.

4

I think I know now the reason why the following warning is coming up in point (a) above:

bash: ccache_cc: command not found

The reason is this definition in toolchain/musl/Makefile :

define Host/Compile
	+$(MAKE) $(HOST_JOBS) $(MUSL_MAKEOPTS) all
endef

Note that this command uses MUSL_MAKEOPTS, which is defined earlier in the same makefile:

MUSL_MAKEOPTS = -C $(HOST_BUILD_DIR) \
	DESTDIR="$(TOOLCHAIN_DIR)/" \
	LIBCC="$(subst libgcc.a,libgcc_initial.a,$(shell $(TARGET_CC) -print-libgcc-file-name))"

The $(shell) command tries to run the compiler for the target. However, during the "make download" phase, the cross-compiler is not built yet, so that command will fail. That is fine as long as no nobody tries to use that Host/Compile function too early.

But take a look at include/host-build.mk :

ifneq ($(if $(HOST_QUILT),,$(CONFIG_AUTOREBUILD)),)
  define HostHost/Autoclean
    $(call rdep,${CURDIR} $(PKG_FILE_DEPENDS),$(HOST_STAMP_PREPARED))
    $(if $(if $(Host/Compile),$(filter prepare,$(MAKECMDGOALS)),1),,$(call rdep,$(HOST_BUILD_DIR),$(HOST_STAMP_BUILT)))
  endef
endif

GNU Make will execute the command above while evaluating the $(if $(Host/Compile) ...) statement.

I guess that $(if ...) statement is just trying to find out whether there is something to build at all.

Could we not replace it with something like this?

ifneq "$(origin Host/Compile)" "undefined"
...

After all, Host/Compile is just a variable as far as GNU Make is concerned.