I'm trying to build OpenWrt in a CI, have a couple questions regarding caching:
What directories I should cache to speed up repeated builds? (Is only caching staging_dir too little or too much?)
Are the tools and the toolchain the only things worth caching?
Should ccache also be enabled?
Does ccache itself use some other directories for caching?
What key should I use for the cache so the next CI build could either reuse or discard the cache? I guess it should depends on the selected target, but even within a target, there should be some criteria to determine if a different commit should rebuild the tools and the toolchain, which I'm not sure what it is.
Really appreciate it if someone could shed some light.
1-2: As long you do not execute a "make clean/distclean" things stay as they are. There is nothing "cached". It's simple there.
3-4: It depends on how often you make changes and compile things. Ccaching makes sense if you need to compile for a bunch of different devices/architectures in order to test. I'm not aware about special directories beside that you must set within config for ccache.
5: Ccache does nothing else then providing a checksum based decission if a file already was compiled successfully. It decides on its own if recompilation is necessary.
There is no mechanism for tools/toolchain to detect if sth. on kernel/package side has changed which would need to trigger a recompilation of toos/toolchain. To determine this you would need to reference every single Makefile. I'm not aware that anybody has written a tool/script doing this.
I'm asking about building in a CI, which starts with a clean slate every time it builds something. I need to specify which directories to cache in order for the next build to contain what's left from the previous build.
Ccache does nothing else then providing a checksum based decission if a file already was compiled successfully.
Thanks for the explanation about ccache. I understand it now. It's probably not something I should use in a CI.
Well what would be the difference in building OpenWrt? CI/CD is just a description of a process what you want to achive. I don't know what your CI/CD process/enviroment is.
You can share your ccache across different builds/architectures. You just have to point ccache to the same directory. It depends what you are doing exactly.
From what you are telling:
Changes on tools/toolchain are rare. So you could easy reuse it/cache it (e. g. snapshot). We don't know if you build on stable, if you are doing driver/kernel stuff, maintaining a few packages, maintaining different devices/architectures or just pushing certain configurations. Depending on your workload ccache can sure make sense because its built already.
I don't know what your CI/CD process/enviroment is.
It's the most common one: when a CI starts, it has no files at all. You need to pull OpenWrt code from somewhere, and run commands after that. Once the toolchain is built, you can specify which directories to cache, so the next build can specify, what directories to get from cache, etc.
Depending on your workload ccache can sure make sense because its built already.
Thanks for the detailed explanation. It seems if I don't run make clean/distclean, then make should do what ccache does? In the CI's case, once it gets files from the build cache previously mentioned, it should skip building what was built, I wonder what benefits does ccache offer?
In principle yes. But there is a big "BUT". As soon as you issue a "git pull" to grep the latest changes I would recommend to issue a "make clean". This is simply because object files could need an update which will not happen if you don't issue a "make clean". That could lead to compile errors e. g. linking fails. So basically you should prefer to do a "make clean".
The more changes you pull into your repo the longer re-compiling will take. The biggest/longes part (beside toolchain) is usally the kernel. It is updated frequently on master. So at least you would have to wait that time.
For me as a private user "maintaining" 7 devices/3 architectures its not worth (tried it in the past). Especially because I decide to compile on what got merged for the device/architecture. For older devices there are not so often changes worth to compile a new image. If i just update the config I'm using the "old" compilation without "make clean".
Only knows how to cache the compilation of a single file. Other types of compilations (multi-file compilation, linking, etc) will silently fall back to running the real compiler.
Some compiler flags are not supported. If such a flag is detected, ccache will silently fall back to running the real compiler.
The fastest mode (the "direct mode") has a corner case which can result in false positive cache hits. This and other minor limitations are listed under caveats in the manual.
Just try out ccache and see if it makes a difference for your usecase.
I think I've written it to short to understand my thought about this. I'll try again:
My thought was that the Makefile is telling make which objects can/have to be deleted and which can be re-used on an re-make (sth. like clean: $(rm) ..o ..ko). So there is a chance that there exist sometimes lets say a buggy Makefile in such a big project like OpenWrt.
So make does not miss that there was an update but does not realize that it cannot/should not re-use an certain object because the Makefile is not telling it.
But my guess is (while writing this up) that one of the Makefiles in the chain from OpenWrt is designed to catch those flaws in saying there was a minor code update: "lets compile the whole package from scratch and make a full cleanup".
Anyway. Sometimes a make clean helps for whatever reason. That's propably the reason why it is recommended everywhere. But it is propably outdated nowdays:
make clean
Deletes contents of the directories /bin and /build_dir . This doesn't remove the toolchain, and it also avoids cleaning architectures/targets other than the one you have selected in your .config . It is a good practice to do make clean before a build to ensure that no outdated artefacts have been left from the previous builds. That may not be necessary always, but as a general rule it helps to ensure quality builds.
@hgl
What did you do finally?
I'm also interested on your setup for CI
I agree with you, a CI (let's say Gitlab) starts with a pull, can inject some cache from previous build then run commands you set
Also, how ASU builds works? https://github.com/openwrt/asu
Does it recompile from scratch every target? Or does it save toolchain? What's the workflow?