I regularly build customized OpenWrt firmware images. The script that I use to automate this process basically does a full build with fresh sources every single time. It boils down to this routine:
Clone the OpenWrt repository in a temporary folder
Update and install the package feeds
Apply my configuration seed (diffconfig) and custom patches
Build everything
Copy generated binaries to my local firmware repository
Delete the temporary folder
While this approach works fine for me, I'm wondering, though, whether there is a safe and more efficient way to do this. After all, with the current approach I always write some 8-10 GB of data to the disk and then wipe it again. And building the toolchain also makes up for most of the compilation time.
I know I can do a "make clean" after a build to keep the toolchain. But how do I know when the toolchain needs to be rebuilt across new builds? Does the build system detect automatically that the sources of some tools in the toolchain changed after I update the source tree? Do I have to manually check for changes in the git history (in which case I would prefer to stick to my current approach)?
How have you automated your workflows? Do you always start from scratch?
Note: When I say, I build firmware images regularly, I mean every few weeks. I usually don't build multiple times a day.
The build system usually detects when sources on either tools or toolchain change, and builds them again at that point.
building tools & toolchain take twice as much time as building just the firmware, so using "make clean" and just building the firmware cuts the build time to 1/3.
I build every few days and use principally just "make clean".
But I always look at the git changes and if I spot major changes for either tools or toolchain, I issue "make dirclean" and rebuild also tools and toolchain. (major changes are e.g. a new musl version, new gcc version, etc.)
Example: of the recent changes, there was the gcc version bump from 730 to 740, which would also leave the multi-gigabyte sized 730-based toolchain intact and build the new 740-based toolchain, so it is better to wipe that directory to get rid of the old 730-based one.
But in nutshell, usually "make clean" is just fine.
Ps.
I have automated quite much of the build process for maintaining my two community builds. But you can't avoid looking at the source code very closely to avoid mishaps.
My scripts can be found in the build threads and in
If I got this right, that means, the only downside of using make clean is that I might end up with multiple outdated versions of tools (taking up diskspace), but the build system will always compile and use the newest versions, if necessary - correct? That would be great acutally, because then I would just to a full wipe from time to time but let my build script reuse the toolchain most of the times and save compilation time (the toolchain compilation actually makes up 75% of the total build time on my desktop).
But in that case, I have one more question: When you apply your own patches, how do you reset and update the git sources without cleaning out everything? (I guess that's more of a git question rather than an OpenWrt, but anyway...) Do you revert each custom patch and then pull the new source?
I looked at your build scripts and couldn't find that part in them (or any instance of 'make clean' for that matter).
If you have patches applied to your buildroot and do a git pull that conflicts with any patch modification, the pull will fail with a message. At that point you will have to back out your patch, update your tree, and rebase the patch (or pull it again).
I git pull the updated sources to my local repo, and if there are no conflicts, nothing else is needed.
If there are conflicts, I usually "git stash" my own changes, git pull updates, "git stash pop" my local changes and then edit the possible conflicts. (It is quite possible that the same file gets modified at a different place, so most of the apparent git pull conflicts are actually automatically handled by "git stash pop". Actual editing is only needed, if there are real changes to the same lines source I have edited.)
(I usually also look at the git commits in the upstream repo, and in case an update there seems to make my own change unnecessary, I may edit my own change away already before git pull.)
I do a couple of things in the way I work with git and OpenWrt that you might find valuable.
First is that I find the worktree feature of git helpful when working on multiple devices or multiple branches. It allows you to check out a branch in a different directory than the .git/ directory resides, but keep a single repo (only one to fetch, backup, what have you). The only downside is that you can only have a specific branch checked out in one directory.
Which is mitigated by having multiple "local" branches. I keep my work on a "devel-ath79" branch and a "devel-ar300m" branch, for example. When it comes time to update from master, I'll typically do something like (in this example the checked-out branch is "devel-ath79"):
# stash or commit any unsaved changes
git fetch # *not* pull
git branch devel-ath79-2018-12-29
git rebase openwrt/master
git tag might also work as the "placeholder", but the branches are habit. This way if there's an unexpected merge issue or something else wonky happens, I can "roll back" to where I was before.
Your local copy of "master" can be updated "in the usual ways" such as checking it out and git pull.
I have now updated my buildscript, so I can reuse the toolchain (and I also implemented a switch so I can wipe everything, so I can start from fresh from time to time). Works fine so far. As for my own patches, I decided to just take the "clean" approach by reverting my patches at the end of the build so the source tree should be clean and up-to-date again after a "make clean" and "git pull".
Thanks jeff for pointing out the worktree feature. I didn't know that. I won't be using it in my buildscript so far, as I currently only have one branch and target to work with, but I have a different project where that comes in handy.