@aparcar has been looking for ways to reduce the number of posts about broken builds against the existing upgrade tools (auc, LuCI Attended Sysupgrade, Firmware Selector), so I added checks for that sort of thing.
A few months back, @daniel mentioned something about rewriting auc in ucode in a PR against auc, so I incorporated auc's basic build-with-installed-packages and added many options. When the ucode-mod-uclient package was published shortly thereafter, I had no excuses.
Here's some details on what I think are interesting features:
The --init-script option allows you to specify a uci-defaults script, just like Firmware Selector does with its 'Script to run on first boot (uci-defaults)' input field. LuCI Attended Sysupgrade sort of does this by looking for /rom/etc/uci-defaults/99-asu-defaults and relaying that implicitly, but it doesn't allow you to delete or change what's already there. owut makes this explicit, if you want the script included in your new image, then you must specify it when you request a build.
The --rootfs-size option allows those of you with expandable file systems (I'm lookin' at you, x86 and armsr users) to specify the size of the root file system. On my x86 test box, I have built the last dozen ext4 images using 128, 256 and 512 MB root file systems with no issues. (The default varies depending on target, but is often 104 MB.)
owut has a config file, so you don't have to remember the above two settings, you can put them there. The values in the config become the command line option defaults, so if you do say owut download --rootfs-size 512, it overrides what's in the config file.
(You can put most of the command line options in the config, but the two in this example seem the most useful to me.)
The owut list command has a --format option that takes one of
fs-user - (the default) produces a package list for use by the Firmware Selector that contains only the top-level, user-installed package modifications. You'd copy'n'paste this after the default list in the FS 'Installed Packages' field.
fs-all - produces a package list for FS containing all top-level packages, which you'd past over the values in the FS 'Installed Packages' field.
config - produces a build .config snippet of user-installed, top-level packages that you can use when doing source builds. Each output line looks like CONFIG_PACKAGE_collectd-mod-thermal=y.
EDIT
Fix config, which now is part of /etc/config/attendedsysupgrade.
This sounds really cool. Thanks @efahl for the work on this!
I'm going to ask the following questions because I'm curious and I suspect others may be, too:
Are there circumstances where one specific tool (Owut, auc, LuCI Attended Sysupgrade) is recommended over another?
Are there plans to deprecate the previous tools in favor of this newest one, or will they continue to co-exist? (and if so, maybe that points to the scenarios above where one each tool is suited to specific situations)?
One of the limitations with auc and /LuCI Attended Sysupgrade is the fact that they do not work with extroot based configurations. Is it possible that the new tool could (in a future version) address that by orchestrating an automated multi-phase installation of some sort?
I'm planning to deprecate auc once owut is fully fit for the task. I anyway wanted to replace auc with a ucode implementation, and now @efahl was quick to implement it with even more features than expected
It will still need some minor fixes here and there, see comments in https://github.com/openwrt/packages/pull/24324, but I expect this to be doable before the next stable release which will then no longer offer auc.
As Daniel says above, apparently we could phase out auc and when it's a bit more mature start using owut in its place. LuCI ASU will remain as the "GUI" version, but I've looked at it a bit and my do some PRs against it to add things like owut's init-script manipulation options, and rootfs_size (when applicable).
As I mentioned at the very start of this thread, upgrading has become sort of my holy grail. I've been working through getting an ASU server instance working so I can set Paul some PRs to make it more robust and add some features to make owut and LuCI ASU easier to maintain (to paraphrase what I say in the PR "make the ASU server our One Stop Shopping center for builds and build metadata").
My hope/goal is to have all of the various tools converge on a common feature set, so you Mr. Support Staff, don't have to remember "well, if you use 'x' do this, and if you use 'y', then do that, but with 'z'...".
Funny you should mention that to a confirmed x86 advocate...
But, I think that falls outside upgrade tools' purview (for me it's like selecting between ext4 or squashfs, the upgrader doesn't care as that is handled elsewhere).
My current belief is that this could be accomplished today without modifying any existing tools, it just needs some extroot user to buckle down and figure out all the permutations. The sysupgrade tool already has pre-install hooks which it uses to create the backup prior to reboot; post-install/first-boot processing could be in a uci-defaults script.
Default package analysis:
Default Provided-by
dnsmasq not installed, possibly replaced by another package
nftables not installed, possibly replaced by another package
wpad-basic-mbedtls not installed, possibly replaced by another package
Would it be possible to match the another package to the actuall installed one?
In my case dnsmasq - dnsmasq-full nftables - nftables-json wpad-basic-mbedtls - wpad-mbedtls
Indeed, what you're looking for is opkg whatprovides <installed> which shows the "functional name" (for lack of a better term) of the package. owut actually did that a couple days ago, but I reworked the source of the package lists to use only ubus calls to make the code agnostic to package manager (APK is right over the horizon!). I consider that a regression for a useful bit of information (now reinforced by your question), and will keep it on my list.
Here's an example of how I want that section to look:
Default package analysis:
dnsmasq - default package replaced/provided by dnsmasq-full
kmod-dwmac-intel - Warning: default package is not present
nftables - default package replaced/provided by nftables-json
Oops, thanks for the review! Looks like rpcd-mod-rpcsys comes free as a transitive dependency of attendedsysupgrade-common, but I do need to add that math module.
Hmm, I'm open to anything, but I'm not sure what you're asking here. Could you explain it in terms of image builder flags or sysupgrade parameters or image selection or whatever?
to be honest it is quite a stretch to the default image concept what i am thinking. let me try to explain.
comparing to plastic routers there are couple of benefits and possibilities in x86/arm platforms which not really leveraged so far. i can understand a common building/upgrade process has many advantage but there were/are a few missed opportunities in my opinion.
like the one you already mentioned and has been supported since not so long ago, i.e. the ridiculously small default root fs size can be changed finally to leverage the much larger storage space available on these platforms.
but with these larger disk space the other missed opportunity in my opinion is that could "easily" setup a proper failover mechanism which is common in other platforms, i.e. the dual image concept, with active and passive root partitions. this could provide a much safer upgrade or configuration experience where the standard boot + root partition schema can be extended with an additional root partition. this could allow to have different versions to be installed or configured differently, and fallback in case of problem.
technically it would mean that the corresponding kernel should be copied to /boot with the right grub entry, and instead of writing the combined.img to disk only write the rootfs.img to the corresponding partition. then user can select during boot which owrt version to run. a simple nextboot tool which would set the default variable in grub.cfg could work for headless hosts also.
sorry, i don't want to hijack your project, i understand currently you are supporting the combined.img. this is just food for thoughts, and understand if this does not fit into your scope.
Yeah, that's good stuff but as you say, sort of out-of-scope for this work. That sounds like it needs some fundamental changes to the core build system...
owut is really just a way to get to the ASU server build api (as are auc, LuCI Attended Sysupgrade and Firmware Selector), and ASU server is really just a fancy wrapper around the downloads site and imagebuilders. None of these actually know anything about "doing builds", they just set imagebuilder values and grab the resulting image.
In fact, you can write your own upgrade tool by just filling in a json blob and POSTing it to https://sysupgrade.openwrt.org/api/v1/build. Conceptually, it's pretty simple.
Yes, that's what a lot of @daniel's comments on it are pushing towards... It will require you to stand up an ASU server instance talking to your build machine, for owut's upstream feed (for one example, see ImmortalWrt). On your router, you'd simply uci set attendedsysupgrade.server.url=http://192.168.1.25 or whatever your ASU server is and off you go.
There are still a few kinks to work out in the ASU server metadata (e.g., adding the upstream build server to the overview), but it's in the pipeline.
EDIT:
PR submitted on that one https://github.com/openwrt/asu/pull/852 and code now behaves as if the PR were merged (with a fallback upstream = "https://downloads.openwrt.org").
rootfs_size_mb integer
example: 100
maximum: 1024
minimum: 1
Ability to specify a custom CONFIG_TARGET_ROOTFS_PARTSIZE for the resulting image. Attaching this optional parameter will cause ImageBuilder to build a rootfs with that size in MB.
Not long ago, this used to be 100mb max. Does the build server accept requests up to 1024mb now ? I'm crying from happiness.
Yes, that is correct as of May 13 (https://github.com/openwrt/asu/issues/826). I've been building my x86 images with owut download --rootfs-size 256 and 512 MB root partitions as test cases for almost a month now...
Report issues, questions, comments in this thread (or as owut github issues if you're more comfortable there)
Suggested first steps:
owut check - Just to see what your installation looks like and get comfortable with things. Does the output match your expected values wrt correct target and arch, correct fs type, versions, et cetera?
owut download - Purely non-destructive build process, to gain familiarity with the build sequence (usually takes 2-5 minutes depending on lots of things). download does everything an auc invocation would, but stops short of the final sysupgrade, so you can look at the result (check in /tmp/firmware*). Also allows you to cp/scp the image, manifest and checksums to somewhere persistent, if you like to archive things.
owut verify - Just for fun, re-do the final verification that download just did. Uses the /tmp/firmware.sha256sums, and does a sysupgrade --test on the image.
You're feeling bold?owut install - Grabs that stuff in /tmp/firmware* and does the sysupgrade. (install again re-does the verify, then does the sysupgrade if verify succeeds.)
owut upgrade - "Just do it." Does the download, verify and install all in one.
Yes, thank you for noticing that! You want to remove that line and also the /usr/bin/owut line, as opkg is now managing the files and that's no longer needed.
Any plans for a luci-app-owut ?