Owut: OpenWrt Upgrade Tool

efahl · June 4, 2024, 4:51pm

This is the primary support thread for

owut - the OpenWrt Upgrade Tool, and for its predecessor
auc - the Attended sysUpgrade Command line tool.

If you are an auc user on 23.05 or earlier, an upgrade to 24.10 or later will replace auc with owut automatically.

I've been pursuing the holy grail of seamless upgrades for some time now, and the results are embodied in a ucode script I've named owut.

owut = OpenWrt Ugrade Tool

owut is available on 24.10 and later, and main SNAPSHOT at this point, no port available for 23.05 or earlier due to dependency on ucode-mod-uclient. For transition from 23.05 to newer versions, you can use the auc tool documented here (tldr, you'll install it and do something like auc -b 24.10 -B 24.10.3).

Your best starting point for using owut is the wiki page: https://openwrt.org/docs/guide-user/installation/sysupgrade.owut

$ opkg update && opkg install owut  ## for 24.10
$ apk update && apk add owut        ## snapshots and after 24.10

$ owut -h
owut - OpenWrt Upgrade Tool version 2024.06.04-r1 (/usr/bin/owut)
...

Code: https://github.com/efahl/owut/
Issues: here or https://github.com/efahl/owut/issues

Example
A typical check, run just now on my x86 testing box, which was upgraded yesterday using owut upgrade, hence the recent version numbers:

$ owut check -v
Board-name     generic
Target         x86/64
Root-FS-type   squashfs
Sys-type       combined-efi
Package-arch   x86_64
Version-from   SNAPSHOT r26504-d4acd05218 (kernel 6.6.32)
Version-to     SNAPSHOT r26515-6e51b363e9 (kernel 6.6.32)
Build-FS-type  squashfs
Build-at       2024-06-04T08:19:41Z
Image-prefix   openwrt-x86-64-generic
Image-file     openwrt-x86-64-generic-squashfs-combined-efi.img.gz
Image-URL      https://downloads.openwrt.org/snapshots/targets/x86/64/openwrt-x86-64-generic-squashfs-combined-efi.img.gz
Installed      273 packages
Top-level       81 packages
Default         46 packages
User-installed  50 packages (top-level only)

Package version changes:
  base-files                   1596~d4acd05218                            1596~6e51b363e9
  owut                         2024.06.04-r1                              missing to-version
1 packages missing in target version, DO NOT UPGRADE!
2 packages are out-of-date.

Default package analysis:
  Default            Provided-by
  dnsmasq            dnsmasq-full
  kmod-dwmac-intel   not-installed
  nftables           nftables-json

There are currently package build failures for SNAPSHOT x86_64:
  Feed: base
    kexec-tools                Mon Jun  3 13:06:48 2024 - Package not installed locally
  Feed: packages
    basicstation               Mon Jun  3 14:10:58 2024 - Package not installed locally
    elektra                    Mon Jun  3 15:05:33 2024 - Package not installed locally
    gatling                    Mon Jun  3 14:30:01 2024 - Package not installed locally
    gcc                        Mon Jun  3 13:18:49 2024 - Package not installed locally
    kadnode                    Mon Jun  3 13:22:19 2024 - Package not installed locally
    libuhttpd                  Mon Jun  3 14:37:34 2024 - Package not installed locally
    micropython                Mon Jun  3 13:27:21 2024 - Package not installed locally
    shadowsocks-libev          Mon Jun  3 14:24:44 2024 - Package not installed locally
    umurmur                    Mon Jun  3 14:48:24 2024 - Package not installed locally
    xtables-addons             Mon Jun  3 15:40:03 2024 - Package not installed locally
  Feed: telephony
    baresip                    Mon Jun  3 16:46:12 2024 - Package not installed locally
    freeswitch                 Mon Jun  3 16:46:14 2024 - Package not installed locally
    freeswitch-mod-bcg729      Mon Jun  3 16:48:53 2024 - Package not installed locally
    freetdm                    Mon Jun  3 16:48:57 2024 - Package not installed locally
Failures don't affect this device, details at
  https://downloads.openwrt.org/snapshots/faillogs/x86_64/

Pings:
@spence got me interested in the code side of things with Detecting user installed pkgs - #8 by spence

@aparcar has been looking for ways to reduce the number of posts about broken builds against the existing upgrade tools (auc, LuCI Attended Sysupgrade, Firmware Selector), so I added checks for that sort of thing.

A few months back, @daniel mentioned something about rewriting auc in ucode in a PR against auc, so I incorporated auc's basic build-with-installed-packages and added many options. When the ucode-mod-uclient package was published shortly thereafter, I had no excuses.

efahl · June 4, 2024, 5:39pm

Here's some details on what I think are interesting features:

The --init-script option allows you to specify a uci-defaults script, just like Firmware Selector does with its 'Script to run on first boot (uci-defaults)' input field. LuCI Attended Sysupgrade sort of does this by looking for /rom/etc/uci-defaults/99-asu-defaults and relaying that implicitly, but it doesn't allow you to delete or change what's already there. owut makes this explicit, if you want the script included in your new image, then you must specify it when you request a build.

The --rootfs-size option allows those of you with expandable file systems (I'm lookin' at you, x86 and armsr users) to specify the size of the root file system. On my x86 test box, I have built the last dozen ext4 images using 128, 256 and 512 MB root file systems with no issues. (The default varies depending on target, but is often 104 MB.)

owut has a config file, so you don't have to remember the above two settings, you can put them there. The values in the config become the command line option defaults, so if you do say owut download --rootfs-size 512, it overrides what's in the config file.

$ tail /etc/config/attendedsysupgrade
...
config owut 'owut'
        option init_script '/root/bin/test-init-script.sh'
        option rootfs_size '256'

(You can put most of the command line options in the config, but the two in this example seem the most useful to me.)

The owut list command has a --format option that takes one of

fs-user - (the default) produces a package list for use by the Firmware Selector that contains only the top-level, user-installed package modifications. You'd copy'n'paste this after the default list in the FS 'Installed Packages' field.
fs-all - produces a package list for FS containing all top-level packages, which you'd past over the values in the FS 'Installed Packages' field.
config - produces a build .config snippet of user-installed, top-level packages that you can use when doing source builds. Each output line looks like CONFIG_PACKAGE_collectd-mod-thermal=y.

EDIT
Fix config, which now is part of /etc/config/attendedsysupgrade.

psherman · June 5, 2024, 12:06am

This sounds really cool. Thanks @efahl for the work on this!

I'm going to ask the following questions because I'm curious and I suspect others may be, too:

Are there circumstances where one specific tool (Owut, auc, LuCI Attended Sysupgrade) is recommended over another?
Are there plans to deprecate the previous tools in favor of this newest one, or will they continue to co-exist? (and if so, maybe that points to the scenarios above where one each tool is suited to specific situations)?
One of the limitations with auc and /LuCI Attended Sysupgrade is the fact that they do not work with extroot based configurations. Is it possible that the new tool could (in a future version) address that by orchestrating an automated multi-phase installation of some sort?

daniel · June 5, 2024, 2:34pm

I'm planning to deprecate auc once owut is fully fit for the task. I anyway wanted to replace auc with a ucode implementation, and now @efahl was quick to implement it with even more features than expected

It will still need some minor fixes here and there, see comments in https://github.com/openwrt/packages/pull/24324, but I expect this to be doable before the next stable release which will then no longer offer auc.

efahl · June 5, 2024, 4:41pm

As Daniel says above, apparently we could phase out auc and when it's a bit more mature start using owut in its place. LuCI ASU will remain as the "GUI" version, but I've looked at it a bit and my do some PRs against it to add things like owut's init-script manipulation options, and rootfs_size (when applicable).

As I mentioned at the very start of this thread, upgrading has become sort of my holy grail. I've been working through getting an ASU server instance working so I can set Paul some PRs to make it more robust and add some features to make owut and LuCI ASU easier to maintain (to paraphrase what I say in the PR "make the ASU server our One Stop Shopping center for builds and build metadata").

My hope/goal is to have all of the various tools converge on a common feature set, so you Mr. Support Staff, don't have to remember "well, if you use 'x' do this, and if you use 'y', then do that, but with 'z'...".

Funny you should mention that to a confirmed x86 advocate...

But, I think that falls outside upgrade tools' purview (for me it's like selecting between ext4 or squashfs, the upgrader doesn't care as that is handled elsewhere).

My current belief is that this could be accomplished today without modifying any existing tools, it just needs some extroot user to buckle down and figure out all the permutations. The sysupgrade tool already has pre-install hooks which it uses to create the backup prior to reboot; post-install/first-boot processing could be in a uci-defaults script.

goetz · June 7, 2024, 12:32pm

@efahl thank you for this tool.
Already in its current state I'm thrilled.
Especially as auc is currently broken https://github.com/openwrt/openwrt/issues/15619

Tested on:

Board-name     netgear_dgnd3800b
Target         bmips/bcm6368
Root-FS-type   squashfs
Sys-type       sysupgrade
Package-arch   mips_mips32

One question from my side.

The default output includes something like this.

Default package analysis:
  Default              Provided-by
  dnsmasq              not installed, possibly replaced by another package
  nftables             not installed, possibly replaced by another package
  wpad-basic-mbedtls   not installed, possibly replaced by another package

Would it be possible to match the another package to the actuall installed one?

In my case
dnsmasq - dnsmasq-full
nftables - nftables-json
wpad-basic-mbedtls - wpad-mbedtls

efahl · June 7, 2024, 2:22pm

Thanks a bunch for testing! If any issues pop up, please don't hesitate to log them. https://github.com/efahl/owut/issues

Hmm, your examples look familiar: https://github.com/efahl/owut/issues/4#issuecomment-2153700505

Indeed, what you're looking for is opkg whatprovides <installed> which shows the "functional name" (for lack of a better term) of the package. owut actually did that a couple days ago, but I reworked the source of the package lists to use only ubus calls to make the code agnostic to package manager (APK is right over the horizon!). I consider that a regression for a useful bit of information (now reinforced by your question), and will keep it on my list.

Here's an example of how I want that section to look:

Default package analysis:
  dnsmasq             - default package replaced/provided by dnsmasq-full
  kmod-dwmac-intel    - Warning: default package is not present
  nftables            - default package replaced/provided by nftables-json

grrr2 · June 7, 2024, 3:43pm

hi,

it looks amazing. just two things please:

i think also need packages rpcd-mod-rpcsys and ucode-mod-math.
would you please consider as a feature request to add support to define target partition to install

thanks

efahl · June 7, 2024, 8:13pm

Oops, thanks for the review! Looks like rpcd-mod-rpcsys comes free as a transitive dependency of attendedsysupgrade-common, but I do need to add that math module.

Hmm, I'm open to anything, but I'm not sure what you're asking here. Could you explain it in terms of image builder flags or sysupgrade parameters or image selection or whatever?

grrr2 · June 7, 2024, 9:12pm

to be honest it is quite a stretch to the default image concept what i am thinking. let me try to explain.

comparing to plastic routers there are couple of benefits and possibilities in x86/arm platforms which not really leveraged so far. i can understand a common building/upgrade process has many advantage but there were/are a few missed opportunities in my opinion.
like the one you already mentioned and has been supported since not so long ago, i.e. the ridiculously small default root fs size can be changed finally to leverage the much larger storage space available on these platforms.

but with these larger disk space the other missed opportunity in my opinion is that could "easily" setup a proper failover mechanism which is common in other platforms, i.e. the dual image concept, with active and passive root partitions. this could provide a much safer upgrade or configuration experience where the standard boot + root partition schema can be extended with an additional root partition. this could allow to have different versions to be installed or configured differently, and fallback in case of problem.

technically it would mean that the corresponding kernel should be copied to /boot with the right grub entry, and instead of writing the combined.img to disk only write the rootfs.img to the corresponding partition. then user can select during boot which owrt version to run. a simple nextboot tool which would set the default variable in grub.cfg could work for headless hosts also.

sorry, i don't want to hijack your project, i understand currently you are supporting the combined.img. this is just food for thoughts, and understand if this does not fit into your scope.

efahl · June 7, 2024, 10:38pm

Yeah, that's good stuff but as you say, sort of out-of-scope for this work. That sounds like it needs some fundamental changes to the core build system...

owut is really just a way to get to the ASU server build api (as are auc, LuCI Attended Sysupgrade and Firmware Selector), and ASU server is really just a fancy wrapper around the downloads site and imagebuilders. None of these actually know anything about "doing builds", they just set imagebuilder values and grab the resulting image.

In fact, you can write your own upgrade tool by just filling in a json blob and POSTing it to https://sysupgrade.openwrt.org/api/v1/build. Conceptually, it's pretty simple.

{
    "client": "owut/2024.06.04",
    "target": "x86/64",
    "profile": "generic",
    "version": "SNAPSHOT",
    "version_code": "r26554-4edde98768",
    "filesystem": "ext4",
    "diff_packages": true,
    "rootfs_size_mb": 512,
    "packages": [
        "auc",
        "base-files",
... all your other packages ...
        "vim-full",
        "wget-ssl"
    ]
}

littleNewton · June 10, 2024, 7:28pm

It looks great!

If the firmware is compiled by myself and with some third-party packages installed, can I still use this upgrading tool?

efahl · June 10, 2024, 8:08pm

Yes, that's what a lot of @daniel's comments on it are pushing towards... It will require you to stand up an ASU server instance talking to your build machine, for owut's upstream feed (for one example, see ImmortalWrt). On your router, you'd simply uci set attendedsysupgrade.server.url=http://192.168.1.25 or whatever your ASU server is and off you go.

There are still a few kinks to work out in the ASU server metadata (e.g., adding the upstream build server to the overview), but it's in the pipeline.

EDIT:

PR submitted on that one https://github.com/openwrt/asu/pull/852 and code now behaves as if the PR were merged (with a fallback upstream = "https://downloads.openwrt.org").

devast · June 14, 2024, 7:28am

Really nice work, congrats!

Also is this true?:

rootfs_size_mb	integer
example: 100
maximum: 1024
minimum: 1

Ability to specify a custom CONFIG_TARGET_ROOTFS_PARTSIZE for the resulting image. Attaching this optional parameter will cause ImageBuilder to build a rootfs with that size in MB.

Not long ago, this used to be 100mb max. Does the build server accept requests up to 1024mb now ? I'm crying from happiness.

efahl · June 14, 2024, 1:43pm

Yes, that is correct as of May 13 (https://github.com/openwrt/asu/issues/826). I've been building my x86 images with owut download --rootfs-size 256 and 512 MB root partitions as test cases for almost a month now...

efahl · June 25, 2024, 5:51pm

Testers invited!

owut has made it into the SNAPSHOT package feed, so

opkg update && opkg install owut

Documentation is at owut wiki page
Source at owut github repo
Report issues, questions, comments in this thread (or as owut github issues if you're more comfortable there)

Suggested first steps:

owut check - Just to see what your installation looks like and get comfortable with things. Does the output match your expected values wrt correct target and arch, correct fs type, versions, et cetera?
owut download - Purely non-destructive build process, to gain familiarity with the build sequence (usually takes 2-5 minutes depending on lots of things). download does everything an auc invocation would, but stops short of the final sysupgrade, so you can look at the result (check in /tmp/firmware*). Also allows you to cp/scp the image, manifest and checksums to somewhere persistent, if you like to archive things.
owut verify - Just for fun, re-do the final verification that download just did. Uses the /tmp/firmware.sha256sums, and does a sysupgrade --test on the image.
You're feeling bold? owut install - Grabs that stuff in /tmp/firmware* and does the sysupgrade. (install again re-does the verify, then does the sysupgrade if verify succeeds.)
owut upgrade - "Just do it." Does the download, verify and install all in one.

ghoffman · June 25, 2024, 7:51pm

i've been using it for about 3 weeks. very helpful information compared to auc. thx

devast · June 25, 2024, 8:57pm

Looking great Any plans for a luci-app-owut ?

ghoffman · June 26, 2024, 12:41am

for those of us who used the .git version -
can we remove this line from /etc/sysupgrade.conf:
/usr/share/ucode/utils/argparse.uc
?

efahl · June 26, 2024, 1:23am

Yes, thank you for noticing that! You want to remove that line and also the /usr/bin/owut line, as opkg is now managing the files and that's no longer needed.