Owut: OpenWrt Upgrade Tool

efahl · October 1, 2025, 7:41pm

No one has chimed in yet on the issue, and no, I still haven't gotten it to work. I dug around in the various makefiles and couldn't figure out how some targets get that initramfs-kernel binary built: some do, some don't, but there's no obvious fingerprint...

antonk · October 1, 2025, 7:50pm

While looking at the code, I noticed a thing which was looking slightly odd. Here:

github.com/efahl/owut

files/owut

f049043ed


      
          		L.log(1, fmt, "Default", "Provided-by");
          		for (let p, a in changes) {
          			L.log(1, fmt, p, a);
          		}
          		L.log(1, "\n");
          	}
          
          	return count;
          }
          
          function check_pkg_builds()
          {
          	// Scraping the failures.html is a total hack.
          	// Let me know if you have an API on downloads (or other build site)
          	// that can give this info.
          	//
          	// The lines we're scraping look like:
          	//    <tr><td class="n"><a href="gummiboot/">gummiboot</a>/</td><td class="s">-</td><td class="d">Tue Apr 23 07:05:36 2024</td></tr>
          
          	let info = regexp(
          		'<td class="n"><a href="[^<]*">([^<]*)</a>/</td>' +

looks like it extracts the list of failed packages from that failures.html file. Wouldn't it be more robust to get a list of packages which succeeded, compare to the list of planned packages and get the failed packages this way?

I'm completely unfamiliar with this whole process and which info is available, so I might be talking nonsense. Just thought it didn't hurt to ask.

efahl · October 1, 2025, 10:43pm

Yup, that one's easy, the ASU server has a call that finds all the feeds package databases and aggregates them into a version- and architecture-specific response. Here's one now: https://sysupgrade.openwrt.org/json/v1/releases/24.10-SNAPSHOT/packages/x86_64-index.json

To understate it a bit, "that's a hard one", because the package list is not defined statically anywhere, it's dynamic depending on what's checked into the git package repos when the build starts.

To build such a list, you have to scan all the feed sources (at the correct git tag/rev), find all the directories, search them for Makefiles. Parse all those Makefiles for define Package/xxx definitions. But then you also have to not just parse the Makefiles, but also process them and all of their includes, to see if the defined package is actually intended to be built, because some are target or profile or architecture specific and only build for certain devices (for an example off the top of my head, we build Intel's hyperscan library for x86-only).

And so you can probably see this is not likely to happen in a standalone tool.

But in fact, what you're suggesting is effectively what the buildbots do: run all the Makefiles and if they succeed, then resulting packages go into the various package indexes, otherwise failed ones get logged in faillogs.

As an aside, since you pointed to that owut code:
The truly bad part about check_pkg_builds is that I'm recursively scraping html, which is a pretty disgusting thing to do even in the best of times. I've had on my list for "a while" to implement an API to put on the downloads site, so you could just ask for, say https://downloads.openwrt.org/releases/faillogs-24.10/aarch64_cortex-a53/failed.json and get all the info (maybe more?) that owut already shows...

antonk · October 2, 2025, 12:13am

I may be misunderstanding something (quite likely) but it seems that we are not talking about the same thing. By planned packages, I mean packages planned for installation on the particular system owut is currently running on. What the owut code is doing:

Get the list of all installed packages, register them in packageDB
Get the list of all packages which failed to build
Loop over the failed packages list and check whether given failed package is in packageDB

What I had in mind:

Get the list of all installed packages, register them in packageDB
Get the list of all packages which were built successfully, register them in, let's call it BuiltPackageDB
Loop over the list of installed packages and check whether each one is in BuiltPackageDB. If a package is not in there then it failed to build or is not in the ASU database.

In most cases, the end result of the 2 approaches will be exactly the same. I am suggesting the latter approach because the first approach may produce unreliable results in some cases, e.g. : html parsing error which incorrectly returned no failed packages; ASU error which incorrectly did not register some failed packages (maybe this is far-fetched); certain installed packages are not in the ASU database at all (some projects may be providing packages independently).

otnert · October 2, 2025, 7:11am

Just reporting, DL-WRX36 snapshot image built yesterday had these 2 packages missing:

iwinfo libustream-openssl.

Just built another image today and all packages have been included!

andybjackson · October 2, 2025, 11:51pm

I think iwinfo is no longer included by default. And libustream-openssl is replaced by libustream-mbedtls.

Anyone able to confirm?

efahl · October 3, 2025, 12:59am

You can dump the profile-specific defaults for your device.

$ owut dump -v -v -v | jsonfilter -e '$.packages.default'
[ "base-files", "ca-bundle", "dnsmasq", ...

Define a one-line version of json_pp utility and you can prettify the output.

cat >json_pp <<THE_CODE
#!/usr/bin/ucode -S
import { stdin } from "fs";
let blob = json(stdin.read("all"));
printf("%.4J\n", blob);
THE_CODE

chmod +x json_pp

then

$ owut dump -v -v -v | jsonfilter -e '$.packages.default' | ./json_pp
[
    "base-files",
    "ca-bundle",
    "dnsmasq",
    ...

efahl · October 3, 2025, 1:07am

Aha, owut is actually doing that, too. That's part of "checking for newer" packages. The big json index file I linked to previously is used not only to see if current-version is less than available-version, but also if the package actually still exists in the upstream feeds. This is manifested in the lines in the package analysis that say missing to-version:

Package version changes:
  base-files          1661~155eea44e7    1663~bf7a4d9d5a
  ca-bundle           20241223-r1        20250419-r1
  jq                  1.7.1-r1           1.8.1-r1
  kmod-amazon-ena     6.6.100-r1         missing to-version
  kmod-amd-xgbe       6.6.100-r1         missing to-version

The failed package analysis is just sugar on top of that, in an attempt to possibly explain why that missing to-version has appeared. Often it's a build failure, but sometimes the missing package is because of repackaging or renaming or something else, and the faillogs just help us in resolving the issue.

efahl · October 3, 2025, 1:10am

I think iwinfo has returned due to hauke's revert of the wifi-script stuff a couple of days ago.

The libustream one looks like you hit my bug in rpcd-mod-rpcsys. Did you upgrade that package prior to your upgrade? (If not, it's probably upgraded now, apk list --installed rpcd-mod-rpcsys should show version 2025.09.30.)

otnert · October 3, 2025, 1:41am

Yes, pre rpcd-mod-rpcsys v2025.09.30 both packages iwinfo libustream-openssl were missing.

Post upgrade, all packages where included!

Dantes · October 3, 2025, 7:10am

@efahl i tried the download today ( i was under the mistaken assumption i could create a second sdcard from it). could that be an option ? at least for the nanopi crowd it could be very handy as we can easily swap cards to test an upgrade. swap back in case of trouble

efahl · October 3, 2025, 7:18am

You'd need to somehow install the downloaded image to "other" sdcard, and then write the sysupgrade backup to it, so I'm not sure how that would work if the card isn't mounted to the device somehow. The sysupgrade code only deals with the live media, there's really no way to stop it in the middle and swap cards...

Dantes · October 3, 2025, 7:26am

I like it!

But my thought was more on generating a full image along side the sysupgrade if possible. So you could retrieve and write to sdcard. Swap and test.

antonk · October 3, 2025, 8:09am

Aha, that's what I missed. Thanks for the clarification. owut code is very nice.

efahl · October 3, 2025, 3:06pm

Yes, that's what download is for, it just produces the build artifacts on-device, verifies them and stops. When you run owut download (or owut upgrade), it creates these three files of interest:

$ owut download
...

$ ll /tmp/firmware*
-rw-r--r-- 1 root  root     43701 Oct  3 08:03 /tmp/firmware-manifest.json
-rw-r--r-- 1 root  root  10904640 Oct  3 08:03 /tmp/firmware.bin
-rw-r--r-- 1 root  root        84 Oct  3 08:03 /tmp/firmware.sha256sums

If you owut download, then you can manually cp/scp them off to persistent storage prior to doing owut install (and don't forget to backup the corresponding config with sysupgrade --create-backup config-backup-blah.tgz).

If you want to automate the archive process during owut upgrade or owut install, take a look at /etc/owut.d/pre-install.sh.

andybjackson · October 6, 2025, 1:06pm

Can confirm that two owut upgrades across my four devices have all worked flawlessly. Thank you @efahl and all.

andybjackson · October 10, 2025, 9:19am

Afraid I have a further apk related niggle. Today's upgrade of apk-mbedtls from 3.0.0_pre20250606-r2 to 3.0.0_pre20251007-r1 is perceived as a downgrade by owut, requiring the use of --force.

My sympathies @efahl - this is the gift that keeps on giving. But at least it isn't hard to work around.

efahl · October 10, 2025, 9:28am

Blah. That's a bug in the already-too-complicated version comparison code. Some regex is probably not taking pre into account or something, I'll take a look. Thanks for reporting...

EDIT: Sure enough, it's a total finger flub. Anything with more than a single digit after the underscore-alpha+ sequence was getting mangled. Here are all the weird versions that I found, the ones with 0 or 1 digit after alpha/beta/pre/rc are fine.

0.11_alpha-r1        { "ver": [ 0, 11 ], "hsh": "_alpha", "rev": 1 }
0.3_beta7-r2         { "ver": [ 0, 3, 7 ], "hsh": null, "rev": 2 }
1.1_pre18-r4         { "ver": [ 1, 1 ], "hsh": "_pre18", "rev": 4 }
1.3.0_rc1-r1         { "ver": [ 1, 3, 0, 1 ], "hsh": null, "rev": 1 }
1.9.17_p2-r1         { "ver": [ 1, 9, 17, 2 ], "hsh": null, "rev": 1 }
10.0_p1-r3           { "ver": [ 10, 0, 1 ], "hsh": null, "rev": 3 }
2.1.0-r1             { "ver": [ 2, 1, 0 ], "hsh": null, "rev": 1 }
3.0.0_pre20250606-r2 { "ver": [ 3, 0, 0 ], "hsh": "_pre20250606", "rev": 2 }
3.0.0_pre20251007-r1 { "ver": [ 3, 0, 0 ], "hsh": "_pre20251007", "rev": 1 }
3.4.5-r1             { "ver": [ 3, 4, 5 ], "hsh": null, "rev": 1 }
3.8.0_rc2-r1         { "ver": [ 3, 8, 0, 2 ], "hsh": null, "rev": 1 }
4.2.8_p18-r3         { "ver": [ 4, 2, 8 ], "hsh": "_p18", "rev": 3 }
4.4.3_p1-r11         { "ver": [ 4, 4, 3, 1 ], "hsh": null, "rev": 11 }

Looking at the regex for that part, it's so obvious, how I didn't see that it was totally mangled before is beyond me:

-const apk_bits = regexp("^_[a-z]+([0-9[+])$", "i");
+const apk_bits = regexp("^_[a-z]+([0-9]+)$",  "i");

Now the bad ones work (and owut check shows proper status for the apk-* update):

1.1_pre18-r4         { "ver": [ 1, 1, 18 ], "hsh": null, "rev": 4 }
3.0.0_pre20250606-r2 { "ver": [ 3, 0, 0, 20250606 ], "hsh": null, "rev": 2 }
3.0.0_pre20251007-r1 { "ver": [ 3, 0, 0, 20251007 ], "hsh": null, "rev": 1 }

Oh yeah, here's how I found those version numbers:
apk list '*' | awk '{print $1}' | grep -E --color '[0-9]_[a-zA-Z]'
opkg list | awk '{print $1" "$3}' | grep -E --color '[0-9]_[a-zA-Z]'

neheb · October 11, 2025, 1:11am

Here’s a fun one:

owut upgrade --verbose
owut - OpenWrt Upgrade Tool 2025.09.03~49e9bce7-r1 (/usr/bin/owut)
ERROR: ucode-mod-uclient requires an SSL library to function:
  Please install one of the libustream-*[ssl|tls] packages as well as
  one of the ca-bundle or ca-certificates packages.
ERROR: Response status (null) while downloading
  http://sysupgrade.openwrt.org/json/v1/overview.json

a previous run of owut upgrade removed libustream-mbedtls.

apk add libustream-mbedtls

results in a broken system.

apk add libustream-mbedtls
( 1/45) Purging libjson-script20250723 (2025.07.23~49056d17-r1)
libjson-script20250723-2025.07.23~49056d17-r1.pre-deinstall: Executing script...
( 2/45) Purging libblobmsg-json20250723 (2025.07.23~49056d17-r1)
libblobmsg-json20250723-2025.07.23~49056d17-r1.pre-deinstall: Executing script...
( 3/45) Purging libubus20250702 (2025.07.02~5952b48e-r1)
libubus20250702-2025.07.02~5952b48e-r1.pre-deinstall: Executing script...
( 4/45) Purging libubox20250723 (2025.07.23~49056d17-r1)
libubox20250723-2025.07.23~49056d17-r1.pre-deinstall: Executing script...
( 5/45) Installing libubox20251004 (2025.10.04~c163d7ab-r1)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/bin/wget)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/lib/libuclient.so)
wget: Error relocating /usr/lib/libuclient.so: md5_end: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_init: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_parse: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_add_field: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/lib/libuclient.so: __calloc_a: symbol not found
wget: Error relocating /usr/lib/libuclient.so: usock_inet_timeout: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_write: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_get_read_buf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_hash: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_consume: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_printf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_begin: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_set: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_fd_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_done: symbol not found
wget: Error relocating /usr/bin/wget: blobmsg_parse: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/bin/wget: uloop_run_timeout: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_set: symbol not found
wget: Error relocating /usr/bin/wget: uloop_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_cancelled: symbol not found
ERROR: wget: exited with error 127
ERROR: libubox20251004-2025.10.04~c163d7ab-r1: remote server returned error (try 'apk update')
( 6/45) Upgrading libuclient20201210 (2024.10.22~88ae8f20-r1 -> 2025.10.03~dc909ca7-r1)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/bin/wget)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/lib/libuclient.so)
wget: Error relocating /usr/lib/libuclient.so: md5_end: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_init: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_parse: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_add_field: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/lib/libuclient.so: __calloc_a: symbol not found
wget: Error relocating /usr/lib/libuclient.so: usock_inet_timeout: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_write: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_get_read_buf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_hash: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_consume: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_printf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_begin: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_set: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_fd_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_done: symbol not found
wget: Error relocating /usr/bin/wget: blobmsg_parse: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/bin/wget: uloop_run_timeout: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_set: symbol not found
wget: Error relocating /usr/bin/wget: uloop_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_cancelled: symbol not found
ERROR: wget: exited with error 127
ERROR: libuclient20201210-2025.10.03~dc909ca7-r1: remote server returned error (try 'apk update')
( 7/45) Installing libubus20251004 (2025.10.04~ad2768bb-r1)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/bin/wget)
wget: Error loading shared library libubox.so.20250723: No such file or directory (needed by /usr/lib/libuclient.so)
wget: Error relocating /usr/lib/libuclient.so: md5_end: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_free: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blob_buf_init: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_parse: symbol not found
wget: Error relocating /usr/lib/libuclient.so: blobmsg_add_field: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/lib/libuclient.so: __calloc_a: symbol not found
wget: Error relocating /usr/lib/libuclient.so: usock_inet_timeout: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_write: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_get_read_buf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_hash: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_consume: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_printf: symbol not found
wget: Error relocating /usr/lib/libuclient.so: md5_begin: symbol not found
wget: Error relocating /usr/lib/libuclient.so: uloop_timeout_set: symbol not found
wget: Error relocating /usr/lib/libuclient.so: ustream_fd_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_done: symbol not found
wget: Error relocating /usr/bin/wget: blobmsg_parse: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_cancel: symbol not found
wget: Error relocating /usr/bin/wget: uloop_run_timeout: symbol not found
wget: Error relocating /usr/bin/wget: uloop_timeout_set: symbol not found
wget: Error relocating /usr/bin/wget: uloop_init: symbol not found
wget: Error relocating /usr/bin/wget: uloop_cancelled: symbol not found
ERROR: wget: exited with error 127
ERROR: libubus20251004-2025.10.04~ad2768bb-r1: remote server returned error (try 'apk update')

efahl · October 11, 2025, 2:13am

The first issue looks like the rpcd bug that got fixed in rpcd-mod-rpcsys version 2025.09.30 or newer. If you hadn't upgraded for a while, various packages were being erroneously excluded. Tons o' details above: Owut: OpenWrt Upgrade Tool - #816 by efahl

apk sure has a problem cleanly upgrading the ABI-versioned dependencies... Is the thing completely broken now? Might have to do a fresh install to get everything consistent again.