Attended system upgrade broken

Installing and Using OpenWrt
1

Impossible package errors again

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-reject
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-reject6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-log
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-log6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-crc32c
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nft-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-flow
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for firewall4:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package firewall4.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-aead
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-md5
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-sha1
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-sha256
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-sha512
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-hw-safexcel
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-crypto-hw-safexcel:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package kmod-crypto-hw-safexcel.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-gpio-button-hotplug
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-hwmon-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-thermal
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-hwmon-pwmfan
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-hwmon-pwmfan:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package kmod-hwmon-pwmfan.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-leds-gpio
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-sha3
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-rng
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-geniv
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-crypto-gf128
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-mac80211
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-mt7915e
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-mt7915e:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package kmod-mt7915e.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-mt7981-firmware
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-phy-aquantia
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-phy-aquantia:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package kmod-phy-aquantia.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-usb-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-usb3
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-usb3:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package kmod-usb3.
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-nf-ipt
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1) for kmod-ipt-core
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package ppp.
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for ppp-mod-pppoe:
 * 	kernel (= 6.6.43~fc44969aed9b15fb1947669c2bddf103-r1)
 * opkg_install_cmd: Cannot install package ppp-mod-pppoe.
make[2]: *** [Makefile:220: package_install] Error 255
make[1]: *** [Makefile:161: _call_manifest] Error 2
make: *** [Makefile:322: manifest] Error 2

This is my device and current SNAPSHOT details:

{
    "url": "https://sysupgrade.openwrt.org",
    "branch": "SNAPSHOT",
    "revision": "r26971-66177c081f",
    "efi": null,
    "advanced_mode": "1",
    "request_hash": "bafd6ade22383121a45931d5f95b7b2d",
    "sha256_unsigned": "",
    "client": "luci/24.171.68415~188de6f",
    "packages": {
        "luci-app-statistics": "24.148.86114~fbbfdb4",
        "libc": "1.2.5-r4",
        "wpad-basic-mbedtls": "2024.03.09~695277a5-r2",
        "busybox": "1.36.1-r1",
        "kmod-hwmon-pwmfan": "6.6.41-r1",
        "nano": "8.1-r1",
        "kmod-phy-aquantia": "6.6.41-r1",
        "procd-ujail": "2024.07.07~f230c117-r1",
        "base-files": "1609~66177c081f",
        "libustream-mbedtls": "2024.04.19~524a76e5-r1",
        "fitblk": "2",
        "firewall4": "2024.05.21~4c01d1eb-r1",
        "uboot-envtools": "2024.07-r1",
        "dnsmasq": "2.90-r2",
        "procd": "2024.07.07~f230c117-r1",
        "kmod-usb3": "6.6.41-r1",
        "odhcp6c": "2023.05.12~bcd28363-r20",
        "uci": "2023.08.10~5781664d-r1",
        "dropbear": "2024.85-r1",
        "curl": "8.8.0-r2",
        "mtd": "26",
        "odhcpd-ipv6only": "2024.05.08~a2988231-r1",
        "procd-seccomp": "2024.07.07~f230c117-r1",
        "kmod-crypto-hw-safexcel": "6.6.41-r1",
        "kmod-mt7915e": "6.6.41.2024.07.13~3b47d9df-r1",
        "urandom-seed": "3",
        "ppp": "2.4.9_git20210104-r5",
        "kmod-leds-gpio": "6.6.41-r1",
        "kmod-gpio-button-hotplug": "6.6.41-r3",
        "logd": "2024.04.26~85f10530-r1",
        "luci-app-attendedsysupgrade": "24.171.68415~188de6f",
        "mt7981-wo-firmware": "20240513-r1",
        "ca-bundle": "20240203-r1",
        "luci-app-nextdns": "24.148.86114~d174bb8",
        "luci": "23.051.66410~a505bb1",
        "owut": "2024.07.14~75faac21-r1",
        "kmod-mt7981-firmware": "6.6.41.2024.07.13~3b47d9df-r1",
        "kernel": "6.6.41~62853ddbb56e6a6aa007966a20cb744c-r1",
        "collectd-mod-thermal": "5.12.0-r53",
        "urngd": "2023.11.01~44365eb1-r1",
        "ppp-mod-pppoe": "2.4.9_git20210104-r5"
    },
    "profile": "glinet,gl-mt3000",
    "target": "mediatek/filogic",
    "version": "SNAPSHOT",
    "diff_packages": true,
    "filesystem": "squashfs"
}
2

Like just related to the kernel version bump from 6.6.41 to 6.6.43 a few hours ago, and build timing in the two buildbot instances.

It is possible for kernel/images and packages be off-sync for a few hours&days after kernel version bumps, as kernel and packages are built by different round-robin schedules by different buildbot instances.

3 Likes
3

Is this the same issue as e.g. Error: Received incorrect version r24021-6edde2b502 (requested r24016-f791ec1f6d) trying to upgrade from 23.05.4 to 23.05-SNAPSHOT?

I've been trying to use either auc or Attended Sysupgrade on 23.05.4 to 23.05-SNAPSHOT or owut to upgrade SNAPSHOT builds and keep running into this same error with different versions. Basically, it seems that SNAPSHOT or VV.XX-SNAPSHOT versions can only be upgraded by downloading sysupgrade images manually these days.

@efahl any thoughts?

1 Like
4

Likely slightly different, as those 5 commits between r24016-f791ec1f6d and r24021-6edde2b502 are normal commits, not kernel updates.

But the underlying reason is somewhat similar: buildbot has already compiled newer image, so the old r24016 binaries are not downloadable any more.
E.g. x86_64:

Snapshots (both in main/master, and in the stable 23.05) are rebuilt rather quickly, roughly once per day if there are new commits.

1 Like
5

But the underlying reason is somewhat similar: buildbot has already compiled newer image, so the old r24016 binaries are not downloadable any more.
E.g. x86_64:

So in these cases, what's the best thing for users to do? It doesn't look like I can force a version for the client to ask for (beyond tag/ branch-snapshot/main-snapshot), and the clients all appear to treat this as a fatal error, not even downloading the image. At some point I thought the keep option allowed at least the download of the build, but now not even -k --force appears to help.

Are SNAPSHOT builds just considered to be too bleeding edge for end-users to upgrade to via these systems? If so, maybe they should be removed from the builds offered for automatic upgrade... e.g. SNAPSHOT is now 30 changes beyond what the upgrade servers know about: Error: Received incorrect version r27032-274b18d105 (requested r27002-a9f58dddb7)... that's super-frustrating to have the infrastructure to upgrade directly from the affected system but having that system fail 99+% of the time!

6

Well, snapshots are a moving target. especially in main/master, but also in stable 23.05.
They are not really meant for static upgrade tools.

When using them, the "normal" easy errorproof way is to build your own firmware from scratch, from sources. I do that. Then you surely have a coherent image from current sources.

Other way is to download and use the imagebuilder in your pc.

Using auc/attendedsysupgrade/owut may be difficult as the tools/servers apparently have some caching, which easily throws cog into wheels, as you are noticing. They are most useful with static releases. (I am not using those tools myself.)

7

@hnyman thanks, I figured I'd do myself and the project a solid by pegging my OpenWRT routers to the branch snapshots and being able to report issues on those if I could do it easily (aka via auc / owut / Attended Sysupgrade). But if I have to set up my own build infrastructure, that probably puts it into the "too difficult" / "too much work" category (at this point, anyway), and even if I have to copy-pasta my package config into the firmware selector each time, that means I'm likely to upgrade much less frequently.

But at least I'm glad to get others' feedback that I'm just not using the tools for their intended purpose... that should probably be made much more clear, since right now the tools look very much like they should work for those use cases, but they don't.

2 Likes
8

Don't take my comments too negatively. The tools likely work most of the time in 23.05 as often there aren't that many commits there and things are static for some days.

1 Like
9

That only works in owut if the error is detected on the client end. If the error is generated at the ASU server, then owut never gets anything to download/keep.

Many of the issues we've been seeing should have been resolved in the past hour or so, as Paul just did a major update on the server internals and restarted it. My owut regression tests are all running again (missing libusb-1.0 and some others). Also did a "live test" upgrading my x86 test machine (had the same 'incorrect version' error this morning, install worked fine 2 minutes ago).

2 Likes
10

Hmm, just tried 23.05.4 -> 23.05-SNAPSHOT on x86 and ipq806x and both get Error: Received incorrect version r24021-6edde2b502 (requested r24016-f791ec1f6d). The difference between client-side checks and server checks does make sense, but only makes it more frustrating since the "use --force to install downloaded packages" workaround no longer applies.

11

Yeah, that's a major source of frustration for people. Since the incorrect version error is only generated on the server side when the client specifies a version_code, I think I'll change just changed owut to not supply the version code and then do the check after the download. That will give you the option of installing or not (at least for SNAPSHOT and future release users...).

Or, for current users of owut, it's a one liner, just comment out the version_code line in the blob function (all the validation is already in place):

1 Like
12

Is owut usable on 23.05.4 as well as main SNAPSHOT builds? Right now I'm running 23.05.4 on all my "production" hardware (ie, the stuff actually providing day-to-day 'net access at home), so still using auc there... but if owut can run there as well, I'll at least standardize on that.

I really do like your idea of not sending the version_code and leaving the check to the client, because as you've suggested (here and previously) that would at least let users automate the build/download and then install with sysupgrade [--whatever-options] /tmp/firmware.bin. It would be awesome to do the same with auc if the answer to the first questions isn't a "yeah, duh, get it via opkg" :wink:

13

No, regrettably. It depends on the quite new ucode_mod_uclient package that is only available on SNAPSHOT right now. I played with backporting, but it was a much bigger task that I wanted to get into, so unless the real devs do it, you're stuck waiting for 24.x...

And I should also add, Daniel deleted auc from SNAPSHOT when I got owut working, so you won't be able to standardize for a while.

1 Like
14

I have to say I love the fact that owut is not a binary, so making that 1-line change to not send the version_code was super easy :grinning: , so I understand everyone wanting to purge auc as soon as possible!

Your suggested definitely helps, though now I realize that with that change, I'm getting the opposite version mismatch than what I see in auc (auc gives "incorrect version NEWER requested OLDER", whereas owut gives "incorrect version OLDER requested NEWER" :confounded:)... but at least now if a newer version is available and has propagated to the build servers, I should be able to download it via owut

1 Like
15

Yeah, that was one of my primary motivators for the whole project, just having ucode available makes many things like this so much more accessible. The only real problem is that owut is ~4x bigger than the auc binary, which bothers me for the older devices.

$ pkg-size -p auc
'auc' files:
     32,916 /usr/sbin/auc
-----------
     32,916 bytes in package 'auc'
===========
     32,916 bytes used by 'auc' and its unique packages

$ pkg-size -p owut
'owut' files:
      1,830 /etc/owut.d/pre-install.sh
        805 /etc/uci-defaults/attendedsysupgrade-owut
         13 /lib/upgrade/keep.d/owut
     59,334 /usr/bin/owut
      9,291 /usr/share/ucode/utils/argparse.uc
-----------
     71,273 bytes in package 'owut'

'ucode-mod-uclient' files via 'owut':
     16,386 /usr/lib/ucode/uclient.so
-----------
     16,386 bytes in package 'ucode-mod-uclient'

'ucode-mod-uloop' files via 'owut':
     28,674 /usr/lib/ucode/uloop.so
-----------
     28,674 bytes in package 'ucode-mod-uloop'
===========
    116,333 bytes used by 'owut' and its unique packages

Oh, and I checked in that version_code change an hour or so ago, it will be in the next version update. I should edit the wiki page and discuss the details there, too...

1 Like
16

so i f understand correctly, if i'm on stable 23.05.4, i can't use auc to upgrade to 23.05-SNAPSHOT? I'd have to flash a snapshot build, since snapshot builds of 23.X contain owut, but current stable version does not?

17

so i f understand correctly, if i'm on stable 23.05.4, i can't use auc to upgrade to 23.05-SNAPSHOT? I'd have to flash a snapshot build, since snapshot builds of 23.X contain owut, but current stable version does not?

Yeah, all my attempts at upgrading to 23.05-SNAPSHOT (x86_64 and ipq806x) result in Error: Received incorrect version r24021-6edde2b502 (requested r24016-f791ec1f6d). Maybe the build infrastructure will settle down, but right now auc seems only really usable for upgrading to stable releases like the recent 23.05.4

Note that owut is only available in main SNAPSHOT builds (ie, the wild west unreleased code that will eventually turn into the next release, aka 24.x), NOT in 23.05-SNAPSHOT (the maintenance branch for 23.05.x that will at some point become 23.05.5), so that won't help you in the short-term either.

18

Just a thought, but you could pack it and then unpack on execute. That should slim it down quite a bit.

19

Good idea, if the size bothers people too much I might try it.

$ tar czf xxx.tgz owut

$ ll xxx.tgz
-rw-r--r--    1 root     root         19857 Jul 30 07:07 xxx.tgz

$ tar tvzf xxx.tgz
-rwxr-xr-x 0/0     60948 2024-07-29 18:28:07 owut

Drops us down by 42k, so ~75k total, which is not too bad...

20

You should be able to do that, you just need to know the magic handshake:

$ auc -c -b 23.05 -B 23.05-SNAPSHOT
auc/0.3.2-1
Server:    https://sysupgrade.openwrt.org
Running:   23.05.4 r24012-d8dd03c46f on x86/64 (generic)
Available: 23.05-SNAPSHOT r24021-6edde2b502
Requesting package lists...
 kmod-usb-storage: 5.15.162-1 -> 5.15.164-1
 kmod-usb-core: 5.15.162-1 -> 5.15.164-1
...