That's the way it should work, but I've experienced, and had too many reports about, losing config when changing rootfs size that I'm sort of surprised when it doesn't break. (When I get bored I read through the sysupgrade code, still trying to figure out all the nuances as big chunks of it are platform-specific so there are a lot of side quests.)
After upgrading 24.10.0-rc6 to 24.10.0 (on RPi CM4) I noticed that several packages are still outdated when rerunning opkg update and owut check after reboot - any advice what I should do to troubleshoot this ?
Summary
root@router:~# owut check
owut - OpenWrt Upgrade Tool 2025.01.29~bced54ad-r1 (/usr/bin/owut)
ASU-Server https://sysupgrade.openwrt.org
Upstream https://downloads.openwrt.org
Target bcm27xx/bcm2711
Profile rpi-4
Package-arch aarch64_cortex-a72
Root-FS-type squashfs
Sys-type sysupgrade
Version-from 24.10.0 r28427-6df0e3d02a (kernel 6.6.73)
Version-to 24.10.0 r28427-6df0e3d02a (kernel 6.6.73)
Build-FS-type squashfs
Build-at 2025-02-03T23:09:37Z (~131 hours ago)
Image-prefix openwrt-24.10.0-bcm27xx-bcm2711-rpi-4
Image-URL https://downloads.openwrt.org/releases/24.10.0/targets/bcm27xx/bcm2711
Image-file openwrt-24.10.0-bcm27xx-bcm2711-rpi-4-squashfs-sysupgrade.img.gz
Installed 183 packages
Top-level 55 packages
Default 42 packages
User-installed 23 packages (top-level only)
Package version changes:
adblock-fast 1.1.2-r20 1.1.3-r1
banip 1.5.0-r3 1.5.0-r6
dnsmasq 2.90-r3 2.90-r4
hostapd-common 2024.09.15~5ace39b0-r1 2024.09.15~5ace39b0-r2
kmod-brcmfmac 6.6.69.6.12.6-r1 6.6.73.6.12.6-r1
kmod-brcmutil 6.6.69.6.12.6-r1 6.6.73.6.12.6-r1
kmod-cfg80211 6.6.69.6.12.6-r1 6.6.73.6.12.6-r1
kmod-crypto-acompress 6.6.69-r1 6.6.73-r1
kmod-crypto-crc32c 6.6.69-r1 6.6.73-r1
kmod-crypto-hash 6.6.69-r1 6.6.73-r1
kmod-fixed-phy 6.6.69-r1 6.6.73-r1
kmod-fs-vfat 6.6.69-r1 6.6.73-r1
kmod-hid 6.6.69-r1 6.6.73-r1
kmod-hid-generic 6.6.69-r1 6.6.73-r1
kmod-hwmon-core 6.6.69-r1 6.6.73-r1
kmod-input-core 6.6.69-r1 6.6.73-r1
kmod-input-evdev 6.6.69-r1 6.6.73-r1
kmod-lib-crc-ccitt 6.6.69-r1 6.6.73-r1
kmod-lib-crc32c 6.6.69-r1 6.6.73-r1
kmod-lib-lzo 6.6.69-r1 6.6.73-r1
kmod-libphy 6.6.69-r1 6.6.73-r1
kmod-mdio-devres 6.6.69-r1 6.6.73-r1
kmod-mii 6.6.69-r1 6.6.73-r1
kmod-mmc 6.6.69-r1 6.6.73-r1
kmod-nf-conntrack 6.6.69-r1 6.6.73-r1
kmod-nf-conntrack6 6.6.69-r1 6.6.73-r1
kmod-nf-flow 6.6.69-r1 6.6.73-r1
kmod-nf-log 6.6.69-r1 6.6.73-r1
kmod-nf-log6 6.6.69-r1 6.6.73-r1
kmod-nf-nat 6.6.69-r1 6.6.73-r1
kmod-nf-reject 6.6.69-r1 6.6.73-r1
kmod-nf-reject6 6.6.69-r1 6.6.73-r1
kmod-nfnetlink 6.6.69-r1 6.6.73-r1
kmod-nft-core 6.6.69-r1 6.6.73-r1
kmod-nft-fib 6.6.69-r1 6.6.73-r1
kmod-nft-nat 6.6.69-r1 6.6.73-r1
kmod-nft-offload 6.6.69-r1 6.6.73-r1
kmod-nls-base 6.6.69-r1 6.6.73-r1
kmod-nls-cp437 6.6.69-r1 6.6.73-r1
kmod-nls-iso8859-1 6.6.69-r1 6.6.73-r1
kmod-nls-utf8 6.6.69-r1 6.6.73-r1
kmod-of-mdio 6.6.69-r1 6.6.73-r1
kmod-phy-microchip 6.6.69-r1 6.6.73-r1
kmod-phy-realtek 6.6.69-r1 6.6.73-r1
kmod-ppp 6.6.69-r1 6.6.73-r1
kmod-pppoe 6.6.69-r1 6.6.73-r1
kmod-pppox 6.6.69-r1 6.6.73-r1
kmod-r8169 6.6.69-r1 6.6.73-r1
kmod-slhc 6.6.69-r1 6.6.73-r1
kmod-sound-arm-bcm2835 6.6.69-r1 6.6.73-r1
kmod-sound-core 6.6.69-r1 6.6.73-r1
kmod-usb-core 6.6.69-r1 6.6.73-r1
kmod-usb-hid 6.6.69-r1 6.6.73-r1
kmod-usb-net 6.6.69-r1 6.6.73-r1
kmod-usb-net-lan78xx 6.6.69-r1 6.6.73-r1
libuci 2023.08.10~5781664d-r1 2025.01.20~16ff0bad-r1
luci 25.006.62535~c9cc773 25.037.68331~42f464c
luci-app-banip 25.024.21113~4d6c9a4 25.037.68331~42f464c
luci-app-firewall 25.006.62535~c9cc773 25.037.68331~42f464c
luci-app-package-manager 25.006.62535~c9cc773 25.037.68331~42f464c
luci-base 25.006.62535~c9cc773 25.037.68331~42f464c
luci-light 25.006.62535~c9cc773 25.037.68331~42f464c
luci-mod-admin-full 25.006.62535~c9cc773 25.037.68331~42f464c
luci-mod-network 25.006.62535~c9cc773 25.037.68331~42f464c
luci-mod-status 25.006.62535~c9cc773 25.037.68331~42f464c
luci-mod-system 25.006.62535~c9cc773 25.037.68331~42f464c
luci-proto-ipv6 25.006.62535~c9cc773 25.037.68331~42f464c
luci-proto-ppp 25.006.62535~c9cc773 25.037.68331~42f464c
luci-ssl 25.006.62535~c9cc773 25.037.68331~42f464c
luci-theme-bootstrap 25.006.62535~c9cc773 25.037.68331~42f464c
uci 2023.08.10~5781664d-r1 2025.01.20~16ff0bad-r1
wpad-basic-mbedtls 2024.09.15~5ace39b0-r1 2024.09.15~5ace39b0-r2
72 packages are out-of-date
Default package analysis:
Default Provided-by
nftables nftables-json
There are currently package build failures for 24.10.0 aarch64_cortex-a72:
Feed: telephony
freetdm Sat Feb 8 18:52:36 2025 - not installed
Failures don't affect this device, details at
https://downloads.openwrt.org/releases/faillogs-24.10/aarch64_cortex-a72/
It is safe to proceed with an upgrade
Thanks for looking into this. Just to clarify both of my GL.iNet MT-6000 devices have the same issue and, as you mentioned, probable server-side issue.
I tried the download command and get the same result.
owut download -v
owut - OpenWrt Upgrade Tool 2025.01.29~bced54ad-r1 (/usr/bin/owut)
ERROR: uclient error code=1
This could be due to the server being down or inaccessible, check
https://sysupgrade.openwrt.org/json/v1/overview.json
ERROR: Response status 0 while downloading
https://sysupgrade.openwrt.org/json/v1/overview.json
Dumb question: tempted to move my spare R4 from 24.10 to snapshot (to play with apk and when it lands, RSS). Running into what I assume are apk related versioning issues when trying with owut -V snapshot. What's the best approach there, use firmware selector? Use a plain vanilla image? Wait some longer?
Yes, that's exactly the problem. All those missing to-version results are a direct result of the incomplete ABI version naming in the apk tooling.
And, yes again, FS using owut list to generate your package list should get you a good, complete image by bypassing all of owut's internal checks for version consistency. Once on SNAPSHOT, owut should work (mostly) fine, you'll see the occasional missing to-version message as those specific packages get upgrades, but usually you can just --remove <pkg> and it works fine.
(And downgrading back from SNAPSHOT to 24.10 should be essentially identical: owut list, FS to 24.10 and go...)
It looks like the sysupgrade didn't actually install the new image (those kmods at the previous kernel version are a dead giveaway). Do you see anything that might give us a hint on the screen just as the device is rebooting in the final steps of the owut upgrade?
I seem to remember seeing this sometime before (took me long enough, sorry, I've been fighting Mr Influenza since Saturday and still feel like shit)... Take a look at Owut: OpenWrt Upgrade Tool - #33 by efahl and see if that looks familiar.
Then try this, let's see if we can pin it on something outside your device:
uclient-fetch -4 https://sysupgrade.openwrt.org/json/v1/overview.json
rm overview.json
uclient-fetch -6 https://sysupgrade.openwrt.org/json/v1/overview.json
Yes, it looks like IPv6 is the issue. I cannot ping the IPv6 address of sysupgrade.openwrt.org and the uciclient command for IPv6 returned the following.
uclient-fetch -6 https://sysupgrade.openwrt.org/json/v1/overview.
json
Downloading 'https://sysupgrade.openwrt.org/json/v1/overview.json'
Connecting to 2001:678:6e1:1001:be24:11ff:fe23:4c6d:443
SSL error: NET - Sending information through the socket failed
Connection error: Connection failed
Is there a workaround?
Thanks for looking into this, now I know what the issue is after all this time.
Nothing through owut itself (the api call used in uclient-fetch that allows the -4 isn't exposed in ucode), but if you disable ipv6 on your WAN temporarily, it might allow things to work.
# Where "eth0" is your WAN nic, change it to match your system:
$ sysctl net.ipv6.conf.eth0.disable_ipv6=1
# Confirm, should be no IPv6 addresses immediately
$ ip a show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:41:9d:01:a0:23 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.20/24 brd 10.1.1.255 scope global eth0
valid_lft forever preferred_lft forever
$ owut ...
If you need IPv6 working after this, use sysctl ...=0 and then do a /etc/init.d/network restart. (Or if you do an upgrade and reboot, it will all be reset automatically.)
I have three Openwrt devices (rockchip, ramips and ath79). Owut has been working only for rockchip for the last two weeks. I receive the following errors when trying to upgrade the other two devices:
Status: validate_manifest
Progress: 24s total = 0s in queue + 24s in build
--
Status: Error: Impossible package selection: libgcc not in manifest
Progress: 26s total = 0s in queue + 26s in build
Build failed in 26s total = 0s in queue + 26s to build:
ASU server error =
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/rq/worker.py", line 1633, in perform_job
return_value = job.perform()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1331, in perform
self._result = self._execute()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1365, in _execute
result = self.func(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/asu/build.py", line 252, in build
report_error(job, err)
File "/app/asu/util.py", line 298, in report_error
raise RuntimeError(msg)
RuntimeError: Impossible package selection: libgcc not in manifest
Any suggestions? Is this an error on ASU server side or is it something wrong with my local configuration?
That's almost always an issue in the image builder itself, but libgcc missing is a very strange one. Was this with snapshot or a release?
All three services are on master snapshots, only one works. The other two are returning the above error.
Ok, let's dig in. Start by posting output from owut blob, which gives me enough to try to replicate the issue.
Following is shown just before the reboot:
Image saved : /tmp/firmware.bin
Manifest : /tmp/firmware-manifest.json
Verifying : /tmp/firmware.bin (16007584 bytes) against /tmp/firmware.sha256sums
Saved sha256 matches
Sat Feb 15 06:54:27 UTC 2025 upgrade: Reading partition table from bootdisk...
zcat: write error: Broken pipe
zcat: write: Broken pipe
Sat Feb 15 06:54:27 UTC 2025 upgrade: Reading partition table from image...
Checks complete, image is valid.
Installing /tmp/firmware.bin and rebooting...
root@router:~# Connection to 192.168.0.1 closed by remote host.
Connection to 192.168.0.1 closed.
Is the zcat: write error possibly indicating an issue with the sdcard?
root@OpenWrt:~# owut blob
{
"client": "owut/2025.01.29~bced54ad-r1",
"target": "ramips/mt7621",
"profile": "xiaomi_mi-router-4a-gigabit",
"version": "SNAPSHOT",
"version_code": "r28819-78b78a4268",
"filesystem": "squashfs",
"diff_packages": true,
"packages_versions": {
"apk-mbedtls": "3.0.0_pre20241130-r2",
"base-files": "1655~78b78a4268",
"busybox": "1.37.0-r4",
"ca-bundle": "20240203-r1",
"dnsmasq": "2.90-r4",
"dropbear": "2024.86-r1",
"firewall4": "2024.12.18~18fc0ead-r1",
"fstools": "2024.12.02~49d36ba2-r1",
"fwtool": "2019.11.12~8f7fe925-r1",
"getrandom": "2024.04.26~85f10530-r1",
"hostapd-common": "2025.02.09~c8c7d56a-r1",
"iwinfo": "2025.02.06~9cec6b4d-r1",
"jansson4": "2.14-r3",
"jshn": "2024.03.29~eb9bcb64-r1",
"jsonfilter": "2024.01.23~594cfa86-r1",
"kernel": "6.6.77~3a8ecf047c5ccf31cf946aa804fad542-r1",
"kmod-crypto-acompress": "6.6.77-r1",
"kmod-crypto-aead": "6.6.77-r1",
"kmod-crypto-authenc": "6.6.77-r1",
"kmod-crypto-crc32c": "6.6.77-r1",
"kmod-crypto-des": "6.6.77-r1",
"kmod-crypto-hash": "6.6.77-r1",
"kmod-crypto-hw-eip93": "6.6.77-r1",
"kmod-crypto-manager": "6.6.77-r1",
"kmod-crypto-md5": "6.6.77-r1",
"kmod-crypto-null": "6.6.77-r1",
"kmod-crypto-sha1": "6.6.77-r1",
"kmod-crypto-sha256": "6.6.77-r1",
"kmod-gpio-button-hotplug": "6.6.77-r5",
"kmod-leds-gpio": "6.6.77-r1",
"kmod-lib-crc-ccitt": "6.6.77-r1",
"kmod-lib-crc32c": "6.6.77-r1",
"kmod-lib-lzo": "6.6.77-r1",
"kmod-mt7603": "6.6.77.2025.01.22~a22d59e4-r1",
"kmod-mt76x2": "6.6.77.2025.01.22~a22d59e4-r1",
"kmod-nf-conntrack": "6.6.77-r1",
"kmod-nf-conntrack6": "6.6.77-r1",
"kmod-nf-flow": "6.6.77-r1",
"kmod-nf-log": "6.6.77-r1",
"kmod-nf-log6": "6.6.77-r1",
"kmod-nf-nat": "6.6.77-r1",
"kmod-nf-reject": "6.6.77-r1",
"kmod-nf-reject6": "6.6.77-r1",
"kmod-nfnetlink": "6.6.77-r1",
"kmod-nft-core": "6.6.77-r1",
"kmod-nft-fib": "6.6.77-r1",
"kmod-nft-nat": "6.6.77-r1",
"kmod-nft-offload": "6.6.77-r1",
"kmod-ppp": "6.6.77-r1",
"kmod-pppoe": "6.6.77-r1",
"kmod-pppox": "6.6.77-r1",
"kmod-slhc": "6.6.77-r1",
"libblobmsg-json20240329": "2024.03.29~eb9bcb64-r1",
"libc": "1.2.5-r4",
"libiwinfo-data": "2025.02.06~9cec6b4d-r1",
"libiwinfo20230701": "2025.02.06~9cec6b4d-r1",
"libjson-c5": "0.18-r1",
"libjson-script20240329": "2024.03.29~eb9bcb64-r1",
"libmbedtls21": "3.6.2-r1",
"libmnl0": "1.0.5-r1",
"libnftnl11": "1.2.8-r1",
"libnl-tiny1": "2023.12.05~965c4bf4-r1",
"libubox20240329": "2024.03.29~eb9bcb64-r1",
"libubus20250102": "2025.01.02~afa57cce-r1",
"libuci20250120": "2025.01.20~16ff0bad-r1",
"libuclient20201210": "2024.10.22~88ae8f20-r1",
"libucode20230711": "2025.02.10~a8a11aea-r1",
"libudebug": "2023.12.06~6d3f51f9",
"libustream-mbedtls20201210": "2024.07.28~99bd3d2b-r1",
"logd": "2024.04.26~85f10530-r1",
"luci-ssl": "25.044.47345~9c3be81",
"mtd": "26",
"netifd": "2024.12.17~ea01ed41-r1",
"nftables-json": "1.1.1-r1",
"odhcp6c": "2024.09.25~b6ae9ffa-r1",
"odhcpd-ipv6only": "2024.05.08~a2988231-r1",
"openwrt-keyring": "2024.11.01~fbae29d7-r1",
"owut": "2025.01.29~bced54ad-r1",
"ppp": "2.5.2-r1",
"ppp-mod-pppoe": "2.5.2-r1",
"procd": "2025.01.30~7fcb5a27-r1",
"procd-seccomp": "2025.01.30~7fcb5a27-r1",
"procd-ujail": "2025.01.30~7fcb5a27-r1",
"ubi-utils": "2.2.1-r1",
"ubox": "2024.04.26~85f10530-r1",
"ubus": "2025.01.02~afa57cce-r1",
"ubusd": "2025.01.02~afa57cce-r1",
"uci": "2025.01.20~16ff0bad-r1",
"uclient-fetch": "2024.10.22~88ae8f20-r1",
"ucode": "2025.02.10~a8a11aea-r1",
"ucode-mod-fs": "2025.02.10~a8a11aea-r1",
"ucode-mod-nl80211": "2025.02.10~a8a11aea-r1",
"ucode-mod-rtnl": "2025.02.10~a8a11aea-r1",
"ucode-mod-ubus": "2025.02.10~a8a11aea-r1",
"ucode-mod-uci": "2025.02.10~a8a11aea-r1",
"ucode-mod-uloop": "2025.02.10~a8a11aea-r1",
"urandom-seed": "3",
"urngd": "2023.11.01~44365eb1-r1",
"usign": "2020.05.23~f1f65026-r1",
"wpad-basic-mbedtls": "2025.02.09~c8c7d56a-r1",
"zlib": "1.3.1-r1",
"libgcc": "13.3.0-r4"
}
}
It might be that owut detects package name as libgcc, while if I look at installed packages it is called libgcc1. Although, the libgcc1 package is also installed on the device that performs owut upgrade without errors.
How to upgrade to snapshot using this tool?
owut upgrade -Vowut upgrade Version-to 24.10-SNAPSHOT
not working for me
Edit:
got it
root@OpenWrt:~# owut upgrade --version-to SNAPSHOT
Edit 2:
root@OpenWrt:~# owut upgrade --version-to 24.10-SNAPSHOT
owut - OpenWrt Upgrade Tool 2025.01.29~bced54ad-r1 (/usr/bin/owut)
ASU-Server https://sysupgrade.openwrt.org
Upstream https://downloads.openwrt.org
Target x86/64
Profile generic
Package-arch x86_64
Root-FS-type ext4
Sys-type combined
Version-from 24.10.0 r28427-6df0e3d02a (kernel 6.6.73)
Version-to 24.10-SNAPSHOT r28459-b7b6ae7424 (kernel 6.6.73)
Build-FS-type ext4
Build-at 2025-02-14T10:31:13Z (~21 hours ago)
Image-prefix openwrt-24.10-snapshot-r28459-b7b6ae7424-x86-64-generic
Image-URL https://downloads.openwrt.org/releases/24.10-SNAPSHOT/targets/x86/64
Image-file openwrt-24.10-snapshot-r28459-b7b6ae7424-x86-64-generic-ext4-combined.img.gz
Installed 182 packages
Top-level 55 packages
Default 44 packages
User-installed 17 packages (top-level only)
Package version changes:
All packages are up-to-date
Default package analysis:
Default Provided-by
nftables nftables-json
There are currently package build failures for 24.10-SNAPSHOT x86_64:
Feed: telephony
freeswitch Sat Feb 15 05:26:23 2025 - not installed
freeswitch-mod-bcg729 Sat Feb 15 05:33:15 2025 - not installed
freetdm Sat Feb 15 05:33:17 2025 - not installed
Failures don't affect this device, details at
https://downloads.openwrt.org/releases/faillogs-24.10/x86_64/
There are no changes to upgrade (see '--force')
Yeah, that's likely a fatal error in sysupgrade. It looks like it's failing to unzip the image during metadata extraction. Seems like it could be caused by a couple things, out-of-memory on reading the file or bad write, but since the Pi has tons o' memory, I'd say you're right to suspect the SD card first.
If you've still got /tmp/firmware.bin around (or just run owut download again to get a new one), maybe try zcat /tmp/firmware.bin > /dev/null and see if you can reproduce a read error on the CLI to get some more meaningful output.
If that fails to show anything, maybe test out the SD card somehow?
Here's where zcat is used in sysupgrade, a bunch of calls to get_image are strewn throughout the rest of that file and also platform.sh...
Haha, yeah about that: https://github.com/efahl/owut/blob/main/files/owut#L988 That's my one super-hacky bit in owut to keep it working on snapshots until we get the ABI versioning working with apk...
It took me an hour or two, but it turns out that, yes, that's exactly the issue. It's an owut bug in handling that specific case, introduced after the hack with the use of packages_versions.
I think for a workaround you could edit /usr/bin/owut and about line 990, stick a return; at the top of the function so it doesn't do anything. I think you'll get warnings about missing default packages, but ignore those and see if it works otherwise.
(Or, just bail out on owut for the build, use owut list output and use the Firmware Selector, which does not deal with the ABI stuff at all, so should work fine.)
Ok, so you want the main snapshot and not the release one? That worked for you?