Attended Upgrade auc error - "killed"

apondarz · June 17, 2024, 8:54am

Hi,
I'm trying to perform auc:

root@OpenWrt:~# auc -b 23.05
auc/0.3.1-1
Server:    https://sysupgrade.openwrt.org
Running:   22.03.2 r19803-9a599fee93 on ipq40xx/generic (asus,rt-ac58u)
Available: 23.05.3 r23809-234f1a2efa

Unfortunately process ends with "killed" message:

Downloading image from https://sysupgrade.openwrt.org/store/b6d5ed5cbf11c3d19c957ed062d5b0ad/openwrt-23.05.3-ed157b4ef461-ipq40xx-generic-asus_rt-ac58u-squashfs-sysupgrade.bin
Writing to 'openwrt-23.05.3-ed157b4ef461-ipq40xx-generic-asus_rt-ac58u-squashfs-sysupgrade.bin'
Killed

Any tips what could be a reason for that? It's not the lack of space in /tmp I think:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.0M      4.0M         0 100% /rom
tmpfs                    57.8M     23.7M     34.1M  41% /tmp

What else should I check?

brada4 · June 17, 2024, 9:17am

What your memory is used for? Like "free" and check ps aw

apondarz · June 17, 2024, 9:49am

free:

root@OpenWrt:~# free -h
              total        used        free      shared  buff/cache   available
Mem:         118372       48312       35288       24228       34772       15992
Swap:             0           0           0

ps:

 PID USER       VSZ STAT COMMAND
    1 root      1488 S    /sbin/procd
    2 root         0 SW   [kthreadd]
    3 root         0 IW<  [rcu_gp]
    4 root         0 IW<  [rcu_par_gp]
    6 root         0 IW<  [kworker/0:0H-ev]
    8 root         0 IW<  [mm_percpu_wq]
    9 root         0 SW   [rcu_tasks_trace]
   10 root         0 SW   [ksoftirqd/0]
   11 root         0 IW   [rcu_sched]
   12 root         0 SW   [migration/0]
   13 root         0 SW   [cpuhp/0]
   14 root         0 SW   [cpuhp/1]
   15 root         0 SW   [migration/1]
   16 root         0 SW   [ksoftirqd/1]
   18 root         0 IW<  [kworker/1:0H-kb]
   19 root         0 SW   [cpuhp/2]
   20 root         0 SW   [migration/2]
   21 root         0 SW   [ksoftirqd/2]
   23 root         0 IW<  [kworker/2:0H-ev]
   24 root         0 SW   [cpuhp/3]
   25 root         0 SW   [migration/3]
   26 root         0 SW   [ksoftirqd/3]
   28 root         0 IW<  [kworker/3:0H-kb]
   29 root         0 IW<  [netns]
   31 root         0 SW   [oom_reaper]
   32 root         0 IW<  [writeback]
   33 root         0 SW   [kcompactd0]
   45 root         0 IW<  [pencrypt_serial]
   46 root         0 IW<  [pdecrypt_serial]
   47 root         0 IW<  [cryptd]
   69 root         0 IW<  [kblockd]
   70 root         0 IW<  [blkcg_punt_bio]
   71 root         0 SW   [watchdogd]
   72 root         0 IW<  [kworker/0:1H-ev]
   78 root         0 SW   [kswapd0]
   80 root         0 IW<  [kthrotld]
   83 root         0 IW   [kworker/0:4-eve]
   84 root         0 SW   [spi0]
  106 root         0 IW<  [ipv6_addrconf]
  107 root         0 IW<  [kworker/2:1H-kb]
  121 root         0 SW   [ubi_bgt0d]
  122 root         0 IW<  [kworker/3:1H-kb]
  124 root         0 IW<  [kworker/1:1H-kb]
  128 root         0 SW   [irq/100-keys]
  129 root         0 SW   [irq/101-keys]
  189 root         0 SW   [ubifs_bgt0_6]
  191 root         0 SW   [scsi_eh_0]
  192 root         0 IW<  [scsi_tmf_0]
  193 root         0 SW   [usb-storage]
  209 ubus      1176 S    /sbin/ubusd
  210 root       744 S    /sbin/askfirst /usr/libexec/login.sh
  250 root       872 S    /sbin/urngd
  318 root         0 IW<  [cfg80211]
  320 root         0 IW<  [ath10k_wq]
  321 root         0 IW<  [ath10k_aux_wq]
  322 root         0 IW<  [ath10k_tx_compl]
  376 root         0 SW   [jbd2/sda1-8]
  377 root         0 IW<  [ext4-rsv-conver]
  473 root         0 IW<  [ath10k_wq]
  474 root         0 IW<  [ath10k_aux_wq]
  475 root         0 IW<  [ath10k_tx_compl]
  722 logd      1164 S    /sbin/logd -S 128
  776 root      2136 S    /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1392 root      2408 S    {hostapd} /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbi
 1393 root      2408 S    {wpa_supplicant} /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json
 1395 network   1984 S    /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1396 network   1808 S    /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1459 root      1612 S    /sbin/netifd
 1514 root      1320 S    /usr/sbin/odhcpd
 1639 root      1856 S    /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p
 1696 root      1460 S    /usr/bin/dbus-daemon --system --nofork
 1797 root         0 IW<  [wg-crypt-vpnwg]
 2188 nobody    1856 S    avahi-daemon: running [OpenWrt.local]
 2790 root      1144 S    udhcpc -p /var/run/udhcpc-eth1.pid -s /lib/netifd/dhcp.script -f -t 0 -i eth1 -x hostname:OpenWrt -C -R -
 3866 root      2408 S    {ntpd} /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/b
 3876 ntp       1144 S    /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.o
 3934 root      4464 S    /usr/bin/ttyd -i br-lan /bin/login
 4250 root      1180 SN   /usr/sbin/nlbwmon -o /opt/nlbwmon -b 524288 -i 24h -r 30s -p /usr/share/nlbwmon/protocols -G 24 -I 1 -L 1
 4465 root       912 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.168.0.1:22 -p 2002:2d0b:c7bc::1:22 -p fd80:b844:4
 5169 root      2408 S    {dnsmasq} /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -
 5172 dnsmasq   1300 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 5497 root       932 R    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.168.0.1:22 -p 2002:2d0b:c7bc::1:22 -p fd80:b844:4
 5498 root      1148 S    -ash
 6090 root      4280 SN   /usr/sbin/collectd -C /tmp/collectd.conf -f
 6098 root      2204 S    /usr/sbin/vnstatd --nodaemon --noadd --noexit
 6099 nobody    3768 S    /usr/sbin/openvpn --syslog openvpn(custom_config) --status /var/run/openvpn.custom_config.status --cd /et
 6160 root     20632 S    /usr/sbin/nmbd -F
 6247 root         0 IW   [kworker/1:2-eve]
 6294 root         0 IW   [kworker/u8:0-fl]
 6296 root         0 IW   [kworker/1:0]
 6297 root         0 IW   [kworker/0:0]
 6329 root         0 IW   [kworker/2:1-eve]
 6340 root         0 IW   [kworker/2:2-eve]
 6341 root         0 IW   [kworker/3:1-eve]
 6342 root         0 IW   [kworker/u8:2-ev]
 6353 root         0 IW   [kworker/2:0-eve]
 6354 root         0 IW   [kworker/3:0-eve]
 6355 root         0 IW   [kworker/u8:1-at]
 6363 root         0 IW   [kworker/3:2-eve]
 6366 root         0 IW   [kworker/2:3-eve]
 6367 root      1152 R    ps w

brada4 · June 17, 2024, 9:54am

Install zram-swap to permit swapping out samba
if that does not help stop samba for upgrade.

apondarz · June 17, 2024, 9:56am

I've just noticed that I have /tmp/firmware created - deleted it and started again. Now the effect is different:

Downloading image from https://sysupgrade.openwrt.org/store/b6d5ed5cbf11c3d19c957ed062d5b0ad/openwrt-23.05.3-ed157b4ef461-ipq40xx-generic-asus_rt-ac58u-squashfs-sysupgrade.bin
Writing to 'openwrt-23.05.3-ed157b4ef461-ipq40xx-generic-asus_rt-ac58u-squashfs-sysupgrade.bin'
Mon Jun 17 11:51:24 CEST 2024 upgrade: The device is supported, but the config is incompatible to the new image (1.0->1.1). Please upgrade without keeping config (sysupgrade -n).
...
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/arm_cortex-a7_neon-vfpv4/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/arm_cortex-a7_neon-vfpv4/telephony/Packages.gz
Updated list of available packages in /builder/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/root-ipq40xx/../../../../builder/dl/openwrt_telephony
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/arm_cortex-a7_neon-vfpv4/telephony/Packages.sig
Signature check passed.
Downloading file:packages/Packages
Updated list of available packages in /builder/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/root-ipq40xx/../../../../builder/dl/imagebuilder
Downloading file:packages/Packages.sig
Signature check passed.
Pseudo file "dev" exists in source filesystem "/builder/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/root-ipq40xx/dev".
Ignoring, exclude it (-e/-ef) to override.
17616+1 records in
17617+0 records out
18039808 bytes (18 MB, 17 MiB) copied, 0.0636498 s, 283 MB/s
/builder/staging_dir/host/bin/tar: Option --mtime: Treating date '@1711145382' as 2024-03-22 22:09:42
done
Invalid argument (22)

Maybe this is the reason:

The device is supported, but the config is incompatible to the new image (1.0->1.1). Please upgrade without keeping config (sysupgrade -n).

?

brada4 · June 17, 2024, 9:59am

Which means breaking change between releases that mandates config reset and building it back from zero.
besides config backup make a copy of installed packages
opkg list-installed | cut -d " " -f 1
and use both as reference to get things back working.

brada4 · June 17, 2024, 3:00pm

small addition if you have not yet started, run auc -f -n and save current image on your pc so that you can sysdowngrade and replay config if something goes faoul

apondarz · June 17, 2024, 3:29pm

Stupid question: where can I find this current image?
There is nothing new in /tmp after auc -f -n ...

brada4 · June 17, 2024, 3:30pm

It prints link i think...

efahl · June 17, 2024, 5:50pm

A better, but much more cumbersome way is

eval $(ubus call rpc-sys packagelist '{"all":false}' | jsonfilter -e 'pkgs=$.packages') && echo $pkgs

as it only shows the "top level" packages and not their dependencies, which often change between releases.

EDIT:
If you want one-per-line output, just change echo $pkgs at the end to printf "%s\n" $pkgs.

Also, some numbers: opkg list-installed shows 282 packages on my particular test machine and the rpc-sys output only has 80...

system · June 27, 2024, 5:51pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.