Sysupgrade not upgrading my router

[doublez13]

I have an Engenius ens202ext running 18.06.01 from the releases section.

This morning I tried upgrading to 18.06.02 using both LuCI and sysupgrade, with no luck.
The device reboots almost immediately back to 18.06.01.
Here is the output of sysupgrade -v

root@OpenWrt:/tmp# sysupgrade -v openwrt-18.06.2-ar71xx-generic-ens202ext-squashfs-sysupgrade.bin 
Image metadata not found
Saving config files...
etc/config/dhcp
etc/config/dropbear
etc/config/dropbear-opkg
etc/config/firewall
etc/config/luci
etc/config/luci-opkg
etc/config/network
etc/config/rpcd
etc/config/system
etc/config/ubootenv
etc/config/ucitrack
etc/config/uhttpd
etc/config/wireless
etc/dropbear/authorized_keys
etc/dropbear/dropbear_rsa_host_key
etc/dropbear/dropbear_rsa_host_key-opkg
etc/fw_env.config
etc/group
etc/hosts
etc/inittab
etc/luci-uploads/.placeholder
etc/opkg/keys/1035ac73cc4e59e3
etc/opkg/keys/5151f69420c3f508
etc/opkg/keys/72a57f2191b211e0
etc/opkg/keys/792d9d9b39f180dc
etc/opkg/keys/9ef4694208102c43
etc/opkg/keys/b26f36ae0f4106d
etc/opkg/keys/b5043e70f9a75cde
etc/opkg/keys/c10b9afab19ee428
etc/opkg/keys/dace9d4df16896bf
etc/opkg/keys/dd6de0d06bbd3d85
etc/passwd
etc/profile
etc/rc.local
etc/shadow
etc/shells
etc/sysctl.conf
etc/uhttpd.crt
etc/uhttpd.key
etc/uhttpd.key
etc/uhttpd.crt
Commencing upgrade. Closing all shell sessions.

Any help is appreciated!!

[moeller0]

Have you tried flashing it immediately again after the first failed flash? On my wndr3700v2, flashing fails after some uptime, but doing it after a reboot so far always worked.

[doublez13]

Yeah I first tried using LuCI, and then immediately tried using sysupgrade.

I have two of these devices. I just tried on the other one (running the same 18.06.01 image), with the same result.

[jeff]

Does sysupgrade -v --test /tmp/desired_upgrade_image provide more insight?

[doublez13]

sysupgrade -v --test /tmp/

root@OpenWrt:/tmp# sysupgrade -v --test /tmp/openwrt-18.06.2-ar71xx-generic-ens202ext-squashfs-sysupgrade.bin 
Image metadata not found

This is all it says. Not sure if the metadata warning is an issue, or safely ignorable?

[trendy]

Have you verified the integrity of the image against the sha256 hash that is published in the download page?

[doublez13]

Yup for sure!

[trendy]

Not sure if connected somehow, but there is another post for the same problem.

Netgear R8000 will not upgrade to 18.06.2

Already I've upgraded WDR3600 from 18.06.1 to 18.06.2 without problems. The WDR3600 runs as simple Access Point to the main device, R8000. The R8000 is running 18.06.1 and has, over the years, been through various OpenWRT and LEDE installs and upgrades. I downloaded and verified checksum for openwrt-18.06.2-bcm53xx-netgear-r8000-squashfs.chk First attempt at upgrade was via LUCI, keep settings checked. After reboot, R8000 was still on 18.06.1. Then tried via commandline sysupgrade -v /tmp…

[reinerotto]

This sounds like another occurence of 'silent' error in sysupgrade (which I consider a dangerous bug), preventing successful upgrade. One likely reason is insufficient RAM for sysupgrade.To verify, just kill the most memory hungry processes before sysupgrade, and disable swap, dismount mass storage devices, if present.
If it works, pls confirm the bug, I reported already here:
https://bugs.openwrt.org/index.php?do=details&task_id=2024&string=sysupgrade&type[0]=&sev[0]=&pri[0]=&due[0]=&reported[0]=&cat[0]=&status[0]=open&percent[0]=&opened=&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=

[moeller0]

For what it is worth, I use swap as well, so next time I will try swapoff ; sysupgrade ... (assuming I remember)

[doublez13]

root@OpenWrt:/tmp# free
             total       used       free     shared    buffers     cached
Mem:         60320      25360      34960       4272       1748      10988
-/+ buffers/cache:      12624      47696
Swap:            0          0          0

I don't have any swap enabled on the router. I've also got ~35MB of free RAM. Anything in particular you'd like me to try?

[geofb]

On the R8000 I killed haproxy and unbound, giving:

             total       used       free     shared    buffers     cached
Mem:        254232      57000     197232       8372       1976      20908
-/+ buffers/cache:      34116     220116
Swap:            0          0          0

Tried "sysupgrade -v .." again.
Still no joy - router comes back as 18.06.1

[reinerotto]

On next try, just gracefully stop all active services before sysupgrade, "/etc/init.d/service stop"
And verify, it exited.
Kill all your user processes, if any.
In summary, to have the system as quiet as possible.

And verify, that the new image really works, when flashed directly.

[geofb]

Stopped services, and killed user processes.
After reboot, still showing 18.06.1

verify, that the new image really works, when flashed directly.

Not sure what I need to do to achieve that - could you let me know what I should do, please?

Image metadata not found
Found CHK image with device board_id U12H315T00_NETGEAR
Found a valid TRX version 1
Saving config files...
...
Commencing upgrade. Closing all shell sessions.

  PID USER       VSZ STAT COMMAND
    1 root      1380 S    /sbin/procd
    2 root         0 SW   [kthreadd]
    4 root         0 IW<  [kworker/0:0H]
    6 root         0 IW<  [mm_percpu_wq]
    7 root         0 SW   [ksoftirqd/0]
    8 root         0 IW   [rcu_sched]
    9 root         0 IW   [rcu_bh]
   10 root         0 SW   [migration/0]
   11 root         0 SW   [cpuhp/0]
   12 root         0 SW   [cpuhp/1]
   13 root         0 SW   [migration/1]
   14 root         0 SW   [ksoftirqd/1]
   15 root         0 IW   [kworker/1:0]
   16 root         0 IW<  [kworker/1:0H]
  117 root         0 SW   [oom_reaper]
  118 root         0 IW<  [writeback]
  119 root         0 IW<  [crypto]
  121 root         0 IW<  [kblockd]
  123 root         0 IW<  [watchdogd]
  150 root         0 SW   [kswapd0]
  214 root         0 IW<  [pencrypt]
  216 root         0 IW<  [pdecrypt]
  338 root         0 SW   [spi0]
  429 root         0 IW   [kworker/1:1]
  478 root         0 IW<  [ipv6_addrconf]
  490 root         0 SW   [ubi_bgt0d]
  495 root         0 IW<  [kworker/0:1H]
  496 root         0 IW<  [kworker/1:1H]
  499 root         0 IW   [kworker/0:2]
  500 root         0 IW   [kworker/0:3]
  516 root         0 SW   [irq/40-gpio-key]
  517 root         0 SW   [irq/41-gpio-key]
  518 root         0 SW   [irq/42-gpio-key]
  519 root         0 SW   [irq/43-gpio-key]
  654 root         0 SW   [ubifs_bgt0_1]
  857 root      1028 S    /sbin/ubusd
  858 root       720 S    /sbin/askfirst /usr/libexec/login.sh
  952 root         0 IW<  [cfg80211]
  967 root         0 SW   [irq/44-brcmf_pc]
  968 root         0 IW<  [msgbuf_txflow]
 1057 root         0 SW   [irq/45-brcmf_pc]
 1058 root         0 IW<  [msgbuf_txflow]
 1077 root         0 SW   [irq/45-brcmf_pc]
 1078 root         0 IW<  [msgbuf_txflow]
 1618 root      1564 S    /sbin/netifd
 3838 root       924 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p
 4032 root       860 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p
 4042 root      1100 S    -ash
 6594 root         0 IW   [kworker/u4:2]
 6595 root         0 IW   [kworker/u4:3]
 6612 root       924 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p
 6613 root      1100 S    -ash
 6982 root      1288 S    /usr/sbin/odhcpd
 7023 root         0 IW   [kworker/0:0]
 7341 root         0 IW   [kworker/1:2]
 7400 root         0 IW   [kworker/1:3]
 7401 root      1644 S    /usr/sbin/hostapd -s -P /var/run/wifi-phy2.pid -B /v
 7423 root         0 IW   [kworker/1:4]
 7426 root         0 IW   [kworker/u4:0]
 7718 root      1664 S    /usr/sbin/hostapd -s -P /var/run/wifi-phy1.pid -B /v
 7865 root      1072 S    /usr/sbin/pppd nodetach ipparam wan ifname pppoe-wan
 8032 root      1664 S    /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /v
 8202 root       852 S    odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 pppoe
 8300 root       924 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p
 8301 root      1100 R    ps

[reinerotto]

I.e.
mtd -r write /tmp/lede-ar71xx-generic-wzr-hp-ag300h-squashfs-sysupgrade.bin firmware

However, all custom configs will be lost. You might consider to do a backup of config before:
https://openwrt.org/docs/guide-user/troubleshooting/backingup

[geofb]

Thanks.

On the R8000:

mtd -r write /tmp/openwrt-18.06.2-bcm53xx-netgear-r8000-squashfs.chk firmware
Image check failed.

[reinerotto]

A reason, at least. Feel free to confirm the bug report, I filed, with your comments.
Bad idea of sysupgrade to 'forget' such erros, reported from lower level routines.
In such case, no boot should happen, and report of detected error in system log.

[slh]

17.01.x changed sysupgrade to work in a detached mode, which is safer (reverting into an initramfs like mode, with sysupgrade taking over the init job), but comes at the expense that debug messages aren't visible over ssh anymore (leaving just serial as an option to observe its actions) and that the device must reboot afterwards.

[geofb]

Thank you. I'm not seeing a bug report yet - maybe it takes time to filter through - or (more likely) I'm looking in the wrong place.

We know, thanks to slh reply, that sysupgrade is unable to supply the mtd error to the user over ssh or on-device log.

But I wonder what is next step - would it be to wait for an updated openwrt-18.06.2-bcm53xx-netgear-r8000-squashfs.chk which does not fail the Image check?

[reinerotto]

Bug report: I mentioned a link up here. Pls, add your observations. Your problem seems to be another manisfestation of the 'silent' errors during sysupgrade, reported nowhere. Which is really dangerous when doing (unattended) remote sysupgrade. The new behaviour, mentioned by slh, seems to be rather buggy and needs serious overhaul.