RPi 4 sysupgrade [21.02.1 squashfs overlay retained]

Installing and Using OpenWrt
12

As one of my additional tests (please see above post) I tried the ext4 image as you suggested.
The unexpected behavior was indeed not present (it was also not present with the upgrade from squashfs RC4 to 21.02.0 stable).

I initially chose the squashfs image as this wiki page said:

ext4-sdcard.img.gz

  • not optimized for flash memory (journaling speed up flash wear) [emphasis mine]
  • sdcard can be easyly mounted externally for modification
  • desktop linux standard

squashfs-sdcard.img.gz

  • newer images include a hidden f2fs filesystem, which is optimized for flash memory [emphasis mine]
  • compressed
  • needs special mount procedure, to externally modify it
  • all changes are done in an overlay partition

Thanks to now having seen the ext4 image I found out it's also missing the "Perform reset" function and has approximately 28(?)MB less space for user-installed packages. Though if I understand it correctly the latter can be remedied by expanding the partition as you suggested.
So I think I still prefer squashfs due to the reset function.

13

Oh I see, I will redo test 2 and report back later.
Since I reformatted the SD card before I did tests 3 and 4 respectively EDIT: and did not change anything regarding the partitions this could indeed be a difference I failed to account for.

As for restoring the settings backup and installing all my usual packages to obtain a controlled initial state:

  1. I think my tests showed that neither did influence the outcome.
  2. In testing it is simply unrealistic to test every single possible combination of variables.

All the images I used are clean official ones by the way.

1 Like
14

I can now say with confidence that there is a bug in the sysupgrade process of RPi 4B sqashfs installations when updating from 21.02.0 to 21.02.1.

Steps to reproduce:

  1. Flash the official 21.02.0-bcm27xx-bcm2711-rpi-4-squashfs-factory.img onto an SD card.
  2. Login to Luci.
  3. Don't change any of the settings. Don't even set a password.
  4. Install a single package. In my tests I used kmod-mii since it is the first of 3 kmods I need to install to get the usb lan adapter to work.
  5. Upload the official 21.02.1-bcm27xx-bcm2711-rpi-4-squashfs-sysupgrade.img.
  6. Unselect the "Keep settings" checkbox.
  7. Update.
  8. After the reboot log into Luci and under System>Software check the versions of the installed kernel and libwolfssl and whether or not kmod-mii is present or not.
    => Expected result: kernel 5.4.154-1, libwolfssl 4.8.1-stable-4, kmod-mii is not present.
    => Actual result: kernel 5.4.143-1, libwolfssl 4.7.0-stable-2, kmod-mii is present.

Workaround: After the update go to System>Backup/Flash Firmware and klick the "Perform reset" button.

EDIT: The sysupgrade process works as expected when upgrading from squashfs 21.02.0 RC4 to squashfs 21.02.0 stable and from ext4 21.02.0 stable to ext4 21.02.1.
EDIT2: Yes, the SD card was properly zeroed before flashing the 21.02.0 factory image.
EDIT3: added link to kmod-mii.

2 Likes
15

i've just repeated and validated your findings...

specifically;

  • when keep settings is unchecked
  • on official squashfs images (21.02.0>21.02.1)

rootfs_data/overlay filesystem is retained leading to the above symptoms...;

  • note: kernel + mii reported info is purely metadata and does not reflect real packages on /rom / system
  • config files/settings are retained

additionally;

  • same behavior is present upgrading from 21.02.1>21.02.1 (unresolved in 21.02.1)
  • rootfs_data/overlay is also retained when keep settings is selected (in otherwords everyone on 21.02.1(upgraded-to) + squashfs + sdcard/rpi?1 has corrupted/stale opkg metadata)
1

thinking back over the changes on this target... you'll probably find this is a 'c' level issue i.e. block_mount
in which case... it would probably effect more than just the rpi4-squashfs

(if as you state rc3>rc4 or whatever was ok)

(but it seems not many people select 'do not keep settings' + use squashfs)

edit: bug is resolved in master branch

hmmm... they all seem to contain the same fs-tools version so maybe not 'c' after all...

1 Like
16

late to the party but it's seems to confirm what i could no understand in my other post here.. this was with squashfs trying to no retain settings.. maybe this is useful otherwise sorry for the noise..

1 Like
17

I encountered exactly the same under exactly the described circumstances, but I too didn't understand what was actually happening at the time. I backup up settings, flashed clean with 21.02.1 and restored the backup and all was well, but overlayfs was definitely in a wonky state after sysupgrade.

1 Like
18

Shouldn't this buggy upgrade be reported directly to Devs so they can fix source ?
I use EXT4, so I never encountered problem...

1 Like
19

Thank you for validating my findings and taking a closer look than I could.

I don't know if it helps but I noticed that the bug only occurs after user-installing a package.
So if you sysupgrade a completely untouched (or only setting a password and changing nothing else) 21.02.0>21.02.1 Luci shows the expected versions of the kernel and libwolfssl (though I did not test installing any 5.4.154 packages at that point).
Only after I had installed kmod-mii before the sysupgrade did the bug occur at which point I stopped testing.

Is it only an issue with the metadata? After all the first symptoms I noticed were the eth1 device (usb lan adapter) being present but not working and not being able to install the 5.4.154 packages after the 21.02.1 upgrade.

Is "Perform reset" and setting up from there ok or is it not a valid workaround?

1 Like
20

not being a squashfs user... i can't really answer that with confidence but in theory;

  • backup> sysupgrate > reset > restore backup
    or
  • backup > remove sdcard > flash factory > boot > restore backup

should both be ok based on the above theory...
(i'd only excluded restoring backups from the tests as it would have complicated things before we knew more about what was happening)

I guess well find out exactly when developers take a look;

afaik;

  • eth1 would not have worked due to no drivers really being present (or thereabouts) (expected)
  • opkg would not have worked due to the metadata telling opkg that different versions of stuff are installed on the system when they are not (expected)

you can further validate these things prior to reset with;

find /overlay/upper
dmesg | grep mount_root
21

Actually, that's the interesting thing - at least in my case, /overlay was empty after the break, yet my configuration was semi-working...almost like squashfs somehow converted itself into an ext4 deployment through sysupgrade. (Yes, it's as ridiculous as it sounds, but the whole thing is a bit hard to believe until you see it.)

1 Like
22

yeah... I did not actually look directly at /overlay/upper but used opkg list-installed which was reporting the package versions from the old system...

i did look in /lib/modules and there was no modules directory there for the previous kernel (mii) so that seems to line up with what you are saying...

also on the effected systems / borked upgrades... there is a noticable absence of this in dmesg;

[    5.348648] mount_root: rootdisk overlay filesystem has not been formatted yet
23

There is a serious concern with squashfs sysupgrade, but what is so wrong going with EXT4 in order to avoid all of these issues ?

Unless Devs can confirm solution is in the process of being fixed...

EXT4 is not optimal for flash storage, however this is much more of an issue with non-removable storage.

Removable sd is easily replaced with inexpensive fresh storage/install in this case with RPi4 in the event of premature ageing due to EXT4 unoptimized flash wear & tear.

Squashfs is key to a long life for my EA/MR8300's as they don't have removable system flash, but what's the real concern here with respect to removable sd ?

24

bug reported on OP behalf;

https://bugs.openwrt.org/index.php?do=details&task_id=4131

2 Likes
25

Thank you, that's appreciated!

I did that and since you also mentioned /lib/modules in another post I also used find /lib/modules just in case.
This was done at four different points:

  1. After the first boot with a fresh 21.02.0 install.
  2. After installing just the kmod-mii package and another reboot.
  3. After the sysupgrade.
  4. Finally after the reset.

I think kmod-mii was present in /overlay/upper after the sysupgrade but since I don't know not what to look for please check the outputs for yourself:

21.02.0 fresh install after first boot

21.02.0 fresh install after first boot:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/boot
root@OpenWrt:~#

root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.143
/lib/modules/5.4.143/brcmfmac.ko
/lib/modules/5.4.143/brcmutil.ko
/lib/modules/5.4.143/cfg80211.ko
/lib/modules/5.4.143/compat.ko
/lib/modules/5.4.143/crc-ccitt.ko
/lib/modules/5.4.143/evdev.ko
/lib/modules/5.4.143/fat.ko
/lib/modules/5.4.143/hid-generic.ko
/lib/modules/5.4.143/hid.ko
/lib/modules/5.4.143/ip6_tables.ko
/lib/modules/5.4.143/ip6t_REJECT.ko
/lib/modules/5.4.143/ip6table_filter.ko
/lib/modules/5.4.143/ip6table_mangle.ko
/lib/modules/5.4.143/ip_tables.ko
/lib/modules/5.4.143/ipt_REJECT.ko
/lib/modules/5.4.143/iptable_filter.ko
/lib/modules/5.4.143/iptable_mangle.ko
/lib/modules/5.4.143/iptable_nat.ko
/lib/modules/5.4.143/nf_conntrack.ko
/lib/modules/5.4.143/nf_defrag_ipv4.ko
/lib/modules/5.4.143/nf_defrag_ipv6.ko
/lib/modules/5.4.143/nf_flow_table.ko
/lib/modules/5.4.143/nf_flow_table_hw.ko
/lib/modules/5.4.143/nf_log_common.ko
/lib/modules/5.4.143/nf_log_ipv4.ko
/lib/modules/5.4.143/nf_log_ipv6.ko
/lib/modules/5.4.143/nf_nat.ko
/lib/modules/5.4.143/nf_reject_ipv4.ko
/lib/modules/5.4.143/nf_reject_ipv6.ko
/lib/modules/5.4.143/nls_cp437.ko
/lib/modules/5.4.143/nls_iso8859-1.ko
/lib/modules/5.4.143/nls_utf8.ko
/lib/modules/5.4.143/ppp_async.ko
/lib/modules/5.4.143/ppp_generic.ko
/lib/modules/5.4.143/pppoe.ko
/lib/modules/5.4.143/pppox.ko
/lib/modules/5.4.143/slhc.ko
/lib/modules/5.4.143/snd-bcm2835.ko
/lib/modules/5.4.143/snd-compress.ko
/lib/modules/5.4.143/snd-hwdep.ko
/lib/modules/5.4.143/snd-mixer-oss.ko
/lib/modules/5.4.143/snd-pcm-dmaengine.ko
/lib/modules/5.4.143/snd-pcm-oss.ko
/lib/modules/5.4.143/snd-pcm.ko
/lib/modules/5.4.143/snd-rawmidi.ko
/lib/modules/5.4.143/snd-seq-device.ko
/lib/modules/5.4.143/snd-timer.ko
/lib/modules/5.4.143/snd.ko
/lib/modules/5.4.143/soundcore.ko
/lib/modules/5.4.143/usbhid.ko
/lib/modules/5.4.143/vfat.ko
/lib/modules/5.4.143/x_tables.ko
/lib/modules/5.4.143/xt_CT.ko
/lib/modules/5.4.143/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.143/xt_LOG.ko
/lib/modules/5.4.143/xt_MASQUERADE.ko
/lib/modules/5.4.143/xt_REDIRECT.ko
/lib/modules/5.4.143/xt_TCPMSS.ko
/lib/modules/5.4.143/xt_comment.ko
/lib/modules/5.4.143/xt_conntrack.ko
/lib/modules/5.4.143/xt_limit.ko
/lib/modules/5.4.143/xt_mac.ko
/lib/modules/5.4.143/xt_mark.ko
/lib/modules/5.4.143/xt_multiport.ko
/lib/modules/5.4.143/xt_nat.ko
/lib/modules/5.4.143/xt_state.ko
/lib/modules/5.4.143/xt_tcpudp.ko
/lib/modules/5.4.143/xt_time.ko
root@OpenWrt:~#

root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    2.218708] bcmgenet: Skipping UMAC reset
[    2.224079] bcmgenet fd580000.ethernet: configuring instance for external RGMII
[    2.231578] bcmgenet fd580000.ethernet eth0: Link is Down
[    6.320916] mount_root: rootdisk overlay filesystem has not been formatted yet
[    7.363901] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    7.372151] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.683247] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    7.691257] mount_root: overlay filesystem has not been fully initialized yet
[    7.698930] mount_root: switching to ext4 overlay
[    7.707330] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    7.744351] urandom-seed: Seed file not found (/etc/urandom.seed)
[    7.805805] bcmgenet fd580000.ethernet eth0: Link is Down
root@OpenWrt:~#
21.02.0 fresh install after kmod-mii installation

21.02.0 fresh install after kmod-mii installation:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/modules.d
/overlay/upper/etc/modules.d/15-mii
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/modules-boot.d
/overlay/upper/etc/modules-boot.d/15-mii
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/usr
/overlay/upper/usr/lib
/overlay/upper/usr/lib/opkg
/overlay/upper/usr/lib/opkg/info
/overlay/upper/usr/lib/opkg/info/kmod-mii.control
/overlay/upper/usr/lib/opkg/info/kmod-mii.postinst
/overlay/upper/usr/lib/opkg/info/kmod-mii.list
/overlay/upper/usr/lib/opkg/info/kmod-mii.prerm
/overlay/upper/usr/lib/opkg/status
/overlay/upper/boot
root@OpenWrt:~#

root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.143
/lib/modules/5.4.143/brcmfmac.ko
/lib/modules/5.4.143/brcmutil.ko
/lib/modules/5.4.143/cfg80211.ko
/lib/modules/5.4.143/compat.ko
/lib/modules/5.4.143/crc-ccitt.ko
/lib/modules/5.4.143/evdev.ko
/lib/modules/5.4.143/fat.ko
/lib/modules/5.4.143/hid-generic.ko
/lib/modules/5.4.143/hid.ko
/lib/modules/5.4.143/ip6_tables.ko
/lib/modules/5.4.143/ip6t_REJECT.ko
/lib/modules/5.4.143/ip6table_filter.ko
/lib/modules/5.4.143/ip6table_mangle.ko
/lib/modules/5.4.143/ip_tables.ko
/lib/modules/5.4.143/ipt_REJECT.ko
/lib/modules/5.4.143/iptable_filter.ko
/lib/modules/5.4.143/iptable_mangle.ko
/lib/modules/5.4.143/iptable_nat.ko
/lib/modules/5.4.143/nf_conntrack.ko
/lib/modules/5.4.143/nf_defrag_ipv4.ko
/lib/modules/5.4.143/nf_defrag_ipv6.ko
/lib/modules/5.4.143/nf_flow_table.ko
/lib/modules/5.4.143/nf_flow_table_hw.ko
/lib/modules/5.4.143/nf_log_common.ko
/lib/modules/5.4.143/nf_log_ipv4.ko
/lib/modules/5.4.143/nf_log_ipv6.ko
/lib/modules/5.4.143/nf_nat.ko
/lib/modules/5.4.143/nf_reject_ipv4.ko
/lib/modules/5.4.143/nf_reject_ipv6.ko
/lib/modules/5.4.143/nls_cp437.ko
/lib/modules/5.4.143/nls_iso8859-1.ko
/lib/modules/5.4.143/nls_utf8.ko
/lib/modules/5.4.143/ppp_async.ko
/lib/modules/5.4.143/ppp_generic.ko
/lib/modules/5.4.143/pppoe.ko
/lib/modules/5.4.143/pppox.ko
/lib/modules/5.4.143/slhc.ko
/lib/modules/5.4.143/snd-bcm2835.ko
/lib/modules/5.4.143/snd-compress.ko
/lib/modules/5.4.143/snd-hwdep.ko
/lib/modules/5.4.143/snd-mixer-oss.ko
/lib/modules/5.4.143/snd-pcm-dmaengine.ko
/lib/modules/5.4.143/snd-pcm-oss.ko
/lib/modules/5.4.143/snd-pcm.ko
/lib/modules/5.4.143/snd-rawmidi.ko
/lib/modules/5.4.143/snd-seq-device.ko
/lib/modules/5.4.143/snd-timer.ko
/lib/modules/5.4.143/snd.ko
/lib/modules/5.4.143/soundcore.ko
/lib/modules/5.4.143/usbhid.ko
/lib/modules/5.4.143/vfat.ko
/lib/modules/5.4.143/x_tables.ko
/lib/modules/5.4.143/xt_CT.ko
/lib/modules/5.4.143/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.143/xt_LOG.ko
/lib/modules/5.4.143/xt_MASQUERADE.ko
/lib/modules/5.4.143/xt_REDIRECT.ko
/lib/modules/5.4.143/xt_TCPMSS.ko
/lib/modules/5.4.143/xt_comment.ko
/lib/modules/5.4.143/xt_conntrack.ko
/lib/modules/5.4.143/xt_limit.ko
/lib/modules/5.4.143/xt_mac.ko
/lib/modules/5.4.143/xt_mark.ko
/lib/modules/5.4.143/xt_multiport.ko
/lib/modules/5.4.143/xt_nat.ko
/lib/modules/5.4.143/xt_state.ko
/lib/modules/5.4.143/xt_tcpudp.ko
/lib/modules/5.4.143/xt_time.ko
root@OpenWrt:~#

root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    2.202955] bcmgenet fd580000.ethernet eth0: Link is Down
[    6.414169] EXT4-fs (loop0): recovery complete
[    6.420540] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.428615] mount_root: switching to ext4 overlay
[    6.434561] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.474368] urandom-seed: Seeding with /etc/urandom.seed
[    6.554075] procd: - early -
root@OpenWrt:~#
21.02.1 after sysupgrade

21.02.1 after sysupgrade:

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/modules.d
/overlay/upper/etc/modules.d/15-mii
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/modules-boot.d
/overlay/upper/etc/modules-boot.d/15-mii
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/usr
/overlay/upper/usr/lib
/overlay/upper/usr/lib/opkg
/overlay/upper/usr/lib/opkg/info
/overlay/upper/usr/lib/opkg/info/kmod-mii.control
/overlay/upper/usr/lib/opkg/info/kmod-mii.postinst
/overlay/upper/usr/lib/opkg/info/kmod-mii.list
/overlay/upper/usr/lib/opkg/info/kmod-mii.prerm
/overlay/upper/usr/lib/opkg/status
/overlay/upper/boot
root@OpenWrt:~#

root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.154
/lib/modules/5.4.154/brcmfmac.ko
/lib/modules/5.4.154/brcmutil.ko
/lib/modules/5.4.154/cfg80211.ko
/lib/modules/5.4.154/compat.ko
/lib/modules/5.4.154/crc-ccitt.ko
/lib/modules/5.4.154/evdev.ko
/lib/modules/5.4.154/fat.ko
/lib/modules/5.4.154/hid-generic.ko
/lib/modules/5.4.154/hid.ko
/lib/modules/5.4.154/ip6_tables.ko
/lib/modules/5.4.154/ip6t_REJECT.ko
/lib/modules/5.4.154/ip6table_filter.ko
/lib/modules/5.4.154/ip6table_mangle.ko
/lib/modules/5.4.154/ip_tables.ko
/lib/modules/5.4.154/ipt_REJECT.ko
/lib/modules/5.4.154/iptable_filter.ko
/lib/modules/5.4.154/iptable_mangle.ko
/lib/modules/5.4.154/iptable_nat.ko
/lib/modules/5.4.154/nf_conntrack.ko
/lib/modules/5.4.154/nf_defrag_ipv4.ko
/lib/modules/5.4.154/nf_defrag_ipv6.ko
/lib/modules/5.4.154/nf_flow_table.ko
/lib/modules/5.4.154/nf_flow_table_hw.ko
/lib/modules/5.4.154/nf_log_common.ko
/lib/modules/5.4.154/nf_log_ipv4.ko
/lib/modules/5.4.154/nf_log_ipv6.ko
/lib/modules/5.4.154/nf_nat.ko
/lib/modules/5.4.154/nf_reject_ipv4.ko
/lib/modules/5.4.154/nf_reject_ipv6.ko
/lib/modules/5.4.154/nls_cp437.ko
/lib/modules/5.4.154/nls_iso8859-1.ko
/lib/modules/5.4.154/nls_utf8.ko
/lib/modules/5.4.154/ppp_async.ko
/lib/modules/5.4.154/ppp_generic.ko
/lib/modules/5.4.154/pppoe.ko
/lib/modules/5.4.154/pppox.ko
/lib/modules/5.4.154/slhc.ko
/lib/modules/5.4.154/snd-bcm2835.ko
/lib/modules/5.4.154/snd-compress.ko
/lib/modules/5.4.154/snd-hwdep.ko
/lib/modules/5.4.154/snd-mixer-oss.ko
/lib/modules/5.4.154/snd-pcm-dmaengine.ko
/lib/modules/5.4.154/snd-pcm-oss.ko
/lib/modules/5.4.154/snd-pcm.ko
/lib/modules/5.4.154/snd-rawmidi.ko
/lib/modules/5.4.154/snd-seq-device.ko
/lib/modules/5.4.154/snd-timer.ko
/lib/modules/5.4.154/snd.ko
/lib/modules/5.4.154/soundcore.ko
/lib/modules/5.4.154/usbhid.ko
/lib/modules/5.4.154/vfat.ko
/lib/modules/5.4.154/x_tables.ko
/lib/modules/5.4.154/xt_CT.ko
/lib/modules/5.4.154/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.154/xt_LOG.ko
/lib/modules/5.4.154/xt_MASQUERADE.ko
/lib/modules/5.4.154/xt_REDIRECT.ko
/lib/modules/5.4.154/xt_TCPMSS.ko
/lib/modules/5.4.154/xt_comment.ko
/lib/modules/5.4.154/xt_conntrack.ko
/lib/modules/5.4.154/xt_limit.ko
/lib/modules/5.4.154/xt_mac.ko
/lib/modules/5.4.154/xt_mark.ko
/lib/modules/5.4.154/xt_multiport.ko
/lib/modules/5.4.154/xt_nat.ko
/lib/modules/5.4.154/xt_state.ko
/lib/modules/5.4.154/xt_tcpudp.ko
/lib/modules/5.4.154/xt_time.ko
root@OpenWrt:~#

root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    6.307982] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    6.316218] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    6.326167] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.336391] mount_root: switching to ext4 overlay
[    6.347177] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.388946] urandom-seed: Seeding with /etc/urandom.seed
[    6.453887] bcmgenet fd580000.ethernet eth0: Link is Down
root@OpenWrt:~#
21.02.1 after reset

21.02.1 after reset

root@OpenWrt:~# find /overlay/upper
/overlay/upper
/overlay/upper/etc
/overlay/upper/etc/urandom.seed
/overlay/upper/etc/board.json
/overlay/upper/etc/uhttpd.crt
/overlay/upper/etc/dropbear
/overlay/upper/etc/dropbear/dropbear_ed25519_host_key
/overlay/upper/etc/dropbear/dropbear_rsa_host_key
/overlay/upper/etc/uci-defaults
/overlay/upper/etc/uci-defaults/50-dnsmasq-migrate-resolv-conf-auto.sh
/overlay/upper/etc/uci-defaults/50_luci-mod-admin-full
/overlay/upper/etc/uci-defaults/10_migrate-shadow
/overlay/upper/etc/uci-defaults/20_migrate-feeds
/overlay/upper/etc/uci-defaults/15_odhcpd
/overlay/upper/etc/uci-defaults/12_network-generate-ula
/overlay/upper/etc/uci-defaults/14_migrate-dhcp-release
/overlay/upper/etc/uci-defaults/30_luci-theme-bootstrap
/overlay/upper/etc/uci-defaults/50-migrate-rpcd-ubus-sock.sh
/overlay/upper/etc/uci-defaults/13_fix-group-user
/overlay/upper/etc/uci-defaults/00_uhttpd_ubus
/overlay/upper/etc/uhttpd.key
/overlay/upper/etc/config
/overlay/upper/etc/config/network
/overlay/upper/etc/config/luci
/overlay/upper/etc/config/uhttpd
/overlay/upper/etc/config/ucitrack
/overlay/upper/etc/config/wireless
/overlay/upper/etc/config/firewall
/overlay/upper/etc/config/dropbear
/overlay/upper/etc/config/system
/overlay/upper/etc/config/dhcp
/overlay/upper/etc/config/rpcd
/overlay/upper/boot
root@OpenWrt:~#

root@OpenWrt:~# find /lib/modules
/lib/modules
/lib/modules/5.4.154
/lib/modules/5.4.154/brcmfmac.ko
/lib/modules/5.4.154/brcmutil.ko
/lib/modules/5.4.154/cfg80211.ko
/lib/modules/5.4.154/compat.ko
/lib/modules/5.4.154/crc-ccitt.ko
/lib/modules/5.4.154/evdev.ko
/lib/modules/5.4.154/fat.ko
/lib/modules/5.4.154/hid-generic.ko
/lib/modules/5.4.154/hid.ko
/lib/modules/5.4.154/ip6_tables.ko
/lib/modules/5.4.154/ip6t_REJECT.ko
/lib/modules/5.4.154/ip6table_filter.ko
/lib/modules/5.4.154/ip6table_mangle.ko
/lib/modules/5.4.154/ip_tables.ko
/lib/modules/5.4.154/ipt_REJECT.ko
/lib/modules/5.4.154/iptable_filter.ko
/lib/modules/5.4.154/iptable_mangle.ko
/lib/modules/5.4.154/iptable_nat.ko
/lib/modules/5.4.154/nf_conntrack.ko
/lib/modules/5.4.154/nf_defrag_ipv4.ko
/lib/modules/5.4.154/nf_defrag_ipv6.ko
/lib/modules/5.4.154/nf_flow_table.ko
/lib/modules/5.4.154/nf_flow_table_hw.ko
/lib/modules/5.4.154/nf_log_common.ko
/lib/modules/5.4.154/nf_log_ipv4.ko
/lib/modules/5.4.154/nf_log_ipv6.ko
/lib/modules/5.4.154/nf_nat.ko
/lib/modules/5.4.154/nf_reject_ipv4.ko
/lib/modules/5.4.154/nf_reject_ipv6.ko
/lib/modules/5.4.154/nls_cp437.ko
/lib/modules/5.4.154/nls_iso8859-1.ko
/lib/modules/5.4.154/nls_utf8.ko
/lib/modules/5.4.154/ppp_async.ko
/lib/modules/5.4.154/ppp_generic.ko
/lib/modules/5.4.154/pppoe.ko
/lib/modules/5.4.154/pppox.ko
/lib/modules/5.4.154/slhc.ko
/lib/modules/5.4.154/snd-bcm2835.ko
/lib/modules/5.4.154/snd-compress.ko
/lib/modules/5.4.154/snd-hwdep.ko
/lib/modules/5.4.154/snd-mixer-oss.ko
/lib/modules/5.4.154/snd-pcm-dmaengine.ko
/lib/modules/5.4.154/snd-pcm-oss.ko
/lib/modules/5.4.154/snd-pcm.ko
/lib/modules/5.4.154/snd-rawmidi.ko
/lib/modules/5.4.154/snd-seq-device.ko
/lib/modules/5.4.154/snd-timer.ko
/lib/modules/5.4.154/snd.ko
/lib/modules/5.4.154/soundcore.ko
/lib/modules/5.4.154/usbhid.ko
/lib/modules/5.4.154/vfat.ko
/lib/modules/5.4.154/x_tables.ko
/lib/modules/5.4.154/xt_CT.ko
/lib/modules/5.4.154/xt_FLOWOFFLOAD.ko
/lib/modules/5.4.154/xt_LOG.ko
/lib/modules/5.4.154/xt_MASQUERADE.ko
/lib/modules/5.4.154/xt_REDIRECT.ko
/lib/modules/5.4.154/xt_TCPMSS.ko
/lib/modules/5.4.154/xt_comment.ko
/lib/modules/5.4.154/xt_conntrack.ko
/lib/modules/5.4.154/xt_limit.ko
/lib/modules/5.4.154/xt_mac.ko
/lib/modules/5.4.154/xt_mark.ko
/lib/modules/5.4.154/xt_multiport.ko
/lib/modules/5.4.154/xt_nat.ko
/lib/modules/5.4.154/xt_state.ko
/lib/modules/5.4.154/xt_tcpudp.ko
/lib/modules/5.4.154/xt_time.ko
root@OpenWrt:~#

root@OpenWrt:~# dmesg | grep -i "mount_root" -C 3
[    6.378193] EXT4-fs (loop0): 6 orphan inodes deleted
[    6.383186] EXT4-fs (loop0): recovery complete
[    6.404678] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
[    6.413026] mount_root: overlay filesystem has not been fully initialized yet
[    6.420421] mount_root: switching to ext4 overlay
[    6.426334] overlayfs: "xino" feature enabled using 32 upper inode bits.
[    6.461169] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    6.472320] urandom-seed: Seed file not found (/etc/urandom.seed)
root@OpenWrt:~#
1 Like
26

yes... in output (3) after not keeping settings (or even with keeping settings) the above should not be present after a sysupgrade...

the stale opkg/status file is the one that's wreaking havoc with opkg install and kernel version mistmatch...

when you 'reset' ... it's just wiping all this...

27

Short update:
Not sure whether the issue stems from this bug or something I did here but at some point the system started losing setting changes, additionally installed packages vanished and uninstalled packages reappeared after a reboot.
So I flashed a stock ext4 21.02.1 factory image on the spare SD card and won't do fancy things anymore (for now).

28

my sincere apologies... it seems that bug tracking system was abandoned and there is conflicting information about whether existing reports will be migrated so someone may have to re-lodge this bug if you want it fixed

A migration of existing issues is not planed

1 Like
29

Hi,

you helped a lot in verifying the issue, reported it in the old system in my stead and now three months later notified me about the move to GitHub I would otherwise probably have missed.
So you went above and beyond and there is absolutely nothing for you to apologize for!

If that's ok with you I plan on opening an issue at GitHub in which I'd just link to the issue you opened and copy your description and the comments of the bug.

30

sounds good... thankyou very much

31

Done (hopefully done correctly, first time I used GitHub).

2 Likes