WS-AP3915i: Settings lost upgrading 23.05.0 -> 23.05.2

I have several WS-AP3915i that I just upgraded from 23.05.0 to 23.05.2. Unfortunately, all devices completely lost their settings during the upgrade.

On the last one (my spare), I managed to capture a serial log.

Upgrading
Mon Oct  9 21:51:58 UTC 2023 upgrade: Sending TERM to remaining processes ...
Mon Oct  9 21:52:02 UTC 2023 upgrade: Sending KILL to remaining processes ...
[  274.771256] stage2 (3157): drop_caches: 3
Mon Oct  9 21:52:09 UTC 2023 upgrade: Switching to ramdisk...
mount: mounting /dev/mtdblock10 on /overlay failed: Resource busy
[  279.671965] VFS: Busy inodes after unmount of jffs2. Self-destruct in 5 seconds.  Have a nice day...
Mon Oct  9 21:52:13 UTC 2023 upgrade: Performing system upgrade...
[  279.779435] do_stage2 (3157): drop_caches: 3
Unlocking firmware ...

Writing from <stdin> to firmware ...     
Appending jffs2 data from /tmp/sysupgrade.tgz to firmware..                                                                                                         
Mon Oct  9 21:53:08 UTC 2023 upgrade: Upgrade completed
Mon Oct  9 21:53:09 UTC 2023 upgrade: Rebooting system...
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy
[
Boot log - all settings gone
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.15.137 (builder@buildhost) (arm-openwrt-linux-muslgnueabi-gcc (OpenWrt GCC 12.3.0 r23630-842932a63d) 12.3.0, GNU ld (GNU Binutils) 2.40.0) #0 SMP Tue Nov 14 13:38:11 2023
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Extreme Networks WS-AP3915i
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x0000000087dfffff]
[    0.000000]   node   0: [mem 0x0000000087e00000-0x0000000087ffffff]
[    0.000000]   node   0: [mem 0x0000000088000000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009fffffff]
[    0.000000] percpu: Embedded 12 pages/cpu s16588 r8192 d24372 u49152
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129920
[    0.000000] Kernel command line: console=ttyMSM0,115200n81 ubi.mtd=0  panic=30 nohlt ro BOOT_KERNEL=primary BOOT_BOOTROM="U-Boot 2012.07.19-r00020.1 (Jul 17 2017 - 17:24:13)"
[    0.000000] Unknown kernel command line parameters "BOOT_KERNEL=primary BOOT_BOOTROM=U-Boot 2012.07.19-r00020.1 (Jul 17 2017 - 17:24:13)", will be passed to user space.
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 506856K/524288K available (6746K kernel code, 610K rwdata, 1020K rodata, 1024K init, 247K bss, 17432K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] arch_timer: cp15 timer(s) running at 48.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb11fd3bfb, max_idle_ns: 440795203732 ns
[    0.000003] sched_clock: 56 bits at 48MHz, resolution 20ns, wraps every 4398046511096ns
[    0.000026] Switching to timer-based delay loop, resolution 20ns
[    0.000305] Calibrating delay loop (skipped), value calculated using timer frequency.. 96.00 BogoMIPS (lpj=480000)
[    0.000333] CPU: Testing write buffer coherency: ok
[    0.000391] pid_max: default: 32768 minimum: 301
[    0.001246] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.001276] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.003291] qcom_scm: convention: smc legacy
[    0.004497] Setting up static identity map for 0x80300000 - 0x8030003c
[    0.004677] rcu: Hierarchical SRCU implementation.
[    0.005284] smp: Bringing up secondary CPUs ...
[    0.008742] smp: Brought up 1 node, 4 CPUs
[    0.008782] SMP: Total of 4 processors activated (384.00 BogoMIPS).
[    0.008797] CPU: All CPU(s) started in SVC mode.
[    0.014836] VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
[    0.015002] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.015036] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.015408] pinctrl core: initialized pinctrl subsystem
[    0.017475] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.017874] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.019253] thermal_sys: Registered thermal governor 'step_wise'
[    0.019679] cpuidle: using governor ladder
[    0.019741] cpuidle: using governor menu
[    0.045842] cryptd: max_cpu_qlen set to 1000
[    0.050595] usbcore: registered new interface driver usbfs
[    0.050709] usbcore: registered new interface driver hub
[    0.050775] usbcore: registered new device driver usb
[    0.050858] pps_core: LinuxPPS API ver. 1 registered
[    0.050872] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.050904] PTP clock support registered
[    0.053119] clocksource: Switched to clocksource arch_sys_counter
[    0.054437] NET: Registered PF_INET protocol family
[    0.054715] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.055849] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    0.055904] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.055928] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.055996] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.056092] TCP: Hash tables configured (established 4096 bind 4096)
[    0.056239] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.056312] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.056756] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.056831] PCI: CLS 0 bytes, default 64
[    0.058937] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[    0.064960] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.064993] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.196056] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    0.199556] bam-dma-engine 8e04000.dma: num-channels unspecified in dt
[    0.199587] bam-dma-engine 8e04000.dma: num-ees unspecified in dt
[    0.201122] tcsr 1949000.tcsr: setting wifi_glb_cfg = 41000000
[    0.201235] tcsr 1953000.ess_tcsr: setting ess interface select = 0
[    0.201330] tcsr 1957000.tcsr: setting wifi_noc_memtype_m0_m2 = 2222222
[    0.201673] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.203871] msm_serial 78af000.serial: msm_serial: detected port #0
[    0.203928] msm_serial 78af000.serial: uartclk = 1843200
[    0.203987] 78af000.serial: ttyMSM0 at MMIO 0x78af000 (irq = 33, base_baud = 115200) is a MSM
[    0.204030] msm_serial: console setup on port #0
[    0.770367] printk: console [ttyMSM0] enabled
[    0.775504] msm_serial: driver initialized
[    0.785356] loop: module loaded
[    0.786459] spi_qup 78b5000.spi: IN:block:16, fifo:64, OUT:block:16, fifo:64
[    0.789144] spi-nor spi0.0: mx25l25635e (32768 Kbytes)
[    0.794723] 9 fixed-partitions partitions found on MTD device spi0.0
[    0.799563] Creating 9 MTD partitions on "spi0.0":
[    0.806089] 0x0000000e0000-0x0000000f0000 : "CFG1"
[    0.811313] 0x0000000f0000-0x000000160000 : "BootBAK"
[    0.816072] 0x000000160000-0x000000170000 : "WINGCFG1"
[    0.821206] 0x000000170000-0x000000180000 : "ART"
[    0.826256] 0x000000180000-0x0000001f0000 : "BootPRI"
[    0.830943] 0x0000001f0000-0x000000200000 : "WINGCFG2"
[    0.836055] 0x000000200000-0x000000280000 : "FS"
[    0.840964] 0x000000280000-0x000001fe0000 : "firmware"
[    0.846113] 2 fit-fw partitions found on MTD device firmware
[    0.850167] 0x000000280000-0x000000690000 : "kernel"
[    0.856618] 0x000000684a6c-0x000001fe0000 : "rootfs"
[    0.860926] mtd: partition "rootfs" doesn't start on an erase/write block boundary -- force read-only
[    0.866423] mtd: setting mtd9 (rootfs) as root device
[    0.875254] 1 squashfs-split partitions found on MTD device rootfs
[    0.880032] 0x000000a50000-0x000001fe0000 : "rootfs_data"
[    0.886828] 0x000001fe0000-0x000001ff0000 : "CFG2"
[    0.947827] i2c_dev: i2c /dev entries driver
[    0.949808] sdhci: Secure Digital Host Controller Interface driver
[    0.951174] sdhci: Copyright(c) Pierre Ossman
[    0.957240] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.965750] NET: Registered PF_INET6 protocol family
[    0.968972] Segment Routing with IPv6
[    0.972329] In-situ OAM (IOAM) with IPv6
[    0.975979] NET: Registered PF_PACKET protocol family
[    0.979883] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    0.985221] 8021q: 802.1Q VLAN Support v1.8
[    0.997962] Registering SWP/SWPB emulation handler
[    1.015397] qca8k-ipq4019 c000000.switch: configuring for fixed/internal link mode
[    1.015736] qca8k-ipq4019 c000000.switch: Link is Up - 1Gbps/Full - flow control rx/tx
[    1.107909] qca8k-ipq4019 c000000.switch lan (uninitialized): PHY [90000.mdio-1:04] driver [Qualcomm QCA8072] (irq=POLL)
[    1.109103] DSA: tree 0 setup
[    1.118605] ubi0: attaching mtd0
[    1.120727] ubi0: MTD device 0 is write-protected, attach in read-only mode
[    1.124692] ubi0: scanning is finished
[    1.130711] ubi0 error: 0xc06ffbf0: the layout volume was not found
[    1.134679] ubi0 error: 0xc0703120: failed to attach mtd0, error -22
[  �[    1.158550] VFS: Mounted root (squashfs filesystem) readonly on device 31:9.
[    1.160581] Freeing unused kernel image (initmem) memory: 1024K
[    1.193339] Run /sbin/init as init process
[    1.753649] init: Console is alive
[    1.754053] init: - watchdog -
[    3.014142] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.187661] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.188958] init: - preinit -
[    4.399511] random: jshn: uninitialized urandom read (4 bytes read)
[    4.444758] random: jshn: uninitialized urandom read (4 bytes read)
[    4.466353] random: jshn: uninitialized urandom read (4 bytes read)
[    4.649172] ipqess-edma c080000.ethernet eth0: configuring for fixed/internal link mode
[    4.649916] qca8k-ipq4019 c000000.switch lan: configuring for phy/psgmii link mode
[    4.656369] ipqess-edma c080000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    5.233138] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[    7.412758] mount_root: no usable overlay filesystem found, using tmpfs overlay
[    7.416346] urandom-seed: Seed file not found (/etc/urandom.seed)
[    7.647492] procd: - early -
[    7.647703] procd: - watchdog -
[    8.295519] procd: - watchdog -
[    8.333156] random: crng init done
[    8.333198] random: 7 urandom warning(s) missed due to ratelimiting
[    8.450106] procd: - ubus -
[    8.635487] procd: - init -
Please press Enter to activate this console.
[    9.406509] kmodloader: loading kernel modules from /etc/modules.d/*
[    9.653846] Loading modules backported from Linux version v6.1.24-0-g0102425ac76b
[    9.653902] Backport generated by backports.git v5.15.92-1-44-gd6ea70fafd36
[    9.782450] PPP generic driver version 2.4.2
[    9.784820] NET: Registered PF_PPPOX protocol family
[   10.316081] urngd: v1.0.2 started.
[   11.247304] ath10k_ahb a000000.wifi: qca4019 hw1.0 target 0x01000000 chip_id 0x003900ff sub 0000:0000
[   11.247375] ath10k_ahb a000000.wifi: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[   11.259279] ath10k_ahb a000000.wifi: firmware ver 10.4b-ct-4019-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 6b2b5c5b
[   11.310970] ath10k_ahb a000000.wifi: Loading BDF type 0
[   11.431326] ath10k_ahb a000000.wifi: board_file api 2 bmi_id 0:20 crc32 d140cd7d
[   12.718560] ath10k_ahb a000000.wifi: 10.4 wmi init: vdevs: 16  peers: 48  tid: 96
[   12.718642] ath10k_ahb a000000.wifi: msdu-desc: 2500  skid: 32
[   12.765274] ath10k_ahb a000000.wifi: wmi print 'P 48/48 V 16 K 144 PH 176 T 186  msdu-desc: 2500  sw-crypt: 0 ct-sta: 0'
[   12.766223] ath10k_ahb a000000.wifi: wmi print 'free: 53252 iram: 13432 sram: 35752'
[   12.903700] ath10k_ahb a000000.wifi: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[   14.474570] ath10k_ahb a800000.wifi: qca4019 hw1.0 target 0x01000000 chip_id 0x003900ff sub 0000:0000
[   14.474658] ath10k_ahb a800000.wifi: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[   14.486600] ath10k_ahb a800000.wifi: firmware ver 10.4b-ct-4019-fW-13-5ae337bb1 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 6b2b5c5b
[   14.538317] ath10k_ahb a800000.wifi: Loading BDF type 0
[   14.550624] ath10k_ahb a800000.wifi: board_file api 2 bmi_id 0:21 crc32 d140cd7d
[   15.839342] ath10k_ahb a800000.wifi: 10.4 wmi init: vdevs: 16  peers: 48  tid: 96
[   15.839414] ath10k_ahb a800000.wifi: msdu-desc: 2500  skid: 32
[   15.886252] ath10k_ahb a800000.wifi: wmi print 'P 48/48 V 16 K 144 PH 176 T 186  msdu-desc: 2500  sw-crypt: 0 ct-sta: 0'
[   15.887212] ath10k_ahb a800000.wifi: wmi print 'free: 53252 iram: 13432 sram: 35752'
[   16.023028] ath10k_ahb a800000.wifi: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[   16.149868] kmodloader: done loading kernel modules from /etc/modules.d/*
[   38.578103] ipqess-edma c080000.ethernet eth0: Link is Down
[   38.583056] ipqess-edma c080000.ethernet eth0: configuring for fixed/internal link mode
[   38.583408] ipqess-edma c080000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   38.588186] qca8k-ipq4019 c000000.switch lan: configuring for phy/psgmii link mode
[   38.600009] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   38.607721] br-lan: port 1(lan) entered blocking state
[   38.612651] br-lan: port 1(lan) entered disabled state
[   38.618479] device lan entered promiscuous mode
[   38.622966] device eth0 entered promiscuous mode
[   38.830602] jffs2_scan_eraseblock(): End of filesystem marker found at 0x20000
[   38.830742] jffs2_build_filesystem(): unlocking the mtd device... 
[   38.830854] done.
[   38.842912] jffs2_build_filesystem(): erasing all blocks after the end marker... 
[   41.764274] qca8k-ipq4019 c000000.switch lan: Link is Up - 1Gbps/Full - flow control rx/tx
[   41.770801] br-lan: port 1(lan) entered blocking state
[   41.778951] br-lan: port 1(lan) entered forwarding state
[   41.784303] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[  136.517547] jffs2: notice: (1905) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[  136.910538] overlayfs: upper fs does not support tmpfile.

Taking a short look at the datasheet of the flash chip I found out that the supported block size of that chip is 4kB, 32kB or 64kB. However, in OpenWrt's image Makefile the JFFS2 padding is generated for a flash chip with 128kB blocksize, see

This results in

[    7.412758] mount_root: no usable overlay filesystem found, using tmpfs overlay

as well as no configuration being restored after a successful sysupgrade.

Most likely the (wrong) BLOCKSIZE was added as a hack to enable flashing OpenWrt when coming from the vendor firmware with precisely that intention: Not keeping the configuration...

The correct solution would be to offer a dedicated factory image to be used when the device is still running vendor firmware, and either remove the BLOCKSIZE statement for the sysupgrade image entirely or state the correct BLOCKSIZE (64k),

This is fixed in openwrt snapshot and 24.10.0-rc5.

See https://github.com/openwrt/openwrt/pull/17305