Powerbeam M5 XW Configuration Loss after reboot

Hi, I tried upgrading to the latest snapshot in my Powerbeam M5 with image builder and after configuring it, the configuration changes were lost after rebooting, I have tried using the master snapshot without luci the same issue is reproducible, and this issue isn't reproducible with 5.10 kernel.

Logs filtered to jffs and overlay:

[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.299228] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    9.050587] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[   46.395338] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   46.402362] jffs2_build_filesystem(): unlocking the mtd device...
[   46.422104] jffs2_build_filesystem(): erasing all blocks after the end marker...
[   48.726475] jffs2: Newly-erased block contained word 0xdeadc0de at offset 0x00000000
[   48.743667] jffs2: notice: (1818) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   49.082673] overlayfs: upper fs does not support tmpfile.
daemon.info mount_root: performing overlay whiteout
daemon.info mount_root: synchronizing overlay
daemon.err mount_root: failed to sync jffs2 overlay

Full dmesg:

[    0.000000] Linux version 5.15.76 (builder@buildhost) (mips-openwrt-linux-musl-gcc (OpenWrt GCC 11.3.0 r21171-46fbe55971) 11.3.0, GNU ld (GNU Binutils) 2.37) #0 Fri Nov 4 15:21:00 2022
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001974c (MIPS 74Kc)
[    0.000000] MIPS: machine is Ubiquiti PowerBeam M5 (XW)
[    0.000000] SoC: Atheros AR9342 rev 3
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16240
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 55268K/65536K available (6179K kernel code, 592K rwdata, 1332K rodata, 1232K init, 217K bss, 10268K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 51
[    0.000000] CPU clock: 535.000 MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7144898866 ns
[    0.000002] sched_clock: 32 bits at 267MHz, resolution 3ns, wraps every 8027976190ns
[    0.008434] Calibrating delay loop... 266.64 BogoMIPS (lpj=1333248)
[    0.095035] pid_max: default: 32768 minimum: 301
[    0.100238] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.108064] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.119570] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build
[    0.131097] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.141646] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.149047] pinctrl core: initialized pinctrl subsystem
[    0.156611] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.163601] thermal_sys: Registered thermal governor 'step_wise'
[    0.183786] clocksource: Switched to clocksource MIPS
[    0.196827] NET: Registered PF_INET protocol family
[    0.202329] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.211094] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.220107] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.228382] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.236575] TCP bind hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.244112] TCP: Hash tables configured (established 1024 bind 1024)
[    0.251037] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.258085] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.265955] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.272039] PCI: CLS 0 bytes, default 32
[    0.280409] workingset: timestamp_bits=14 max_order=14 bucket_order=0
[    0.292929] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.299228] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.311025] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.323063] pinctrl-single 1804002c.pinmux: 544 pins, size 68
[    0.330490] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.340196] printk: console [ttyS0] disabled
[    0.344919] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 2500000) is a 16550A
[    0.354160] printk: console [ttyS0] enabled
[    0.363059] printk: bootconsole [early0] disabled
[    0.396119] spi-nor spi0.0: mx25l6405d (8192 Kbytes)
[    0.401261] 5 fixed-partitions partitions found on MTD device spi0.0
[    0.407817] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.414628] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.421742] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.428565] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.435757] Creating 5 MTD partitions on "spi0.0":
[    0.440652] 0x000000000000-0x000000040000 : "u-boot"
[    0.451930] 0x000000040000-0x000000050000 : "u-boot-env"
[    0.458685] 0x000000050000-0x0000007b0000 : "firmware"
[    0.467614] 2 uimage-fw partitions found on MTD device firmware
[    0.473656] Creating 2 MTD partitions on "firmware":
[    0.478757] 0x000000000000-0x000000270000 : "kernel"
[    0.485099] 0x000000270000-0x000000760000 : "rootfs"
[    0.493337] mtd: device 4 (rootfs) set to be root filesystem
[    0.499284] 1 squashfs-split partitions found on MTD device rootfs
[    0.505623] 0x0000005c0000-0x000000760000 : "rootfs_data"
[    0.513505] 0x0000007b0000-0x0000007f0000 : "cfg"
[    0.519633] 0x0000007f0000-0x000000800000 : "art"
[    1.014448] ag71xx 19000000.eth: connected to PHY at mdio.0:04 [uid=004dd072, driver=Qualcomm Atheros AR8035]
[    1.025349] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode: rgmii-id
[    1.032597] i2c_dev: i2c /dev entries driver
[    1.039771] NET: Registered PF_INET6 protocol family
[    1.057429] Segment Routing with IPv6
[    1.061264] In-situ OAM (IOAM) with IPv6
[    1.065484] NET: Registered PF_PACKET protocol family
[    1.070717] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.083912] 8021q: 802.1Q VLAN Support v1.8
[    1.101467] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.116724] Freeing unused kernel image (initmem) memory: 1232K
[    1.122757] This architecture does not have kernel memory protection.
[    1.129339] Run /sbin/init as init process
[    1.133500]   with arguments:
[    1.133506]     /sbin/init
[    1.133512]   with environment:
[    1.133518]     HOME=/
[    1.133524]     TERM=linux
[    1.962353] init: Console is alive
[    1.966554] init: - watchdog -
[    3.214455] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.313176] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.331202] init: - preinit -
[    5.410642] random: jshn: uninitialized urandom read (4 bytes read)
[    5.897940] random: jshn: uninitialized urandom read (4 bytes read)
[    5.998590] random: jshn: uninitialized urandom read (4 bytes read)
[    6.414204] random: jshn: uninitialized urandom read (4 bytes read)
[    6.566414] random: procd: uninitialized urandom read (4 bytes read)
[    9.050587] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[    9.063199] urandom-seed: Seed file not found (/etc/urandom.seed)
[    9.374076] procd: - early -
[    9.377480] procd: - watchdog -
[   10.133070] procd: - watchdog -
[   10.137076] procd: - ubus -
[   10.315081] random: ubusd: uninitialized urandom read (4 bytes read)
[   10.325413] random: ubusd: uninitialized urandom read (4 bytes read)
[   10.332595] random: ubusd: uninitialized urandom read (4 bytes read)
[   10.348852] procd: - init -
[   11.681698] random: jshn: uninitialized urandom read (4 bytes read)
[   11.869424] random: ubusd: uninitialized urandom read (4 bytes read)
[   11.945309] kmodloader: loading kernel modules from /etc/modules.d/*
[   12.493406] urngd: v1.0.2 started.
[   12.602655] Loading modules backported from Linux version v5.15.74-0-ga3f2f5ac9d61
[   12.610430] Backport generated by backports.git v5.15.74-1-0-ge2d78967
[   13.107746] random: crng init done
[   13.111228] random: 25 urandom warning(s) missed due to ratelimiting
[   13.292306] ath: EEPROM regdomain: 0x0
[   13.292337] ath: EEPROM indicates default country code should be used
[   13.292344] ath: doing EEPROM country->regdmn map search
[   13.292364] ath: country maps to regdmn code: 0x3a
[   13.292374] ath: Country alpha2 being used: US
[   13.292382] ath: Regpair used: 0x3a
[   13.307149] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   13.309841] ieee80211 phy0: Atheros AR9340 Rev:3 mem=0xb8100000, irq=12
[   13.424767] kmodloader: done loading kernel modules from /etc/modules.d/*
[   46.395338] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   46.402362] jffs2_build_filesystem(): unlocking the mtd device...
[   46.413860] done.
[   46.422104] jffs2_build_filesystem(): erasing all blocks after the end marker...
[   48.266963] br-lan: port 1(eth0) entered blocking state
[   48.279964] br-lan: port 1(eth0) entered disabled state
[   48.285658] device eth0 entered promiscuous mode
[   48.726475] jffs2: Newly-erased block contained word 0xdeadc0de at offset 0x00000000
[   48.741668] done.
[   48.743667] jffs2: notice: (1818) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   49.082673] overlayfs: upper fs does not support tmpfile.
[   64.905265] eth0: link up (100Mbps/Full duplex)
[   64.909943] br-lan: port 1(eth0) entered blocking state
[   64.915308] br-lan: port 1(eth0) entered forwarding state
[   64.924328] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready

Thanks, Regards.

That looks like only 4 blocks for the jffs, which will not allow formatting a writeable filesystem. Confirm with cat /proc/mtd. If that is the case you will need to take some packages out of the build to make space in the flash.

This issue is reproducible with a snapshot build with no packages, I even tried to remove as many as possible packages from the build image with image-builder (IPv6, USB Modules, PPPoE, and Opkg) but I still had configuration loss, I've downgraded it at the moment so I can't check the mtd output

I tried trimming most of the packages and I still reproduced this, here's the mtd output

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00040000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00760000 00010000 "firmware"
mtd3: 00270000 00010000 "kernel"
mtd4: 004f0000 00010000 "rootfs"
mtd5: 00220000 00010000 "rootfs_data"
mtd6: 00040000 00010000 "cfg"
mtd7: 00010000 00010000 "art"
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.299212] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    8.476950] jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000
[    8.484359] jffs2_build_filesystem(): unlocking the mtd device...
[    8.492633] jffs2_build_filesystem(): erasing all blocks after the end marker...
[    8.495187] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00210000
[    8.513147] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00200000
[    8.523519] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001f0000
[    8.533921] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001e0000
[    8.544295] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001d0000
[    8.554669] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001c0000
[    8.565063] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001b0000
[    8.575442] jffs2: Newly-erased block contained word 0x19852003 at offset 0x001a0000
[    8.585814] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00190000
[    8.596199] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00180000
[    8.606565] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00170000
[    8.616940] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00160000
[    8.627314] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00150000
[    8.637689] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00140000
[    8.648063] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00130000
[    8.658439] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00120000
[    8.668814] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00110000
[    8.679188] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00100000
[    8.689556] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000f0000
[    8.699932] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000e0000
[    8.710320] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000d0000
[    8.720692] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000c0000
[    8.731066] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000b0000
[    8.741441] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000a0000
[    8.751816] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00090000
[    8.762193] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00080000
[    8.772567] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00070000
[    8.782958] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00060000
[    8.793338] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00050000
[    8.803710] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00040000
[    8.814122] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00030000
[    8.824499] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00020000
[    8.834867] jffs2: Newly-erased block contained word 0xdeadc0de at offset 0x00010000
[    8.844726] jffs2: notice: (376) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[    8.867791] mount_root: switching to jffs2 overlay
[    8.873933] mount_root: switching to jffs2 failed - fallback to ramoverlay

The same configuration works fine with 22.03.2 build

It looks to me like either the flash chip failed, or maybe it is clocked too fast, as newly-erased erase block should contain only binary ones, and here you have some data. Is it reproducible with older builds, like, say, 19.07?

This issue isn't reproducible in 19.07 and 22.03, there are another issue in 22.03, which is consistent high latency in ath9k but it's out of scope of this issue, and the reason that drove me trying snapshot to begin with

Same problem here while adding support with the current snapshot for a Nanobeam 5AC Gen2 XC. Flash chip is mx25l12805d.

Kernel messages:
jffs2: Newly-erased block contained word ..............

no settings and no sysupgrade work

I have the same problem, tried a few different ubnt devices, same result with 23.05-rc1 and a few older snapshots of it. Works fine in 22.03. Here is my boot dmesg output after a executing a firstboot+reboot. https://textbin.net/eo6yuvznph
This if from a nanobeam 5AC WA. So this seems to apply to all ubnts.

Hi, I have the same problem on 23.05-RC1 with Ubiquiti Nanostation AC loco.

Hi, did anyone manage to revert to a previous (working) firmware version? Neither "sysupgrade" "mtd -r write" seem to be able to flash a new version...

Update: I can put my device in recovery mode and flash the OEMs firmware: Hold the reset button and plug the power / PoE (for 25-30 seconds, light did not flash differently). Then "put" the file to 192.168.1.20 via TFTP, see UniFi Recovery Mode – Ubiquiti Support and Help Center

Got the same filesystem issues with Ubiquiti Nanostation Loco M (XW) trying to upgrade to 23.05.0-rc3. Config is lost on reboot. Everything works again after a downgrade back to 22.03.5.

Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.304562] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.317582] pinctrl-single 1804002c.pinmux: 544 pins, size 68
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.325303] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.335653] printk: console [ttyS0] disabled
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.340309] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 2500000) is a 16550A
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.349589] printk: console [ttyS0] enabled
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.358509] printk: bootconsole [early0] disabled
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.392882] spi-nor spi0.0: mx25l6405d (8192 Kbytes)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.398115] 5 fixed-partitions partitions found on MTD device spi0.0
Sat Aug 19 14:01:17 2023 kern.err kernel: [    0.404655] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
Sat Aug 19 14:01:17 2023 kern.err kernel: [    0.411429] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
Sat Aug 19 14:01:17 2023 kern.err kernel: [    0.418573] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
Sat Aug 19 14:01:17 2023 kern.err kernel: [    0.425388] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.432617] Creating 5 MTD partitions on "spi0.0":
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.437563] 0x000000000000-0x000000040000 : "u-boot"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.449388] 0x000000040000-0x000000050000 : "u-boot-env"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.456250] 0x000000050000-0x0000007b0000 : "firmware"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.465438] 2 uimage-fw partitions found on MTD device firmware
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.471479] Creating 2 MTD partitions on "firmware":
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.476581] 0x000000000000-0x000000240000 : "kernel"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.482962] 0x000000240000-0x000000760000 : "rootfs"
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.491561] mtd: setting mtd4 (rootfs) as root device
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.496888] 1 squashfs-split partitions found on MTD device rootfs
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.503182] 0x0000005c0000-0x000000760000 : "rootfs_data"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.510262] 0x0000007b0000-0x0000007f0000 : "cfg"
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    0.518560] 0x0000007f0000-0x000000800000 : "art"
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.939720] ag71xx 19000000.eth: connected to PHY at mdio.0:01 [uid=004dd023, driver=Qualcomm Atheros AR8032]
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.950730] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.957704] i2c_dev: i2c /dev entries driver
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.965455] NET: Registered PF_INET6 protocol family
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.988926] Segment Routing with IPv6
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.992767] In-situ OAM (IOAM) with IPv6
Sat Aug 19 14:01:17 2023 kern.info kernel: [    0.996980] NET: Registered PF_PACKET protocol family
Sat Aug 19 14:01:17 2023 kern.info kernel: [    1.002254] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Sat Aug 19 14:01:17 2023 kern.info kernel: [    1.015780] 8021q: 802.1Q VLAN Support v1.8
Sat Aug 19 14:01:17 2023 kern.info kernel: [    1.031373] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
Sat Aug 19 14:01:17 2023 kern.info kernel: [    1.046350] Freeing unused kernel image (initmem) memory: 1184K
Sat Aug 19 14:01:17 2023 kern.warn kernel: [    1.052381] This architecture does not have kernel memory protection.
Sat Aug 19 14:01:17 2023 kern.info kernel: [    1.058959] Run /sbin/init as init process
Sat Aug 19 14:01:17 2023 kern.debug kernel: [    1.063115]   with arguments:
Sat Aug 19 14:01:17 2023 kern.debug kernel: [    1.063121]     /sbin/init
Sat Aug 19 14:01:17 2023 kern.debug kernel: [    1.063129]   with environment:
Sat Aug 19 14:01:17 2023 kern.debug kernel: [    1.063134]     HOME=/
Sat Aug 19 14:01:17 2023 kern.debug kernel: [    1.063141]     TERM=linux
Sat Aug 19 14:01:17 2023 user.info kernel: [    1.835454] init: Console is alive
Sat Aug 19 14:01:17 2023 user.info kernel: [    1.839598] init: - watchdog -
Sat Aug 19 14:01:17 2023 user.info kernel: [    3.263753] kmodloader: loading kernel modules from /etc/modules-boot.d/*
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.367185] usbcore: registered new interface driver usbfs
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.372851] usbcore: registered new interface driver hub
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.378433] usbcore: registered new device driver usb
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.396256] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.405227] fsl-ehci: Freescale EHCI Host controller driver
Sat Aug 19 14:01:17 2023 kern.info kernel: [    3.414087] ehci-platform: EHCI generic platform driver
Sat Aug 19 14:01:17 2023 user.info kernel: [    3.424286] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
Sat Aug 19 14:01:17 2023 user.info kernel: [    3.442316] init: - preinit -
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    5.223814] random: jshn: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    5.711961] random: jshn: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    5.814922] random: jshn: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    6.224428] random: jshn: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    6.383244] random: procd: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.info kernel: [    8.506506] eth0: link up (100Mbps/Full duplex)
Sat Aug 19 14:01:17 2023 kern.info kernel: [    8.511173] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sat Aug 19 14:01:17 2023 user.notice kernel: [    8.677179] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
Sat Aug 19 14:01:17 2023 user.warn kernel: [    8.689973] urandom-seed: Seed file not found (/etc/urandom.seed)
Sat Aug 19 14:01:17 2023 kern.info kernel: [    8.847479] eth0: link down
Sat Aug 19 14:01:17 2023 user.info kernel: [    8.867381] procd: - early -
Sat Aug 19 14:01:17 2023 user.info kernel: [    8.870762] procd: - watchdog -
Sat Aug 19 14:01:17 2023 user.info kernel: [    9.679639] procd: - watchdog -
Sat Aug 19 14:01:17 2023 user.info kernel: [    9.683624] procd: - ubus -
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    9.821091] random: ubusd: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    9.849556] random: ubusd: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [    9.864538] random: ubusd: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 user.info kernel: [    9.877676] procd: - init -
Sat Aug 19 14:01:17 2023 kern.notice kernel: [   11.142538] random: jshn: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 kern.notice kernel: [   11.184034] random: ubusd: uninitialized urandom read (4 bytes read)
Sat Aug 19 14:01:17 2023 user.info kernel: [   11.387882] kmodloader: loading kernel modules from /etc/modules.d/*
Sat Aug 19 14:01:17 2023 user.info kernel: [   11.817650] urngd: v1.0.2 started.
Sat Aug 19 14:01:17 2023 kern.info kernel: [   12.033223] Loading modules backported from Linux version v6.1.24-0-g0102425ac76b
Sat Aug 19 14:01:17 2023 kern.info kernel: [   12.040926] Backport generated by backports.git v5.15.92-1-44-gd6ea70fafd36
Sat Aug 19 14:01:17 2023 kern.notice kernel: [   12.051982] random: crng init done
Sat Aug 19 14:01:17 2023 kern.notice kernel: [   12.055534] random: 24 urandom warning(s) missed due to ratelimiting
Sat Aug 19 14:01:17 2023 kern.info kernel: [   12.612217] PPP generic driver version 2.4.2
Sat Aug 19 14:01:17 2023 kern.info kernel: [   12.621965] NET: Registered PF_PPPOX protocol family
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689215] ath: EEPROM regdomain: 0x0
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689243] ath: EEPROM indicates default country code should be used
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689252] ath: doing EEPROM country->regdmn map search
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689271] ath: country maps to regdmn code: 0x3a
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689281] ath: Country alpha2 being used: US
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.689290] ath: Regpair used: 0x3a
Sat Aug 19 14:01:17 2023 kern.debug kernel: [   12.703933] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
Sat Aug 19 14:01:17 2023 kern.info kernel: [   12.706619] ieee80211 phy0: Atheros AR9340 Rev:2 mem=0xb8100000, irq=12
Sat Aug 19 14:01:17 2023 user.info kernel: [   12.718538] kmodloader: done loading kernel modules from /etc/modules.d/*
Sat Aug 19 14:01:19 2023 user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses!
Sat Aug 19 14:01:37 2023 authpriv.info dropbear[1321]: Not backgrounding
Sat Aug 19 14:01:41 2023 daemon.notice wpa_supplicant[1429]: Successfully initialized wpa_supplicant
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: bonding
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: 8021ad
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: 8021q
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: macvlan
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: veth
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: bridge
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: Network device
Sat Aug 19 14:01:42 2023 user.notice : Added device handler type: tunnel
Sat Aug 19 14:01:47 2023 user.notice ucitrack: Setting up /etc/config/dhcp reload dependency on /etc/config/network
Sat Aug 19 14:01:47 2023 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/wireless
Sat Aug 19 14:01:47 2023 user.notice ucitrack: Setting up /etc/config/luci-splash reload dependency on /etc/config/firewall
Sat Aug 19 14:01:48 2023 user.notice ucitrack: Setting up /etc/config/qos reload dependency on /etc/config/firewall
Sat Aug 19 14:01:48 2023 user.notice ucitrack: Setting up /etc/config/miniupnpd reload dependency on /etc/config/firewall
Sat Aug 19 14:01:48 2023 user.notice ucitrack: Setting up /etc/config/odhcpd reload dependency on /etc/config/dhcp
Sat Aug 19 14:01:49 2023 user.notice ucitrack: Setting up non-init /etc/config/fstab reload handler: /sbin/block mount
Sat Aug 19 14:01:49 2023 user.notice ucitrack: Setting up /etc/config/system reload trigger for non-procd /etc/init.d/led
Sat Aug 19 14:01:50 2023 user.notice ucitrack: Setting up /etc/config/luci_statistics reload dependency on /etc/config/system
Sat Aug 19 14:01:50 2023 user.notice ucitrack: Setting up /etc/config/dhcp reload dependency on /etc/config/system
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.777659] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.803924] jffs2_build_filesystem(): unlocking the mtd device...
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.804019] done.
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.812245] jffs2_build_filesystem(): erasing all blocks after the end marker...
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.829376] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00190000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.882976] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00180000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.930653] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00170000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.956365] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00160000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   55.992104] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00150000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.033914] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00140000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.066348] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00130000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.096360] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00120000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.116701] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00110000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.136708] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00100000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.156810] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000f0000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.176712] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000e0000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.196716] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000d0000
Sat Aug 19 14:01:51 2023 kern.warn kernel: [   56.216720] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000c0000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.243911] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000b0000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.263994] jffs2: Newly-erased block contained word 0x19852003 at offset 0x000a0000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.286348] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00090000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.306713] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00080000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.326731] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00070000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.346760] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00060000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.366716] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00050000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.386719] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00040000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.413912] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00030000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.433914] jffs2: Newly-erased block contained word 0x19852003 at offset 0x00020000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.456326] jffs2: Newly-erased block contained word 0xdeadc0de at offset 0x00010000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.476716] jffs2: Newly-erased block contained word 0xdeadc0de at offset 0x00000000
Sat Aug 19 14:01:52 2023 kern.warn kernel: [   56.494272] done.
Sat Aug 19 14:01:52 2023 kern.notice kernel: [   56.496263] jffs2: notice: (1970) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Aug 19 14:01:52 2023 daemon.notice procd: /etc/rc.d/S95done: cp: can't create directory '/rom/overlay/upper': No space left on device
Sat Aug 19 14:01:52 2023 daemon.notice procd: /etc/rc.d/S95done: cp: can't create directory '/rom/overlay/work': No space left on device
Sat Aug 19 14:01:52 2023 daemon.err mount_root: failed - cp -a /tmp/root/* /rom/overlay: Not a tty

I'd take a look into kernel changes regarding Macronix NOR flash. I recall their chips being quirky, and sensitive for misdetection or misselection. For example, in flashrom, this caused the erase to fail, for example if MX25L6405 was selected instead of MX26L6406, AND flashrom could not tell the two models apart by itself, so one has to specify this manually.
Logs from working scenario would be helpful, for comparison.

From working 22.03.5. It looks like both detected mx25l6405d.

[    0.340738] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.353699] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.365326] pinctrl-single 1804002c.pinmux: 544 pins, size 68
[    0.372735] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.382548] printk: console [ttyS0] disabled
[    0.387194] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 2500000) is a 16550A
[    0.396455] printk: console [ttyS0] enabled
[    0.405373] printk: bootconsole [early0] disabled
[    0.441558] spi-nor spi0.0: mx25l6405d (8192 Kbytes)
[    0.446774] 5 fixed-partitions partitions found on MTD device spi0.0
[    0.453319] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.460068] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.467383] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.474205] OF: Bad cell count for /ahb/spi@1f000000/flash@0/partitions
[    0.481632] Creating 5 MTD partitions on "spi0.0":
[    0.486520] 0x000000000000-0x000000040000 : "u-boot"
[    0.495385] 0x000000040000-0x000000050000 : "u-boot-env"
[    0.502190] 0x000000050000-0x0000007b0000 : "firmware"
[    0.510878] 2 uimage-fw partitions found on MTD device firmware
[    0.516981] Creating 2 MTD partitions on "firmware":
[    0.522064] 0x000000000000-0x000000230000 : "kernel"
[    0.529374] 0x000000230000-0x000000760000 : "rootfs"
[    0.535663] mtd: setting mtd4 (rootfs) as root device
[    0.542076] 1 squashfs-split partitions found on MTD device rootfs
[    0.548380] 0x0000005d0000-0x000000760000 : "rootfs_data"
[    0.555186] 0x0000007b0000-0x0000007f0000 : "cfg"
[    0.563393] 0x0000007f0000-0x000000800000 : "art"
[    0.967246] ag71xx 19000000.eth: connected to PHY at mdio.0:01 [uid=004dd023, driver=Qualcomm Atheros AR8032]
[    0.978187] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
[    0.985028] i2c /dev entries driver
[    0.991217] NET: Registered protocol family 10
[    1.011039] Segment Routing with IPv6
[    1.015008] NET: Registered protocol family 17
[    1.019633] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.032832] 8021q: 802.1Q VLAN Support v1.8
[    1.049152] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.064298] Freeing unused kernel memory: 1200K
[    1.068907] This architecture does not have kernel memory protection.
[    1.075484] Run /sbin/init as init process
[    1.079639]   with arguments:
[    1.079645]     /sbin/init
[    1.079650]   with environment:
[    1.079656]     HOME=/
[    1.079661]     TERM=linux
[    1.924715] init: Console is alive
[    1.928797] init: - watchdog -
[    3.371519] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.438018] usbcore: registered new interface driver usbfs
[    3.443786] usbcore: registered new interface driver hub
[    3.449309] usbcore: registered new device driver usb
[    3.464761] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    3.475159] fsl-ehci: Freescale EHCI Host controller driver
[    3.483155] ehci-platform: EHCI generic platform driver
[    3.493262] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.502627] init: - preinit -
[    5.565077] random: jshn: uninitialized urandom read (4 bytes read)
[    6.087690] random: jshn: uninitialized urandom read (4 bytes read)
[    6.194382] random: jshn: uninitialized urandom read (4 bytes read)
[    6.665321] random: jshn: uninitialized urandom read (4 bytes read)
[    6.792359] random: procd: uninitialized urandom read (4 bytes read)
[    8.903163] eth0: link up (100Mbps/Full duplex)
[    8.907823] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   11.286959] jffs2: notice: (495) jffs2_build_xattr_subsystem: complete building xattr subsystem, 29 of xdatum (5 unchecked, 20 orphan) and 34 of xref (4 dead, 18 orphan) found.
[   11.305181] mount_root: switching to jffs2 overlay
[   11.320508] overlayfs: upper fs does not support tmpfile.

Logs look similar, just without jffs2 rebuilding the filesystem

I took a look into the internal photos submitted to FCC, and found a Winbond flash there:

So they probably use >1 vendor. I wonder what the actual label on the flash is. I took a quick grep at the kernel source and documentation - and there is no distinction for MX25L6406* chips, so maybe the kernel can figure out the valid erase algorithm by itself, and a bug has sneaked in. I would ask on LKML.

Also, just now I found this patch: https://github.com/openwrt/openwrt/blob/master/target/linux/generic/pending-6.1/498-mtd-spi-nor-locking-support-for-MX25L6405D.patch - which seems oddly relevant. Bootloader locking (or failing to unlock) parts of flash for writing can have this exact effect, too, especially if the blocks failed to be erased start from an even boundary.

Hello,
A few comments on this old thread, since AFAICT, no solution was found yet.

  • I got hit by this issue too, on a Nanostation AC loco. I had to do the same dance as @humaita to recover - go back to Ubiquiti's firmware and then reflash 23.05.
  • This has been reported as #13750 in github as well.
  • I'd be willing to spent some time to get this one closer to a resolution, but beside bisecting, I'm kind of out of ideas - and a bisection that need to go back to the OEM firmware every other round is kind of tiresome. Especially on a device mounted outdoor. Any ideas?
  • If anyone is refraining from upgrading to newer 22.03.x because of this, I just bumped mine to 22.03.6 without issues.

I got my hands on Nanostation loco M5 (XW). Maybe I can try to reproduce that next week.

That isn't necessary if you have serial connection you can TFTP boot an OpenWrt initramfs of a version that can write to flash. Use it to sysupgrade a test build then observe the boot log to see if "Previously erased block contains..." errors occur.

I have a Nanostation AC loco indoors on the bench so I can execute this process quickly. I'm not sure exactly what to try though.

I have done a few other tests:

The erase and write commands in the bootloader itself cannot erase or write the flash (including saveenv). The only way the bootloader will successfully write flash is in the TFTP recovery process-- which requires a signed firmware.

I made an initram with no spi-nor driver at all, exposing the bus to the flash chip directly as a spidev. With this it is possible to access the registers in the chip according to the datasheet.

The datasheet describes an "Advanced Sector Protection" mode. I can confirm that the Advanced mode has not been activated (such activation is not reversible), and the chip is using the more or less standard sector protection based on five bits in the Status Register.

The bootloader writes the Status Register to 0x3c during every boot, which protects all sectors in the chip. The protect command in the bootloader appears to do nothing.

Thank you both for your replies.

Given that this is a regression between 22.03 and 23.05, I still believe that a bisection would provide useful info. @mk24, you seem to have a much better setup than me to do so.
I suspect that it'll end up pointing to a kernel version change, which would need to be bisected as well.
I'm aware that the core issue probably predates the regression (there are old reports of the same issue), and therefore that there's something that we don't understand about the flash handling in ubnt devices, but I believe that understanding the regression is our best entry point.

One more thing: in the github bug entry, I asked if this could be documented as a known issue in the 23.05 release notes, but that hasn't happened yet. I still believe this would be appropriate.

Wasn't that signature checking actually very lax, and was fixed by https://github.com/openwrt/openwrt/commit/d42a7c4699098507b459fe3ed323d829901eee9e?

I had the latest available XW firmware loaded on my NS Loco M5 XW, and OpenWrt 23.05 successfully flashed from U-boot TFTP recovery.

While at that, I wonder how it relates to airOS version, which contains U-boot. Maybe older versions didn't do that?

My Loco M5 XW also has Macronix flash, I'll try to play with it. I installed on top of XW.v6.3.11.33396.230425.1644.bin
Edit: it reproduced on my device. But one thing puzzles me: why sysupgrade works, and only jffs2 cannot properly unlock the blocks for rootfs_data?

Update from the field:

I built two images from this commit: https://github.com/openwrt/openwrt/tree/a24e07700d0f884ff82e565110adef4ad2155db4 - which is the last one before removing kernel 5.10 from ath79 subtarget.
On kernel 5.10 - working OK, on 5.15 - issue reproduced. So this definitely lies somewhere in the kernel, and thus we can skip bisecting the OpenWrt tree. And given the fact, that the sysupgrade works, I'd take a look into jffs2 support.