19.07.xx: Strange kernel jffs2 bug

The link to glinet came up googling the jffs erase error message because I wanted to know (as openwrt user) if anyone else had the same or similar message in their dmesg. When I had read this, I thought this could also be a pointer to the root cause why it's happening with openwrt reproducibly on all of my 5 devices. Before finding that link with the exact same offset and wrong word mentioned I was also thinking the flash might be faulty. Now I know for sure that 5 devices cannot have the same "hardware fault" at the exact same offset by accident. It might be a kernel driver issue. Just wanted to give this ray of light to someone probably reading this and concluding "aha, it might be the spi read driver stuff for us at openwrt as well". I'm not arguing that both projects have a codebase in common where the other should care about. I'm only interested to see this problem fixed in openwrt in some future.

1 Like

(Moved to the For Developers section.)

1 Like

For me on ubnt device (e.g. nanostation ac loco) the jffs2 is working on 4.19 but I have the same errors on 5.4... :confused:

1 Like

@PolynomialDivision thanks for making a PR :slight_smile:

1 Like

Does it fix your issue? :smiley:

@PolynomialDivision Is there a nightly test build or CI build I could try, then I'd love to do my test process again and report back :-). Still have a test device at spare. (Archer c7v5)

The fix is already included upstream in the git.
But it always takes sometime until images are rebuilt, so not sure if the fix is already included.

The download is here:
https://downloads.openwrt.org/snapshots/targets/ath79/generic/openwrt-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin

1 Like

Well , I'll give it a shot :-).

1 Like

I'm very interested to hear from the result.

1 Like

First attempt: Upgrading via CLI sysupgrade -v from 19.07.4 to "openwrt-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin"

  • Preparation: Filled 2 MBytes of IPK data to /root/packages (as the bug only occured before when pretty much data needed to be persisted, not only some Kilobytes)
  • Set "/etc/sysupgrade.conf" to persist the /root/
  • Ran sysupgrade
  • First boot after sysupgrade, used SSH and grabbed "dmesg | grep jffs2". Looks fine, because there isn't a message "jffs2: Newly-erased block contained word xxxxxxx at offset xxxxxxx".
[    0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
[    0.231495] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[   12.011482] jffs2_scan_eraseblock(): End of filesystem marker found at 0x200000
[   12.019079] jffs2_build_filesystem(): unlocking the mtd device...
[   12.027488] jffs2_build_filesystem(): erasing all blocks after the end marker...
[   35.728549] jffs2: notice: (601) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   36.747548] mount_root: switching to jffs2 overlay
  • Rebooted device
  • Second boot after flash: Still no jffs2 error in the dmesg. Still looks fine.

Second attempt: Same procedure, but sysupgrade -v from "snapshot to the same snapshot" ("openwrt-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin")

  • First boot: jffs2 log shows no errors, all files were persisted correctly.
  • Second boot: Still everything fine, config and files are there like they should. No jffs2 errors in dmesg.

Third attempt: Same procedure as in second attempt. I even did overload the jffs2 overlay partition (which one shouldn't do) to 88% used space. And still everything is FINE. No jffs2 errors, first boot after flash okay, second boot and so on okay.

Here's the situation before my third attempt, just for reference.

df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.0M      3.0M         0 100% /rom
tmpfs                    60.1M    100.0K     60.0M   0% /tmp
/dev/mtdblock9           10.0M      8.4M      1.6M  84% /overlay
overlayfs:/overlay       10.0M      8.4M      1.6M  84% /
tmpfs                   512.0K         0    512.0K   0% /dev

Even my "pre-download IPK package files and place them on /root/packages/" together with the "/root/opkg_reinstall.sh" scripts do work FINE now. After upgrading to a new OpenWrt (Snapshot) image, the scripts automaticaly take care of reinstalling the OPKG's and bring the non-wired access point back online. @PolynomialDivision You're a rock star! Thanks for solving the jffs2 newly erased block issue :-).

1 Like

Just for completenes: the new snapshot linked by @PolynomialDivision solves the jffs problem.

I've also tried to downgrade to the (problematic prior) 19.07.4 openwrt image and when the old version made its first boot after flash, the jffs2 newly erased block at offset error immediately came back (as expected).

That is nice to hear! :slight_smile: But I got help from a very expierenced OpenWrt dev. :slight_smile:

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.