[19.07] EX2700/ramips, overlay not persistent despite squashfs image <4mb

I understand this device is no longer supported because of its 4/32 specs, but I'm hoping I can get some advice anyway since 19.07 in itself after all does work perfectly fine on this device as long as I don't reboot.

The problem is the issue mentioned in the release notes, that some images have grown too large to be able to support the overlay file system with the consequence that no configuration is persistent. I built 19.07 manually (not using the Image Builder) and after several iterations and tests of removing things I didn't need I eventually landed with a lean 3.1 mb squashfs image. This is well smaller than the pre-built images for 18.06.4 and .6, but the overlay file system is still missing and I cannot understand why.

I have clearly misunderstood the file systems and how the flash memory in the device is laid out. Any advice on what needs to be tuned to be able to mount the overlayfs? Is it some "hidden" dependency gone missing?

Output from df(1) showing the various allocations (18.06 included just as a comparison): https://pastebin.com/raw/MUMwWPYB

Best include in your post what you already checked and tried, like we discussed on IRC, so people know what has been covered already.

So far all I've done is to remove things according to official and inofficial suggestions: printk support, sysrq support, swap support, ipv6 support, disabling various debug features, pppoe, opkg, dropbear, odhcpd, dnsmasq and a few non-critical utilities included with busybox (traceroute, hexdump mkswap/swapon/swapoff).

There was a patch on the mailing list for ar71xx or ath79 to allow smaller erase sectors. Maybe that could help you. I might already have said that (or might have been to someone else, don't remember).

I assume you already tried to issue the 'firstboot' command to reformat your overlay?

Better bump this thread instead of the generic one.

Don't know if it helps you, but that's how it looks like on my working ex2700 on OpenwWrt 19.07.0:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2048      2048         0 100% /rom
tmpfs                    13700       240     13460   2% /tmp
/dev/mtdblock5             288        48       240  17% /overlay
overlayfs:/overlay         288        48       240  17% /
tmpfs                      512         0       512   0% /dev

My self compiled squashfs-sysupgrade.bin has a size of 3.7 MB

My hunch is this.

Looking at your 18.06 comparison data: there you only have overlay of 112 or 48 kB, which is laughably little. The overlay requires 4 (or 5??) erase block of 64 kB, = 256 kB, if I remember right.

You may really need to use some "small erase block" magic.

Otherwise, you need to have either (4 or 5) x 64 kB blocks for overlay, = 256 or 320 kB...

This is probably the patch that @Borromini mentioned:
But I have no idea if that is applicable for your device.

Easiest would be to remove some 256 kB of stuff from your image.

You should show us what error messages you get in the kernel & system logs. I am suite sure that the overlay / jffs2 complains something...
Something like [ 19.743591] jffs2: Too few erase blocks (4)

Just do logread | grep jffs2 and logread | grep overlay after a reboot, so that you capture the boot process.

For reference, this is from a ar71xx WNDR3700v2:

root@router2:~# logread | grep overlay
Sat Jan 25 10:22:50 2020 user.info kernel: [   18.275547] mount_root: loading kmods from internal overlay
Sat Jan 25 10:22:50 2020 user.info kernel: [   71.750296] mount_root: overlay filesystem has not been fully initialized yet
Sat Jan 25 10:22:50 2020 user.info kernel: [   71.758690] mount_root: switching to jffs2 overlay
Sat Jan 25 10:22:50 2020 kern.warn kernel: [   71.821436] overlayfs: upper fs does not support tmpfile.

root@router2:~# logread | grep jffs2
Sat Jan 25 10:22:50 2020 kern.info kernel: [    2.954517] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
Sat Jan 25 10:22:50 2020 kern.warn kernel: [   18.619533] jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000
Sat Jan 25 10:22:50 2020 kern.warn kernel: [   18.626774] jffs2_build_filesystem(): unlocking the mtd device...
Sat Jan 25 10:22:50 2020 kern.warn kernel: [   18.634869] jffs2_build_filesystem(): erasing all blocks after the end marker...
Sat Jan 25 10:22:50 2020 kern.notice kernel: [   71.306972] jffs2: notice: (483) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jan 25 10:22:50 2020 kern.notice kernel: [   71.515051] jffs2: notice: (481) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jan 25 10:22:50 2020 user.info kernel: [   71.758690] mount_root: switching to jffs2 overlay
1 Like

Yeah I checked it out, and noticed that the same setting was already activated for the ramips target.

The setting that patch relates to is already activated for the ramips target. The 18.06 image is the pre-built vanilla offering downloaded directly from openwrt.org, and while I don't know if the overlay size there is sufficient or not it at least functions correctly and my entire config manages to persist through rebooting 18.06.

Could you elaborate on the "small erase block magic" part? I'll try again from a clean slate after doing the "firstboot" stuff one more time.

Would you mind sharing a description (or config diff) of that build so that I can try replicate it?

# CONFIG_PACKAGE_dnsmasq is not set
# CONFIG_PACKAGE_firewall is not set
# CONFIG_PACKAGE_ip6tables is not set
# CONFIG_PACKAGE_iptables is not set
# CONFIG_PACKAGE_kmod-ip6tables is not set
# CONFIG_PACKAGE_kmod-ipt-conntrack is not set
# CONFIG_PACKAGE_kmod-ipt-nat is not set
# CONFIG_PACKAGE_kmod-ipt-offload is not set
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-flow is not set
# CONFIG_PACKAGE_kmod-nf-ipt6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
# CONFIG_PACKAGE_kmod-ppp is not set
# CONFIG_PACKAGE_libip4tc is not set
# CONFIG_PACKAGE_libip6tc is not set
# CONFIG_PACKAGE_libpthread is not set
# CONFIG_PACKAGE_libuclient is not set
# CONFIG_PACKAGE_libxtables is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
# CONFIG_PACKAGE_opkg is not set
# CONFIG_PACKAGE_ppp is not set
# CONFIG_PACKAGE_uclient-fetch is not set
CONFIG_TARGET_OPTIMIZATION="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kec"
1 Like

@accelerate Use ./scripts/diffconfig.sh to print only the changes against the default configuration. That makes easier to spot differences.

Edit: nevermind me!

This is already my diff config.

The command mount will show more information on what you have mounted.

Usually the squashfs rom filesyste is mount at /rom and and read write filesystem at /overlay. These are then combined in / as an overlay fs, with /overlay/upper overlaying /rom.

Now what you should note is that you have no /overlay mounted. This is why you can't write. Usually this is specifyied in /etc/fstab - but it may be different in openwrt due to cicrular definition, (e.g. can only mount the overlay fs after /rom and /overlay are mounted, but /overlay and /rom are subdirectories of / which is what the overlay fs mounts). This circular problem may be solved in intramfs - and we'll need a high level openwrt to explain.

Now /overlay should be mounted from the flash, you can check what flash you have access to in /proc/mtd (IIRC), or it may be /sys/class/mtd. But be aware where the flash partitions used to be defined on the linux command line (at boot). These days its is probably specified in the device tree, e.g. look at:
ex2700.dtsi device tree - so look if your device tree specifies the flash partition, if these are correct (e.g. correspond to your hardware), and if they are large enough for the various partitions.

Anyway this should be enough for you to dig.

Ah - this explains how the file system gets mounted: https://openwrt.org/docs/techref/preinit_mount

I finally managed to get the overlay file system to manifest on my 19.07.0 build through a curious installation/upgrade path:

  1. install original Netgear firmware using nmrpflash (https://github.com/jclehner/nmrpflash)
  2. from the Netgear firmware, install 18.06.4
  3. do a "hard reset" (unmount /overlay, run jffs2reset, reboot)
  4. upgrade from 18.06.4 to 19.07.0, without "keep settings"

Omitting or shortcutting any steps, by nmrpflash'ing 18.06.4 at once, or by installing 18.06.6 with its much smaller overlay allocation compared to 18.06.4, would not work. I'm not sure what's happening but I suspect step 1 + 3 were key to somehow "unf*ck" the partitioning.

Thanks for all the advice and suggestions, people.

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.