I understand this device is no longer supported because of its 4/32 specs, but I'm hoping I can get some advice anyway since 19.07 in itself after all does work perfectly fine on this device as long as I don't reboot.
The problem is the issue mentioned in the release notes, that some images have grown too large to be able to support the overlay file system with the consequence that no configuration is persistent. I built 19.07 manually (not using the Image Builder) and after several iterations and tests of removing things I didn't need I eventually landed with a lean 3.1 mb squashfs image. This is well smaller than the pre-built images for 18.06.4 and .6, but the overlay file system is still missing and I cannot understand why.
I have clearly misunderstood the file systems and how the flash memory in the device is laid out. Any advice on what needs to be tuned to be able to mount the overlayfs? Is it some "hidden" dependency gone missing?
So far all I've done is to remove things according to official and inofficial suggestions: printk support, sysrq support, swap support, ipv6 support, disabling various debug features, pppoe, opkg, dropbear, odhcpd, dnsmasq and a few non-critical utilities included with busybox (traceroute, hexdump mkswap/swapon/swapoff).
There was a patch on the mailing list for ar71xx or ath79 to allow smaller erase sectors. Maybe that could help you. I might already have said that (or might have been to someone else, don't remember).
I assume you already tried to issue the 'firstboot' command to reformat your overlay?
Better bump this thread instead of the generic one.
Looking at your 18.06 comparison data: there you only have overlay of 112 or 48 kB, which is laughably little. The overlay requires 4 (or 5??) erase block of 64 kB, = 256 kB, if I remember right.
You may really need to use some "small erase block" magic.
Otherwise, you need to have either (4 or 5) x 64 kB blocks for overlay, = 256 or 320 kB...
Easiest would be to remove some 256 kB of stuff from your image.
You should show us what error messages you get in the kernel & system logs. I am suite sure that the overlay / jffs2 complains something...
Something like [ 19.743591] jffs2: Too few erase blocks (4)
Just do logread | grep jffs2 and logread | grep overlay after a reboot, so that you capture the boot process.
For reference, this is from a ar71xx WNDR3700v2:
root@router2:~# logread | grep overlay
Sat Jan 25 10:22:50 2020 user.info kernel: [ 18.275547] mount_root: loading kmods from internal overlay
Sat Jan 25 10:22:50 2020 user.info kernel: [ 71.750296] mount_root: overlay filesystem has not been fully initialized yet
Sat Jan 25 10:22:50 2020 user.info kernel: [ 71.758690] mount_root: switching to jffs2 overlay
Sat Jan 25 10:22:50 2020 kern.warn kernel: [ 71.821436] overlayfs: upper fs does not support tmpfile.
root@router2:~# logread | grep jffs2
Sat Jan 25 10:22:50 2020 kern.info kernel: [ 2.954517] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
Sat Jan 25 10:22:50 2020 kern.warn kernel: [ 18.619533] jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000
Sat Jan 25 10:22:50 2020 kern.warn kernel: [ 18.626774] jffs2_build_filesystem(): unlocking the mtd device...
Sat Jan 25 10:22:50 2020 kern.warn kernel: [ 18.634869] jffs2_build_filesystem(): erasing all blocks after the end marker...
Sat Jan 25 10:22:50 2020 kern.notice kernel: [ 71.306972] jffs2: notice: (483) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jan 25 10:22:50 2020 kern.notice kernel: [ 71.515051] jffs2: notice: (481) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
Sat Jan 25 10:22:50 2020 user.info kernel: [ 71.758690] mount_root: switching to jffs2 overlay
The setting that patch relates to is already activated for the ramips target. The 18.06 image is the pre-built vanilla offering downloaded directly from openwrt.org, and while I don't know if the overlay size there is sufficient or not it at least functions correctly and my entire config manages to persist through rebooting 18.06.
Could you elaborate on the "small erase block magic" part? I'll try again from a clean slate after doing the "firstboot" stuff one more time.
CONFIG_TARGET_ramips=y
CONFIG_TARGET_ramips_mt7620=y
CONFIG_TARGET_ramips_mt7620_DEVICE_ex2700=y
CONFIG_DEVEL=y
CONFIG_IMAGEOPT=y
CONFIG_PACKAGE_cgi-io=y
# CONFIG_PACKAGE_dnsmasq is not set
CONFIG_PACKAGE_ebtables=y
# CONFIG_PACKAGE_firewall is not set
# CONFIG_PACKAGE_ip6tables is not set
# CONFIG_PACKAGE_iptables is not set
CONFIG_PACKAGE_kmod-ebtables=y
CONFIG_PACKAGE_kmod-ebtables-ipv4=y
CONFIG_PACKAGE_kmod-ebtables-ipv6=y
# CONFIG_PACKAGE_kmod-ip6tables is not set
# CONFIG_PACKAGE_kmod-ipt-conntrack is not set
# CONFIG_PACKAGE_kmod-ipt-nat is not set
# CONFIG_PACKAGE_kmod-ipt-offload is not set
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-flow is not set
# CONFIG_PACKAGE_kmod-nf-ipt6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
# CONFIG_PACKAGE_kmod-ppp is not set
# CONFIG_PACKAGE_libip4tc is not set
# CONFIG_PACKAGE_libip6tc is not set
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
# CONFIG_PACKAGE_libpthread is not set
CONFIG_PACKAGE_libubus-lua=y
# CONFIG_PACKAGE_libuclient is not set
# CONFIG_PACKAGE_libxtables is not set
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
# CONFIG_PACKAGE_opkg is not set
# CONFIG_PACKAGE_ppp is not set
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
# CONFIG_PACKAGE_uclient-fetch is not set
CONFIG_PACKAGE_uhttpd=y
# CONFIG_PER_FEED_REPO is not set
CONFIG_TARGET_OPTIMIZATION="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kec"
CONFIG_TARGET_OPTIONS=y
# CONFIG_TARGET_ROOTFS_INITRAMFS is not set
The command mount will show more information on what you have mounted.
Usually the squashfs rom filesyste is mount at /rom and and read write filesystem at /overlay. These are then combined in / as an overlay fs, with /overlay/upper overlaying /rom.
Now what you should note is that you have no /overlay mounted. This is why you can't write. Usually this is specifyied in /etc/fstab - but it may be different in openwrt due to cicrular definition, (e.g. can only mount the overlay fs after /rom and /overlay are mounted, but /overlay and /rom are subdirectories of / which is what the overlay fs mounts). This circular problem may be solved in intramfs - and we'll need a high level openwrt to explain.
Now /overlay should be mounted from the flash, you can check what flash you have access to in /proc/mtd (IIRC), or it may be /sys/class/mtd. But be aware where the flash partitions used to be defined on the linux command line (at boot). These days its is probably specified in the device tree, e.g. look at: ex2700.dtsi device tree - so look if your device tree specifies the flash partition, if these are correct (e.g. correspond to your hardware), and if they are large enough for the various partitions.
do a "hard reset" (unmount /overlay, run jffs2reset, reboot)
upgrade from 18.06.4 to 19.07.0, without "keep settings"
Omitting or shortcutting any steps, by nmrpflash'ing 18.06.4 at once, or by installing 18.06.6 with its much smaller overlay allocation compared to 18.06.4, would not work. I'm not sure what's happening but I suspect step 1 + 3 were key to somehow "unf*ck" the partitioning.
Thanks for all the advice and suggestions, people.