After Upgrade to 18.06.2 from ChaosCalmer, each reboot clears data

I have run Chaos Calmer 15.05 on this device for years, and finally decided to upgrade (I know, long time, sorry). If it matters, this is on a On Networks (NetGear) N150R

The upgrade to 18.06.2 was painless enough. I used:
/tmp# wget http://downloads.openwrt.org/releases/18.06.2/targets/ar71xx/tiny/openwrt-18.06.2-ar71xx-tiny-n150r-squashfs-sysupgrade.bin
/tmp# sysupgrade -v openwrt-18.06.2-ar71xx-tiny-n150r-squashfs-sysupgrade.bin

Upon reboot, the IP address reverted to 192.168.1.1, the SSH key had changed, the password was wiped... and it appears that the filesystem is completely "factory reset".

Ok... Not the worst thing in the world, I have backups. But...

On reboot, the filesystem is reset to factory defaults again.
I change root password, it's cleared at next reboot.
I change network settings, it's cleared at next reboot.
I touch a file in /root, and it's gone at next reboot.

Every reboot, nothing I have changed stays.

I'm not sure what to do? It seems as if there is a "reset the filesystem" script that is run every time the device boots.

Is there anything that would make sense? Anything from 15.05->18.06 that might stick around and cause this behavior?

Thank you in advance!
Matt

Use the 18.06 to flash 18.06 again. Then check with mount and df to see that you have a writeable overlay partition.

"Tiny" models have almost no space in the flash to store anything other than a few settings.

Usability issues

Insufficient RAM for stable operation

Extensibility issues

Barely enough Flash to accommodate OpenWrt firmware image

Your device was already on "life support" being on the "tiny" target. It has probably been very useful to you over its life, but it is time to select and purchase a replacement. Good-quality devices with at least 16 MB flash and 128 MB RAM are available starting at ~US$20.

A self-built custom image may allow you to get another year or so out of it, but most likely without LuCI or any other packages.

1 Like

Thank you.

It looks like there's writeable overlay partition:

root@OpenWrt:/etc/config# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2304      2304         0 100% /rom
tmpfs                    13928        76     13852   1% /tmp
tmpfs                    13928        56     13872   0% /tmp/root
overlayfs:/tmp/root      13928        56     13872   0% /
tmpfs                      512         0       512   0% /dev
root@OpenWrt:/etc/config# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /tmp/root type tmpfs (rw,noatime,mode=755)
overlayfs:/tmp/root on / type overlay (rw,noatime,lowerdir=/,upperdir=/tmp/root/upper,workdir=/tmp/root/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

I understand that the device is limited, but I'm trying to figure out what, in OpenWRT causes this behavior, and learn something in the process. Thanks!

Matt

Thank you, @jeff.
So it's not that I'm having the filesystem reset, it's that the overlay partition isn't being stored correctly due to insufficient space?

Thanks. I'll downgrade to 17.01. Thanks! I have a number of these, so it's helpful not to have to replace them all at once. Plus, I learned something!

I appreciate all the help, everyone.
Matt

Yes, what is likely happening is that the build is too large for the overlay system to use the flash, so it uses a tempfs (RAM backed). You typically need at least 3 erase blocks, or close to 200 kB free to be able to use JFFS2 (flash backed) for the overlay.

You probably should learn how to build or assemble images as 17.01 is about to be EOL, with no further patches, including security updates.

https://openwrt.org/docs/guide-user/additional-software/saving_space

(This is nothing new, the end of the road for 4/32 devices has been foreseen for several years, with the "tiny" target being a stop-gap measure too give people time to either upgrade, or learn the skills to be able to run these devices with custom builds and within their limited capabilities.)

1 Like

Thanks again! These devices have more than paid for themselves. Looks like it's time for an upgrade. I'll still upgrade them to 17.01 to keep them as toys.

2 Likes

Hmmm... 17.01 doesn't work either. Is it possible that I can reduce something (like the initial needs of the overlay) to make this work?

You can likely use 18.06 if you use the Image Builder to remove a few things. Features many people don't need are ipv6, ppp and LuCI.

2 Likes

One thing that can be puzzling at first is that "deleting" packages from an installed ROM doesn't free up any space. That's because a ROM is, well, read-only, and all that happens is that they get "Wite-Out" over them so you don't see them.

The Image Builder is a way to assemble pre-compiled packages into a ROM. The full build system is used to build from source and doesn't require any more packages to be installed on your Linux-based build machine, is just about as easy to use, and it even more flexible than the Image Builder.

1 Like

I do understand the idea of ROM/Overlays, I guess I'm just confused because it seemed like this model should work with 17.01.5 but I'm having issues. Perhaps I'm mistaken and it just doesn't work with LEDE-17.05.1

You're in the position of needing to build your own image if the pre-built ones don't have enough space for a functional overlay. There isn't any reasonable way to modify a pre-built image. The limitation of three, free, erase blocks is from how JFFS2 works.

1 Like

Sure there is... The image builder. That's exactly what it's for.

@kurios First thing to strip would be LuCI, that gives you some headroom. Other things to look at are PPP if you don't need it (and IPv6 support if need be).

1 Like

The advantage of Image Builder vs. a complete build from source code-- as well as the disadvantage-- is that it uses a pre-built kernel.

The advantage is that the kernel exactly matches the kernel required by the packages in the server repository. Thus you could (if you had the space, which you likely don't) install packages later online with opkg. Which reminds me, if you are building a custom image and won't be needing to install any more packages online later, you can remove opkg.

The disadvantage is that you can't make the kernel smaller by removing kmods that are built into the kernel itself, which I think includes a large part of the ppp and IPv6 support. You can remove user-space packages related to ppp and IPv6.

1 Like