Overlay & jffs no longer used on Ubiquiti EdgeRouter build?

Does the 19.07.8 build for the Ubiquiti EdgeRouter-Lite (ER-3) no longer use the overlay mounts to store changes in JFFS?

The story behind the question:

I've been happily operating the EdgeRouter-Lite (ER-3) under 19.07.7 for several months.
Filesystems layout looked like the documentation (JFFS via overlay+rom=rootfs)

19.07.8. is released.
I build a new thumbdrive (this device uses a removable USB stick as its filesystem) using 19.07.8.
Boot the device using the new version.
Runs fine, but now the filesystem layout looks like a conventional Linux system, with a rootfs partition mounted directly at /, no overlay, no jffs to be seen.
Did I mess up, or is this by design?

There have been no changes to the file defining the root image for that device in years. For example it is identical to the one used in 18.06
old https://github.com/openwrt/openwrt/blob/openwrt-18.06/target/linux/octeon/image/Makefile
current https://github.com/openwrt/openwrt/blob/openwrt-19.07/target/linux/octeon/image/Makefile

From the installation instructions, this device always had a smaller partition on the flash drive for the kernel and a larger partition for the rootfs

I'm not sure how can you follow those instructions and end up with a rom+overlay.
It's literally extracting kernel, moving it to first partition, then extracting rootfs contents in rootfs partition.
This device does not seem to have overlay + rom, just a ext4 rootfs.

Note that this type of setup is normal for many OpenWrt devices that use SD cards or eMMC for storage. Or for x86 target (normal PCs) where you can boot OpenWrt from a USB flash drive.

Also JFFS is not a valid filesystem for a USB flash drive.

HM, I had a look at sysupgrade (firmware upgrade) procedure here https://github.com/openwrt/openwrt/blob/openwrt-19.07/target/linux/octeon/base-files/lib/upgrade/platform.sh
I'm not sure I'm understanding it correctly but it seems the rom+overlay may be created when you do a sysupgrade (normal firmware upgrade) from Luci web interface or from commandline.

Try doing a firmware upgrade and see if something changes.
If rom+overlay does not appear then this device never had this feature, and I read the above code wrong.

I'm running 19.07.7 on an ER-Lite. The file system is exactly what x86 squashfs does. The root ROM is a squashfs; the writeable part is a f2fs overlay mounted in the same second partition after the squashfs.

2 Likes

Hm, so it's probably like I was speculating, the manual install instructions make you do a system without a rom+overlay because it is much easier to do for end user, but the first time you run a firmware upgrade the sysupgrade will create the rom+overlay.

Thanks, both of you. Those are very helpful insights.
(BTW, I mistakenly said 'jffs' when I meant 'f2fs', sorry for the misleading detail)

Yes, I installed .07.8 as the docs say, with rootfs going on an ext4 partition using rsync (as files) not 'dd' as an image.
And I seem to recall those same instructions (for .07.7, which still called it 'LEDE') saying to dd the 'root' image over the top of the 2nd partition.
That might explain why under .07.7 I saw a mount table like this:
(19.07.7:)

/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/loop0 on /overlay type f2fs (rw,lazytime,noatime,background_gc=on,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

and why under newly-built .07.8, with the instructions now having me mount 'root' as a loopback, then rsync its contents onto the ext4 filesystem.
The mount table at runtime looks like / is on an ext4, just like on a RasPi, x86, etc.

19.07.8
root@border:~# mount
/dev/root on / type ext4 (rw,noatime,data=ordered)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

I'm thinking the install instructions changed on the ER-Lite's page between .7 and .8?

While everyone can edit the wiki, edits are logged and versioned. You can go back in time in that article from this page (it is accessible by clicking on the clock spinner icon in the side bar on the right) https://openwrt.org/toh/ubiquiti/edgerouter_lite?do=revisions

Going back in time to
2021/04/20 14:33 Show differences to current revisions Ubiquiti EdgeRouter Lite – ↷ Page name changed from toh:ubiquiti:edgerouter.lite to toh:ubiquiti:edgerouter_lite tmomas ( )

we see that there are two sets of inconsistent install instructions, the first one is Installation and ends with

sudo dd if=sysupgrade-erlite/root of=/dev/sdc2 bs=4096

which is the command that flashes raw the "root" that you remember.

yet in the general summary above that code block it says you are supposed to rsync from root to the ext4 partition.

The instruction set after that, "Installing Lede 17.01.2 using Ext4" is the one that is more or less the same as the current instructions in the page, and is actually doing the resync.

So as you can see from the changes after that, the page was edited by jeremynd01 that has dropped the first set of instructions and made it a single thing.

I do not know what instructions are correct, I do not own the device.

As I said above I think that just doing a firmware upgrade from Luci or command line should fix the problem and restore the root/overlay setup.

If someone of you can test and write down here a set of instructions that make sense and try also a firmware upgrade (sysupgrade) from Luci web interface or command line, I can edit the wiki to fix the documentation.
Or you can also do it yourself if you have an account.

1 Like

Thanks. I'm quite happy with it the way it is: / on ext4.
I can pull the USB media out and duplicate it for testing, examine it offline, etc.
Hoping that because this device is running on a USB thumbdrive (actually, an adapter holding a C10 SDcard), it's not as vulnerable to wear as a flash chip might be.
I'd be happy with just letting it continue to be as it is.

I have a SDcard prepped with .07.8 with no local configuration applied, no mods or changes - it's as the docs currently say to do it.
It runs with / on an ext4 filesystem, no overlays or f2fs.
I will boot that up tomorrow, do a Luci web 'upgrade' on it (to the same version, since nothing's newer) and report here whether it changes / to overlay+f2fs, or remains as an ext4.

If you have some time to experiment on the router, could you see if booting up the initramfs then doing a sysupgrade will make a proper install? If that works it would be a very simple way to install since you only need to write two files (kernel and md5) to the FAT partition on the drive. OpenWrt itself would take care of building out the rootfs and overlay.

Since I have only the one device, and it's my production border router, it's hard to pull it out of service for very long for extensive testing.
Though I'm pretty technical by some standards, tracing the actions during the initramfs stages of boot is a bit outside my familiarity. I may not be experienced enough in that realm to provide the answer you're asking for.

Upon re-re-rereading this thread, and considering bobafetthotmail's findings in the docs (I think he found the root cause of my confusion), I'm of the belief that it was always the author's intent to have this platform running with root on an ext4 partition, and not use f2fs or overlays (since this isn't a conventional flash-based device), and that the use of 'dd' to plant the squashfs on the SDcard was a remnant that someone had forgotten to remove when documenting a setup for this device).
The outdated instruction block that's since been removed is what led me to build it as f2fs for 19.07.7, but everything I've done since those instructions were updated for 07.8 has worked perfectly, makes sense, and results in an ext4 root filesystem, all of which seems appropriate for this device's storage technology.
I'll still check to see if the Luci-invoked upgrade scripts convert the layout to the overlay+f2fs layout used on flash-based devices, but that may be something the person(s) who manage this device's build want to look into - it may be unintended. (just my guess)

I did this test:
SDcard prepared with 19.07.8 according to the docs in place as of this date.
Booted system. Runs with / mounted on sda2, ext4 fs.
Installed upgrade using Luci, using openwrt-19.07.8-octeon-erlite-squashfs-sysupgrade.tar
Upgrade accepted, device self-reboots.
Now the filesystem layout resembles that of flash-based devices, with /rom, /overlay, overlayfs composing the / filesystem.
As mentioned above, I suspect (but am unqualified to assert more strongly) that this outcome is not the intent of those who maintain this device's build, but rather just the outcome assumed by the upgrade script author(s) which may need an exception for this USB-storage based device.

Have a Lite myself here and I don't remember doing any rsyncing whatsoever. Just extracting kernel + rootfs onto the USB stick Ubiquiti shoved inside the enclosure.

Ok, thanks for doing the testing. So the instructions as current in the wiki are not wrong, but result in a situation that does not survive the firmware update.

One last test, to see what is current filesystem. You can do easily by just connecting with ssh or from Luci interface you click on System and then on Mount Points and look at the Mount Points table.

This is an example of a x86 router (a mini PC) using a similar setup as said by mk24, which is one of the standard setups for OpenWrt.
"The root ROM is a squashfs; the writeable part is a f2fs overlay mounted in the same second partition after the squashfs"

Mount Points
Mount Points define at which point a memory device will be attached to the filesystem
Enabled	Device	                            Mount point	     Filesystem	Mount options	Run filesystem check	
UUID: 48d22ccb-bbb3-40be-910a-719ff25cd473 (/dev/loop0, 462.56 MB)	/overlay	auto (f2fs)	defaults	No	
UUID: 84173db5-fa99-e35a-95c6-28613cc79ea9 (/dev/sda1, 64.00 MB)	/boot	auto (ext4)	defaults	No	
UUID: be39bc79-c2f51995-23635f58-f8213e30 (/dev/sda2, 512.00 MB)	/rom	auto (squashfs)	defaults	No	

and this is the same thing, but connected from ssh commandline and sending block info command


root@OpenMPTCProuter:~# block info
/dev/loop0: UUID="48d22ccb-bbb3-40be-910a-719ff25cd473" LABEL="rootfs_data" VERSION="1.14" MOUNT="/overlay" TYPE="f2fs"
/dev/sda1: UUID="84173db5-fa99-e35a-95c6-28613cc79ea9" LABEL="kernel" VERSION="1.0" MOUNT="/boot" TYPE="ext4"
/dev/sda2: UUID="be39bc79-c2f51995-23635f58-f8213e30" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"

and this is the result of mount command

root@OpenMPTCProuter:~# mount
/dev/root on /rom type squashfs (ro,relatime)
devtmpfs on /rom/dev type devtmpfs (rw,relatime,size=1023232k,nr_inodes=217962,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/loop0 on /overlay type f2fs (rw,lazytime,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=reuse,fsync_mode=posix)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
/dev/sda1 on /boot type ext4 (rw,noatime)
/dev/sda1 on /boot type ext4 (rw,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
mountd(pid9136) on /tmp/run/blockd type autofs (rw,relatime,fd=7,pgrp=1,timeout=30,minproto=5,maxproto=5,indirect)

In case you were wondering where did the f2fs overlay /dev/loop0 come from, it is physically in the second partition (/dev/sda2) appended at the end of the squashfs filesystem (that is much smaller than the 512 Mb partition) in a non-standard way so the tools don't see it.
It's usually created by the OpenWrt system on first boot or after firmware upgrade, so you do not need to create it.

The updgrade script is part of OpenWrt's support for this device and must be in sync with how OpenWrt generates a firmware image for this device.
The same people maintain that.

People updating the wiki may or may not have fully understood how OpenWrt works, sometimes it's just people trying to clean up confusion and old information.

It seems the right install instructions were the ones where you just copy the kernel to the boot folder and then flash the "root" file aka the "/rom" to the second partition like the older (if a bit confusing) instructions.
This means that on boot the OpenWrt system will notice that there is no read-write partition and create it.

I'll have to edit the wiki to fix that

1 Like

Thank you for continuing to look into this (and for helping me understand the reasons for what it's doing).
Below is from the layout which resulted from following the 07.8 (current) install instructions, and which is running in prod at the moment.

...
 -----------------------------------------------------
 OpenWrt 19.07.8, r11364-ef56c85848
 -----------------------------------------------------
root@border:~# mount
/dev/root on / type ext4 (rw,noatime,data=ordered)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
root@border:~# block info
/dev/sda1: UUID="801D-C9AB" VERSION="FAT32" TYPE="vfat"
/dev/sda2: UUID="f294e4c7-9311-4345-8016-e8eefd910fff" LABEL="rootfs" VERSION="1.0" MOUNT="/" TYPE="ext4"

(FWIW, I'm quite happy with this layout, and would prefer to just leave it as-is since it makes offline inspection far simpler. Since this machine uses removable USB storage instead of flash, I'm unclear what advantage may result from changing it to the f2fs+overlay scheme, but that's perhaps a different topic that I need to go learn more about.
And yes, I was definitely interested to know from where did the f2fs overlay /dev/loop0 come from. I'd suspected it was using the remainder of partition2 after the squashfs portion, but could not find the script that calculates where in that partition the /overlay portion began, and/or how it avoided trampling the squashfs portion.)

You can run your device like this, but if you want to keep this after you do a firmware upgrade from Luci or commandline (i.e. you do not pull the USB drive and reinstall manually), something must be changed in the OpenWrt build system.

The image building and firmware upgrade scripts will have to be updated to provide (and do proper firmware upgrade with) an additional firmware image with ext4, like it is done for x86 target for example, where you can choose between these two types.
So I assume it's relatively easy as most things can just be copy-pasted from there.
But since an important feature does not work with ext4 (see below), this may not be worth doing, or can be rejected by OpenWrt developers.

The main advantage is that you can do failsafe mode or firmware reset, by pressing the reset button, similar to what you do on a normal router (and to what the stock firmware did on this device).
You can enter failsafe mode if you are connected to console/serial port during boot too, you will see a "press a key now to enter Failsafe mode" in the console.
You can also do a "factory reset" from Luci web interface or from command line.

That works because the /rom aka squashfs is a read-only filesystem that contains all defaults and all default applications as they were when you installed (or did the firmware upgrade). So OpenWrt can just boot and ignore (or delete and re-create) the /overlay to return to a "known good" state temporarily with failsafe (or permanently with factory reset).

With a fully writeable setup like ext4, that does not work and if you screw up config or break the system in other ways you need to pull the USB drive and do maintenance or reinstall.

EDIT:
an addendum: the f2fs filesystem was created to be used on the type of managed flash used in USB drives, SD cards and SSDs (and eMMC which is a miniaturized SSD used for smartphones and other embedded devices), so it's not "wrong", it should be better than ext4, although OpenWrt does not write much to disk so either are fine.

f2fs is not in the same category as JFFS2 and UBI/UBIFS, that are designed to work on raw (unmanaged) flash like it is used in most embedded devices.

https://f2fs.wiki.kernel.org/

3 Likes

Thanks again for taking the time to provide such a thorough and helpful explanation. I believe this thread will be valuable to others who want to better understand how this scheme(s) benefits their system operation.

Yes, I would very much like the option (e.g. just a checkbox during self-upgrade) that allows one to:
a) keep the fs layout as-is, ext4 (and care for it yourself), or
b) convert it to the overlay scheme (and have the ability to 'reset to defaults' with a button).

Easy fallback to 'factory' settings is a great feature. We're all accustomed to having it in embedded-system devices like a Linksys box.
But in the case of this machine (which to me resembles a RasPi more than a Linksys, due to its removable media) I enjoy being able to pull the thumbdrive and swap a different one in with ease, even though the risk of goofing up the production filesystem would mean I may have to rebuild from scratch.
Having lived with that reality on every other computer I've managed, it feels quite familiar here, too. Perhaps others would appreciate having the choice as well.

A sidebar-question: How does the script(s) discover &/or create the two-volume sub-partition within the 2nd partition that I create during install? That is, under the previous install instructions where one would 'dd' the squashfs onto (e.g.) sdx2 during initial setup, but then somehow the running system carves sda2 into two more partitions. When I examine the previous OS version while running, it references UUIDs for the /rom and /overlay partitions that I am not able to see when I examine the media offline using my laptop (e.g. 'blkid's report).

I understand, but since it's still something that requires some work, testing and also convincing the devs to merge the contribution, I'll give this a pass.
I don't have or care much about that device.
If you want to ask for volunteers to do and contribute this change you should open a thread in the Development section (linking back to this thread for more info about the issue).

My "official" role is wiki admin so I'll fix the instructions on the wiki shortly, so that people know how to install this "the proper way" and that there is a "cheeky" way to install it in ext4 mode, with the limitations we discussed.

It's not a script, shell scripts are not able to do these kinds of "hack the planet" tricks.
It is done by a C application called mount_root and part of the package fstools which is also developed by OpenWrt developers.
This tool reads the partition, finds where the squashfs filesystem ends, calculates the space occupied and applies this offset to create a loop device /dev/loop0 that is in fact mapped to the free space left in that partition.
Then if it's empty calls the mkfs.f2fs to format it.
Then mounts it as overlay.
This application does a bunch of other disk-related operations during firmware reset or firmware upgrade and has also code for doing similar tricks on embedded devices as well. Fun and games.

This tool's source is not on the github mirror, there you find only the package definition and the init scripts https://github.com/openwrt/openwrt/tree/master/package/system/fstools
The source is on the git server of OpenWrt, https://git.openwrt.org/?p=project/fstools.git;a=tree
where you also find other custom software written for OpenWrt like the init system Procd.

I have limited understanding of how the whole application actually works under the hood since I only contributed a tiny bit to it many years ago https://git.openwrt.org/?p=project/fstools.git;a=commit;h=94a5b0ad8d53f024f036c3526b48c34ebbd66a2f
and I'm not going to invest the time to actually understand all of it properly unless I REALLY need to do that, so bear with me.

In this part of the code https://git.openwrt.org/?p=project/fstools.git;a=blob;f=libfstools/rootdisk.c;h=9f2317f14e8d8f12c71b30944138d7a6c877b406;hb=HEAD
it has functions to detect the space available after the squashfs
look at this function code
static struct volume *rootdisk_volume_find(char *name)
it is clearly written to be called in a loop by other code and it first filters out block devices with names like rootfs_data, mtdblock and ubiblock (they are used by OpenWrt to mean specific things), then after it has found the right partition it will calculate an offset, the space occupied by the squashfs filesystem and then return this value to its caller

 125         p->offset = le64_to_cpu(sb.bytes_used);
 126         p->offset = ((p->offset + (ROOTDEV_OVERLAY_ALIGN - 1)) &
 127                      ~(ROOTDEV_OVERLAY_ALIGN - 1));
 129         return &p->v;

Then in a function after it, there is the code to create the loop device, aka /dev/loop0
148 static int rootdisk_create_loop(struct rootdev_volume *p)
and we see again our old friend p->offset which in the code above was the variable containing the offset, aka space occupied by current squashfs filesystem.

Then in this other file in the source code https://git.openwrt.org/?p=project/fstools.git;a=blob;f=libfstools/common.c;hb=d4f01298105bb8c97e7ac0cad0e78f0ffe261354
we find the function used to actually format the read-write file system
100 int block_volume_format(struct volume *v, uint64_t offset, const char *bdev)

and here we see it's just deciding on f2fs vs ext4 and then sending plain console commands to call mkfs tools to do the job.

 110         case FS_NONE:
 111                 ULOG_INFO("overlay filesystem in %s has not been formatted yet\n", v->blk);
 112                 if (use_f2fs(v, offset, bdev))
 113                         snprintf(str, sizeof(str), "mkfs.f2fs -q -l rootfs_data %s", v->blk);
 114                 else
 115                         snprintf(str, sizeof(str), "mkfs.ext4 -q -L rootfs_data %s", v->blk);
 116
3 Likes

I restate my gratitude:
You folks have been more than kind in not only answering my post's questions, but also in going the extra mile to further explain what relative newcomers like myself would otherwise have a difficult time learning by other means.
I'm in your debt.

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.