How can I setup a tmpfs overlay for the root filesystem (to go on top of the existing ubifs overlay)?

I am trying to figure out how to setup a tmpfs overlay mounted to /. This would be on top of the ubifs overlay mounted to /. The contents stored in the tmpfs overlay need to not survive a reboot.

The goal here is to be able to tweak things on the router and, just in case these tweaks screw everything up, have to ability to get everything back to the same state it was when the tmpfs overlay got mounted by power cycling the router.

The device is a netgear R9000 running firmware built from egorenar's openwrt fork. Heres the relevant output from mount:

/dev/ubiblock0_0 on /rom type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime,compr=zlib,assert=read-only,ubi=0,vol=1)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)

I tried the running the following:

mkdir -p /tmp/overlay_tmpfs/upper /tmp/overlay_tmpfs/work
mount -t overlay -o rw,noatime,\
workdir=/tmp/overlay_tmpfs/work \
overlayfs:/overlay_tmpfs / 

These commands run but,unfortunately, dont do anything. Everything written to / gets saved in /overlay/upper...../tmp/overlay_tmpfs/{upper,work} have nothing in them.

Any idea what the problem is?

NOTE1: If I manually set up a layered overlay (e.g., using the commands below) then things work like they are supposed to. It is just when trying to mount the overlay to / that it doesnt work.

mkdir -p tmp/overlay_tmpfs
cd /tmp/overlay_tmpfs
mkdir -p lower upper work upper1 work1

mount -t overlay -o rw,noatime,\
workdir=/tmp/overlay_tmpfs/work \
overlayfs:/overlay_tmpfs0 /tmp/overlay/lower 

echo 'test' > /tmp/overlay_tmpfs/lower/test0
find /tmp/overlay_tmpfs -name 'test0'    # shows up at  /tmp/overlay_tmpfs/lower/test0 and at /tmp/overlay_tmpfs/upper/test0 

mount -t overlay -o rw,noatime,\
workdir=/tmp/overlay_tmpfs/work1 \
overlayfs:/overlay_tmpfs1 /tmp/overlay/lower

echo 'test' > /tmp/overlay_tmpfs/lower/test1
find /tmp/overlay_tmpfs -name 'test1'    # shows up at  /tmp/overlay_tmpfs/lower/test1and at /tmp/overlay_tmpfs/upper1/test1. Is not present at /tmp/overlay_tmpfs/upper/test

NOTE2: This behavior has been documented a few other times via questions on the openwrt forum (here and here). Unfortunately these questions never got a solution....The closest thing to a solution is a suggestion that fiddling with "something" in the preinit source files and then recompiling might work.

Thank you for pointing this solution out to me...I saw "extroot" and sort of glanced right by it.

Not gunna lie im not crazy about the whole "overlay is on a NSF share" thing. BUT, this has given me an idea that I think just might work.

# create directories and setup a dedicated tmpfs for the overlay
mkdir -p /newroot /oldroot /overlay_tmpfs
mount -t tmpfs tmpfs /overlay_tmpfs
chmod 755 /newroot /oldroot /overlay_tmpfs
mkdir -p /overlay_tmpfs/rom /overlay_tmpfs/lower /overlay_tmpfs/upper /overlay_tmpfs/work

# setup read-only bind mounts of '/rom' and '/overlay/upper' into '/overlay_tmpfs'
mount -o bind,ro /rom /overlay_tmpfs/rom
mount -o bind,ro /overlay/upper /overlay_tmpfs/lower

# setup tmpfs overlay (tmpfs <-- /overlay/upper <-- /rom) at '/newroot'
mount -t overlay -o rw,noatime,lowerdir=/overlay_tmpfs/lower:/overlay_tmpfs/rom,upperdir=/overlay_tmpfs/upper,workdir=/overlay_tmpfs/work/ overlayfs:/overlay_tmpfs /newroot

# bind-mount everything else into '/newroot'
cat /proc/mounts | awk '{print $2}' | grep -Fv '/newroot' | grep -Ev '^\/$' | while read -r nn; do mkdir -p "/newroot${nn}"; findmnt "/newroot${nn}" 1>/dev/null 2>/dev/null || mount -o bind "${nn}" "/newroot${nn}"; done

# pivot root to '/newroot'
pivot_root /newroot /newroot/oldroot
wifi up
/etc/init.d/wpad restart
/etc/init.d/network restart

# pivot root to '/oldroot'
pivot_root /oldroot /oldroot/newroot
wifi up
/etc/init.d/wpad restart
/etc/init.d/network restart

I havent tried the pivot_root yet (i wont have router access untill a couple of people wake up tomorrow morning, So im going to wait until then in case something goes wrong and I need to manually power cycle the router).

Everything else seems to work though. Since everything is bind-mounted, there are identical filesystems (excluding the /newroot and /oldroot directories) under / and under /newroot, except that stuff written to /* ends up at /overlay/upper and stuff written to /newroot/* ends up at /overlay_tmpfs/upper. This has the nice benefit that when you do the pivot_root, the only thing that the kernel needs to modify is the mount point path - the lowerdir=<...>, upperdir=<...> and workdir=<...> paths dont change.

Assuming everything works like i think/hope it will, you should also be able to switcch back and forth freely (or, rath, freely enable/disable the tmpfs overlay). pivot_root /newroot /newroot/oldroot enabled the tmpfs overlay, and pivot_root /oldroot /oldroot/newroot should disable it. I think.

EDIT: it worked!