Can I stack on a second overlayfs from RAM to temporarily install more packages?

I have a WPA8630P device that has 128MB of RAM but only 8MB of Flash, which isn't partitioned generously at all! I only have ~100KB of total free space available for anything.

This device serves as just an access point and so doesn't normally need extra packages anyway, but I would like to be able to make packages like tcpdump-mini or open-plc-utils available from time to time. I noticed that:

  • when the router boots up, the opkg lists are empty
  • downloading the lists doesn't take up free space
  • installing packages does take up free space

My guess is that the package lists get downloaded to RAM, and just fetched from the latest when needed. Whereas it looks like the installed packages get stored in an "overlay" filesystem that is persisted to Flash.

My question is: can I add a third overlay filesystem, but based in memory? I'm imagining a stack like:

  1. default root "squashfs" install
  2. normal tiny NVRAM "overlay"
  3. additional RAM "overlay" that I can turn on before (temporarily) installing bigger packages

Is it possible to (a) have a RAM-based overlay, and (b) stack up multiple overlays simultaneously? Is there a sequence of commands that would let me do this without rebooting? Or at least an example fstab that I could switch to on next boot?

(I don't want to have to roll my own image every time there's an update — I just want something I can occasionally paste in to put my root filesystem into
a temporary "steal space from RAM" mode from time to time.)

1 Like

That's a somewhat useful article but I don't have any USB ports (hence wanting to use RAM) and I don't want to maintain a custom build (hence not just adding the required packages directly to one) so I'm not quite following how it's relevant to my question?

I tried:

mkdir /overlay-mem
mount -t tmpfs -o size=2097152 tmp-overlay /overlay-mem
mkdir /overlay-mem/upper /overlay-mem/work
mount -t overlay -o remount,lowerdir=/,upperdir=/overlay-mem/upper,workdir=/overlay-mem/work overlayfs:tmpfs /

Which didn't error, but didn't work. When I start I have [discarding irrelevant e.g. dev/proc/tmp entries]:

/dev/root on /rom type squashfs (ro,relatime)
/dev/mtdblock5 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmp-overlay on /overlay-mem type tmpfs (rw,relatime,size=2048k)

and running the final mount doesn't change anything despite echo $? showing 0!

If I drop the remount option and run instead:

mount -t overlay -o lowerdir=/,upperdir=/overlay-mem/upper,workdir=/overlay-mem/work overlayfs:tmpfs /

My mtab does change:

/dev/root on /rom type squashfs (ro,relatime)
/dev/mtdblock5 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmp-overlay on /overlay-mem type tmpfs (rw,relatime,size=2048k)
overlayfs:tmpfs on / type overlay (rw,relatime,lowerdir=/,upperdir=/overlay-mem/upper,workdir=/overlay-mem/work)

But if I touch /testing123 it still creates a file "/overlay/upper/testing123" instead of "/overlay-mem/upper/testing123". And df still shows the same tiny available size for both my overlayfs:tmpfs and the original overlayfs:/overlay:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 3840      3840         0 100% /rom
tmpfs                    61700      1100     60600   2% /tmp
/dev/mtdblock5             320       236        84  74% /overlay
overlayfs:/overlay         320       236        84  74% /
tmpfs                      512         0       512   0% /dev
tmp-overlay               2048         0      2048   0% /overlay-mem
overlayfs:tmpfs            320       236        84  74% /

When I umount / my own futile attempt goes away, but subsequent calls just happily and silently do nothing. Why does it let me mount my "overlayfs:tmpfs" but then not really use it? Why does trying with the remount option not do anything at all?

Opkg package manager > Non-standard installation destinations

DIR="/etc /etc/init.d /bin /sbin /usr/bin /usr/sbin /usr/lib /usr/lib64"
for DIR in ${DIR}
do find /tmp${DIR} -mindepth 1 -maxdepth 1 -exec \
ln -s {} ${DIR} \; 2> /dev/null
done
2 Likes

I think I tracked down why "remount" does work. As quoted at https://unix.stackexchange.com/a/404008/42732:

[the remount option] does not change device or mount point.

I.e. I think all you can do with remount is tweak other options like ro/noatime. Of course the lowerdir/upperdir settings are options but maybe they don't count because they are rather more ambitious options?

I didn't go that route initially because it sounded unlikely to work based on the notes there. But I went ahead and got:

# opkg -d ram install tcpdump-mini
Installing tcpdump-mini (4.9.3-3) to ram...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc4/packages/mips_24kc/base/tcpdump-mini_4.9.3-3_mips_24kc.ipk
Installing libpcap1 (1.9.1-3) to ram...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc4/packages/mips_24kc/base/libpcap1_1.9.1-3_mips_24kc.ipk
Configuring libpcap1.
Configuring tcpdump-mini.
Collected errors:
 * opkg_conf_write_status_files: Can't open status file //usr/lib/opkg/status: No such file or directory.

Despite the errors, I did end up with an executable at "/tmp/usr/sbin/tcpdump". But it is a rather unhappy executable:

# /tmp/usr/sbin/tcpdump
Error loading shared library libpcap.so.1: No such file or directory (needed by /tmp/usr/sbin/tcpdump)
[…snipped several dozen errors about specific pcap symbols missing…]

I can run it via LD_LIBRARY_PATH=/tmp/usr/lib /tmp/usr/sbin/tcpdump though. So if I can convince Wireshark sshdump to do the same from my main machine then I might be able to live with this. Still curious if there's any way to remount / with a different overlay though — it's unclear to me how the ExtRoot stuff in fstab is doing anything different than I should be able to do from the command line?

The above method with symlinks should basically work.
Remounting may require to switch_root or pivot_root.
That probably needs to reinit, or run at the preinit stage.

1 Like

you may wish to checkout this for inspiration

what you are attempting to do is fairly specialized so anticipate doing most of the gruntwork yourself...

1 Like

Ah, that's probably the key difference then.

Not going to try figure out the exact OpenWrt boot sequence (there's some general Linux discussion at e.g. https://unix.stackexchange.com/questions/107828/how-is-etc-fstab-accessed-before-root-is-mounted and https://unix.stackexchange.com/questions/242747/what-is-the-purpose-of-listing-the-rootfs-in-etc-fstab one could start with in a pinch) but as the default /etc/fstab is completely empty it's probably safe to assume the initial overlayfs gets set up at a special time and setting up a new one is unlikely once the system is up and running.

Just to re-iterate and followup in the context of (one of) my original goals:

It is possible to use this half-installed tcpdump via a Wireshark sshdump session. The trick is that the "Remote capture command" sort of overrides a lot of the other options (at least in the macOS GUI of Wireshark v3.8.4) so you need a more complete command with all the interface/filter/output stuff baked in like:

LD_LIBRARY_PATH=/tmp/usr/lib /tmp/usr/sbin/tcpdump -U -i 'eth0'  -w - 'not ((host 10.0.0.142 or host clientmachine::whatever::42) and port 22)'

Or, the symlink trick added into Can I stack on a second overlayfs from RAM to temporarily install more packages? - #5 by vgaetera might avoid the need for this. Pick your poison I guess :laughing:

21.02 release builds are failing for WPA8630Pv2 due to small available space (6MiB), so it looks like the only option is to move it to the tiny target. This works surprisingly well with almost no loss in functionality, with 1MB free space: https://github.com/openwrt/openwrt/pull/4500 . Hopefully it can be backported to 21.02...

1 Like

are you sure you can't expand root partition by adding other non-contiguous partitions with mtd-concat driver?
see here for mtd-concat https://github.com/openwrt/openwrt/blob/ec6293febc244d187e71a6e54f44920be679cde4/target/linux/generic/pending-5.4/497-mtd-mtdconcat-add-dt-driver-for-concat-devices.patch

1 Like

This 6MB available already includes the os-image and file-system.

mtd-concat looks very cool though, I don't know why it wasn't considered in the initial support for this device.

Here are the partition layouts of all the models. There is a 1MB region for the PLC firmware (default-nvm and default-pib which could be used by OpenWRT. They are included in all known firmware files, so they are recoverable.

However one problem is that the only known way of reverting back to stock for this unit without opening it up is by using tplink-safeloader convert_firmware to make a sysupgrade image from the OEM firmwares. Currently that only supports restoring os-image and file-system and partition-table, so that would need to be extended to include those extra partitions (e.g. by specifying for each device). Maybe there is a way of triggering the OEM HTTP restore using uboot-envtools (which would be awesome), but noone has found it yet.

In contrast, the tiny target looks easy to achieve giving 1MB free without too many changes, so it's probably more likely at this stage. mtd-concat is worth looking into though...

afaik it is an OpenWrt kernel patch, so it is not well-known, many contributors don't know it exists. Which is why I always ask when I see these situations.

I don't think you need to add these partitions to the tplink-safeloader.
If I understand correctly the wiki entry, the powerline will still work fine since it has its own SPI chip with the firmware https://openwrt.org/toh/tp-link/tl-wpa8630p_v2#powerline_firmware_and_configuration
So the device will still be able to go back to stock firmware and work correctly without modifying the tplink-safeloader.
Those partitions will now contain garbage but according to wiki they are only used to store new PLC firmware during stock firmware upgrade procedure (WTF TP-Link?! Why waste flash like that).
During stock firmware upgrade they will be overwritten with the new PLC firmware blobs from the new firmware image anyway, regardless of previous contents.

But I don't have the device, so I can't guarantee this.

afaik you can have both at the same time, and they are two different changes so you can make two independent PRs.

1 Like

Yes, that's worth testing. If there being garbage in the PLC partitions temporarily doesn't cause any issues until the next OEM flash (e.g. PLC bricking from either the firmware or PIB?). The PLC has its own SPI flash, so it should be safe to backup everything and test this out.

1 Like

To use mtd-concat, it looks like the main blocker is that tplink-safeloader build_image() doesn't yet support splitting up the OpenWRT firmware across multiple non-contiguous partitions. build_image(), read_file() and maybe a few others would need to be changed. Looks doable but tricky.

Glad to see many others interested in this device and the storage space situation. I still haven't gotten to 100% proving this out but fwiw I did spent some more time looking into how the "ram-root" code mentioned here works:

I think the scripts in that download are more ambitious/flexible than than they strictly "need" to be for my purposes (quick spin up more space temporarily) but iiuc they do show the answer to the walls I hit earlier with just mount itself. I think the key trick is to use pivot_root, something roughly like:

mount -o noatime,nodiratime,move /proc $NEW_ROOT/proc
pivot_root $NEW_ROOT $NEW_ROOT/$OLD_ROOT
for mount_dir in /dev /sys /tmp; do
  mount -o noatime,nodiratime,move /$OLD_ROOT$mount_dir $mount_dir
done
mount -o noatime,nodiratime,move $NEW_OVERLAY /overlay

Again the above is untested by me and just cribbed from the ram-root.sh from the Extroot using (z)ram drive with the backup&restore option over NFS share - #2 by lleachii thread @anon50098793 linked to.

(Oh, and I see Can I stack on a second overlayfs from RAM to temporarily install more packages? - #8 by vgaetera mentions this trick also.)

1 Like

Here is an unofficial BETA build of 21.02.1 using the ath79-tiny patch mentioned above. This should be identical to the official release images if the patch gets merged in (and 21.02.2 is released). After installing, there is 1MB free space, and that includes LuCi.

Keep in mind this is a "BETA" release, so you if you don't want to risk anything you can wait until its officially included.

Release note:

If you are sysupgrading from the earlier ath79-generic snapshot images to this new ath79-tiny image, you will see this message and should follow its instructions:

The flash erase blocksize has changed to 4k from the 64k in previous versions, so the JFFS2 settings partition MUST be reformatted to avoid data corruption. Backup your settings before upgrading, then during sysupgrade, de-select "Keep settings" and select "Force" to continue (equivilant to "sysupgrade -n -F)

If you forget to do this and your system becomes unstable after upgrading, you can do this to format the partition and recover:

Reboot
Press RESET when Power LED blinks during boot to enter Failsafe mode
SSH to 192.168.1.1
Run "firstboot" and reboot

If anyone tests it, let me know how you get on!

The images should be safe to use, and definitely much safer than making custom cut-down images with ImageBuilder, which can be a loaded foot-gun as removing the wrong package can make the unit inaccessible. (Happened to me before...)

1 Like

I've tested your unofficial 21.02.1 build on a TP-Link WPA8630P v2.1 (EU) and it works well. Thank you!

(I upgraded from a previous development build to I haven't tested the factory image)