Shrinking overlay FS

The free space in my overlay filesystem has been shrinking without any additional packages installed (opkg free space went from 63% to 0% without any additional packages). I have a Netgear R7500v2 router and it comes with 256 mb flash.

There are two things going on which may all be related: 1 - I installed a USB drive. Not using Extroot but rather for storing videos from security cameras. I formatted it as EXT4 and had some challenges getting the system to see it. I did not explicitly create a partition since I believe that is part of the formatting process since it is created as a single partition.

I discovered that I could not df and see the filesystem and there was a bad sector 0 error on boot. However, at this point I still had plenty of space in OPKG. I ran fdsk.ext4 -c to clean it up. It recreated the journal. Rebooted and the file system was able to df although syslog bad sector errors remained (fdsk didn’t identify any errors). Immediately after this is when the overlay fs became exhausted. I assume this is where opkg is stored because that ran out of space at the same time.

2 - The router has 256 MB of flash, but only 20 MB is associated with Overlay. Most of the remaining is in tempfs, almost none of which is being used. I was wondering why that is the case and if I can/should re-allocate some of the tempfs space. That said, I believe I need to deal with issue 1 first.

tmpfs is normally ramdisk. Nothing to do with flash.

run "cat /proc/mtd" to see how the flash partitions are arranged.
E.g. in R7800 only 19 MB isnormally in use, and quite much is left unused as "reserve" as LEDE mimics the OEM firmware and does not change partitions.

Thanks @hnyman. The mtd data follows as well as a df.

root@SchaubBoarder:/sys/kernel/debug/usb/ohci# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00c80000 00020000 "qcadata"
mtd1: 00500000 00020000 "APPSBL"
mtd2: 00080000 00020000 "APPSBLENV"
mtd3: 00140000 00020000 "art"
mtd4: 00140000 00020000 "artbak"
mtd5: 00200000 00020000 "kernel"
mtd6: 01e00000 00020000 "ubi"
mtd7: 04480000 00020000 "netgear"
mtd8: 00700000 00020000 "reserve"
mtd9: 02000000 00020000 "firmware"

and

root@SchaubBoarder:/sys/kernel/debug/usb/ohci# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 3.0M      3.0M         0 100% /rom
tmpfs                   234.2M      1.9M    232.3M   1% /tmp
/dev/ubi0_1              19.6M     19.6M         0 100% /overlay
overlayfs:/overlay       19.6M     19.6M         0 100% /
tmpfs                   512.0K         0    512.0K   0% /dev

Finally, mount looks like this

root@SchaubBoarder:/# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

Interesting finding: I removed my usb disk from fstab, rebooted, and now I have 11.2 GB available. I’d really like to understand why unmounting that drive created the free space.

Make that 11.2 MB

Okay - very strange now. I'm back to 0 available on Overlay, I've stopped/disabled vsftpd which writes to the USB drive and also stopped the logging that goes to it. This makes no sense to me.

I think I finally have this figured out, but don’t know how to fix it. It appears that data on the usb was getting cached on /overlay/upper. I removed the drive, erased the cached files, and now I’m good.

That said, I wasn’t expecting (or wanting) that cache. Does anyone know how to stop that? I need to reattach the drive and can’t have overlay eaten up this way.

did you figure out how to fix it?