JFFS2 getting filled 100%, 0% free

[mrgenie]

I'm having a problem with the Linksys WRTxxxx routers. I noticed this recently and I think I upgraded the firmware a few months ago, like 2 months ago. Before that been using openWRT for years without ever experiencing this.

1900ACS OpenWrt SNAPSHOT r8917-ae622c93b3 / LuCI Master (git-18.363.34501-1d3a873)

For example the 1900ACS has over 500Megs of memory. According to my LUCI 60% or more is always free.

/dev/root has 22MB mounted on rom
tmpfs has 255MB mounted on tmp
/dev/ubi0_1 has 6MB mounted on overlay
overlayfs:/overlay 6MB mounted on /
ubi1:syscfg 30MB mounted on /tmp/syscfg
tmpfs 0.5MB mounted on /dev

When I reboot within the first 2-.3 minutes out of the 6MB for the overlay 90% is free.

Internet comes up, vpn comes up, 90% free

But after 10 minutes 60% is free

after 30 minutes 0%is free and I basically can't do anymore can't change firewall settings or anything else because the jffs2 (overlay) is 100% used.

I also upgraded to a newer release thought maybe it's a bug and got fixed meanwhile.

But still there even with latest release.

I can't remember I had this problem with previous releases.

So a few questions come to my mind:
is this normal?
Overlay over time getting filed supposed todo that?
Why only 6MB if I have hundreds of megs for free anyway? Why not using the free space.

Any possible reason a package might cause this? if so, which package would be the first to go after?

Any way maybe a script could flush the overlay every 5minutes to keep it from getting filled?

I need to add: even doing something like:

vi /etc/config/openvpn
delete a few lines
:wq

I get an error like:
"/etc/config/openvpn" E509: Cannot create backup file (add ! to override)

And when I do :wq!

I get an error like:
"/etc/config/openvpn" E514: write error (file system full?)
WARNING: Original file may be lost or damaged
don't quit the editor until the file is successfully written!

again, the tmpfs is 4% used and memory 60% empty as well..

Just the jffs2, overlay is getting full over time preventing me from doing anything.

[hnyman]

@mrgenie

is this normal?
Overlay over time getting filed supposed todo that?
Why only 6MB if I have hundreds of megs for free anyway? Why not using the free space.

Any possible reason a package might cause this? if so, which package would be the first to go after?

No.
No.
You are mixing RAM memory and flash. Your router does not have "over 500 megs" of flash.
First guess is overzealous logging by some package.
No idea about which package to suspect, as you have not told anything about your installed packages. But using "du" on /overlay should pretty quickly give you clues which files grow and fill the overlay.

Regarding flash size, 1900acs has 128 MB that is divided to two dual firmwares plus syscfg

[mrgenie]

Additional Information:

On a wrt1200AC: LEDE Reboot SNAPSHOT r5113-2af10c3 / LuCI Master (git-17.291.51172-a98548e)

Same memory, same packages
doing df gives me:
/dev/root 8MB mounted on /rom
tmpfs 256MB mounted on /tmp
/dev/ubi0_1 19MB mounted on /overlay
overlayfs:/overlay 19MB mounted on /
ubi1:syscfg 30MB mounted on /tmp/syscfg
tmpfs 0.5MB mounted on /dev

So why do I have a much larger overlay on the old 1200AC vs the newer 1900ACS?
And why it only takes 2.5MB from the older initially and 2.5MB after 1 day but 2.5MB on the 1900ACS initially but 100% 6MB after several minutes?

It looks like some "leak"

[mrgenie]

ahh, logging of information doesn't go to the tmpfs?

Thanks for clearing that one. I thought "of course" logging goes to tmp file system...

Indeed I have some extra logging going on on VPN so I'll switch that off to see if that's it.

[hnyman]

@mrgenie

ahh, logging of information doesn't go to the tmpfs?

Depends on your config.
For many packages you can select the log directory.
I assume that you have enabled verbose debug logging for some pakcage that writes its log to flash (overlay) instead of /tmp

[hnyman]

@mrgenie

1900ACS OpenWrt SNAPSHOT r8917...
/dev/root has 22MB mounted on rom
overlayfs:/overlay 6MB mounted on /

@mrgenie

On a wrt1200AC...
Same memory, same packages
/dev/root 8MB mounted on /rom
overlayfs:/overlay 19MB mounted on /

Based on your data, the router do NOT have the same packages.
On your 1900acs, the firmware image takes 22MB which leaves only 6 MB for overlay,
while on your 1200AC the firmware only takes 8 MB leaving 19 MB for overlay.

(note that the total 22+6 or 8+19 is roughly the same)

[hnyman]

Please also look at wiki for your device, as it explains how the 128 MB flash gets actually divided into several smaller pieces and only about 30 MB is usable for rom+overlay as "rootfs":

https://openwrt.org/toh/linksys/linksys_wrt1900acs#flash_layout

[mrgenie]

hnyman thank you for your very extensive explanation to such a trivial problem I had.

You're 100% absolutely right with everything you answered!

I told openvpn to log to /tmp/log/openvpn analog to default logging of ddns/samba/netdata logging and problem solved

Thank you AGAIN for helping me out. Somehow you always push me in the right direction.

PS from old compilation logs I found out the 1200AC and 1900ACS indeed as you suggested have different packages.. thought they were roughly the same but as you suggested correctly a lot of unused stuff I wasn't aware about got into my recent compilations..