Filesystem is full

Hi Folks

Been using a WNDR3700v1 flawlessly for many many years so you didn't hear from me on this forum before.

Right now I am unable to install or uninstall anything from Luci app or even opkg cmdline on ssh. - the overlayfs is almost full and read-only.

Filesystem                Size      Used Available Use% Mounted on
/dev/root                        2.5M      2.5M         0 100% /rom
tmpfs                           29.1M      3.0M     26.1M  10% /tmp
/dev/mtdblock5              3.5M      3.4M     92.0K  97% /overlay
overlayfs:/overlay          3.5M      3.4M     92.0K  97% /
tmpfs                          512.0K         0    512.0K   0% /dev

A few months back I installed unbound to experiment with DNS over TLS for a quick review but I started seeing freezes and hangs with crash messages (especially when tried to reverse name resolve all the IPs in the connection pool) . Also I could not track the TLS connections and I suspected it wasn't working. So I went back to dnsmasq. Installed/uninstalled twice like this and I suspect this was the trigger for the problems I having today. The 3rd attempt at using unbound a few weeks later was also similar but I have been unable to uninstall ever since. Now I wanted to install packages to samba share files via USB when I noticed the problem. Nothing can be uninstalled or installed because it is read-only.

/tmp# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock5 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (**ro**,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

mount -orw -oremount /overlay does not make it rw.

I went through the older threads but since uninstalling unwanted apps like unbound is not an option. I tried manually deleting files from /overlay/upper/usr/lib/unbound but it only removed 200k.

du -h -d1
440.0K  ./bin
0       ./dev
706.5K  ./etc
3.1M    ./lib
0       ./mnt
6.5M    ./overlay
0       ./proc
7.1M    ./rom
0       ./root
433.0K  ./sbin
0       ./sys
3.0M    ./tmp
8.1M    ./usr
674.5K  ./www
30.0M   .

Running the latest 19.0.7 lede 4.14.171 #0 Thu Feb 27 21:05:12 2020 mips GNU/Linux

I dont have too many apps installed really and not sure if a reset will help either. I just use dnsmasq and dhcp and collectd to report cpu/mem/net/wifi etc. Rebooting has not helped. Sysupgrade is also failing:

/tmp# sysupgrade ./openwrt-19.07.3-ath79-generic-netgear_wndr3700-squashfs-sysupgrade.bin
Image check failed.

What to do now?

Thanks for your advice on this.

Hi there, welcome to the community.
You might not have noticed but your post is not very readable. Could you please edit it and put the console outputs into code blocks </>

2nd, could you provide the output from dmesg in a code block as well - best guess I have at the moment is there are bad blocks in flash, but the real reason why it’s mounted RO should be in there somewhere

Mysteriously a second reboot cleared 200kb more from /overlay then I was able execute uninstall some unbound-* utilities and collectd-mod-uptime. This made the free space jump drastically to 1.9mb - especially the uptime removal did something. This is the first time I installed uptime a few weeks back and it seems to have flooded my free space along with unbound.

After this a reset was performed and the sysupgrade was able to read the bin file without an image check fail. I am now on and installing the collectd utils has again shrunk my free space on /overlay to 700kb. But I know how to control this now to an extent.

Thanks for replying and offering to help.

1 Like

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.

The only surefire way to recover from overfilling the jffs is to re-flash a clean install, or do a firstboot aka. reset to defaults. You can first use the sysupgrade backup to have a tar file of the settings, or even tar the entire /overlay directory to have a record of what you had.

One last question - will building my own firmware and adding the packages I want in the build give me more free space in the /overlay in the custom distro within the 8mb limit of the hardware? From a quick glance it looks like with a custom roll the built-in packages reside in another part of the filesystem...

Is that a correct understading?


Yes. Assuming there’s significantly large enough packages in the official images that you are ok with leaving out.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.