Filesystem is full

Installing and Using OpenWrt
1

Hi Folks

Been using a WNDR3700v1 flawlessly for many many years so you didn't hear from me on this forum before.

Right now I am unable to install or uninstall anything from Luci app or even opkg cmdline on ssh. - the overlayfs is almost full and read-only.

Filesystem                Size      Used Available Use% Mounted on
/dev/root                        2.5M      2.5M         0 100% /rom
tmpfs                           29.1M      3.0M     26.1M  10% /tmp
/dev/mtdblock5              3.5M      3.4M     92.0K  97% /overlay
overlayfs:/overlay          3.5M      3.4M     92.0K  97% /
tmpfs                          512.0K         0    512.0K   0% /dev

A few months back I installed unbound to experiment with DNS over TLS for a quick review but I started seeing freezes and hangs with crash messages (especially when tried to reverse name resolve all the IPs in the connection pool) . Also I could not track the TLS connections and I suspected it wasn't working. So I went back to dnsmasq. Installed/uninstalled twice like this and I suspect this was the trigger for the problems I having today. The 3rd attempt at using unbound a few weeks later was also similar but I have been unable to uninstall ever since. Now I wanted to install packages to samba share files via USB when I noticed the problem. Nothing can be uninstalled or installed because it is read-only.

/tmp# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock5 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (**ro**,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)

mount -orw -oremount /overlay does not work.to make it rw.

I went through the older threads but since uninstalling unwanted apps like unbound is not an option. I tried manually deleting files from /overlay/upper/usr/lib/unbound but it only removed 200k.

du -h -d1
440.0K  ./bin
0       ./dev
706.5K  ./etc
3.1M    ./lib
0       ./mnt
6.5M    ./overlay
0       ./proc
7.1M    ./rom
0       ./root
433.0K  ./sbin
0       ./sys
3.0M    ./tmp
8.1M    ./usr
674.5K  ./www
30.0M   .

Running the latest 19.0.7 lede 4.14.171 #0 Thu Feb 27 21:05:12 2020 mips GNU/Linux

I dont have too many apps installed really and not sure if a reset will help either. I just use dnsmasq and dhcp and collectd to report cpu/mem/net/wifi etc. Rebooting has not helped. Sysupgrade is also failing:

/tmp# sysupgrade ./openwrt-19.07.3-ath79-generic-netgear_wndr3700-squashfs-sysupgrade.bin
Image check failed.

What to do now?

Thanks for your advice on this.

2

Hi there, welcome to the community.
You might not have noticed but your post is not very readable. Could you please edit it and put the console outputs into code blocks </>

2nd, could you provide the output from dmesg in a code block as well - best guess I have at the moment is there are bad blocks in flash, but the real reason why it’s mounted RO should be in there somewhere

3

Mysteriously a second reboot cleared 200kb more from /overlay then I was able execute uninstall some unbound-* utilities and collectd-mod-uptime. This made the free space jump drastically to 1.9mb - especially the uptime removal did something. This is the first time I installed uptime a few weeks back and it seems to have flooded my free space along with unbound.

After this a reset was performed and the sysupgrade was able to read the bin file without an image check fail. I am now on 19.0.7.3 and installing the collectd utils has again shrunk my free space on /overlay to 700kb. But I know how to control this now to an extent.

Thanks for replying and offering to help.

1 Like
4

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.

5

The only surefire way to recover from overfilling the jffs is to re-flash a clean install, or do a firstboot aka. reset to defaults. You can first use the sysupgrade backup to have a tar file of the settings, or even tar the entire /overlay directory to have a record of what you had.

6

One last question - will building my own firmware and adding the packages I want in the build give me more free space in the /overlay in the custom distro within the 8mb limit of the hardware? From a quick glance it looks like with a custom roll the built-in packages reside in another part of the filesystem...

Is that a correct understading?

Thanks

7

Yes. Assuming there’s significantly large enough packages in the official images that you are ok with leaving out.

8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.