/overlay 100% used, and doesnt show less usage after deleting files from /tmp

Installing and Using OpenWrt
1

I am facig an issue that after 6 or 7 days of up time, asterisk behaves in a starnge way giving these errors

[Jun 11 11:24:54] ERROR[24680]: res_pjsip_registrar.c:626 register_aor_core: Unable to bind contact 'sip:104@192.168.1.104:5060' to AOR '104'

On debuging, I found that df -h results in this

Filesystem                Size      Used Available Use% Mounted on
/dev/root                30.8M     30.8M         0 100% /rom
tmpfs                   501.7M    196.0K    501.5M   0% /tmp
/dev/loop0              223.4M    157.4M         0 100% /overlay
overlayfs:/overlay      223.4M    157.4M         0 100% /
tmpfs                   512.0K         0    512.0K   0% /dev

and du -s -h /* gives

1.2M	/bin
0	/dev
1.1M	/etc
3.8M	/lib
0	/mnt
1.4M	/overlay
0	/proc
65.7M	/rom
4.0K	/root
453.0K	/sbin
0	/sys
26.7M	/tmp
60.3M	/usr
0	/var
0	/www

on deleting some files from /tmp response for abovementioned command becomes

1.2M	/bin
0	/dev
1.1M	/etc
3.8M	/lib
0	/mnt
1.4M	/overlay
0	/proc
65.7M	/rom
4.0K	/root
453.0K	/sbin
0	/sys
232.0K	/tmp
60.3M	/usr
0	/var
0	/www

but response to df -h is same shwoing 100% usage of overlay

Filesystem                Size      Used Available Use% Mounted on
/dev/root                30.8M     30.8M         0 100% /rom
tmpfs                   501.7M    240.0K    501.4M   0% /tmp
/dev/loop0              223.4M    157.4M         0 100% /overlay
overlayfs:/overlay      223.4M    157.4M         0 100% /
tmpfs                   512.0K         0    512.0K   0% /dev

Asterisk is having the same problems, If I reboot the system then overlay would show to be using 11% or so and asterisk will behave properly

How to solve this issue whitout periodic reboots.

2

after reboot

 du -s -h /*
1.2M	/bin
0	/dev
1.1M	/etc
3.8M	/lib
0	/mnt
1.4M	/overlay
du: /proc/5790: No such file or directory
du: /proc/5801: No such file or directory
du: /proc/5803: No such file or directory
0	/proc
65.7M	/rom
4.5K	/root
453.0K	/sbin
0	/sys
84.0K	/tmp
60.3M	/usr
0	/var
0	/www

and

df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                30.8M     30.8M         0 100% /rom
tmpfs                   501.7M     84.0K    501.6M   0% /tmp
/dev/loop0              223.4M     17.6M    139.9M  11% /overlay
overlayfs:/overlay      223.4M     17.6M    139.9M  11% /
tmpfs                   512.0K         0    512.0K   0% /dev
3

Files in /tmp are in RAM. /var is a link to /tmp.

Investigate /usr, which is in flash. Likely an application is writing log files there.

3 Likes
4

As the space used reported by df and du differs so much, I would suspect some large files was deleted while kept open. Please take a look at this and see if it helps.

5

Many of the file systems that OpenWrt runs with are either pre-compressed, or can use compression-on-the-fly. As a result, df and du will show different numbers.

(Though I do agree that an improperly closed file or a power-off condition can cause disk space to "vanish" unexpectedly.)

6

Yes, you are right, the values should differ due to compression, which df will show the actual space used while du will show the uncompressed size. In that case, the values shown by du should be larger than df, which is not the case shown above.

1 Like
7 
find /overlay -exec ls -ld {} \;

at a "good" and "bad" time might provide additional insight.