Problem Statement
Router sometimes fails to resolve hostname, results in "no internet error" when browsing internet (.e.g, accessing www.youtube.com ).
However, the client, including the router are still able to ping the Ipv4 address of the internet server.
OpenWRT Version:
AC1750 running Powered by LuCI openwrt-22.03 branch (git-22.245.77528-487e58a) / OpenWrt 22.03.0 r19685-512e76967f
Trouble shoot
Using a windows laptop with manually configured DNS '8.8.8.8', I can ping youtube.com and it will automatically resolve the hostname.
C:\>ping youtube.com
Pinging youtube.com [142.250.191.46] with 32 bytes of data:
Reply from 142.250.191.46: bytes=32 time=11ms TTL=116
Reply from 142.250.191.46: bytes=32 time=12ms TTL=116
Reply from 142.250.191.46: bytes=32 time=12ms TTL=116
Reply from 142.250.191.46: bytes=32 time=14ms TTL=116
Ping statistics for 142.250.191.46:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 11ms, Maximum = 14ms, Average = 12ms
C:\>nslookup youtube.com
Server: dns.google
Address: 8.8.8.8
Non-authoritative answer:
Name: youtube.com
Addresses: 2607:f8b0:4005:80e::200e
142.251.214.142
Troubleshooting in OpenWRT, nslookup results in server refused.
root@GenericRouter:~# ping 142.250.192.46
PING 142.250.192.46 (142.250.192.46): 56 data bytes
64 bytes from 142.250.192.46: seq=0 ttl=58 time=237.418 ms
64 bytes from 142.250.192.46: seq=1 ttl=58 time=238.128 ms
64 bytes from 142.250.192.46: seq=2 ttl=58 time=236.965 ms
64 bytes from 142.250.192.46: seq=3 ttl=58 time=237.119 ms
64 bytes from 142.250.192.46: seq=4 ttl=58 time=238.349 ms
^C
--- 142.250.192.46 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 236.965/237.595/238.349 ms
root@GenericRouter:~# ping youtube.com
ping: bad address 'youtube.com'
root@GenericRouter:~# nslookup youtube.com
Server: 127.0.0.1
Address: 127.0.0.1:53
** server can't find youtube.com: REFUSED
** server can't find youtube.com: REFUSED
Comments
This problem will resolve itself if I reboot the router, but after a few days (sometimes may be weeks), it comes back again.
root@GenericRouter:~# cat /etc/resolv.conf
search lan
nameserver 127.0.0.1
nameserver ::1
Comment
I tired enabling DNS queries log https://openwrt.org/docs/guide-user/base-system/dhcp_configuration#logging_dns_queries
it tells me no space left on device:
uci set dhcp.@dnsmasq[0].logqueries="1"
uci commit dhcp
/etc/init.d/dnsmasq restart
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: no lease, failing
sh: write error: No space left on device
Could the DNS problem related to no space left? But that doesn't make sense.
I have a 64Gb USB drive mounted for logs and SWAP
root@GenericRouter:~# free
total used free shared buff/cache available
Mem: 122328 45168 37972 8048 39188 28972
Swap: 422284 55352 366932
root@GenericRouter:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.0M 4.0M 0 100% /rom
tmpfs 59.7M 59.7M 0 100% /tmp
/dev/mtdblock9 8.9M 7.0M 1.9M 78% /overlay
overlayfs:/overlay 8.9M 7.0M 1.9M 78% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 56.0G 992.0K 53.1G 0% /mnt/sda1
root@GenericRouter:~# uci show fstab; block info
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='5'
fstab.@global[0].check_fs='0'
fstab.@global[0].anon_mount='1'
fstab.@mount[0]=mount
fstab.@mount[0].target='/mnt/sda1'
fstab.@mount[0].uuid='c2594417-e1ef-4d38-985d-9a7583666e39'
fstab.@mount[0].enabled='1'
fstab.@swap[0]=swap
fstab.@swap[0].device='/dev/sda2'
fstab.@swap[0].enabled='1'
/dev/mtdblock8: UUID="246e008f-cce5fe3e-a4b92bef-be0c6219" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock9: MOUNT="/overlay" TYPE="jffs2"
/dev/sda1: UUID="c2594417-e1ef-4d38-985d-9a7583666e39" VERSION="1.0" MOUNT="/mnt/sda1" TYPE="ext4"
/dev/sda2: UUID="ae3c1777-7a70-4fd0-a14d-0aefef420711" LABEL="SWAP" VERSION="1" TYPE="swap"
root@GenericRouter:~#
powtrix
December 12, 2022, 2:00am
5
No left space ..
Define log to external disk and restart the router:
mkdir -pv /mnt/sda1/logs
uci set system.@system[0].log_proto='udp'
uci set system.@system[0].conloglevel='8'
uci set system.@system[0].cronloglevel='5'
uci set system.@system[0].log_file='/mnt/sda1/logs/system.log'
uci commit
reboot
powtrix
December 12, 2022, 2:05am
6
You should use this disk as root:
1 Like
I already configured log files to be saved on /mnt/sda1
But regardless, I did a reboot and it started to work ok
root@GenericRouter:~# cat /etc/resolv.conf
search lan
nameserver 127.0.0.1
nameserver ::1
root@GenericRouter:~# nslookup youtube.com
Server: 127.0.0.1
Address: 127.0.0.1:53
Non-authoritative answer:
Name: youtube.com
Address: 142.250.189.238
Non-authoritative answer:
Name: youtube.com
Address: 2607:f8b0:4005:812::200e
root@GenericRouter:~# free
total used free shared buff/cache available
Mem: 122328 35652 68356 232 18320 52832
Swap: 422284 0 422284
root@GenericRouter:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.0M 4.0M 0 100% /rom
tmpfs 59.7M 232.0K 59.5M 0% /tmp
/dev/mtdblock9 8.9M 7.0M 1.9M 78% /overlay
overlayfs:/overlay 8.9M 7.0M 1.9M 78% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 56.0G 992.0K 53.1G 0% /mnt/sda1
root@GenericRouter:~# uci show dhcp.@dnsmasq[0]
dhcp.cfg01411c=dnsmasq
dhcp.cfg01411c.domainneeded='1'
dhcp.cfg01411c.localise_queries='1'
dhcp.cfg01411c.rebind_protection='1'
dhcp.cfg01411c.rebind_localhost='1'
dhcp.cfg01411c.local='/lan/'
dhcp.cfg01411c.domain='lan'
dhcp.cfg01411c.expandhosts='1'
dhcp.cfg01411c.authoritative='1'
dhcp.cfg01411c.readethers='1'
dhcp.cfg01411c.leasefile='/tmp/dhcp.leases'
dhcp.cfg01411c.localservice='1'
dhcp.cfg01411c.resolvfile='/tmp/resolv.conf.d/resolv.conf.auto'
dhcp.cfg01411c.sequential_ip='1'
dhcp.cfg01411c.dhcpleasemax='30'
After removing the manually configured DNS on windows PC, nslookup used the router's DNS server successfully,
C:\>nslookup youtube.com
Server: GenericRouter.lan
Address: 10.0.0.1
Non-authoritative answer:
Name: youtube.com
Addresses: 2607:f8b0:4005:80b::200e
142.250.188.14
And this time, there is no issue restarting dnsmasq
root@GenericRouter:~# uci set dhcp.@dnsmasq[0].logqueries="1"
root@GenericRouter:~# uci commit dhcp
root@GenericRouter:~# /etc/init.d/dnsmasq restart
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: no lease, failing
root@GenericRouter:~#
Very strange.
I'll look into extboot should it continues to happen.
ahh,
the tmpf was completely eaten up before the reboot.
Before reboot
root@GenericRouter:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.0M 4.0M 0 100% /rom
tmpfs 59.7M 59.7M 0 100% /tmp
/dev/mtdblock9 8.9M 7.0M 1.9M 78% /overlay
overlayfs:/overlay 8.9M 7.0M 1.9M 78% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 56.0G 992.0K 53.1G 0% /mnt/sda1
After reboot
root@GenericRouter:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.0M 4.0M 0 100% /rom
tmpfs 59.7M 232.0K 59.5M 0% /tmp
/dev/mtdblock9 8.9M 7.0M 1.9M 78% /overlay
overlayfs:/overlay 8.9M 7.0M 1.9M 78% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 56.0G 992.0K 53.1G 0% /mnt/sda1
Let's see what is filling up the tmp folder
root@GenericRouter:~# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 4.0M 4.0M 0 100% /rom
tmpfs 59.7M 1.2M 58.6M 2% /tmp
/dev/mtdblock9 8.9M 7.0M 1.9M 78% /overlay
overlayfs:/overlay 8.9M 7.0M 1.9M 78% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/sda1 56.0G 992.0K 53.1G 0% /mnt/sda1
root@GenericRouter:~# du /tmp | sort -n
0 /tmp/.uci
0 /tmp/dnsmasq.d
0 /tmp/etc/uhttpd
0 /tmp/extroot
0 /tmp/lib
0 /tmp/lib/misc
0 /tmp/lock
0 /tmp/overlay
0 /tmp/run/collectd
0 /tmp/run/hostapd
0 /tmp/run/rpcd
0 /tmp/run/ubus
0 /tmp/run/wpa_supplicant
0 /tmp/shm
0 /tmp/spool
0 /tmp/spool/cron
0 /tmp/tmp
4 /tmp/hosts
4 /tmp/resolv.conf.d
4 /tmp/run/dnsmasq
8 /tmp/etc
8 /tmp/sysinfo
12 /tmp/state
40 /tmp/run
1088 /tmp/log
1180 /tmp
powtrix
December 12, 2022, 2:41am
9
consider using external disk as root, 1.9mb is eminent risk
1 Like
Turns out it was ulogd that's eating the ram
root@GenericRouter:/tmp# find /tmp -size +10k | xargs ls -lh
-rw-r--r-- 1 root root 4.4M Dec 11 19:30 /tmp/log/ulogd_syslogemu.log
-r--r--r-- 1 root root 57.0K Dec 11 18:56 /tmp/luci-modulecache/6C7563692E64697370617463686572
-r--r--r-- 1 root root 19.0K Dec 11 18:56 /tmp/luci-modulecache/6C7563692E68747470
-r--r--r-- 1 root root 24.9K Dec 11 18:56 /tmp/luci-modulecache/6C7563692E737973
system
Closed
December 22, 2022, 3:36am
11
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.