Dnsmasq-full gets killed by out of memory on 21.02 (64 MB RAM)

Installing and Using OpenWrt
1

Hi, I see dnsmasq causing OOM and getting killed at least once a day after upgrading to 21.02 from 19.07. Sometimes I see multiple OOM messages in a row (with the same timestamp) but different dnsmasq PID.
My router is Asus N14U with MT7620/64 MB RAM/16 MB flash. I have dnsmasq-full which is required for domain based vpn redirection by vpn-policy-routing and openvpn. I also have stubby for DoT and adblock with ~60k rules. Every time I do 'free -m' on my router I see 12 MB available.

root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          58700       27824       22348        2964        8528       12844
Swap:             0           0           0
part of normal top output w/ parameters deleted 
Mem: 36628K used, 22072K free, 2964K shrd, 1348K buff, 6488K cached
CPU:   0% usr   0% sys   0% nic  99% idle   0% io   0% irq   0% sirq
Load average: 0.00 0.00 0.00 1/43 26958
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
26958 26910 root     R     1252   2%   0% top
26909  2150 root     S     1208   2%   0% /usr/sbin/dropbear 
26473     1 dnsmasq  S    12880  22%   0% /usr/sbin/dnsmasq 
20133     1 stubby   S     4884   8%   0% /usr/sbin/stubby
18643     1 nobody   S     4176   7%   0% /usr/sbin/openvpn 
 2011     1 root     S     3668   6%   0% /usr/sbin/uhttpd 
 1265     1 root     S     3648   6%   0% /usr/sbin/hostapd 
 1266     1 root     S     3552   6%   0% /usr/sbin/wpa_supplicant
  969     1 root     S     2288   4%   0% /sbin/rpcd
 1328     1 root     S     1788   3%   0% /sbin/netifd

I actually had ~ 8-9MB available on 19.07 and it never caused OOM. I didn't change any settings after upgrading to 21.02. OOM seem to happen randomly, even when I'm asleep and don't use internet at all. After getting killed dnsmasq just restarts and continues to work normally though so it's not really noticable for me.

OOM log example 
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.049669] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.068916] CPU: 0 PID: 26472 Comm: dnsmasq Not tainted 5.4.143 #0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.081385] Stack : 00000240 8005ab38 805c0000 8055f9a0 00000000 00000000 00000000 00000000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.098194]         00000000 00000000 00000000 00000000 00000000 00000001 83679bb0 f163a48b
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.115045]         83679c48 00000000 00000000 00000000 00000038 804eb6c4 00000010 2e342e35
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.131849]         00000000 00002d77 00000000 203a6d6d 80000000 83679b90 83e76be0 00000000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.148642]         00000000 805c4aa4 00000000 00100cca 00000010 802ab06c 00000000 806f0000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.166284]         ...
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.172172] Call Trace:
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.172195] [<8005ab38>] 0x8005ab38
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.184636] [<804eb6c4>] 0x804eb6c4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.191891] [<802ab06c>] 0x802ab06c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.199063] [<80009790>] 0x80009790
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.206180] [<80009798>] 0x80009798
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.213315] [<800cf130>] 0x800cf130
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.220465] [<800cfbcc>] 0x800cfbcc
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.227607] [<800d0380>] 0x800d0380
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.234723] [<8011042c>] 0x8011042c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.241887] [<800cb464>] 0x800cb464
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.249002] [<8012bd64>] 0x8012bd64
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.256176] [<800cbf3c>] 0x800cbf3c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.263304] [<800f6a40>] 0x800f6a40
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.270434] [<804e85a8>] 0x804e85a8
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.277551] [<800c88a4>] 0x800c88a4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.284721] [<800f6584>] 0x800f6584
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.292167] [<800fb570>] 0x800fb570
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.299546] [<8006938c>] 0x8006938c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.306742] [<80010234>] 0x80010234
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.314011] [<800784c4>] 0x800784c4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.321167] [<8012e024>] 0x8012e024
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.328304] [<80015d4c>] 0x80015d4c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.335437] [<801301f8>] 0x801301f8
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.342555]
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.345689] Mem-Info:
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] active_anon:8927 inactive_anon:32 isolated_anon:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411]  active_file:3 inactive_file:2 isolated_file:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411]  unevictable:0 dirty:0 writeback:0 unstable:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411]  slab_reclaimable:216 slab_unreclaimable:1821
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411]  mapped:2 shmem:741 pagetables:94 bounce:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411]  free:2039 free_pcp:2 free_cma:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.413936] Node 0 active_anon:35708kB inactive_anon:128kB active_file:12kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:2964kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.458053] Normal free:8156kB min:8192kB low:10240kB high:12288kB active_anon:35708kB inactive_anon:128kB active_file:12kB inactive_file:8kB unevictable:0kB writepending:0kB present:65536kB managed:58700kB mlocked:0kB kernel_stack:336kB pagetables:376kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.511821] lowmem_reserve[]: 0 0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.518601] Normal: 95*4kB (UE) 50*8kB (UME) 17*16kB (UME) 8*32kB (ME) 3*64kB (UE) 6*128kB (UE) 3*256kB (UM) 0*512kB 1*1024kB (U) 2*2048kB (ME) 0*4096kB = 8156kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.547514] 746 total pagecache pages
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.554967] 0 pages in swap cache
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.561731] Swap cache stats: add 0, delete 0, find 0/0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.572290] Free swap  = 0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.578187] Total swap = 0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.584089] 16384 pages RAM
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.589814] 0 pages HighMem/MovableOnly
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.597691] 1709 pages reserved
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.604138] Tasks state (memory values in pages):
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.613663] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.630977] [    575]    81   575      320       23    20480        0             0 ubusd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.647418] [    576]     0   576      235        9    16384        0             0 askfirst
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.664375] [    917]   514   917      326       43    20480        0             0 logd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.680643] [    969]     0   969      572       80    16384        0             0 rpcd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.696913] [   1265]     0  1265      912       42    20480        0             0 hostapd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.713867] [   1266]     0  1266      888       27    24576        0             0 wpa_supplicant
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.731897] [   1328]     0  1328      447       39    20480        0             0 netifd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.748512] [   2011]     0  2011      917      169    20480        0             0 uhttpd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.765128] [   2150]     0  2150      287       11    16384        0             0 dropbear
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.782087] [   3604]     0  3604      314       10    16384        0             0 ntpd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.798354] [  16348]     0 16348      313       10    12288        0             0 udhcpc
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.815107] [  18643] 65534 18643     1044      140    24576        0             0 openvpn
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.831921] [  20133]   410 20133     1221      299    24576        0             0 stubby
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.848535] [  26017]   453 26017     3221     2532    36864        0             0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.865321] [  26471]   453 26471     3221     2530    36864        0             0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.882105] [  26472]   453 26472     3221     2530    36864        0             0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.898886] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=dnsmasq,pid=26017,uid=453
Wed Sep 15 08:15:58 2021 kern.err kernel: [233940.920423] Out of memory: Killed process 26017 (dnsmasq) total-vm:12884kB, anon-rss:10120kB, file-rss:4kB, shmem-rss:4kB, UID:453 pgtables:36kB oom_score_adj:0
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.966512] oom_reaper: reaped process 26017 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
dnsmasq config 
root@OpenWrt:~# cat /var/etc/dnsmasq.conf.cfg01411c 
# auto-generated config file from /etc/config/dhcp
conf-file=/etc/dnsmasq.conf
dhcp-authoritative
domain-needed
no-resolv
localise-queries
read-ethers
enable-ubus=dnsmasq
expand-hosts
bind-dynamic
local-service
domain=lan
local=/lan/
server=127.0.0.1#5453
addn-hosts=/tmp/hosts
dhcp-leasefile=/tmp/dhcp.leases
stop-dns-rebind
rebind-localhost-ok
dhcp-broadcast=tag:needs-broadcast
conf-dir=/tmp/dnsmasq.d
user=dnsmasq
group=dnsmasq

dhcp-host=XX:XX:XX:XX:XX:XX,XXX.XXX.X.XXX

dhcp-ignore-names=tag:dhcp_bogus_hostname
conf-file=/usr/share/dnsmasq/dhcpbogushostname.conf


bogus-priv
conf-file=/usr/share/dnsmasq/rfc6761.conf
dhcp-range=set:lan,192.168.1.100,192.168.1.249,255.255.255.0,12h
dhcp-option=lan,252,"\n"
no-dhcp-interface=eth0.2
enable-ra
quiet-ra
2

Have you seen

adblock with 60k+ rules sounds risky.

2 Likes
3

I used to use Adblock on a BT Hub5 but ran into the same issues that Dnsmasq would shit itself and fall over.

In the end i swapped to using AdGuardHome and that solved the OOMs. However i have 128mb of ram and your 64mb is kinda tight. (I'm also using about 140k blocks.) It also does DNS over HTTP via Cloudflare. (you can set it up with any provider)

Now, your case of needing dnsmasq-full is going to be an issue. While AdGuardHome does do DNS they arent fully there with their dns implimentation. You would have to use their edge 107 client and NOT the 104 opkg version (its a shitshow and should be removed from opkg). Even then i dont know if they have implemented all the functions in their DNS that you require. (in particular their DHCP implementation is lacking and needs some polish. Its pencilled in for their 108 release)

However what you could do is do similar to what i did. Make AdGuard the primary DNS and move the onboard dns/dhcp via dnsmasq to a seperate port and forward from that. The other route is to move your requirements into AdGuard and then remove dnsmasq completely and only use AdGuard. Both are possible (I have my dnsmasq running on an alt port "just in case")

Right now i'm using 87mb from 120 reported with 28mb free. I dont know if you can squeeze what you need into 64mb of ram however. You might be better off running a raspberry pi with pihole or even adguardhome on that instead of running it on your main router?

But if you are interested in trying it:

Start there and work down (its a REALLY long thread and tbh i maybe should start a new one and type up all my experiences and get them to sticky it to replace this older thread.)

4

dnsmasq 2.86 has improved (reduced) memory usage with regards (ab)use of 'address=foo/server=' config lines for ad blocking. That should help.

There's a gotcha in that dnsmasq forks copies of itself to handle TCP queries, so try adding/increasing swap on your system - it most likely won't be used but it gives linux' VM sub-system a warm fuzzy feeling of somewhere to go if it did need to swap dnsmasq out somewhere.

2 Likes
5

There's a workaround/patch for it, but you need to compile yourself unless it's been accepted.

1 Like