Hi, I see dnsmasq causing OOM and getting killed at least once a day after upgrading to 21.02 from 19.07. Sometimes I see multiple OOM messages in a row (with the same timestamp) but different dnsmasq PID.
My router is Asus N14U with MT7620/64 MB RAM/16 MB flash. I have dnsmasq-full which is required for domain based vpn redirection by vpn-policy-routing and openvpn. I also have stubby for DoT and adblock with ~60k rules. Every time I do 'free -m' on my router I see 12 MB available.
root@OpenWrt:~# free -m
total used free shared buff/cache available
Mem: 58700 27824 22348 2964 8528 12844
Swap: 0 0 0
part of normal top output w/ parameters deleted
Mem: 36628K used, 22072K free, 2964K shrd, 1348K buff, 6488K cached
CPU: 0% usr 0% sys 0% nic 99% idle 0% io 0% irq 0% sirq
Load average: 0.00 0.00 0.00 1/43 26958
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
26958 26910 root R 1252 2% 0% top
26909 2150 root S 1208 2% 0% /usr/sbin/dropbear
26473 1 dnsmasq S 12880 22% 0% /usr/sbin/dnsmasq
20133 1 stubby S 4884 8% 0% /usr/sbin/stubby
18643 1 nobody S 4176 7% 0% /usr/sbin/openvpn
2011 1 root S 3668 6% 0% /usr/sbin/uhttpd
1265 1 root S 3648 6% 0% /usr/sbin/hostapd
1266 1 root S 3552 6% 0% /usr/sbin/wpa_supplicant
969 1 root S 2288 4% 0% /sbin/rpcd
1328 1 root S 1788 3% 0% /sbin/netifd
I actually had ~ 8-9MB available on 19.07 and it never caused OOM. I didn't change any settings after upgrading to 21.02. OOM seem to happen randomly, even when I'm asleep and don't use internet at all. After getting killed dnsmasq just restarts and continues to work normally though so it's not really noticable for me.
OOM log example
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.049669] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.068916] CPU: 0 PID: 26472 Comm: dnsmasq Not tainted 5.4.143 #0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.081385] Stack : 00000240 8005ab38 805c0000 8055f9a0 00000000 00000000 00000000 00000000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.098194] 00000000 00000000 00000000 00000000 00000000 00000001 83679bb0 f163a48b
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.115045] 83679c48 00000000 00000000 00000000 00000038 804eb6c4 00000010 2e342e35
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.131849] 00000000 00002d77 00000000 203a6d6d 80000000 83679b90 83e76be0 00000000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.148642] 00000000 805c4aa4 00000000 00100cca 00000010 802ab06c 00000000 806f0000
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.166284] ...
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.172172] Call Trace:
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.172195] [<8005ab38>] 0x8005ab38
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.184636] [<804eb6c4>] 0x804eb6c4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.191891] [<802ab06c>] 0x802ab06c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.199063] [<80009790>] 0x80009790
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.206180] [<80009798>] 0x80009798
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.213315] [<800cf130>] 0x800cf130
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.220465] [<800cfbcc>] 0x800cfbcc
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.227607] [<800d0380>] 0x800d0380
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.234723] [<8011042c>] 0x8011042c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.241887] [<800cb464>] 0x800cb464
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.249002] [<8012bd64>] 0x8012bd64
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.256176] [<800cbf3c>] 0x800cbf3c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.263304] [<800f6a40>] 0x800f6a40
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.270434] [<804e85a8>] 0x804e85a8
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.277551] [<800c88a4>] 0x800c88a4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.284721] [<800f6584>] 0x800f6584
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.292167] [<800fb570>] 0x800fb570
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.299546] [<8006938c>] 0x8006938c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.306742] [<80010234>] 0x80010234
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.314011] [<800784c4>] 0x800784c4
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.321167] [<8012e024>] 0x8012e024
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.328304] [<80015d4c>] 0x80015d4c
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.335437] [<801301f8>] 0x801301f8
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.342555]
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.345689] Mem-Info:
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] active_anon:8927 inactive_anon:32 isolated_anon:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] active_file:3 inactive_file:2 isolated_file:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] unevictable:0 dirty:0 writeback:0 unstable:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] slab_reclaimable:216 slab_unreclaimable:1821
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] mapped:2 shmem:741 pagetables:94 bounce:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.350411] free:2039 free_pcp:2 free_cma:0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.413936] Node 0 active_anon:35708kB inactive_anon:128kB active_file:12kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:2964kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.458053] Normal free:8156kB min:8192kB low:10240kB high:12288kB active_anon:35708kB inactive_anon:128kB active_file:12kB inactive_file:8kB unevictable:0kB writepending:0kB present:65536kB managed:58700kB mlocked:0kB kernel_stack:336kB pagetables:376kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.511821] lowmem_reserve[]: 0 0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.518601] Normal: 95*4kB (UE) 50*8kB (UME) 17*16kB (UME) 8*32kB (ME) 3*64kB (UE) 6*128kB (UE) 3*256kB (UM) 0*512kB 1*1024kB (U) 2*2048kB (ME) 0*4096kB = 8156kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.547514] 746 total pagecache pages
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.554967] 0 pages in swap cache
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.561731] Swap cache stats: add 0, delete 0, find 0/0
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.572290] Free swap = 0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.578187] Total swap = 0kB
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.584089] 16384 pages RAM
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.589814] 0 pages HighMem/MovableOnly
Wed Sep 15 08:15:58 2021 kern.warn kernel: [233940.597691] 1709 pages reserved
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.604138] Tasks state (memory values in pages):
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.613663] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.630977] [ 575] 81 575 320 23 20480 0 0 ubusd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.647418] [ 576] 0 576 235 9 16384 0 0 askfirst
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.664375] [ 917] 514 917 326 43 20480 0 0 logd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.680643] [ 969] 0 969 572 80 16384 0 0 rpcd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.696913] [ 1265] 0 1265 912 42 20480 0 0 hostapd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.713867] [ 1266] 0 1266 888 27 24576 0 0 wpa_supplicant
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.731897] [ 1328] 0 1328 447 39 20480 0 0 netifd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.748512] [ 2011] 0 2011 917 169 20480 0 0 uhttpd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.765128] [ 2150] 0 2150 287 11 16384 0 0 dropbear
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.782087] [ 3604] 0 3604 314 10 16384 0 0 ntpd
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.798354] [ 16348] 0 16348 313 10 12288 0 0 udhcpc
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.815107] [ 18643] 65534 18643 1044 140 24576 0 0 openvpn
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.831921] [ 20133] 410 20133 1221 299 24576 0 0 stubby
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.848535] [ 26017] 453 26017 3221 2532 36864 0 0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.865321] [ 26471] 453 26471 3221 2530 36864 0 0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.882105] [ 26472] 453 26472 3221 2530 36864 0 0 dnsmasq
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.898886] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=dnsmasq,pid=26017,uid=453
Wed Sep 15 08:15:58 2021 kern.err kernel: [233940.920423] Out of memory: Killed process 26017 (dnsmasq) total-vm:12884kB, anon-rss:10120kB, file-rss:4kB, shmem-rss:4kB, UID:453 pgtables:36kB oom_score_adj:0
Wed Sep 15 08:15:58 2021 kern.info kernel: [233940.966512] oom_reaper: reaped process 26017 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
dnsmasq config
root@OpenWrt:~# cat /var/etc/dnsmasq.conf.cfg01411c
# auto-generated config file from /etc/config/dhcp
conf-file=/etc/dnsmasq.conf
dhcp-authoritative
domain-needed
no-resolv
localise-queries
read-ethers
enable-ubus=dnsmasq
expand-hosts
bind-dynamic
local-service
domain=lan
local=/lan/
server=127.0.0.1#5453
addn-hosts=/tmp/hosts
dhcp-leasefile=/tmp/dhcp.leases
stop-dns-rebind
rebind-localhost-ok
dhcp-broadcast=tag:needs-broadcast
conf-dir=/tmp/dnsmasq.d
user=dnsmasq
group=dnsmasq
dhcp-host=XX:XX:XX:XX:XX:XX,XXX.XXX.X.XXX
dhcp-ignore-names=tag:dhcp_bogus_hostname
conf-file=/usr/share/dnsmasq/dhcpbogushostname.conf
bogus-priv
conf-file=/usr/share/dnsmasq/rfc6761.conf
dhcp-range=set:lan,192.168.1.100,192.168.1.249,255.255.255.0,12h
dhcp-option=lan,252,"\n"
no-dhcp-interface=eth0.2
enable-ra
quiet-ra