Opening Taxi App - Oom_reaper kills dnsmasq

Curious to know a couple of things about this if you could maybe dig in a bit...

  • Exactly how many times does dnsmasq have to fork before the router is oom?
  • Could you post your /etc/dnsmasq.conf
  • With only one dnsmasq process running, can you post a screen shot of your memory usage (used, buffers, cache)
  • Then the same thing when dnsmasq cause oom errors...
  • What are your exact router specs? Precise make and model...

I just noticed you changed your dnsmasq cache size in /etc/dnsmasq.conf. You should try removing this option cachesize '1000' line (and removing the --max-procs= value) and see if the problem still occurs.

Router is Netgear R7800:

root@R7800:~# ubus call system board
{
	"kernel": "5.4.106",
	"hostname": "R7800",
	"system": "ARMv7 Processor rev 0 (v7l)",
	"model": "Netgear Nighthawk X4S R7800",
	"board_name": "netgear,r7800",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r16345-d71424a085",
		"target": "ipq806x/generic",
		"description": "OpenWrt SNAPSHOT r16345-d71424a085"
	}
}
/etc/dnsmasq.conf (max-procs commented out for below testing)
root@R7800:~# cat /etc/dnsmasq.conf
# Change the following lines if you want dnsmasq to serve SRV
# records.
# You may add multiple srv-host lines.
# The fields are <name>,<target>,<port>,<priority>,<weight>

# A SRV record sending LDAP for the example.com domain to
# ldapserver.example.com port 289
#srv-host=_ldap._tcp.example.com,ldapserver.example.com,389

# Two SRV records for LDAP, each with different priorities
#srv-host=_ldap._tcp.example.com,ldapserver.example.com,389,1
#srv-host=_ldap._tcp.example.com,ldapserver.example.com,389,2

# A SRV record indicating that there is no LDAP server for the domain
# example.com
#srv-host=_ldap._tcp.example.com

# The following line shows how to make dnsmasq serve an arbitrary PTR
# record. This is useful for DNS-SD.
# The fields are <name>,<target>
#ptr-record=_http._tcp.dns-sd-services,"New Employee Page._http._tcp.dns-sd-services"

# Change the following lines to enable dnsmasq to serve TXT records.
# These are used for things like SPF and zeroconf.
# The fields are <name>,<text>,<text>...

#Example SPF.
#txt-record=example.com,"v=spf1 a -all"

#Example zeroconf
#txt-record=_http._tcp.example.com,name=value,paper=A4

# Provide an alias for a "local" DNS name. Note that this _only_ works
# for targets which are names from DHCP or /etc/hosts. Give host
# "bert" another name, bertrand
# The fields are <cname>,<target>
#cname=bertand,bert

#max-procs=1
root@R7800:~#

Below memory usage after commenting out option cachesize '1000' and reboot:

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      133436      257800       12472       52940      253460
Swap:             0           0           0

Notes:

  • I use adblock and have +300k blocked domains
Tue Mar 30 20:22:53 2021 user.info adblock-4.1.0[1278]: adblock instance started ::: action: start, priority: 0, pid: 1278
Tue Mar 30 20:24:15 2021 user.info adblock-4.1.0[1278]: blocklist with overall 309334 blocked domains loaded successfully (Netgear Nighthawk X4S R7800, OpenWrt SNAPSHOT r16345-d71424a085)
  • Slighly cut the memory to get pstore/ramoops activated
[    0.000000] Kernel command line: mem=0x1C000000 ramoops.mem_address=0x5F000000 ramoops.mem_size=0x2C000 ramoops.record_size=0x4000
[    0.018995] ramoops: using module parameters
[    0.019360] pstore: Registered ramoops as persistent store backend
[    0.019378] ramoops: using 0x2c000@0x5f000000, ecc: 0

For below testing of dnsmasq oom I used netcat as when using Taxi App oo-reaper clears all the extra dnsmasq processes and memory consumption returns normal.

Mem after 1st netcat -t 127.0.0.1 53 &

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      185020      206048       12472       53108      201792
Swap:             0           0           0

Mem after 2nd netcat -t 127.0.0.1 53 &

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      232116      158928       12472       53132      154684
Swap:             0           0           0

Mem after 3rd netcat -t 127.0.0.1 53 &

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      283340      107692       12472       53144      103456
Swap:             0           0           0

Mem after 4th netcat -t 127.0.0.1 53 &

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      338756       52232       12472       53188       48016
Swap:             0           0           0

Mem after 5th netcat -t 127.0.0.1 53 &

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      386996       26696       12472       30484       11088
Swap:             0           0           0

Mem after 6th netcat -t 127.0.0.1 53 & --> dnsmasq oom

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176      389360       29004       12472       25812       11100
Swap:             0           0           0
syslog - dnsmasq oom
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.820556] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.820590] CPU: 0 PID: 9121 Comm: dnsmasq Not tainted 5.4.106 #0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.829094] Hardware name: Generic DT based system
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.835182] [<c0311068>] (unwind_backtrace) from [<c030c4e0>] (show_stack+0x14/0x20)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.839867] [<c030c4e0>] (show_stack) from [<c0923098>] (dump_stack+0x94/0xa8)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.847768] [<c0923098>] (dump_stack) from [<c0422338>] (dump_header+0x54/0x1a8)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.854794] [<c0422338>] (dump_header) from [<c0422bb4>] (oom_kill_process+0x190/0x194)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.862347] [<c0422bb4>] (oom_kill_process) from [<c04235c8>] (out_of_memory+0x268/0x324)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.870076] [<c04235c8>] (out_of_memory) from [<c0467dd8>] (__alloc_pages_nodemask+0xb9c/0x1260)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.878410] [<c0467dd8>] (__alloc_pages_nodemask) from [<c044e8f4>] (wp_page_copy+0x64/0x77c)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.887262] [<c044e8f4>] (wp_page_copy) from [<c04513a8>] (do_wp_page+0xb0/0x5b0)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.895678] [<c04513a8>] (do_wp_page) from [<c04529a0>] (handle_mm_fault+0x64c/0xb6c)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.903147] [<c04529a0>] (handle_mm_fault) from [<c0311fd4>] (do_page_fault+0x124/0x2c4)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.910960] [<c0311fd4>] (do_page_fault) from [<c0312328>] (do_DataAbort+0x48/0xd4)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.919117] [<c0312328>] (do_DataAbort) from [<c0301ddc>] (__dabt_usr+0x3c/0x40)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.926492] Exception stack(0xda22dfb0 to 0xda22dff8)
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.934132] dfa0:                                     00000000 00000000 04145f10 ffffffff
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.939089] dfc0: 00000000 004b2f58 00000026 00000000 004b2f58 004b2fa0 004b2f3c 05425990
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.947245] dfe0: bef34b20 bef34b60 00482f84 0048311c 20000010 ffffffff
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.955529] Mem-Info:
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836] active_anon:83316 inactive_anon:770 isolated_anon:0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836]  active_file:14 inactive_file:50 isolated_file:1
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836]  unevictable:0 dirty:1 writeback:3 unstable:0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836]  slab_reclaimable:1994 slab_unreclaimable:3315
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836]  mapped:3 shmem:3118 pagetables:336 bounce:0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.961836]  free:4015 free_pcp:33 free_cma:0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.974227] Node 0 active_anon:333264kB inactive_anon:3080kB active_file:92kB inactive_file:200kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:4kB writeback:8kB shmem:12472kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1760.996533] Normal free:16060kB min:16384kB low:20480kB high:24576kB active_anon:332796kB inactive_anon:3080kB active_file:40kB inactive_file:284kB unevictable:0kB writepending:12kB present:458752kB managed:444176kB mlocked:0kB kernel_stack:856kB pagetables:1344kB bounce:0kB free_pcp:132kB local_pcp:0kB free_cma:0kB
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.024640] lowmem_reserve[]: 0 0 0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.046894] Normal: 165*4kB (UE) 119*8kB (UME) 43*16kB (UE) 12*32kB (UE) 17*64kB (UE) 19*128kB (UME) 5*256kB (UE) 1*512kB (M) 0*1024kB 2*2048kB (UE) 1*4096kB (M) = 16188kB
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.050128] 3199 total pagecache pages
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.065440] 0 pages in swap cache
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.069211] Swap cache stats: add 0, delete 0, find 0/0
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.072579] Free swap  = 0kB
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.077674] Total swap = 0kB
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.080738] 114688 pages RAM
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.083671] 0 pages HighMem/MovableOnly
Tue Mar 30 20:50:39 2021 kern.warn kernel: [ 1761.086466] 3644 pages reserved
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.090041] Tasks state (memory values in pages):
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.093168] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.098087] [    190]    81   190      260       23    10240        0             0 ubusd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.106606] [    191]     0   191      176       10     6144        0             0 askfirst
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.114853] [    235]     0   235      204       12     6144        0             0 urngd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.123283] [    614]   514   614      267       42    10240        0             0 logd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.131412] [    615]     0   615      293       23     8192        0             0 logread
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.139564] [    667]     0   667      500       64    12288        0             0 rpcd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.147655] [    736]   323   736      252       33     8192        0             0 chronyd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.155996] [    954]     0   954      215       11     6144        0             0 dropbear
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.164056] [   1052]     0  1052     1027       68    12288        0             0 hostapd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.172675] [   1053]     0  1053     1003       51    12288        0             0 wpa_supplicant
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.180811] [   1115]     0  1115      393       37     8192        0             0 netifd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.189829] [   1311]     0  1311      211       14     6144        0             0 odhcp6c
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.198076] [   1336]     0  1336      270       10     8192        0             0 udhcpc
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.206324] [   1462]     0  1462      315       31     6144        0             0 odhcpd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.214665] [   1626]     0  1626      271       10     8192        0             0 crond
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.222848] [   1813]     0  1813     1016      152    12288        0             0 uhttpd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.231161] [   2000]     0  2000     1045      160    10240        0             0 collectd
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.239313] [   2089] 65534  2089     1043      317    12288        0             0 https-dns-proxy
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.247904] [   2090] 65534  2090     1080      353    14336        0             0 https-dns-proxy
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.256683] [   4161]   453  4161    13047    12443    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.265711] [   4536]     0  4536      229       22     8192        0             0 dropbear
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.274052] [   4555]     0  4555      272       13     8192        0             0 ash
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.282659] [   8868]     0  8868      185       15     8192        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.290454] [   8869]   453  8869    13064    12444    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.298764] [   8883]     0  8883      185       15     8192        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.307011] [   8884]   453  8884    13064    12444    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.315352] [   8890]     0  8890      185       15     8192        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.323600] [   8891]   453  8891    13064    12444    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.331878] [   8900]     0  8900      185       15     8192        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.340187] [   8901]   453  8901    13064    12444    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.348528] [   9046]     0  9046      185       15    10240        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.356776] [   9047]   453  9047    13064    12444    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.365087] [   9120]     0  9120      185       15    10240        0             0 netcat
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.373282] [   9121]   453  9121    13047    12442    61440        0             0 dnsmasq
Tue Mar 30 20:50:39 2021 kern.info kernel: [ 1761.381671] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=dnsmasq,pid=9047,uid=453
Tue Mar 30 20:50:39 2021 kern.err kernel: [ 1761.389935] Out of memory: Killed process 9047 (dnsmasq) total-vm:52256kB, anon-rss:49776kB, file-rss:0kB, shmem-rss:0kB, UID:453 pgtables:60kB oom_score_adj:0

and if you'd do the same with the block list disabled/removed ?

I'm curious if it creates a full copy of the block list in RAM, for every child is starts.

Seems like it does.

Memory usage after disabling adblock.

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176       86876      329344         288       27956      318604
Swap:             0           0           0

Memory usage after max 20 dnsmasq instances running and cannot oom:

root@R7800:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         444176       89320      326792         288       28064      316104
Swap:             0           0           0
root@R7800:~# ps |grep dnsmasq
13232 dnsmasq   2564 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13832 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13916 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13918 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13920 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13922 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13931 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13933 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
13940 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14032 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14039 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14041 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14043 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14045 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14047 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14056 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14058 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14065 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14067 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14163 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14228 dnsmasq   2632 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14253 root      1080 R    grep dnsmasq
root@R7800:~#

Ok,

ideally the children would use the dns buffer from the mother, but it seems they don't :expressionless:
Work as designed I'm sure, but in this case the culprit is the huge block list of yours.

Based on your first test, you shouldn't let it create more than 4 or 5 childs.

Not sure in what way the conclusion helps you thought :wink:

Heh... I'll use patch as long as it's working as cannot replace the Taxi App.
And still going to use the adblock as well as that's one of the main apps in Openwrt for my use case.

Anyhow this is kind of a border case unless DNS over TCP usage will grow and others will start facing similar issues.

Are those TCP queries really important?
Can the client failover to UDP if you reject DNS over TCP?
Perhaps you should just reject TCP queries from that specific client, or reject them entirely?

Otherwise, consider to use 2 Dnsmasq instances.
The second instance can be lightweight running without Adblock.
You can redirect TCP queries or all queries from that client to the second instance.

I would be interested to know if running "sysctl -w vm.overcommit_memory=1" reduces the instances of OOM killer.

  • Working Taxi App is really important as it is only viable App for the purpose here and needed on daily basis during working days.
  • After blocking IPv4/IPv6 TCP:53, Taxi App shows "Oops! You're Offline". So doesn't help.
  • In wireshark it'll just receive RST (mobile is .110) for the 1st SYN msg.
  • That's a food for thought...

  • To be noted, ultimately I can just disable mobile wireless and use mobile data for Taxi App whenever I need it (as did before all the findings in this thread).

  • When I let the TCP:53 through and the App works, it seems like it is resolving required addresses later using IPv6 UDP DNS.

  • So I really don't understand for what is this App using the hardcoded IPv4 DNS query over TCP to google DNS server. Seems it's only SYN/ACK/FIN ACK msg in-between and no data, but then I don't know what to expect in this communication...

1 Like

You have posted the TCP SYN and RST captures, but not the actual query.
Is this really a DNS query and not just a workaround to check internet connectivity?
Try to capture and extract specifically the DNS query message if any.

1 Like

Keeping all the above config the same when Taxi App "works", unfortunately dnsmasq will still Oom.

root@R7800:~# date
Wed Mar 31 23:16:12 +08 2021
root@R7800:~# sysctl -w vm.overcommit_memory=1
vm.overcommit_memory = 1
syslog - dnsmasq Oom
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.816450] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.816483] CPU: 0 PID: 29692 Comm: dnsmasq Not tainted 5.4.106 #0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.824986] Hardware name: Generic DT based system
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.831077] [<c0311068>] (unwind_backtrace) from [<c030c4e0>] (show_stack+0x14/0x20)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.835850] [<c030c4e0>] (show_stack) from [<c0923098>] (dump_stack+0x94/0xa8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.843751] [<c0923098>] (dump_stack) from [<c0422338>] (dump_header+0x54/0x1a8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.850777] [<c0422338>] (dump_header) from [<c0422bb4>] (oom_kill_process+0x190/0x194)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.858328] [<c0422bb4>] (oom_kill_process) from [<c04235c8>] (out_of_memory+0x268/0x324)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.866058] [<c04235c8>] (out_of_memory) from [<c0467dd8>] (__alloc_pages_nodemask+0xb9c/0x1260)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.874390] [<c0467dd8>] (__alloc_pages_nodemask) from [<c044e8f4>] (wp_page_copy+0x64/0x77c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.883243] [<c044e8f4>] (wp_page_copy) from [<c04513a8>] (do_wp_page+0xb0/0x5b0)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.891662] [<c04513a8>] (do_wp_page) from [<c04529a0>] (handle_mm_fault+0x64c/0xb6c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.899129] [<c04529a0>] (handle_mm_fault) from [<c0311fd4>] (do_page_fault+0x124/0x2c4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.906941] [<c0311fd4>] (do_page_fault) from [<c0312328>] (do_DataAbort+0x48/0xd4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.915099] [<c0312328>] (do_DataAbort) from [<c0301ddc>] (__dabt_usr+0x3c/0x40)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.922473] Exception stack(0xd559ffb0 to 0xd559fff8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.930113] ffa0:                                     00000000 00000000 03639fa0 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.935070] ffc0: 00000000 00532f58 00000026 00000000 00532f58 00532fa0 00532f3c 04fb9ba0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.943227] ffe0: be8ccb20 be8ccb60 00502f84 0050311c 20000010 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.951538] Mem-Info:
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887] active_anon:84821 inactive_anon:772 isolated_anon:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887]  active_file:44 inactive_file:82 isolated_file:35
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887]  unevictable:0 dirty:76 writeback:0 unstable:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887]  slab_reclaimable:2014 slab_unreclaimable:3414
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887]  mapped:18 shmem:3380 pagetables:402 bounce:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.957887]  free:3965 free_pcp:0 free_cma:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.970182] Node 0 active_anon:339284kB inactive_anon:3088kB active_file:332kB inactive_file:252kB unevictable:0kB isolated(anon):0kB isolated(file):12kB mapped:72kB dirty:304kB writeback:0kB shmem:13520kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5242.993020] Normal free:15860kB min:16384kB low:20480kB high:24576kB active_anon:339208kB inactive_anon:3088kB active_file:88kB inactive_file:464kB unevictable:0kB writepending:304kB present:458752kB managed:444176kB mlocked:0kB kernel_stack:840kB pagetables:1608kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.021065] lowmem_reserve[]: 0 0 0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.043276] Normal: 89*4kB (UME) 28*8kB (UME) 14*16kB (UME) 20*32kB (UME) 10*64kB (UE) 4*128kB (UE) 9*256kB (U) 15*512kB (UM) 3*1024kB (UM) 0*2048kB 0*4096kB = 15652kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.046441] 3547 total pagecache pages
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.061427] 0 pages in swap cache
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.065175] Swap cache stats: add 0, delete 0, find 0/0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.068612] Free swap  = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.073595] Total swap = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.076718] 114688 pages RAM
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.079639] 0 pages HighMem/MovableOnly
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.082449] 3644 pages reserved
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.086007] Tasks state (memory values in pages):
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.089200] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.094012] [    189]    81   189      262       24     8192        0             0 ubusd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.102574] [    190]     0   190      176       10     8192        0             0 askfirst
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.110820] [    234]     0   234      204       12     8192        0             0 urngd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.119318] [    613]   514   613      268       43     8192        0             0 logd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.127421] [    614]     0   614      293       23    10240        0             0 logread
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.135496] [    666]     0   666      500       64    10240        0             0 rpcd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.143622] [    735]   323   735      252       33    10240        0             0 chronyd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.151963] [    953]     0   953      215       11     6144        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.160054] [   1051]     0  1051     1027       68    12288        0             0 hostapd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.168708] [   1052]     0  1052     1003       51    10240        0             0 wpa_supplicant
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.176730] [   1114]     0  1114      393       37    12288        0             0 netifd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.185827] [   1266]     0  1266      313       30    10240        0             0 odhcpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.194076] [   1362]     0  1362      270       10     8192        0             0 udhcpc
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.202324] [   1391]     0  1391      211       14     8192        0             0 odhcp6c
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.210569] [   1490]     0  1490      271       10     8192        0             0 crond
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.218816] [   1577]     0  1577     1077      213    10240        0             0 uhttpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.227077] [   1808]     0  1808     1059      174    14336        0             0 collectd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.235312] [   2006] 65534  2006      996      227    12288        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.243902] [   2007] 65534  2007      873      154    10240        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.252649] [   4846]     0  4846      229       22    10240        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.261678] [   4866]     0  4866      272       13    10240        0             0 ash
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.270363] [  15537]   453 15537    13137    12532    61440        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.278080] [  29686]   453 29686    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.286364] [  29687]   453 29687    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.294762] [  29688]   453 29688    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.303102] [  29689]   453 29689    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.311413] [  29690]   453 29690    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.319753] [  29691]   453 29691    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.328094] [  29692]   453 29692    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.336365] [  29693]   453 29693    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.344745] [  29694]   453 29694    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.353083] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=dnsmasq,pid=15537,uid=453
Wed Mar 31 23:18:02 2021 kern.err kernel: [ 5243.361436] Out of memory: Killed process 15537 (dnsmasq) total-vm:52548kB, anon-rss:50124kB, file-rss:4kB, shmem-rss:0kB, UID:453 pgtables:60kB oom_score_adj:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.374667] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.388813] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.406714] oom_reaper: reaped process 15537 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.407296] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.414951] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.422146] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.547286] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.547319] CPU: 0 PID: 29692 Comm: dnsmasq Not tainted 5.4.106 #0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.555822] Hardware name: Generic DT based system
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.561915] [<c0311068>] (unwind_backtrace) from [<c030c4e0>] (show_stack+0x14/0x20)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.566684] [<c030c4e0>] (show_stack) from [<c0923098>] (dump_stack+0x94/0xa8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.574590] [<c0923098>] (dump_stack) from [<c0422338>] (dump_header+0x54/0x1a8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.581614] [<c0422338>] (dump_header) from [<c0422bb4>] (oom_kill_process+0x190/0x194)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.589165] [<c0422bb4>] (oom_kill_process) from [<c04235c8>] (out_of_memory+0x268/0x324)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.596905] [<c04235c8>] (out_of_memory) from [<c0467dd8>] (__alloc_pages_nodemask+0xb9c/0x1260)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.605226] [<c0467dd8>] (__alloc_pages_nodemask) from [<c041df0c>] (pagecache_get_page+0x120/0x34c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.614078] [<c041df0c>] (pagecache_get_page) from [<c041e708>] (filemap_fault+0x5d0/0x94c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.623194] [<c041e708>] (filemap_fault) from [<c044e6b8>] (__do_fault+0x40/0x12c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.631265] [<c044e6b8>] (__do_fault) from [<c0452b3c>] (handle_mm_fault+0x7e8/0xb6c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.638907] [<c0452b3c>] (handle_mm_fault) from [<c0311fd4>] (do_page_fault+0x124/0x2c4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.646805] [<c0311fd4>] (do_page_fault) from [<c03123fc>] (do_PrefetchAbort+0x48/0xa4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.654964] [<c03123fc>] (do_PrefetchAbort) from [<c0302088>] (ret_from_exception+0x0/0x18)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.662686] Exception stack(0xd559ffb0 to 0xd559fff8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.671019] ffa0:                                     00000000 00000000 03551fa0 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.676236] ffc0: 00000000 00532f58 00000026 00000000 00532f58 00532fa0 00532f3c 04fb9ba0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.684394] ffe0: be8ccb20 be8ccb60 00502f84 0050311c 20000010 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.692746] Mem-Info:
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045] active_anon:84540 inactive_anon:772 isolated_anon:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045]  active_file:38 inactive_file:28 isolated_file:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045]  unevictable:0 dirty:66 writeback:2 unstable:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045]  slab_reclaimable:2003 slab_unreclaimable:3415
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045]  mapped:16 shmem:3380 pagetables:367 bounce:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.699045]  free:3987 free_pcp:80 free_cma:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.711367] Node 0 active_anon:338184kB inactive_anon:3088kB active_file:108kB inactive_file:116kB unevictable:0kB isolated(anon):0kB isolated(file):48kB mapped:32kB dirty:260kB writeback:12kB shmem:13520kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.734188] Normal free:16272kB min:16384kB low:20480kB high:24576kB active_anon:338124kB inactive_anon:3088kB active_file:156kB inactive_file:0kB unevictable:0kB writepending:272kB present:458752kB managed:444176kB mlocked:0kB kernel_stack:832kB pagetables:1468kB bounce:0kB free_pcp:320kB local_pcp:12kB free_cma:0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.762298] lowmem_reserve[]: 0 0 0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.784539] Normal: 128*4kB (UME) 53*8kB (UME) 27*16kB (UME) 30*32kB (UME) 11*64kB (UME) 4*128kB (UE) 11*256kB (UM) 14*512kB (U) 3*1024kB (UM) 0*2048kB 0*4096kB = 16600kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.787945] 3467 total pagecache pages
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.803191] 0 pages in swap cache
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.806862] Swap cache stats: add 0, delete 0, find 0/0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.810312] Free swap  = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.815280] Total swap = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.818465] 114688 pages RAM
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.821269] 0 pages HighMem/MovableOnly
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5243.824133] 3644 pages reserved
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.827744] Tasks state (memory values in pages):
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.830821] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.835702] [    189]    81   189      262       24     8192        0             0 ubusd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.844276] [    190]     0   190      176       10     8192        0             0 askfirst
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.852521] [    234]     0   234      204       12     8192        0             0 urngd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.861020] [    613]   514   613      268       43     8192        0             0 logd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.869080] [    614]     0   614      293       23    10240        0             0 logread
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.877232] [    666]     0   666      500       64    10240        0             0 rpcd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.885257] [    735]   323   735      252       33    10240        0             0 chronyd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.893664] [    953]     0   953      215       11     6144        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.901725] [   1051]     0  1051     1027       68    12288        0             0 hostapd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.910409] [   1052]     0  1052     1003       51    10240        0             0 wpa_supplicant
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.918468] [   1114]     0  1114      393       37    12288        0             0 netifd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.927497] [   1266]     0  1266      313       30    10240        0             0 odhcpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.935691] [   1362]     0  1362      270       10     8192        0             0 udhcpc
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.943992] [   1391]     0  1391      211       14     8192        0             0 odhcp6c
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.952238] [   1490]     0  1490      271       10     8192        0             0 crond
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.960487] [   1577]     0  1577     1077      213    10240        0             0 uhttpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.968828] [   1808]     0  1808     1059      174    14336        0             0 collectd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.976923] [   2006] 65534  2006      996      227    12288        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.985572] [   2007] 65534  2007      873      154    10240        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5243.994350] [   4846]     0  4846      229       22    10240        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.003379] [   4866]     0  4866      272       13    10240        0             0 ash
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.012067] [  29686]   453 29686    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.019780] [  29687]   453 29687    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.028121] [  29688]   453 29688    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.036384] [  29689]   453 29689    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.044772] [  29690]   453 29690    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.053113] [  29691]   453 29691    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.061454] [  29692]   453 29692    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.069795] [  29693]   453 29693    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.078105] [  29694]   453 29694    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.086381] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=dnsmasq,pid=29694,uid=453
Wed Mar 31 23:18:02 2021 kern.err kernel: [ 5244.094797] Out of memory: Killed process 29694 (dnsmasq) total-vm:52548kB, anon-rss:50124kB, file-rss:0kB, shmem-rss:0kB, UID:453 pgtables:58kB oom_score_adj:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5244.108122] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5244.122193] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5244.124204] oom_reaper: reaped process 29694 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5244.129817] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5244.145867] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5244.152994] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.887722] dnsmasq invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.887766] CPU: 0 PID: 29687 Comm: dnsmasq Not tainted 5.4.106 #0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.896265] Hardware name: Generic DT based system
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.902363] [<c0311068>] (unwind_backtrace) from [<c030c4e0>] (show_stack+0x14/0x20)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.907132] [<c030c4e0>] (show_stack) from [<c0923098>] (dump_stack+0x94/0xa8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.915025] [<c0923098>] (dump_stack) from [<c0422338>] (dump_header+0x54/0x1a8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.922052] [<c0422338>] (dump_header) from [<c0422bb4>] (oom_kill_process+0x190/0x194)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.929605] [<c0422bb4>] (oom_kill_process) from [<c04235c8>] (out_of_memory+0x268/0x324)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.937333] [<c04235c8>] (out_of_memory) from [<c0467dd8>] (__alloc_pages_nodemask+0xb9c/0x1260)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.945664] [<c0467dd8>] (__alloc_pages_nodemask) from [<c044e8f4>] (wp_page_copy+0x64/0x77c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.954518] [<c044e8f4>] (wp_page_copy) from [<c04513a8>] (do_wp_page+0xb0/0x5b0)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.962935] [<c04513a8>] (do_wp_page) from [<c04529a0>] (handle_mm_fault+0x64c/0xb6c)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.970404] [<c04529a0>] (handle_mm_fault) from [<c0311fd4>] (do_page_fault+0x124/0x2c4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.978216] [<c0311fd4>] (do_page_fault) from [<c0312328>] (do_DataAbort+0x48/0xd4)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.986374] [<c0312328>] (do_DataAbort) from [<c0301ddc>] (__dabt_usr+0x3c/0x40)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5245.993748] Exception stack(0xd4c09fb0 to 0xd4c09ff8)
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.001388] 9fa0:                                     00000000 00000000 02ea6f60 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.006345] 9fc0: 00000000 00532f58 00000026 00000000 00532f58 00532fa0 00532f3c 04fb9ba0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.014502] 9fe0: be8ccb20 be8ccb60 00502f84 0050311c 20000010 ffffffff
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.022821] Mem-Info:
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141] active_anon:84915 inactive_anon:772 isolated_anon:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141]  active_file:28 inactive_file:57 isolated_file:32
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141]  unevictable:0 dirty:14 writeback:3 unstable:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141]  slab_reclaimable:2001 slab_unreclaimable:3415
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141]  mapped:48 shmem:3380 pagetables:338 bounce:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.029141]  free:4019 free_pcp:0 free_cma:0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.041448] Node 0 active_anon:339660kB inactive_anon:3088kB active_file:112kB inactive_file:228kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:192kB dirty:56kB writeback:12kB shmem:13520kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.064629] Normal free:16076kB min:16384kB low:20480kB high:24576kB active_anon:339660kB inactive_anon:3088kB active_file:112kB inactive_file:228kB unevictable:0kB writepending:68kB present:458752kB managed:444176kB mlocked:0kB kernel_stack:832kB pagetables:1352kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.092489] lowmem_reserve[]: 0 0 0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.114730] Normal: 133*4kB (UME) 39*8kB (UME) 18*16kB (UME) 29*32kB (UME) 11*64kB (UE) 4*128kB (UE) 10*256kB (UM) 14*512kB (U) 3*1024kB (UM) 0*2048kB 0*4096kB = 16076kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.117948] 3496 total pagecache pages
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.133225] 0 pages in swap cache
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.136798] Swap cache stats: add 0, delete 0, find 0/0
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.140252] Free swap  = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.145216] Total swap = 0kB
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.148406] 114688 pages RAM
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.151205] 0 pages HighMem/MovableOnly
Wed Mar 31 23:18:02 2021 kern.warn kernel: [ 5246.154071] 3644 pages reserved
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.157685] Tasks state (memory values in pages):
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.160757] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.165631] [    189]    81   189      262       24     8192        0             0 ubusd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.174183] [    190]     0   190      176       10     8192        0             0 askfirst
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.182430] [    234]     0   234      204       12     8192        0             0 urngd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.190959] [    613]   514   613      268       43     8192        0             0 logd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.199019] [    614]     0   614      293       23    10240        0             0 logread
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.207172] [    666]     0   666      500       64    10240        0             0 rpcd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.215192] [    735]   323   735      252       33    10240        0             0 chronyd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.223573] [    953]     0   953      215       11     6144        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.231664] [   1051]     0  1051     1027       68    12288        0             0 hostapd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.240349] [   1052]     0  1052     1003       51    10240        0             0 wpa_supplicant
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.248409] [   1114]     0  1114      393       37    12288        0             0 netifd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.257437] [   1266]     0  1266      313       30    10240        0             0 odhcpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.265625] [   1362]     0  1362      270       10     8192        0             0 udhcpc
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.273932] [   1391]     0  1391      211       45     8192        0             0 odhcp6c
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.282179] [   1490]     0  1490      271       26     8192        0             0 crond
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.290427] [   1577]     0  1577     1077      213    10240        0             0 uhttpd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.298768] [   1808]     0  1808     1059      174    14336        0             0 collectd
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.306858] [   2006] 65534  2006      996      227    12288        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.315513] [   2007] 65534  2007      873      154    10240        0             0 https-dns-proxy
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.324291] [   4846]     0  4846      229       22    10240        0             0 dropbear
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.333319] [   4866]     0  4866      272       13    10240        0             0 ash
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.341976] [  29686]   453 29686    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.349721] [  29687]   453 29687    13137    12547    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.358061] [  29688]   453 29688    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.366320] [  29689]   453 29689    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.374714] [  29690]   453 29690    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.383053] [  29691]   453 29691    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.391395] [  29692]   453 29692    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.399704] [  29693]   453 29693    13137    12531    59392        0             0 dnsmasq
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.408043] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=dnsmasq,pid=29687,uid=453
Wed Mar 31 23:18:02 2021 kern.err kernel: [ 5246.416335] Out of memory: Killed process 29687 (dnsmasq) total-vm:52548kB, anon-rss:50124kB, file-rss:64kB, shmem-rss:0kB, UID:453 pgtables:58kB oom_score_adj:0
Wed Mar 31 23:18:02 2021 kern.info kernel: [ 5246.436459] oom_reaper: reaped process 29687 (dnsmasq), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
root@R7800:~# date
Wed Mar 31 23:18:07 +08 2021
root@R7800:~# sysctl vm.overcommit_memory
vm.overcommit_memory = 1

I suspect this might be the case as I cannot find any address query from message transfers between the mobile and google DNS server.
--> Any advise what should I look for as an actual "DNS query" msg when using DNS over TCP?

1 Like

Capturing DNS over TCP:

# Client command
dig -q google.com @8.8.8.8 +vc
nslookup -vc google.com 8.8.8.8

# Wireshark > Filter
dns && tcp

# Wireshark > Packet Details
Frame ...
Linux cooked capture ...
Internet Protocol ...
Transmission Control Protocol ...
Domain Name System (query)
    Length: 28
    Transaction ID: 0x0928
    Flags: 0x0100 Standard query
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 0
    Queries
        google.com: type A, class IN
            Name: google.com
            [Name Length: 10]
            [Label Count: 2]
            Type: A (Host Address) (1)
            Class: IN (0x0001)
    [Response In: 31995]

Possible workaround for OpenWrt:

uci -q delete firewall.dns_tcp
uci set firewall.dns_tcp="redirect"
uci set firewall.dns_tcp.name="Dummy-DNS+TCP"
uci set firewall.dns_tcp.src="lan"
uci set firewall.dns_tcp.src_dip="8.8.8.8"
uci set firewall.dns_tcp.src_dport="53"
uci set firewall.dns_tcp.dest_port="80"
uci set firewall.dns_tcp.proto="tcp"
uci set firewall.dns_tcp.target="DNAT"
uci reorder firewall.dns_tcp="0"
uci commit firewall
/etc/init.d/firewall restart

You can limit the rule scope to a specific MAC address and/or redirect it as-is to a public DNS.

Thanks for trying that. I'll go back to the drawing board :slight_smile:

Thanks for the above. Can confirm I can filter and find TCP DNS query.

Also can confirm there's no DNS query in messages from mobile to TCP 8.8.8.8:53 when using Taxi App. So most likely it's used for kind of a connectivity check as you suspected as no real data is transferred in between.

As for the possible firewall rule workaround, seems it doesn't have effect when I'm using https-dns-proxy. It hits https-dns-proxy own firewall rule before;

For the sake of testing and understanding, I stopped https-dns-proxy and set dns manually to 1.1.1.1 in /etc/config/dhcp.
Now it hits the firewall rule and it is working.

Question:
Why it works when forwarding request to port 80 (uhttpd) but when forwarding the same request to port 53 (dnsmasq, and the rule below it) it does not work at all ("Oops! You're Offline") ?

  • Assuming http server is able to respond those messages at port 80 but what happens at dnsmasq when it receives these...
1 Like

The iptables targets such as DNAT and REDIRECT are terminating in the current chain.
If traffic matches the first rule, it won't be processed by the following rules in the chain.

You can disable the automatic redirects that mess your rule order:

uci set https-dns-proxy.config.force_dns="0"
uci commit https-dns-proxy
/etc/init.d/https-dns-proxy restart

And set up a manual redirect instead:
https://openwrt.org/docs/guide-user/firewall/fw3_configurations/intercept_dns

uHTTPd should process those connections more gracefully compared to Dnsmasq.
Apparently it has no need to fork or consume too much RAM to load blocklists.

1 Like

Noted with thanks. Will try later.

Edit:
Am I able to see current parameter value for https-dns-proxy.config.force_dns parameter?
I cannot find just by uci show command.

1 Like

Check out the default settings:

1 Like

Thanks for pointing out. Seem I had an old config from https://docs.openwrt.melmac.net/https-dns-proxy/ that did not have that option force_dns line.
I added it and set it to '0' in the config and got rid of the related firewall rules.

Thanks once again.

1 Like