Netgear R6220 eating through ram and crashing

Firmware version: OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.247.75781-0d0ab01

Not sure if this is a proper "memory leak" or something else, runs about 30 hours (the first time I thought to measure it anyhow) and then the internet goes out, it refuses SSH connections and drops SSH if it had a session going, web interface shows error page "Internal Server Error Failed to create CGI process: Out of Memory", power cycle brings everything back up. If I watch in htop I can see ram use creeping up throughout the day (cold boot is something like 25MB) but no processes are shown eating it up. Router is connected to a bridged DSL gateway box acting as a dumb modem, PPPoE handled on the router, I only have the 2.4ghz wifi radio running (N is plenty with a 7/1 link, lol), config mostly default aside from forwarding a few ports, only extra packages installed are SQM (which works great and was my reason for buying the router and installing OpenWrt) and htop. Problem has always existed.

I know I could bodge this with a 5AM reboot script but would like to diagnose it properly, I have some linux experience but not enough to to work this deep under the hood without some guidance.

If you have IPv6 connectivity on WAN, please disable it for a couple of days' testing - there might be an open issue with odhcp6c (at least in master) going rampant under some conditions.

WAN appears to be v4 only (Network>Interfaces, WAN interface "pppoe-wan" only has a public v4 address on it). LAN has it enabled and there are some v6 addressed devices on the LAN/WLAN, I could put that at V4 only, I considered it during initial setup (becasue IPv6 is new and scary to me) but decided to leave well enough alone.

There is no reason to be afraid of IPv6, and even less a reason to avoid it - I was just considering a potentially 'easy' issue here, sadly it doesn't seem to apply to your situation.

In order to debug this further, it would probably help to do some further logging (e.g. keeping logread -f open over an ssh session, to gather some more information what (might) happens when you get closer to your OOM situation, keeping top/ htop running in an open ssh session might also help to identify the culprit). If you can, it's always useful to give a current snapshot build a test as well (be aware that luci isn't preinstalled in snapshots - and that the switch framework has been migrated over to DSA for mt7621, so you can't retain your configuration over the upgrade to snapshots either).

Ok, got a log of "logread -f" going as suggested and will report back when it crashes again, hopefully there's something good in there. On htop I've seen it grow 40MB, but like I said there's nothing in the process list using it, in fact htop is the biggest consumer of memory.

Maybe related to this?:

Interesting, I was unable to get a log of the crash because it has stopped doing it, or at least slowed down a lot (up 40 hours and using 37M/121M). I don't know what changed other than one member of the house comes and goes and was here when I set up the router, and left about the same time I started investigating the problem. So it would have to be some traffic from them triggering the leak or at least the majority of it. I'll have to wait for their return in a few weeks.

I don't think they have anything super out of the ordinary on the network, xbox, phone, the only smart bulb in the house (guess I could go flip that switch), one of those Amazon assistant box things, and a pair of laptops personal and work.

Log will never show any problem. Use a snapshot build or build it yourself

Could this be related? Hasn't crashed yet, everything else in log is just ordinary looking sporadic DHCP stuff.

Thu Dec  3 02:35:52 2020 kern.warn kernel: [351531.655185] vlan egress tag control neither untag nor tag.
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.447377] ------------[ cut here ]------------
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.456788] WARNING: CPU: 1 PID: 0 at /builder/shared-workdir/build/build_dir/target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2020-03-10-08054d5a/mt7603/mac.c:204 0x86fe4320 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.492437] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.633611]  cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.690111] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.195 #0
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.702399] Stack : 00000000 00000000 00000000 00000001 00000000 00000000 00000000 00000000
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.719203]         00000000 00000000 00000000 00000000 00000000 00000001 87c0bc78 53261622
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.736020]         87c0bd10 00000000 00000000 00005e80 00000038 8049c858 00000007 00000000
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.752915]         00000000 80550000 000a87bf 70617773 87c0bc58 00000000 00000000 86fe8bb4
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.769719]         86fe4320 000000cc 86ff0000 00000001 00000008 802ad210 00000004 806b0004
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.786522]         ...
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.791553] Call Trace:
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.791610] [<8049c858>] 0x8049c858
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.803741] [<86fe4320>] 0x86fe4320 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.815338] [<86ff0000>] 0x86ff0000 [ppp_async@86ff0000+0x1b90]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.827280] [<802ad210>] 0x802ad210
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.834394] [<8000c1a0>] 0x8000c1a0
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.841495] [<8000c1a8>] 0x8000c1a8
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.848599] [<804856b4>] 0x804856b4
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.855705] [<86fe4320>] 0x86fe4320 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.867299] [<80072a84>] 0x80072a84
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.874399] [<8002e608>] 0x8002e608
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.881507] [<86fe4320>] 0x86fe4320 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.893110] [<8002e6f0>] 0x8002e6f0
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.900217] [<8034ce7c>] 0x8034ce7c
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.907331] [<86fe4320>] 0x86fe4320 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.918927] [<8046da9c>] 0x8046da9c
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.926033] [<87770274>] 0x87770274 [mt76@87770000+0x79e0]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.937139] [<86fe4488>] 0x86fe4488 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.948738] [<80017f6c>] 0x80017f6c
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.955951] [<877742ec>] 0x877742ec [mt76@87770000+0x79e0]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.967099] [<86fe18d4>] 0x86fe18d4 [mt7603e@86fe0000+0x9100]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.978727] [<87773b80>] 0x87773b80 [mt76@87770000+0x79e0]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465120.989838] [<87770dec>] 0x87770dec [mt76@87770000+0x79e0]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.000959] [<80364238>] 0x80364238
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.008088] [<804a3658>] 0x804a3658
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.015191] [<80032fb4>] 0x80032fb4
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.022292] [<8025a5f0>] 0x8025a5f0
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.029403] [<80007488>] 0x80007488
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.036508]
Fri Dec  4 10:09:02 2020 kern.warn kernel: [465121.039766] ---[ end trace a2dc59334870a4c4 ]---

It starting to freak out at ~95/128MB tons of daemon.err dnsmasq-dhcp[3049]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s) in the log. Then I got the crash about a day later.

Sun Dec  6 17:50:01 2020 daemon.err dnsmasq-dhcp[3049]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Sun Dec  6 17:51:08 2020 kern.warn kernel: [665641.464661] odhcpd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.489103] CPU: 0 PID: 1219 Comm: odhcpd Tainted: G        W       4.14.195 #0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.503824] Stack : 00000000 00000000 00000000 00000019 00000000 00000000 00000000 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.520661]         00000000 00000000 00000000 00000000 00000000 00000001 8707fae8 53261622
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.537511]         8707fb80 00000000 00000000 000067c0 00000038 8049c858 00000007 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.554340]         00000000 80550000 0007768f 20202020 8707fac8 80000000 00000000 804e0448
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.571162]         00000002 00200000 ffffffff 00000019 00000003 802ad210 00000000 806b0000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.587996]         ...
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.593048] Call Trace:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.593136] [<8049c858>] 0x8049c858
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.605295] [<802ad210>] 0x802ad210
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.612432] [<8000c1a0>] 0x8000c1a0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.619559] [<8000c1a8>] 0x8000c1a8
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.626672] [<804856b4>] 0x804856b4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.633778] [<80072a84>] 0x80072a84
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.640943] [<800c8150>] 0x800c8150
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.648122] [<800c739c>] 0x800c739c
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.655258] [<800c7ff0>] 0x800c7ff0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.662370] [<800cc960>] 0x800cc960
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.669517] [<800d22a4>] 0x800d22a4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.676708] [<800c48d4>] 0x800c48d4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.683845] [<800f4468>] 0x800f4468
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.690989] [<800c22f8>] 0x800c22f8
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.698159] [<800f0124>] 0x800f0124
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.705291] [<800f4cac>] 0x800f4cac
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.712466] [<800189d0>] 0x800189d0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.719600] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.726734] [<80051820>] 0x80051820
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.733894] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.741010] [<80094500>] 0x80094500
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.748120] [<8001e588>] 0x8001e588
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.755237]
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.758653] Mem-Info:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397] active_anon:550 inactive_anon:13 isolated_anon:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397]  active_file:115 inactive_file:127 isolated_file:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397]  unevictable:0 dirty:0 writeback:0 unstable:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397]  slab_reclaimable:396 slab_unreclaimable:1863
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397]  mapped:22 shmem:90 pagetables:69 bounce:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.763397]  free:3977 free_pcp:73 free_cma:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.827676] Node 0 active_anon:2200kB inactive_anon:52kB active_file:476kB inactive_file:512kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:92kB dirty:0kB writeback:0kB shmem:360kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.872038] Normal free:16260kB min:16384kB low:20480kB high:24576kB active_anon:2200kB inactive_anon:52kB active_file:436kB inactive_file:852kB unevictable:0kB writepending:0kB present:131072kB managed:123880kB mlocked:0kB kernel_stack:528kB pagetables:276kB bounce:0kB free_pcp:292kB local_pcp:148kB free_cma:0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.927434] lowmem_reserve[]: 0 0 0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.934578] Normal: 226*4kB (UME) 219*8kB (UME) 148*16kB (ME) 70*32kB (ME) 24*64kB (UME) 14*128kB (UME) 3*256kB (ME) 4*512kB (M) 3*1024kB (ME) 0*2048kB 0*4096kB = 16480kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.965209] 344 total pagecache pages
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.972697] 0 pages in swap cache
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.979485] Swap cache stats: add 0, delete 0, find 0/0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.990084] Free swap  = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665641.995991] Total swap = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665642.001916] 32768 pages RAM
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665642.007660] 0 pages HighMem/MovableOnly
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665642.015472] 1798 pages reserved
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.021922] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.039117] [  539]     0   539      316       10       3       0        0             0 ubusd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.056485] [  540]     0   540      232        7       3       0        0             0 askfirst
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.074395] [  557]     0   557      258        8       4       0        0             0 urngd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.091758] [  999]     0   999      323       41       5       0        0             0 logd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.108939] [ 1030]     0  1030      561       84       5       0        0             0 rpcd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.126126] [ 1126]     0  1126      271        7       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.144013] [ 1183]     0  1183      437       42       5       0        0             0 netifd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.161560] [ 1219]     0  1219      363       26       5       0        0             0 odhcpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.179099] [ 1639]     0  1639      338       28       3       0        0             0 uhttpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.196629] [ 1709]     0  1709      310       29       5       0        0             0 pppd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.213816] [ 2514]     0  2514      305       10       4       0        0             0 ntpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.231014] [ 3049]   453  3049      342       29       5       0        0             0 dnsmasq
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.248736] [ 3154]     0  3154      451       40       4       0        0             0 hostapd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.266421] [15069]     0 15069      288       26       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.284265] [15070]     0 15070      306       12       4       0        0             0 ash
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.301281] [25000]     0 25000      343       21       4       0        0             0 logread
Sun Dec  6 17:51:10 2020 kern.err kernel: [665642.318968] Out of memory: Kill process 1030 (rpcd) score 2 or sacrifice child
Sun Dec  6 17:51:10 2020 kern.err kernel: [665642.333593] Killed process 1030 (rpcd) total-vm:2244kB, anon-rss:336kB, file-rss:0kB, shmem-rss:0kB
Sun Dec  6 17:51:10 2020 kern.info kernel: [665642.451109] oom_reaper: reaped process 1030 (rpcd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.775138] hostapd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.800474] CPU: 1 PID: 3154 Comm: hostapd Tainted: G        W       4.14.195 #0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.815371] Stack : 00000000 00000000 00000000 00000019 00000000 00000000 00000000 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.832215]         00000000 00000000 00000000 00000000 00000000 00000001 86149ae8 53261622
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.849039]         86149b80 00000000 00000000 00007bf8 00000038 8049c858 00000007 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.865882]         00000000 80550000 000c36da 20202020 86149ac8 80000000 00000000 804e0448
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.882714]         00000001 00200000 ffffffff 00000019 00000000 802ad210 00000004 806b0004
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.899539]         ...
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.904576] Call Trace:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.904603] [<8049c858>] 0x8049c858
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.916744] [<802ad210>] 0x802ad210
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.923854] [<8000c1a0>] 0x8000c1a0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.930964] [<8000c1a8>] 0x8000c1a8
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.938079] [<804856b4>] 0x804856b4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.945189] [<80072a84>] 0x80072a84
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.952352] [<800c8150>] 0x800c8150
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.959495] [<800c739c>] 0x800c739c
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.966612] [<800c7ff0>] 0x800c7ff0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.973724] [<800cc960>] 0x800cc960
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.980865] [<800d22a4>] 0x800d22a4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.988029] [<800c48d4>] 0x800c48d4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665643.995168] [<800f4468>] 0x800f4468
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.002334] [<800f0124>] 0x800f0124
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.009449] [<800f4cac>] 0x800f4cac
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.016559] [<8005adb8>] 0x8005adb8
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.023738] [<800189d0>] 0x800189d0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.030851] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.037975] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.045080] [<8033fb90>] 0x8033fb90
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.052192] [<8001e588>] 0x8001e588
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.059312]
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.065222] Mem-Info:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709] active_anon:466 inactive_anon:13 isolated_anon:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709]  active_file:97 inactive_file:74 isolated_file:32
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709]  unevictable:0 dirty:0 writeback:0 unstable:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709]  slab_reclaimable:396 slab_unreclaimable:1878
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709]  mapped:16 shmem:90 pagetables:64 bounce:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.070709]  free:4214 free_pcp:40 free_cma:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.136460] Node 0 active_anon:1864kB inactive_anon:52kB active_file:388kB inactive_file:296kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:64kB dirty:0kB writeback:0kB shmem:360kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.182117] Normal free:16856kB min:16384kB low:20480kB high:24576kB active_anon:1864kB inactive_anon:52kB active_file:228kB inactive_file:412kB unevictable:0kB writepending:0kB present:131072kB managed:123880kB mlocked:0kB kernel_stack:520kB pagetables:256kB bounce:0kB free_pcp:160kB local_pcp:160kB free_cma:0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.239469] lowmem_reserve[]: 0 0 0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.247151] Normal: 200*4kB (UME) 244*8kB (UME) 145*16kB (ME) 79*32kB (UM) 25*64kB (ME) 13*128kB (ME) 4*256kB (ME) 4*512kB (M) 3*1024kB (ME) 0*2048kB 0*4096kB = 17008kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.278151] 360 total pagecache pages
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.285619] 0 pages in swap cache
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.293161] Swap cache stats: add 0, delete 0, find 0/0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.304444] Free swap  = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.310820] Total swap = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.317161] 32768 pages RAM
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.322886] 0 pages HighMem/MovableOnly
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665644.331227] 1798 pages reserved
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.338360] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.356320] [  539]     0   539      316       10       3       0        0             0 ubusd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.374572] [  540]     0   540      232        7       3       0        0             0 askfirst
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.393288] [  557]     0   557      258        8       4       0        0             0 urngd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.411533] [  999]     0   999      323       41       5       0        0             0 logd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.429630] [ 1126]     0  1126      271        7       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.448572] [ 1183]     0  1183      437       43       5       0        0             0 netifd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.466098] [ 1219]     0  1219      363       26       5       0        0             0 odhcpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.483699] [ 1639]     0  1639      338       28       3       0        0             0 uhttpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.501255] [ 1709]     0  1709      310       29       5       0        0             0 pppd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.518449] [ 2514]     0  2514      305       10       4       0        0             0 ntpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.535669] [ 3049]   453  3049      342       29       5       0        0             0 dnsmasq
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.555056] [ 3154]     0  3154      451       40       4       0        0             0 hostapd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.573133] [15069]     0 15069      288       26       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.591838] [15070]     0 15070      306       12       4       0        0             0 ash
Sun Dec  6 17:51:10 2020 kern.info kernel: [665644.609930] [25000]     0 25000      343       21       4       0        0             0 logread
Sun Dec  6 17:51:10 2020 kern.err kernel: [665644.629098] Out of memory: Kill process 1183 (netifd) score 1 or sacrifice child
Sun Dec  6 17:51:10 2020 kern.err kernel: [665644.644514] Killed process 1709 (pppd) total-vm:1240kB, anon-rss:116kB, file-rss:0kB, shmem-rss:0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.057416] odhcpd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null),  order=0, oom_score_adj=0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.082133] CPU: 1 PID: 1219 Comm: odhcpd Tainted: G        W       4.14.195 #0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.096856] Stack : 00000000 00000000 00000000 00000019 00000000 00000000 00000000 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.113690]         00000000 00000000 00000000 00000000 00000000 00000001 8707fae8 53261622
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.130532]         8707fb80 00000000 00000000 00008f38 00000038 8049c858 00000007 00000000
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.147367]         00000000 80550000 000140d5 20202020 8707fac8 80000000 00000000 804e0448
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.164197]         00000001 00200000 ffffffff 00000019 00000003 802ad210 00000004 806b0004
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.181028]         ...
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.186076] Call Trace:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.186155] [<8049c858>] 0x8049c858
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.198317] [<802ad210>] 0x802ad210
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.205472] [<8000c1a0>] 0x8000c1a0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.212585] [<8000c1a8>] 0x8000c1a8
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.219695] [<804856b4>] 0x804856b4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.226805] [<80072a84>] 0x80072a84
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.233960] [<800c8150>] 0x800c8150
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.241141] [<800c739c>] 0x800c739c
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.248282] [<800c7ff0>] 0x800c7ff0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.255395] [<800cc960>] 0x800cc960
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.262539] [<800d22a4>] 0x800d22a4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.269704] [<800c48d4>] 0x800c48d4
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.276846] [<800f4468>] 0x800f4468
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.284048] [<800f0124>] 0x800f0124
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.291186] [<800f4cac>] 0x800f4cac
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.298301] [<8005ae10>] 0x8005ae10
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.305512] [<800189d0>] 0x800189d0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.312636] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.319745] [<80051820>] 0x80051820
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.326887] [<8049eac0>] 0x8049eac0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.334002] [<80094500>] 0x80094500
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.341109] [<800955a0>] 0x800955a0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.348219] [<8001e588>] 0x8001e588
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.355337]
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.360667] Mem-Info:
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741] active_anon:441 inactive_anon:13 isolated_anon:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741]  active_file:152 inactive_file:147 isolated_file:32
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741]  unevictable:0 dirty:0 writeback:0 unstable:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741]  slab_reclaimable:406 slab_unreclaimable:1886
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741]  mapped:34 shmem:90 pagetables:64 bounce:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.365741]  free:4073 free_pcp:52 free_cma:0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.430636] Node 0 active_anon:1764kB inactive_anon:52kB active_file:608kB inactive_file:588kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:136kB dirty:0kB writeback:0kB shmem:360kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.475939] Normal free:16276kB min:16384kB low:20480kB high:24576kB active_anon:1864kB inactive_anon:52kB active_file:772kB inactive_file:392kB unevictable:0kB writepending:0kB present:131072kB managed:123880kB mlocked:0kB kernel_stack:520kB pagetables:256kB bounce:0kB free_pcp:208kB local_pcp:64kB free_cma:0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.531195] lowmem_reserve[]: 0 0 0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.538358] Normal: 178*4kB (UME) 180*8kB (UME) 142*16kB (ME) 80*32kB (UME) 26*64kB (ME) 12*128kB (ME) 5*256kB (ME) 4*512kB (M) 3*1024kB (ME) 0*2048kB 0*4096kB = 16584kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.569737] 417 total pagecache pages
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.577459] 0 pages in swap cache
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.584316] Swap cache stats: add 0, delete 0, find 0/0
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.595008] Free swap  = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.600940] Total swap = 0kB
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.606862] 32768 pages RAM
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.612583] 0 pages HighMem/MovableOnly
Sun Dec  6 17:51:10 2020 kern.warn kernel: [665645.620397] 1798 pages reserved
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.626853] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.644032] [  539]     0   539      316       10       3       0        0             0 ubusd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.661376] [  540]     0   540      232        7       3       0        0             0 askfirst
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.679881] [  557]     0   557      258        8       4       0        0             0 urngd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.697266] [  999]     0   999      323       41       5       0        0             0 logd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.714475] [ 1126]     0  1126      271        7       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.732352] [ 1183]     0  1183      437       42       5       0        0             0 netifd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.749873] [ 1219]     0  1219      363       26       5       0        0             0 odhcpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.767419] [ 1639]     0  1639      338       28       3       0        0             0 uhttpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.785885] [ 2514]     0  2514      305       47       4       0        0             0 ntpd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.803583] [ 3049]   453  3049      342       29       5       0        0             0 dnsmasq
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.821311] [ 3154]     0  3154      451       40       4       0        0             0 hostapd
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.839000] [15069]     0 15069      288       26       3       0        0             0 dropbear
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.856862] [15070]     0 15070      306       12       4       0        0             0 ash
Sun Dec  6 17:51:10 2020 kern.info kernel: [665645.875411] [25000]     0 25000      343       21       4       0        0             0 logread
Sun Dec  6 17:51:10 2020 kern.err kernel: [665645.893478] Out of memory: Kill process 1183 (netifd) score 1 or sacrifice child
Sun Dec  6 17:51:10 2020 kern.err kernel: [665645.908527] Killed process 1183 (netifd) total-vm:1748kB, anon-rss:168kB, file-rss:0kB, shmem-rss:0kB
Sun Dec  6 17:51:10 2020 kern.info kernel: [665646.200092] oom_reaper: reaped process 1183 (netifd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sun Dec  6 17:51:10 2020 daemon.err dnsmasq-dhcp[3049]: failed to write /tmp/dhcp.leases: No space left on device (retry in 60s)
Sun Dec  6 17:51:10 2020 daemon.notice hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Not found