A few days ago upgraded this device from a 23.x version. However noticed that from time to time is hanging up (ping via LAN but not Luci neither ssh, neither reachable by WiFi).
I setup a syslog server to be able to track the problem and today noticed it resets with hostapd invoked oom-killer.
Some of the logs:
Mar 8 04:45:30 Paco-Salon collectd[2342]: Sleeping only 2s because the next interval is 41.436 seconds in the past!
Mar 8 04:46:06 Paco-Salon hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Mar 8 04:46:10 Paco-Salon hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Mar 8 04:49:36 Paco-Salon kernel: [158983.725019] hostapd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Mar 8 04:49:36 Paco-Salon kernel: [158983.736265] CPU: 0 UID: 101 PID: 1909 Comm: hostapd Tainted: G O 6.12.71 #0
Mar 8 04:49:36 Paco-Salon kernel: [158983.745008] Tainted: [O]=OOT_MODULE
Mar 8 04:49:36 Paco-Salon kernel: [158983.748693] Hardware name: TP-Link Archer C7 v4
Mar 8 04:49:36 Paco-Salon kernel: [158983.753462] Stack : 807b561c 800c9d00 00000001 00000004 00000000 00000000 00000000 00000000
Mar 8 04:49:36 Paco-Salon kernel: [158983.762213] 00000000 00000000 00000000 00000000 00000000 00000001 828d1b98 248e86ee
Mar 8 04:49:36 Paco-Salon kernel: [158983.770959] 828d1c30 00000000 00000000 828d1ad0 00000038 8075a624 ffffffea 00000115
Mar 8 04:49:36 Paco-Salon kernel: [158983.779712] 828d1adc 00000115 8085f610 ffffffff 80774c58 828d1b78 80774c58 00000800
Mar 8 04:49:36 Paco-Salon kernel: [158983.788467] 00000001 00000840 00000000 00000011 00000000 80406fac 00000000 80a20000
Mar 8 04:49:36 Paco-Salon kernel: [158983.797221] ...
Mar 8 04:49:36 Paco-Salon kernel: [158983.799837] Call Trace:
Mar 8 04:49:36 Paco-Salon kernel: [158983.799844] [<800c9d00>] 0x800c9d00
Mar 8 04:49:36 Paco-Salon kernel: [158983.806312] [<8075a624>] 0x8075a624
Mar 8 04:49:36 Paco-Salon kernel: [158983.810112] [<80406fac>] 0x80406fac
Mar 8 04:49:36 Paco-Salon kernel: [158983.813909] [<80066294>] 0x80066294
Mar 8 04:49:36 Paco-Salon kernel: [158983.817606] [<8006629c>] 0x8006629c
Mar 8 04:49:36 Paco-Salon kernel: [158983.821297] [<8072e7c4>] 0x8072e7c4
Mar 8 04:49:36 Paco-Salon kernel: [158983.825091] [<801a1610>] 0x801a1610
Mar 8 04:49:36 Paco-Salon kernel: [158983.828940] [<801a1ef0>] 0x801a1ef0
Mar 8 04:49:36 Paco-Salon kernel: [158983.832684] [<801a24e8>] 0x801a24e8
Mar 8 04:49:36 Paco-Salon kernel: [158983.836377] [<801fc20c>] 0x801fc20c
Mar 8 04:49:36 Paco-Salon kernel: [158983.840328] [<801fcc0c>] 0x801fcc0c
Mar 8 04:49:36 Paco-Salon kernel: [158983.844120] [<8019dc64>] 0x8019dc64
Mar 8 04:49:36 Paco-Salon kernel: [158983.848375] [<8019e574>] 0x8019e574
Mar 8 04:49:36 Paco-Salon kernel: [158983.852065] [<800b4410>] 0x800b4410
Mar 8 04:49:36 Paco-Salon kernel: [158983.855757] [<800b4fdc>] 0x800b4fdc
Mar 8 04:49:36 Paco-Salon kernel: [158983.859807] [<801d7768>] 0x801d7768
Mar 8 04:49:36 Paco-Salon kernel: [158983.863651] [<801debe4>] 0x801debe4
Mar 8 04:49:36 Paco-Salon kernel: [158983.867700] [<8006f3c4>] 0x8006f3c4
Mar 8 04:49:36 Paco-Salon kernel: [158983.871395] [<8007444c>] 0x8007444c
Mar 8 04:49:36 Paco-Salon kernel: [158983.875102]
Mar 8 04:49:36 Paco-Salon kernel: [158983.877930] Mem-Info:
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] active_anon:2568 inactive_anon:2956 isolated_anon:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] active_file:545 inactive_file:0 isolated_file:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] unevictable:0 dirty:0 writeback:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] slab_reclaimable:471 slab_unreclaimable:2556
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] mapped:63 shmem:873 pagetables:909
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] sec_pagetables:0 bounce:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] kernel_misc_reclaimable:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.880442] free:4268 free_pcp:8 free_cma:0
Mar 8 04:49:36 Paco-Salon kernel: [158983.920508] Node 0 active_anon:10272kB inactive_anon:11824kB active_file:2180kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:252kB dirty:0kB writeback:0kB shmem:3492kB writeback_tmp:0kB kernel_stack:1960kB pagetables:3636kB sec_pagetables:0kB all_unreclaimable? yes
Mar 8 04:49:36 Paco-Salon kernel: [158983.948356] Normal free:17072kB boost:0kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:10272kB inactive_anon:11824kB active_file:2180kB inactive_file:0kB unevictable:0kB writepending:0kB present:131072kB managed:120760kB mlocked:0kB bounce:0kB free_pcp:32kB local_pcp:32kB free_cma:0kB
Mar 8 04:49:36 Paco-Salon kernel: [158983.977226] lowmem_reserve[]: 0 0
Mar 8 04:49:36 Paco-Salon kernel: [158983.981039] Normal: 254*4kB (UME) 225*8kB (UME) 47*16kB (UME) 38*32kB (UME) 22*64kB (UME) 7*128kB (UM) 1*256kB (E) 3*512kB (UE) 2*1024kB (UM) 1*2048kB (M) 1*4096kB (E) 0*8192kB = 17072kB
Mar 8 04:49:36 Paco-Salon kernel: [158983.999055] 1418 total pagecache pages
Mar 8 04:49:36 Paco-Salon kernel: [158984.003224] 0 pages in swap cache
Mar 8 04:49:36 Paco-Salon kernel: [158984.006752] Free swap = 0kB
Mar 8 04:49:36 Paco-Salon kernel: [158984.010041] Total swap = 0kB
Mar 8 04:49:36 Paco-Salon kernel: [158984.013122] 32768 pages RAM
Mar 8 04:49:36 Paco-Salon kernel: [158984.016096] 0 pages HighMem/MovableOnly
Mar 8 04:49:36 Paco-Salon kernel: [158984.020269] 2578 pages reserved
Mar 8 04:49:36 Paco-Salon kernel: [158984.023614] Tasks state (memory values in pages):
Mar 8 05:31:24 Paco-Salon kernel: [161375.188714] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=collectd,pid=8016,uid=0
Mar 8 05:31:24 Paco-Salon kernel: [161375.188790] Out of memory: Killed process 8016 (collectd) total-vm:6556kB, anon-rss:988kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:20kB oom_score_adj:0
Mar 8 05:31:24 Paco-Salon kernel: [161377.240740] oom_reaper: reaped process 8016 (collectd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Mar 8 05:31:46 Paco-Salon hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
Mar 8 05:32:01 Paco-Salon hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
Mar 8 05:40:25 Paco-Salon collectd[8104]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "cpu" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "df" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "disk" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "entropy" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "interface" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "irq" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "iwinfo" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "load" successfully loaded.
Mar 8 05:40:25 Paco-Salon collectd[8104]: plugin_load: plugin "memory" successfully loaded.
Mar 8 05:40:26 Paco-Salon collectd[8104]: plugin_load: plugin "processes" successfully loaded.
Mar 8 05:40:32 Paco-Salon collectd[8104]: plugin_load: plugin "rrdtool" successfully loaded.
Mar 8 05:40:32 Paco-Salon collectd[8104]: rrdtool plugin: RRASingle = true: creating only AVERAGE RRAs
Mar 8 05:40:33 Paco-Salon collectd[8104]: plugin_load: plugin "tcpconns" successfully loaded.
Mar 8 05:40:33 Paco-Salon collectd[8104]: plugin_load: plugin "uptime" successfully loaded.
Mar 8 05:40:48 Paco-Salon hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Mar 8 05:40:49 Paco-Salon collectd[8104]: utils_taskstats: CTRL_CMD_GETFAMILY("TASKSTATS"): No such file or directory
Mar 8 05:40:49 Paco-Salon collectd[8104]: utils_taskstats: get_family_id() = No such file or directory
Mar 8 05:40:49 Paco-Salon collectd[8104]: processes plugin: Creating taskstats handle failed.
Mar 8 05:40:49 Paco-Salon collectd[8104]: Initialization complete, entering read-loop.
Mar 8 05:41:05 Paco-Salon collectd[8104]: tcpconns plugin: conn_read_netlink: Received error -122.
Mar 8 05:41:05 Paco-Salon collectd[8104]: tcpconns plugin: Reading from netlink failed. Will read from /proc from now on.
Mar 8 05:43:46 Paco-Salon hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Mar 8 05:44:34 Paco-Salon hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
Mar 6 08:47:00 Paco-Salon logread[1426]: Logread connected to 10.10.9.7:514 via udp
Mar 6 08:47:01 Paco-Salon netifd: radio0 (2193): wifi-scripts: Starting
This is the mem status after booting:
root@Paco-Salon:~# free total used free shared buff/cache available Mem: 120760 39888 51236 1136 29636 40912 Swap: 0 0 0
About 6 hours later, don’t looks to me like there is a too high memory usage:
root@Paco-Salon:~# free total used free shared buff/cache available Mem: 120760 49344 40348 1920 31068 30348 Swap: 0 0 0 root@Paco-Salon:~# echo 3 > /proc/sys/vm/drop_caches root@Paco-Salon:~# free total used free shared buff/cache available Mem: 120760 50056 59008 1920 11696 39324 Swap: 0 0 0
I’m not really sure if the cause is the hostapd or the collectd, or both …
Any ideas?