Switch crashes on Turris Omnia

I have a couple of Turris Omnia and installed OpenWrt 23.05.0-rc4 on them. In one of them, the switch crashes after a couple of days while the others keep working. The only difference is that this particular system has a lxc instance running. The log looks always the same and there seems to be a kernel crash. I dont know how to debug this further, where should i start? There is no specific task running at that time of the day. Happened in rc3 as well. Any help appreciated.

Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.606007] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.614521] CPU: 1 PID: 5310 Comm: netifd Not tainted 5.15.132 #0
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.620721] Hardware name: Marvell Armada 380/385 (Device Tree)
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.626743] Function entered at [<c010e150>] from [<c01099a4>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.632677] Function entered at [<c01099a4>] from [<c04c16e4>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.638610] Function entered at [<c04c16e4>] from [<c0219a74>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.644543] Function entered at [<c0219a74>] from [<c021a1d8>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.650475] Function entered at [<c021a1d8>] from [<c021abc0>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.656407] Function entered at [<c021abc0>] from [<c02638f0>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.662339] Function entered at [<c02638f0>] from [<c0263e70>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.668270] Function entered at [<c0263e70>] from [<c02f45b8>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.674202] Function entered at [<c02f45b8>] from [<c029b220>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.680135] Function entered at [<c029b220>] from [<c028e010>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.686067] Function entered at [<c028e010>] from [<c0100060>]
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.691999] Exception stack(0xc4d27fa8 to 0xc4d27ff0)
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.697149] 7fa0:                   0000000e b6e59b44 be8cdd3c be8cdd20 0000000e be8cdd14
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.705434] 7fc0: 0000000e b6e59b44 be8cdd20 00000055 b6e59a40 b6e5fbc0 000387b4 be8cdd74
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.713718] 7fe0: 00037e68 be8cdd10 00017a0c b6f1234c
Fri Oct  6 23:17:00 2023 kern.warn kernel: [173598.718881] Mem-Info:
Fri Oct  6 23:17:00 2023 kern.info kernel: [173599.212691] [   8758]   101  8758     4668     2269    26624        0             0 systemd-resolve
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.221852] [   8760]     0  8760      513      206    10240        0             0 ntpd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.230055] [   8761]   123  8761      275      139    10240        0             0 ntpd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.238258] [   8778]     0  8778     8975     3633    36864        0             0 unattended-upgr
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.247419] [   8781]     0  8781     1627      377    16384        0             0 agetty
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.255791] [   8782]     0  8782     2303     1239    16384        0         -1000 sshd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.263995] [   9156]     0  9156    19556     4480    61440        0             0 smbd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.289647] [   9190]     0  9190    19414     3775    59392        0             0 samba-bgqd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.298375] [  14876]     0 14876      276      141     8192        0             0 sleep
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.306665] [  14881]     0 14881      275      123    10240        0             0 sleep
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.314951] [  14896]     0 14896    11934     1487    20480        0          -900 snapperd
Fri Oct  6 23:17:01 2023 kern.info kernel: [173599.323501] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/lxc.payload.zeus/system.slice/smbd.service,task=smbd,pid=9156,uid=0
Fri Oct  6 23:17:01 2023 kern.err kernel: [173599.339562] Out of memory: Killed process 9156 (smbd) total-vm:78224kB, anon-rss:2200kB, file-rss:14824kB, shmem-rss:896kB, UID:0 pgtables:60kB oom_score_adj:0
Fri Oct  6 23:17:01 2023 kern.warn kernel: [173600.211770] Mem-Info:
Fri Oct  6 23:17:02 2023 kern.warn kernel: [173600.308443] HighMem free:850112kB min:512kB low:7612kB high:14712kB reserved_highatomic:0KB active_anon:30064kB inactive_anon:752kB active_file:73996kB inactive_file:346820kB unevictable:0kB writepending:24kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:232kB local_pcp:0kB free_cma:0kB
Fri Oct  6 23:17:02 2023 kern.warn kernel: [173600.353987] HighMem: 3827*4kB (UM) 1703*8kB (UM) 888*16kB (UM) 369*32kB (UM) 94*64kB (UM) 42*128kB (UM) 17*256kB (UM) 13*512kB (UM) 5*1024kB (UM) 3*2048kB (UM) 186*4096kB (M) = 850468kB
Fri Oct  6 23:17:02 2023 kern.warn kernel: [173600.370669] 240399 total pagecache pages
Fri Oct  6 23:17:02 2023 kern.info kernel: [173600.421772] [    815]     0   815      173      107     8192        0             0 askfirst
Fri Oct  6 23:17:02 2023 kern.info kernel: [173600.498491] [   2005]     0  2005      277      192     6144        0             0 crond
Fri Oct  6 23:17:02 2023 kern.info kernel: [173600.748753] [   8782]     0  8782     2303     1239    16384        0         -1000 sshd
Fri Oct  6 23:17:02 2023 kern.info kernel: [173600.756958] [  14876]     0 14876      276      141     8192        0             0 sleep
Fri Oct  6 23:17:02 2023 kern.info kernel: [173600.782085] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=lxc.payload.zeus,mems_allowed=0,global_oom,task_memcg=/lxc.payload.zeus/system.slice/unattended-upgrades.service,task=unattended-upgr,pid=8778,uid=0
Fri Oct  6 23:17:02 2023 kern.err kernel: [173600.801719] Out of memory: Killed process 8778 (unattended-upgr) total-vm:35900kB, anon-rss:4900kB, file-rss:9632kB, shmem-rss:0kB, UID:0 pgtables:36kB oom_score_adj:0
Fri Oct  6 23:17:03 2023 kern.warn kernel: [173601.718739] Normal free:15956kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:4824kB inactive_file:535436kB unevictable:0kB writepending:40944kB present:786432kB managed:755888kB mlocked:0kB bounce:0kB free_pcp:440kB local_pcp:240kB free_cma:0kB
Fri Oct  6 23:17:03 2023 kern.warn kernel: [173601.745938] lowmem_reserve[]: 0 10240 10240
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.855660] [    814]    81   814      218      181    10240        0             0 ubusd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.863994] [    815]     0   815      173      107     8192        0             0 askfirst
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.872553] [    855]     0   855      190      152     8192        0             0 urngd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.880848] [   1291]   514  1291      247      193    10240        0             0 logd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.889051] [   1345]     0  1345     1521      884    16384        0             0 rpcd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.897254] [   1616]     0  1616      513      166    10240        0             0 hostapd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.905714] [   1617]     0  1617      513      182    10240        0             0 wpa_supplicant
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.914786] [   1634]   101  1634      671      526     8192        0             0 wpa_supplicant
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.923859] [   1640]   101  1640      738      607    10240        0             0 hostapd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.932322] [   1930]     0  1930      250      196    10240        0             0 odhcpd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.940698] [   2005]     0  2005      277      192     6144        0             0 crond
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.948987] [   2059]     0  2059     1145     1015    12288        0         -1000 sshd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.957190] [   2125]     0  2125      401      354     8192        0             0 uhttpd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.965563] [   2214]     0  2214      579      525    10240        0             0 vnstatd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.974097] [   2577]     0  2577     1193     1095    14336        0             0 openvpn
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.982581] [   2578]     0  2578     1170     1119    14336        0             0 openvpn
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.991180] [   5310]     0  5310      312      260    10240        0             0 netifd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173601.999632] [   5428]     0  5428      283      247    10240        0             0 watchcat.sh
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.008465] [   5517]     0  5517      513      206    10240        0             0 dnsmasq
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.016941] [   5522]   453  5522      896      690    10240        0             0 dnsmasq
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.025417] [   6111]     0  6111     1463      476    12288        0             0 lxc-start
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.034069] [   6116]     0  6116     8031     1848    26624        0             0 systemd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.042541] [   7305]     0  7305      193      144    10240        0             0 odhcp6c
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.051023] [   7311]     0  7311      275      139    10240        0             0 udhcpc
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.059413] [   7708]     0  7708      354      322    10240        0             0 dynamic_dns_upd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.068587] [   8701]     0  8701    11382     6940    51200        0          -250 systemd-journal
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.077756] [   8739]   100  8739     3509     1073    22528        0             0 systemd-network
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.086932] [   8742]     0  8742    10305     1281    32768        0             0 accounts-daemon
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.096103] [   8743]     0  8743     1944      501    16384        0             0 cron
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.104303] [   8744]   103  8744     1555      800    14336        0          -900 dbus-daemon
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.113129] [   8751]     0  8751     5928     3098    30720        0             0 networkd-dispat
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.122300] [   8752]   104  8752     6403      757    18432        0             0 rsyslogd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.130864] [   8753]     0  8753     1823      969    16384        0             0 smartd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.139256] [   8754]     0  8754     3254     1324    20480        0             0 systemd-logind
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.182323] [   8782]     0  8782     2303     1239    16384        0         -1000 sshd
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.190569] [  14876]     0 14876      276      141     8192        0             0 sleep
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.198879] [  14881]     0 14881      275      123    10240        0             0 sleep
Fri Oct  6 23:17:03 2023 kern.info kernel: [173602.207186] [  14896]     0 14896    11934     1487    20480        0          -900 snapperd
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.419260] kthreadd invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=1, oom_score_adj=0
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.429852] CPU: 0 PID: 2 Comm: kthreadd Not tainted 5.15.132 #0
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.435963] Hardware name: Marvell Armada 380/385 (Device Tree)
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.441986] Function entered at [<c010e150>] from [<c01099a4>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.447919] Function entered at [<c01099a4>] from [<c04c16e4>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.453853] Function entered at [<c04c16e4>] from [<c0219a74>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.459785] Function entered at [<c0219a74>] from [<c021a1d8>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.465718] Function entered at [<c021a1d8>] from [<c021abc0>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.471649] Function entered at [<c021abc0>] from [<c02638f0>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.477581] Function entered at [<c02638f0>] from [<c0126370>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.483514] Function entered at [<c0126370>] from [<c0127e58>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.489445] Function entered at [<c0127e58>] from [<c01282a8>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.495377] Function entered at [<c01282a8>] from [<c014a908>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.501309] Function entered at [<c014a908>] from [<c0100134>]
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.507240] Exception stack(0xc105ffb0 to 0xc105fff8)
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.512391] ffa0:                                     00000000 00000000 00000000 00000000
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.520675] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.528958] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
Fri Oct  6 23:17:04 2023 kern.warn kernel: [173602.535782] Mem-Info:
Fri Oct  6 23:17:04 2023 kern.info kernel: [173602.771207] [   1345]     0  1345     1521      884    16384        0             0 rpcd
Fri Oct  6 23:17:04 2023 kern.info kernel: [173602.796992] [   1634]   101  1634      671      526     8192        0             0 wpa_supplicant
Fri Oct  6 23:17:04 2023 kern.info kernel: [173602.898957] [   5522]   453  5522      896      690    10240        0             0 dnsmasq
Fri Oct  6 23:17:04 2023 kern.info kernel: [173602.907423] [   6111]     0  6111     1463      476    12288        0             0 lxc-start
Fri Oct  6 23:17:04 2023 kern.info kernel: [173602.916065] [   6116]     0  6116     8031     1848    26624        0             0 systemd
Fri Oct  6 23:17:06 2023 kern.info kernel: [173604.692148] [   1640]   101  1640      738      607    10240        0             0 hostapd
Fri Oct  6 23:17:06 2023 kern.info kernel: [173604.733998] [   2214]     0  2214      579      525    10240        0             0 vnstatd
Fri Oct  6 23:17:06 2023 kern.info kernel: [173604.742487] [   2577]     0  2577     1193     1095    14336        0             0 openvpn
Fri Oct  6 23:17:06 2023 kern.info kernel: [173604.827604] [   7708]     0  7708      354      322    10240        0             0 dynamic_dns_upd
Fri Oct  6 23:17:06 2023 kern.info kernel: [173604.999888] [  14912]     0 14912      194      152     8192        0             0 modprobe
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.744562] lowmem_reserve[]: 0 10240 10240
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.748856] HighMem free:859800kB min:512kB low:7612kB high:14712kB reserved_highatomic:0KB active_anon:17924kB inactive_anon:732kB active_file:74072kB inactive_file:346932kB unevictable:0kB writepending:660kB present:1310720kB managed:1310720kB mlocked:0kB bounce:0kB free_pcp:1940kB local_pcp:340kB free_cma:0kB
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.776739] lowmem_reserve[]: 0 0 0
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.780329] Normal: 375*4kB (UME) 286*8kB (UME) 97*16kB (UME) 61*32kB (ME) 48*64kB (UME) 25*128kB (UM) 9*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15868kB
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.794836] HighMem: 2830*4kB (UM) 1836*8kB (UM) 965*16kB (UM) 412*32kB (UM) 134*64kB (UM) 68*128kB (UM) 26*256kB (UM) 15*512kB (UM) 5*1024kB (UM) 3*2048kB (UM) 186*4096kB (M) = 859368kB
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.811608] 240897 total pagecache pages
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.827347] Total swap = 0kB
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.830320] 524288 pages RAM
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.833292] 327680 pages HighMem/MovableOnly
Fri Oct  6 23:17:07 2023 kern.warn kernel: [173605.837664] 7636 pages reserved
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.840901] Tasks state (memory values in pages):
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.845707] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.854482] [    814]    81   814      218      181    10240        0             0 ubusd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.862783] [    815]     0   815      173      107     8192        0             0 askfirst
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.871341] [    855]     0   855      190      152     8192        0             0 urngd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.879639] [   1291]   514  1291      247      193    10240        0             0 logd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.887844] [   1345]     0  1345     1521      884    16384        0             0 rpcd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.896048] [   1616]     0  1616      513      166    10240        0             0 hostapd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.904508] [   1617]     0  1617      513      182    10240        0             0 wpa_supplicant
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.913584] [   1634]   101  1634      671      526     8192        0             0 wpa_supplicant
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.922660] [   1640]   101  1640      738      607    10240        0             0 hostapd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.931131] [   1930]     0  1930      250      196    10240        0             0 odhcpd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.939512] [   2005]     0  2005      277      192     6144        0             0 crond
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.947803] [   2059]     0  2059     1145     1015    12288        0         -1000 sshd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.956041] [   2125]     0  2125      401      354     8192        0             0 uhttpd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.964416] [   2214]     0  2214      579      525    10240        0             0 vnstatd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.972883] [   2577]     0  2577     1193     1095    14336        0             0 openvpn
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.981350] [   2578]     0  2578     1170     1119    14336        0             0 openvpn
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.989817] [   5310]     0  5310      312      260    10240        0             0 netifd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173605.998195] [   5428]     0  5428      283      247    10240        0             0 watchcat.sh
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.007009] [   5517]     0  5517      513      206    10240        0             0 dnsmasq
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.015470] [   5522]   453  5522      896      690    10240        0             0 dnsmasq
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.023934] [   6111]     0  6111     1463      476    12288        0             0 lxc-start
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.032574] [   6116]     0  6116     8031     1848    26624        0             0 systemd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.041041] [   7305]     0  7305      193      144    10240        0             0 odhcp6c
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.049506] [   7311]     0  7311      275      139    10240        0             0 udhcpc
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.057925] [   7708]     0  7708      354      322    10240        0             0 dynamic_dns_upd
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.067092] [   8701]     0  8701    11382     6944    51200        0          -250 systemd-journal
Fri Oct  6 23:17:07 2023 kern.info kernel: [173606.076256] [   8739]   100  8739     3509     1073    22528        0             0 systemd-network
Fri Oct  6 23:17:08 2023 kern.info kernel: [173606.238009] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=lxc.payload.zeus,mems_allowed=0,global_oom,task_memcg=/lxc.payload.zeus,task=systemd,pid=6116,uid=0
Fri Oct  6 23:17:08 2023 kern.err kernel: [173606.253376] Out of memory: Killed process 6116 (systemd) total-vm:32124kB, anon-rss:1692kB, file-rss:5700kB, shmem-rss:0kB, UID:0 pgtables:26kB oom_score_adj:0
Fri Oct  6 23:17:08 2023 daemon.notice netifd: Network device 'vethvm1' link is down
Fri Oct  6 23:17:08 2023 kern.info kernel: [173607.134538] br-lan: port 6(vethvm1) entered disabled state
Fri Oct  6 23:17:08 2023 kern.info kernel: [173607.184249] device vethvm1 left promiscuous mode
Fri Oct  6 23:17:08 2023 kern.info kernel: [173607.189027] br-lan: port 6(vethvm1) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.619446] mv88e6085 f1072004.mdio-mii:10 lan4: Link is Down
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.686143] mv88e6085 f1072004.mdio-mii:10 lan0: Link is Down
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.692220] br-lan: port 5(lan4) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.697701] br-lan: port 4(lan3) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.703139] br-lan: port 3(lan2) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.708607] br-lan: port 2(lan1) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.714066] br-lan: port 1(lan0) entered disabled state
Fri Oct  6 23:17:09 2023 kern.info kernel: [173607.727382] mvneta f1070000.ethernet eth0: Link is Down
Fri Oct  6 23:17:09 2023 daemon.notice netifd: Network device 'lan4' link is down
Fri Oct  6 23:17:09 2023 daemon.notice netifd: Network device 'lan0' link is down
Fri Oct  6 23:17:09 2023 daemon.notice netifd: Network device 'eth0' link is down

Out of memory. There are several oom errors in the log snippet.

Seems to me that the lxc instance is eating up all your memory. Remove it and it should resolve the problem.

2 Likes

LXC has a memory limit of <40% of total memory (2G) set. Usually the system uses only 300MB out of 2G. I used the LXC Container for >3 years on that machine and never had issues with it, those just appeared with rc4. Any other ideas?

I'd monitor free to see what is using the memory.

i used free to monitor free

RAM was never fully used and OOM triggered anyway. I did some research and found this:

Indeed disabling MGLRU (which was backportet to 5.15) fixes these OOM killer triggers

echo 0 | sudo tee /sys/kernel/mm/lru_gen/enabled

This happend in my case while syncing big Files (>50 Gbyte) with rsync.

Okay, that looks like it is interesting. Could you please verify that it works for you with enabled MGLRU since kernel 6.1.54? I see that some fixes landed (referenced in the 1st issue), there.
If it works, I'll work on backporting the recent fixes for kernel 5.15.

Is there a precompiled snapshot image with that kernel version?

Nope, there isnt, because kernel 6.1 is for mvebu labeled as testing.

Will be away for 2 weeks now. But afterwards will be able to compile and test.