The USB setup for this router in 23.05.2 seems to be not quite right.
To get USB drives to detect at all, the router needs the following pkg :-
opkg install kmod-usb-storage
Once USB storage is detected and mounted, it works OK for a short time, and will read data ok.
A minute or two after writing to the USB disk however, the system starts to OOM-kill processes repeatedly, until the dropbear server is killed and the ssh session disconnects. This usually results in a reboot of the whole router.
Getting a log of this event has been a pain, as the dead ssh server also closes the ssh client window on both my windows and ubuntu clients.
I have managed to preserve the stacktrace in system.log however, by setting system.log to be recorded on the USB disk. Writing to the log causes the OOM, which is recorded in the log. It's stupid, but it works.
Below is the stacktrace for the first OOM in the chain, this is followed by successive OOMs called from kthreadd, logread, procd, then 'top' which I happened to be running in another ssh session, then logread again, and finally the router kicks me off and reboots.
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.227663] netifd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.236138] CPU: 0 PID: 1383 Comm: netifd Not tainted 5.15.137 #0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.242262] Hardware name: BCM5301X
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.245765] Function entered at [<c0109580>] from [<c0105070>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.251610] Function entered at [<c0105070>] from [<c041bbc0>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.257458] Function entered at [<c041bbc0>] from [<c021517c>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.263309] Function entered at [<c021517c>] from [<c02158fc>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.269155] Function entered at [<c02158fc>] from [<c021633c>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.274997] Function entered at [<c021633c>] from [<c0262560>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.280847] Function entered at [<c0262560>] from [<c0262af4>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.286694] Function entered at [<c0262af4>] from [<c02fa1b4>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.292536] Function entered at [<c02fa1b4>] from [<c029cd6c>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.298385] Function entered at [<c029cd6c>] from [<c028f080>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.304234] Function entered at [<c028f080>] from [<c0100040>]
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.310073] Exception stack(0xc1ecdfa8 to 0xc1ecdff0)
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.315134] dfa0: 0000000e b6ee6174 bee47d4c bee47d30 0000000e bee47d24
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.323333] dfc0: 0000000e b6ee6174 bee47d30 00000055 b6ee6250 b6eec190 000397b4 bee47d84
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.331530] dfe0: 00038e70 bee47d20 000179d0 b6f5de38
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.336667] Mem-Info:
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] active_anon:2075 inactive_anon:535 isolated_anon:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] active_file:2828 inactive_file:14989 isolated_file:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] unevictable:0 dirty:1642 writeback:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] slab_reclaimable:5229 slab_unreclaimable:2873
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] mapped:2754 shmem:862 pagetables:203 bounce:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] kernel_misc_reclaimable:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.338959] free:93321 free_pcp:2119 free_cma:0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.374992] Node 0 active_anon:8300kB inactive_anon:2140kB active_file:11312kB inactive_file:59956kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:11016kB dirty:6568kB writeback:0kB shmem:3448kB writeback_tmp:0kB kernel_stack:768kB pagetables:812kB all_unreclaimable? no
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.400334] Normal free:15888kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:3980kB inactive_file:56740kB unevictable:0kB writepending:6568kB present:131072kB managed:117108kB mlocked:0kB bounce:0kB free_pcp:332kB local_pcp:96kB free_cma:0kB
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.427323] lowmem_reserve[]: 0 3072 3072
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.431562] HighMem free:357396kB min:384kB low:14136kB high:27888kB reserved_highatomic:0KB active_anon:8300kB inactive_anon:2140kB active_file:7332kB inactive_file:3216kB unevictable:0kB writepending:0kB present:393216kB managed:393216kB mlocked:0kB bounce:0kB free_pcp:8140kB local_pcp:6548kB free_cma:0kB
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.458947] lowmem_reserve[]: 0 0 0
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.462479] Normal: 92*4kB (UME) 114*8kB (UME) 1*16kB (M) 2*32kB (UE) 28*64kB (UME) 21*128kB (ME) 19*256kB (UME) 8*512kB (UE) 1*1024kB (U) 0*2048kB 0*4096kB = 15824kB
Mon Feb 5 22:52:34 2024 kern.warn kernel: [ 559.477489] HighMem: 1*4kB (M) 2*8kB (UM) 8*16kB (UM) 4*32kB (U) 6*64kB (U) 3*128kB (U) 0*256kB 0*512kB 2*1024kB (UM) 1*2048kB (U) 86*4096kB (M) = 357396kB
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.491550] 18679 total pagecache pages
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.495409] 0 pages in swap cache
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.498758] Swap cache stats: add 0, delete 0, find 0/0
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.504000] Free swap = 0kB
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.506916] Total swap = 0kB
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.509806] 131072 pages RAM
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.512694] 98304 pages HighMem/MovableOnly
Mon Feb 5 22:52:35 2024 kern.warn kernel: [ 559.516913] 3491 pages reserved
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.520067] Tasks state (memory values in pages):
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.524784] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.533452] [ 541] 81 541 226 190 10240 0 0 ubusd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.541689] [ 542] 0 542 185 132 10240 0 0 askfirst
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.550178] [ 577] 0 577 202 164 8192 0 0 urngd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.558410] [ 887] 514 887 243 189 10240 0 0 logd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.566551] [ 953] 0 953 466 413 10240 0 0 rpcd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.574668] [ 1211] 0 1211 227 191 10240 0 0 dropbear
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.583153] [ 1318] 0 1318 525 218 10240 0 0 hostapd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.591560] [ 1319] 0 1319 525 218 8192 0 0 wpa_supplicant
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.600568] [ 1345] 101 1345 685 520 10240 0 0 wpa_supplicant
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.609582] [ 1347] 101 1347 735 581 12288 0 0 hostapd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.617982] [ 1383] 0 1383 335 241 8192 0 0 netifd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.626295] [ 1502] 0 1502 261 194 8192 0 0 odhcpd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.634587] [ 1620] 0 1620 383 287 10240 0 0 uhttpd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.642897] [ 1669] 0 1669 319 276 10240 0 0 dbus-daemon
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.651652] [ 1715] 65534 1715 383 322 8192 0 0 avahi-daemon
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.660497] [ 2448] 0 2448 270 216 10240 0 0 pppd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.668639] [ 2786] 0 2786 6201 2107 32768 0 0 smbd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.676778] [ 2787] 0 2787 3930 1213 24576 0 0 nmbd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.684896] [ 2848] 0 2848 525 218 8192 0 0 ntpd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.693031] [ 2877] 123 2877 288 221 10240 0 0 ntpd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.701177] [ 2933] 0 2933 5852 730 28672 0 0 smbd-notifyd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.710008] [ 2934] 0 2934 5850 683 26624 0 0 smbd-cleanupd
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.718935] [ 3423] 0 3423 525 218 10240 0 0 dnsmasq
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.727361] [ 3430] 453 3430 318 266 10240 0 0 dnsmasq
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.735764] [ 3541] 0 3541 243 190 10240 0 0 logread
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.744144] [ 3927] 0 3927 232 178 8192 0 0 dropbear
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.752631] [ 3928] 0 3928 289 238 10240 0 0 ash
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.760686] [ 3936] 0 3936 289 221 10240 0 0 tail
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.768822] [ 3937] 0 3937 232 194 10240 0 0 dropbear
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.777317] [ 3938] 0 3938 289 254 10240 0 0 ash
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.785350] [ 3946] 0 3946 288 220 10240 0 0 top
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.793402] [ 3947] 0 3947 232 194 10240 0 0 dropbear
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.801893] [ 3948] 0 3948 289 238 10240 0 0 ash
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.809951] [ 3956] 0 3956 292 225 8192 0 0 chown
Mon Feb 5 22:52:35 2024 kern.info kernel: [ 559.818177] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=smbd,pid=2786,uid=0
Mon Feb 5 22:52:35 2024 kern.err kernel: [ 559.830545] Out of memory: Killed process 2786 (smbd) total-vm:24804kB, anon-rss:1308kB, file-rss:5204kB, shmem-rss:1916kB, UID:0 pgtables:32kB oom_score_adj:0
Mon Feb 5 22:52:36 2024 kern.warn kernel: [ 560.933052] kthreadd invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=1, oom_score_adj=0
And here's a list of the USB drivers and firmware installed
root@OpenWrt:/mnt/sda1# opkg list-installed |grep usb
brcmfmac-firmware-usb - 20230804-1
kmod-phy-bcm-ns-usb2 - 5.15.137-1
kmod-phy-bcm-ns-usb3 - 5.15.137-1
kmod-usb-bcma - 5.15.137-1
kmod-usb-core - 5.15.137-1
kmod-usb-storage - 5.15.137-1