So you are using wsl... Wsl1 or 2? Can't you read the error and search it? You have to disable windows PATH merge in wsl... Again search on Google... Also use wsl2 and not wsl1
3 Likes
@robimarko btw to me it seems the ath11k leak is present in the rx path... and i bet it's dma memory not freed... fun thing is that with nss offload the amount of ring to handle is really low so it's incredible that it's still present... (it's in rx path as before the nss fixes the rx speed was very low so i assume the affected function was called less times)
That would be just great, I will bet that they fixed that in wlan-open in the RX throughput improvement patch by accident
It's what Ansuel says about wsl2.
You can also paste the path (without win part) before you do make:
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib"
1 Like
btw the wiki.... https://openwrt.org/docs/guide-developer/toolchain/wsl
@sqrwv news with our special firmware?
2 Likes
i test it ... and borked/crashed the router with fast.com + ipferf ... unfortunately was not capturing any logs but saying this much better throughput on ath11k unsurprisingly ... would tuning the sysctl settings help?
Sorry for stupid question, but after installing openwrt on ax3600 I cannot run opkg_download. and install any package beside already installed by default. Ii should be cause openwrt currently not officially supported on this router? Or I should be able install packages ?
well yes but they should be already set to the correct values for 512mb memory profile... I'm using a revision of the 512mb patch that tweak also the ring values... wonder if that was the cause of the crash... In any case the main problem here is qcom permitting 512mb on these soc...
I'm starting to think what we are missing that would workaround the low ram problem is the tlv 2kb patch but it's very big and require lots of rework... not really maintainable
would you mind to share the patch or push into your repo?
done just now
This is the patch that should reduce skb size
2 Likes
Also i notice many
Thu May 19 14:41:05 2022 daemon.info hostapd: wlan2: STA 28:c2:1f:ce:b2:1d RADIUS: starting accounting session 91933A64C85B4806
Thu May 19 14:41:05 2022 daemon.info hostapd: wlan2: STA 28:c2:1f:ce:b2:1d WPA: pairwise key handshake completed (RSN)
Thu May 19 14:41:05 2022 daemon.notice hostapd: wlan2: EAPOL-4WAY-HS-COMPLETED 28:c2:1f:ce:b2:1d
Thu May 19 14:41:31 2022 kern.warn kernel: [ 170.740681] ath11k c000000.wifi: failed peer find by addr 28:c2:1f:ce:b2:1d on vdev 0 after creation
Thu May 19 14:41:31 2022 kern.warn kernel: [ 170.740739] ath11k c000000.wifi: Failed to add peer: 28:c2:1f:ce:b2:1d for VDEV: 0
Thu May 19 14:41:31 2022 kern.warn kernel: [ 170.748898] ath11k c000000.wifi: Failed to add station: 28:c2:1f:ce:b2:1d for VDEV: 0
Thu May 19 14:41:31 2022 daemon.notice hostapd: wlan1: STA 28:c2:1f:ce:b2:1d IEEE 802.11: Could not add STA to kernel driver
Thu May 19 14:42:12 2022 kern.warn kernel: [ 211.858349] ath11k c000000.wifi: failed peer find by addr 54:8d:5a:63:b9:c4 on vdev 0 after creation
Thu May 19 14:42:12 2022 kern.warn kernel: [ 211.858407] ath11k c000000.wifi: Failed to add peer: 54:8d:5a:63:b9:c4 for VDEV: 0
Thu May 19 14:42:12 2022 kern.warn kernel: [ 211.866600] ath11k c000000.wifi: Failed to add station: 54:8d:5a:63:b9:c4 for VDEV: 0
Thu May 19 14:42:12 2022 daemon.notice hostapd: wlan1: STA 54:8d:5a:63:b9:c4 IEEE 802.11: Could not add STA to kernel driver
Thu May 19 14:42:12 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 IEEE 802.11: authenticated
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 IEEE 802.11: associated (aid 1)
Thu May 19 14:42:13 2022 daemon.notice hostapd: wlan1: Prune association for 54:8d:5a:63:b9:c4
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 RADIUS: starting accounting session 41BC4B44CB307AAF
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 WPA: pairwise key handshake completed (RSN)
Thu May 19 14:42:13 2022 daemon.notice hostapd: wlan2: EAPOL-4WAY-HS-COMPLETED 54:8d:5a:63:b9:c4
No idea if it's related to nss but in theory my patch doesn't touch peer addition logic so i wonder if it's a bug present in current ath11k
I mean the error is that a peer is tried to be added while it's already present... the function return error and the add fails... no idea if it should be handled this way tho
Your special FW is a beast! 
With wifi offload, wifi cpu usage is (almost) non existent now, avg 6% only.
It frees a lot of cpu needed for the other tasks.
Wifi<->WiredLAN speed now does ~940Mbits Tx or Rx (only limited by 1Gbit Wired LAN).
Wifi <-> router I got ~1.5Gbits (this depends on channel interference by neighbors).
I saw you included the nss-firmware file in last build, so people don't have to copy it now.
Thanks
PS: I didn't had memory problems here, will use it and see how it goes in the next days.
3 Likes
That was really cool
1 Like
thing is as soon as you start using other stuff (example i use unbound) ram gets filled real fast... and ath11k or nss expect them to be present... and i'm still convinced that with a very specific load ath11k starts to miss memory like crazy...
3 Likes
I see what you are saying in my other router that has more uptime:
(Will load your latest build now (2022-05-19), I had the previous one here.)
curious why you are using unbound vs stubby?
mhhh i always notice that to get a good stubby configuration you need to use additional process and the packets needs to go to multiple place... with unbound i can just use odhcpd and unbound and that's it
so I applied your patches ... still crashes but yes good throughput until it dies ...this time kept a logread -f
warning long log ...
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237724] netifd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237758] CPU: 2 PID: 3381 Comm: netifd Not tainted 5.15.34 #0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237766] Hardware name: Xiaomi AX3600 (DT)
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237771] Call trace:
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237773] dump_backtrace+0x0/0x160
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237791] show_stack+0x18/0x30
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237797] dump_stack_lvl+0x7c/0xa0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237807] dump_stack+0x18/0x34
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237813] dump_header+0x48/0x184
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237819] oom_kill_process+0x1e0/0x1ec
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237828] out_of_memory+0x1cc/0x320
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237833] __alloc_pages_slowpath.constprop.0+0x73c/0xa34
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237844] __alloc_pages+0x220/0x24c
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237850] pagecache_get_page+0x100/0x400
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237856] filemap_fault+0x5e0/0x800
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237861] __do_fault+0x3c/0x110
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237871] __handle_mm_fault+0x540/0xa70
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237878] handle_mm_fault+0x130/0x180
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237885] do_page_fault+0x14c/0x380
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237895] do_translation_fault+0xa0/0xb0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237900] do_mem_abort+0x40/0xb0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237906] el0_ia+0x8c/0xe0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237916] el0t_64_sync_handler+0x124/0x130
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237920] el0t_64_sync+0x184/0x188
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237928] Mem-Info:
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] active_anon:84 inactive_anon:4074 isolated_anon:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] active_file:1 inactive_file:5 isolated_file:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] unevictable:0 dirty:0 writeback:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] slab_reclaimable:2017 slab_unreclaimable:74578
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] mapped:678 shmem:1249 pagetables:398 bounce:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] kernel_misc_reclaimable:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237932] free:3365 free_pcp:31 free_cma:0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237947] Node 0 active_anon:336kB inactive_anon:16296kB active_file:4kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2712kB dirty:0kB writeback:0kB shmem:4996kB writeback_tmp:0kB kernel_stack:2384kB pagetables:1592kB all_unreclaimable? no
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237959] DMA free:13460kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:336kB inactive_anon:16296kB active_file:4kB inactive_file:20kB unevictable:0kB writepending:0kB present:524288kB managed:417312kB mlocked:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237974] lowmem_reserve[]: 0 0 0 0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.237987] DMA: 1323*4kB (UME) 627*8kB (UME) 88*16kB (UME) 51*32kB (UE) 3*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13540kB
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238029] 1283 total pagecache pages
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238032] 0 pages in swap cache
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238035] Swap cache stats: add 0, delete 0, find 0/0
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238039] Free swap = 0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238041] Total swap = 0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238044] 131072 pages RAM
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238046] 0 pages HighMem/MovableOnly
Thu May 19 18:13:36 2022 kern.warn kernel: [ 361.238049] 26744 pages reserved
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238051] Tasks state (memory values in pages):
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238054] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238082] [ 720] 81 720 317 33 32768 0 0 ubusd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238092] [ 721] 0 721 215 9 28672 0 0 askfirst
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238102] [ 756] 0 756 249 16 32768 0 0 urngd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238121] [ 2148] 0 2148 1772 207 40960 0 0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238130] [ 2180] 0 2180 1836 272 40960 0 0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238139] [ 2181] 0 2181 1843 277 40960 0 0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238148] [ 2182] 0 2182 1836 270 40960 0 0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238158] [ 2184] 0 2184 1835 252 40960 0 0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238167] [ 2226] 0 2226 644 54 32768 0 0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238176] [ 2396] 514 2396 308 21 32768 0 0 logd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238186] [ 2450] 0 2450 571 77 32768 0 0 rpcd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238195] [ 2797] 0 2797 313 18 32768 0 0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238205] [ 3164] 0 3164 396 86 32768 0 0 mwan3track
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238215] [ 3165] 0 3165 399 85 28672 0 0 mwan3track
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238225] [ 3166] 0 3166 448 132 32768 0 0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238234] [ 3167] 0 3167 447 131 28672 0 0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238244] [ 3244] 0 3244 352 25 32768 0 0 ip
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238253] [ 3246] 0 3246 453 137 28672 0 0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238262] [ 3281] 0 3281 647 26 36864 0 0 hostapd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238271] [ 3284] 0 3284 647 25 36864 0 0 wpa_supplicant
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238281] [ 3300] 0 3300 352 24 32768 0 0 ip
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238291] [ 3301] 0 3301 454 138 32768 0 0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238300] [ 3325] 101 3325 1530 137 45056 0 0 hostapd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238311] [ 3327] 101 3327 1402 33 40960 0 0 wpa_supplicant
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238319] [ 3381] 0 3381 473 61 32768 0 0 netifd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238328] [ 3541] 0 3541 375 40 36864 0 0 odhcpd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238337] [ 3603] 0 3603 329 15 28672 0 0 crond
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238347] [ 3854] 0 3854 328 9 32768 0 0 udhcpc
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238356] [ 3877] 0 3877 254 10 28672 0 0 odhcp6c
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238365] [ 4012] 0 4012 2457 812 49152 0 0 collectd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238375] [ 5613] 0 5613 647 28 36864 0 0 ntpd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238384] [ 5686] 123 5686 328 9 28672 0 0 ntpd
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238394] [ 6434] 0 6434 254 10 28672 0 0 odhcp6c
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238404] [ 6438] 0 6438 328 9 28672 0 0 udhcpc
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238414] [ 6967] 0 6967 647 28 40960 0 0 dnsmasq
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238423] [ 6972] 453 6972 798 47 36864 0 0 dnsmasq
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238432] [ 7055] 410 7055 1604 366 40960 0 0 stubby
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238443] [ 7181] 0 7181 318 20 32768 0 0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238452] [ 7187] 0 7187 332 18 28672 0 0 ash
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238462] [ 7255] 0 7255 345 25 32768 0 0 logread
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238472] [ 7338] 0 7338 318 20 32768 0 0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238481] [ 7367] 0 7367 851 192 32768 0 0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238490] [ 7368] 0 7368 844 70 32768 0 0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238499] [ 7383] 0 7383 332 18 28672 0 0 ash
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238508] [ 7391] 0 7391 345 26 28672 0 0 logread
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238518] [ 12004] 0 12004 328 10 28672 0 0 sleep
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238527] [ 12025] 0 12025 329 11 28672 0 0 sleep
Thu May 19 18:13:36 2022 kern.info kernel: [ 361.238536] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=collectd,pid=4012,uid=0
Thu May 19 18:13:36 2022 kern.err kernel: [ 361.238630] Out of memory: Killed process 4012 (collectd) total-vm:9828kB, anon-rss:1192kB, file-rss:2052kB, shmem-rss:4kB, UID:0 pgtables:48kB oom_score_adj:0
as I said I just need to run a fast.com test ... reverted back to Robi repo with but with 500/mbs vs 800/mbs wiht your repo until it crashes with oom
I did tests at 80 and 160 MHz, before the ath11k driver, it just crashed, and a local reboot was required, now the router reboots itself, most likely the panic kernel panic.