Adding OpenWrt support for Xiaomi AX3600

So you are using wsl... Wsl1 or 2? Can't you read the error and search it? You have to disable windows PATH merge in wsl... Again search on Google... Also use wsl2 and not wsl1

3 Likes

@robimarko btw to me it seems the ath11k leak is present in the rx path... and i bet it's dma memory not freed... fun thing is that with nss offload the amount of ring to handle is really low so it's incredible that it's still present... (it's in rx path as before the nss fixes the rx speed was very low so i assume the affected function was called less times)

That would be just great, I will bet that they fixed that in wlan-open in the RX throughput improvement patch by accident

It's what Ansuel says about wsl2.
You can also paste the path (without win part) before you do make:
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib"

1 Like

btw the wiki.... https://openwrt.org/docs/guide-developer/toolchain/wsl

@sqrwv news with our special firmware?

2 Likes

i test it ... and borked/crashed the router with fast.com + ipferf ... unfortunately was not capturing any logs but saying this much better throughput on ath11k unsurprisingly ... would tuning the sysctl settings help?

Sorry for stupid question, but after installing openwrt on ax3600 I cannot run opkg_download. and install any package beside already installed by default. Ii should be cause openwrt currently not officially supported on this router? Or I should be able install packages ?

well yes but they should be already set to the correct values for 512mb memory profile... I'm using a revision of the 512mb patch that tweak also the ring values... wonder if that was the cause of the crash... In any case the main problem here is qcom permitting 512mb on these soc...

I'm starting to think what we are missing that would workaround the low ram problem is the tlv 2kb patch but it's very big and require lots of rework... not really maintainable

would you mind to share the patch or push into your repo?

done just now

This is the patch that should reduce skb size

2 Likes

Also i notice many

Thu May 19 14:41:05 2022 daemon.info hostapd: wlan2: STA 28:c2:1f:ce:b2:1d RADIUS: starting accounting session 91933A64C85B4806
Thu May 19 14:41:05 2022 daemon.info hostapd: wlan2: STA 28:c2:1f:ce:b2:1d WPA: pairwise key handshake completed (RSN)
Thu May 19 14:41:05 2022 daemon.notice hostapd: wlan2: EAPOL-4WAY-HS-COMPLETED 28:c2:1f:ce:b2:1d
Thu May 19 14:41:31 2022 kern.warn kernel: [  170.740681] ath11k c000000.wifi: failed peer find by addr 28:c2:1f:ce:b2:1d on vdev 0 after creation
Thu May 19 14:41:31 2022 kern.warn kernel: [  170.740739] ath11k c000000.wifi: Failed to add peer: 28:c2:1f:ce:b2:1d for VDEV: 0
Thu May 19 14:41:31 2022 kern.warn kernel: [  170.748898] ath11k c000000.wifi: Failed to add station: 28:c2:1f:ce:b2:1d for VDEV: 0
Thu May 19 14:41:31 2022 daemon.notice hostapd: wlan1: STA 28:c2:1f:ce:b2:1d IEEE 802.11: Could not add STA to kernel driver
Thu May 19 14:42:12 2022 kern.warn kernel: [  211.858349] ath11k c000000.wifi: failed peer find by addr 54:8d:5a:63:b9:c4 on vdev 0 after creation
Thu May 19 14:42:12 2022 kern.warn kernel: [  211.858407] ath11k c000000.wifi: Failed to add peer: 54:8d:5a:63:b9:c4 for VDEV: 0
Thu May 19 14:42:12 2022 kern.warn kernel: [  211.866600] ath11k c000000.wifi: Failed to add station: 54:8d:5a:63:b9:c4 for VDEV: 0
Thu May 19 14:42:12 2022 daemon.notice hostapd: wlan1: STA 54:8d:5a:63:b9:c4 IEEE 802.11: Could not add STA to kernel driver
Thu May 19 14:42:12 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 IEEE 802.11: authenticated
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 IEEE 802.11: associated (aid 1)
Thu May 19 14:42:13 2022 daemon.notice hostapd: wlan1: Prune association for 54:8d:5a:63:b9:c4
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 RADIUS: starting accounting session 41BC4B44CB307AAF
Thu May 19 14:42:13 2022 daemon.info hostapd: wlan2: STA 54:8d:5a:63:b9:c4 WPA: pairwise key handshake completed (RSN)
Thu May 19 14:42:13 2022 daemon.notice hostapd: wlan2: EAPOL-4WAY-HS-COMPLETED 54:8d:5a:63:b9:c4

No idea if it's related to nss but in theory my patch doesn't touch peer addition logic so i wonder if it's a bug present in current ath11k

I mean the error is that a peer is tried to be added while it's already present... the function return error and the add fails... no idea if it should be handled this way tho

Your special FW is a beast! :slight_smile: :clap: :clap: :clap:
With wifi offload, wifi cpu usage is (almost) non existent now, avg 6% only.
It frees a lot of cpu needed for the other tasks.

Wifi<->WiredLAN speed now does ~940Mbits Tx or Rx (only limited by 1Gbit Wired LAN).
Wifi <-> router I got ~1.5Gbits (this depends on channel interference by neighbors).

I saw you included the nss-firmware file in last build, so people don't have to copy it now.

Thanks

PS: I didn't had memory problems here, will use it and see how it goes in the next days.

3 Likes

That was really cool :slight_smile:

1 Like

thing is as soon as you start using other stuff (example i use unbound) ram gets filled real fast... and ath11k or nss expect them to be present... and i'm still convinced that with a very specific load ath11k starts to miss memory like crazy...

3 Likes

I see what you are saying in my other router that has more uptime:
(Will load your latest build now (2022-05-19), I had the previous one here.)

curious why you are using unbound vs stubby?

mhhh i always notice that to get a good stubby configuration you need to use additional process and the packets needs to go to multiple place... with unbound i can just use odhcpd and unbound and that's it

so I applied your patches ... still crashes but yes good throughput until it dies ...this time kept a logread -f

warning long log ...

Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237724] netifd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237758] CPU: 2 PID: 3381 Comm: netifd Not tainted 5.15.34 #0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237766] Hardware name: Xiaomi AX3600 (DT)
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237771] Call trace:
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237773]  dump_backtrace+0x0/0x160
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237791]  show_stack+0x18/0x30
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237797]  dump_stack_lvl+0x7c/0xa0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237807]  dump_stack+0x18/0x34
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237813]  dump_header+0x48/0x184
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237819]  oom_kill_process+0x1e0/0x1ec
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237828]  out_of_memory+0x1cc/0x320
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237833]  __alloc_pages_slowpath.constprop.0+0x73c/0xa34
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237844]  __alloc_pages+0x220/0x24c
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237850]  pagecache_get_page+0x100/0x400
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237856]  filemap_fault+0x5e0/0x800
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237861]  __do_fault+0x3c/0x110
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237871]  __handle_mm_fault+0x540/0xa70
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237878]  handle_mm_fault+0x130/0x180
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237885]  do_page_fault+0x14c/0x380
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237895]  do_translation_fault+0xa0/0xb0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237900]  do_mem_abort+0x40/0xb0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237906]  el0_ia+0x8c/0xe0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237916]  el0t_64_sync_handler+0x124/0x130
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237920]  el0t_64_sync+0x184/0x188
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237928] Mem-Info:
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932] active_anon:84 inactive_anon:4074 isolated_anon:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  active_file:1 inactive_file:5 isolated_file:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  unevictable:0 dirty:0 writeback:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  slab_reclaimable:2017 slab_unreclaimable:74578
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  mapped:678 shmem:1249 pagetables:398 bounce:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  kernel_misc_reclaimable:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237932]  free:3365 free_pcp:31 free_cma:0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237947] Node 0 active_anon:336kB inactive_anon:16296kB active_file:4kB inactive_file:20kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2712kB dirty:0kB writeback:0kB shmem:4996kB writeback_tmp:0kB kernel_stack:2384kB pagetables:1592kB all_unreclaimable? no
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237959] DMA free:13460kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:336kB inactive_anon:16296kB active_file:4kB inactive_file:20kB unevictable:0kB writepending:0kB present:524288kB managed:417312kB mlocked:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237974] lowmem_reserve[]: 0 0 0 0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.237987] DMA: 1323*4kB (UME) 627*8kB (UME) 88*16kB (UME) 51*32kB (UE) 3*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13540kB
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238029] 1283 total pagecache pages
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238032] 0 pages in swap cache
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238035] Swap cache stats: add 0, delete 0, find 0/0
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238039] Free swap  = 0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238041] Total swap = 0kB
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238044] 131072 pages RAM
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238046] 0 pages HighMem/MovableOnly
Thu May 19 18:13:36 2022 kern.warn kernel: [  361.238049] 26744 pages reserved
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238051] Tasks state (memory values in pages):
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238054] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238082] [    720]    81   720      317       33    32768        0             0 ubusd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238092] [    721]     0   721      215        9    28672        0             0 askfirst
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238102] [    756]     0   756      249       16    32768        0             0 urngd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238121] [   2148]     0  2148     1772      207    40960        0             0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238130] [   2180]     0  2180     1836      272    40960        0             0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238139] [   2181]     0  2181     1843      277    40960        0             0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238148] [   2182]     0  2182     1836      270    40960        0             0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238158] [   2184]     0  2184     1835      252    40960        0             0 nginx
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238167] [   2226]     0  2226      644       54    32768        0             0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238176] [   2396]   514  2396      308       21    32768        0             0 logd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238186] [   2450]     0  2450      571       77    32768        0             0 rpcd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238195] [   2797]     0  2797      313       18    32768        0             0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238205] [   3164]     0  3164      396       86    32768        0             0 mwan3track
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238215] [   3165]     0  3165      399       85    28672        0             0 mwan3track
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238225] [   3166]     0  3166      448      132    32768        0             0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238234] [   3167]     0  3167      447      131    28672        0             0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238244] [   3244]     0  3244      352       25    32768        0             0 ip
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238253] [   3246]     0  3246      453      137    28672        0             0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238262] [   3281]     0  3281      647       26    36864        0             0 hostapd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238271] [   3284]     0  3284      647       25    36864        0             0 wpa_supplicant
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238281] [   3300]     0  3300      352       24    32768        0             0 ip
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238291] [   3301]     0  3301      454      138    32768        0             0 mwan3rtmon
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238300] [   3325]   101  3325     1530      137    45056        0             0 hostapd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238311] [   3327]   101  3327     1402       33    40960        0             0 wpa_supplicant
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238319] [   3381]     0  3381      473       61    32768        0             0 netifd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238328] [   3541]     0  3541      375       40    36864        0             0 odhcpd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238337] [   3603]     0  3603      329       15    28672        0             0 crond
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238347] [   3854]     0  3854      328        9    32768        0             0 udhcpc
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238356] [   3877]     0  3877      254       10    28672        0             0 odhcp6c
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238365] [   4012]     0  4012     2457      812    49152        0             0 collectd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238375] [   5613]     0  5613      647       28    36864        0             0 ntpd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238384] [   5686]   123  5686      328        9    28672        0             0 ntpd
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238394] [   6434]     0  6434      254       10    28672        0             0 odhcp6c
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238404] [   6438]     0  6438      328        9    28672        0             0 udhcpc
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238414] [   6967]     0  6967      647       28    40960        0             0 dnsmasq
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238423] [   6972]   453  6972      798       47    36864        0             0 dnsmasq
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238432] [   7055]   410  7055     1604      366    40960        0             0 stubby
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238443] [   7181]     0  7181      318       20    32768        0             0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238452] [   7187]     0  7187      332       18    28672        0             0 ash
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238462] [   7255]     0  7255      345       25    32768        0             0 logread
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238472] [   7338]     0  7338      318       20    32768        0             0 dropbear
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238481] [   7367]     0  7367      851      192    32768        0             0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238490] [   7368]     0  7368      844       70    32768        0             0 uwsgi
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238499] [   7383]     0  7383      332       18    28672        0             0 ash
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238508] [   7391]     0  7391      345       26    28672        0             0 logread
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238518] [  12004]     0 12004      328       10    28672        0             0 sleep
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238527] [  12025]     0 12025      329       11    28672        0             0 sleep
Thu May 19 18:13:36 2022 kern.info kernel: [  361.238536] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=collectd,pid=4012,uid=0
Thu May 19 18:13:36 2022 kern.err kernel: [  361.238630] Out of memory: Killed process 4012 (collectd) total-vm:9828kB, anon-rss:1192kB, file-rss:2052kB, shmem-rss:4kB, UID:0 pgtables:48kB oom_score_adj:0

as I said I just need to run a fast.com test ... reverted back to Robi repo with but with 500/mbs vs 800/mbs wiht your repo until it crashes with oom

@rmandrad you have too much stuff and too low mem that's all... it's just an oom panic

I did tests at 80 and 160 MHz, before the ath11k driver, it just crashed, and a local reboot was required, now the router reboots itself, most likely the panic kernel panic.