Adding OpenWrt support for Xiaomi AX3600 (Part 1)

Just for comparison, this is what 8 year old Wave1 802.11ac TP-Link Archer C7 v2 running as dumb AP (80MHz channel) is capable of (iperf3 client on iPhone , iperf3 server on x86 wired router):
At this speed, its single-core CPU is at 98% (I have overclocked it from 720MHz to 1GHz).

I understand that 802.11ax is better when there are many clients, but when it comes to single client throughout, it does not seem to be massively better. Not many hand-held things (phones, many laptops) are able to run more than 2x2 80MHz anyway so it seems that we are looking at ~700Mbit/sec instead of ~500Mbit/sec by going to from ac to ax for typical consumer device. 3x3 160MHz is cool and all, but more a curiosity in real world.

Testing AX6 running old "restart" branch with Galaxy S10+ (3 year old phone) connected to 5GHz AX 80Mhz.
Upload speed is limited by single core, but download speed is a lot faster than AC.
I think it saturates LAN-LAN speed which is 949 Mbits/sec.


htop1
htop2

Yes, it is a 1x1 AC device. The problem is not the speed (or the coverage), but how high CPU load it induces. It is also interesting, that the download is spread across 3 cores, the upload on the other hand single core limited.

can you share the output of cat /proc/interrupts

root@XAX6:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
 11:   10114902   10103636   10094542   10386234     GIC-0  20 Level     arch_timer
 13:          0          0          0          0     GIC-0  39 Level     arch_mem_timer
 16:          2          0          0          0     GIC-0 354 Edge      smp2p
 17:          0          0          0          0     GIC-0 216 Level     4a9000.thermal-sensor
 18:          0          0          0          0     GIC-0 239 Level     bam_dma
 21:          0          0          0          0     GIC-0 270 Level     bam_dma
 22:          4          0          0          0     GIC-0 340 Level     msm_serial0
 23:      43810          0          0          0     GIC-0 178 Level     bam_dma
 24:          0          0          0          0     GIC-0  35 Edge      wdt_bark
 25:          0          0          0          0     GIC-0 357 Edge      q6v5 wdog
 29:          5          0          0          0     GIC-0 348 Edge      ce0
 30:     687700          0          0          0     GIC-0 347 Edge      ce1
 31:      40507          0          0          0     GIC-0 346 Edge      ce2
 32:      56265          0          0          0     GIC-0 343 Edge      ce3
 34:          1          0          0          0     GIC-0 443 Edge      ce5
 36:      65705          0          0          0     GIC-0  72 Edge      ce7
 37:          0          0          0          0     GIC-0  71 Edge      ce8
 38:          0          0          0          0     GIC-0 334 Edge      ce9
 39:          0          0          0          0     GIC-0 333 Edge      ce10
 40:          0          0          0          0     GIC-0  69 Edge      ce11
 47:          0          0          0          0     GIC-0 323 Edge      reo2ost-exception
 48:      58054          0          0          0     GIC-0 322 Edge      wbm2host-rx-release
 49:        162          0          0          0     GIC-0 321 Edge      reo2host-status
 50:     172020          0          0          0     GIC-0 320 Edge      reo2host-destination-ring4
 51:     120069          0          0          0     GIC-0 271 Edge      reo2host-destination-ring3
 52:      83247          0          0          0     GIC-0 268 Edge      reo2host-destination-ring2
 53:     127124          0          0          0     GIC-0 267 Edge      reo2host-destination-ring1
 57:     188894          0          0          0     GIC-0 263 Edge      ppdu-end-interrupts-mac3
 58:          0          0          0          0     GIC-0 262 Edge      ppdu-end-interrupts-mac2
 59:     155666          0          0          0     GIC-0 261 Edge      ppdu-end-interrupts-mac1
 60:          1          0          0          0     GIC-0 260 Edge      rxdma2host-monitor-status-ring-mac3
 61:          0          0          0          0     GIC-0 256 Edge      rxdma2host-monitor-status-ring-mac2
 62:          1          0          0          0     GIC-0 255 Edge      rxdma2host-monitor-status-ring-mac1
 63:          4          0          0          0     GIC-0 235 Edge      host2rxdma-host-buf-ring-mac3
 64:          0          0          0          0     GIC-0 215 Edge      host2rxdma-host-buf-ring-mac2
 65:          1          0          0          0     GIC-0 212 Edge      host2rxdma-host-buf-ring-mac1
 66:          0          0          0          0     GIC-0 211 Edge      rxdma2host-destination-ring-mac3
 67:          0          0          0          0     GIC-0 210 Edge      rxdma2host-destination-ring-mac2
 68:          0          0          0          0     GIC-0 209 Edge      rxdma2host-destination-ring-mac1
 73:      49957          0          0          0     GIC-0 191 Edge      wbm2host-tx-completions-ring3
 74:      58488          0          0          0     GIC-0 190 Edge      wbm2host-tx-completions-ring2
 75:     166253          0          0          0     GIC-0 189 Edge      wbm2host-tx-completions-ring1
 81:     704998          0          0          0     GIC-0 409 Edge      nss_empty_buf_sos
 82:    1227543          0          0          0     GIC-0 410 Edge      nss_empty_buf_queue
 83:          0          0          0          0     GIC-0 411 Edge      nss-tx-unblock
 84:   44257916          0          0          0     GIC-0 412 Edge      nss_queue0
 85:          0     289473          0          0     GIC-0 413 Edge      nss_queue1
 86:          0          0     740846          0     GIC-0 414 Edge      nss_queue2
 87:          0          0          0     235961     GIC-0 415 Edge      nss_queue3
 88:          0          0          0          0     GIC-0 416 Edge      nss_coredump_complete
 89:          0          0          0          0     GIC-0 417 Edge      nss_paged_empty_buf_sos
 90:          0          0          0          0     GIC-0 418 Edge      nss_profile_dma
 91:     388618          0          0          0     GIC-0 422 Edge      nss_empty_buf_sos
 92:          0          0          0          0     GIC-0 423 Edge      nss_empty_buf_queue
 93:          0          0          0          0     GIC-0 424 Edge      nss-tx-unblock
 94:   40020789          0          0          0     GIC-0 425 Edge      nss_queue0
 95:          0          0          0          0     GIC-0 426 Edge      nss_queue1
 96:          0          0          0          0     GIC-0 427 Edge      nss_queue2
 97:          0          0          0          0     GIC-0 428 Edge      nss_queue3
 98:          0          0          0          0     GIC-0 429 Edge      nss_coredump_complete
 99:          0          0          0          0     GIC-0 430 Edge      nss_paged_empty_buf_sos
100:          0          0          0          0     smp2p   0 Edge      q6v5 fatal
101:          1          0          0          0     smp2p   1 Edge      q6v5 ready
102:          0          0          0          0     smp2p   2 Edge      q6v5 handover
103:          0          0          0          0     smp2p   3 Edge      q6v5 stop
104:          0          0          0          0   msmgpio  34 Edge      keys
105:         63          0          0          0     GIC-0 353 Edge      glink-native
IPI0:    181128     718406     684166     521938       Rescheduling interrupts
IPI1:      2819       4041       3447       3281       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:    397487     947324     884762    1000326       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

Thanks. Can you add this into /etc/rc.local and reboot then do iPerf again?

#assign 4 rx interrupts to each cores
echo 8 > /proc/irq/50/smp_affinity
echo 4 > /proc/irq/51/smp_affinity
echo 2 > /proc/irq/52/smp_affinity
echo 1 > /proc/irq/53/smp_affinity

#assign 3 tcl completions to 3 CPUs
echo 4 > /proc/irq/73/smp_affinity
echo 2 > /proc/irq/74/smp_affinity
echo 1 > /proc/irq/75/smp_affinity

EDIT: it's based on this smp_affinity_settings

4 Likes

Yep, uplink traffic is now spread across 3 cores instead of 1 core.

MOD: non the less, after the affinity mods nss_queue0 is still the only one that has interrupts (see 94):

 81:     763705          0          0          0     GIC-0 409 Edge      nss_empty_buf_sos
 82:    2495785          0          0          0     GIC-0 410 Edge      nss_empty_buf_queue
 83:          0          0          0          0     GIC-0 411 Edge      nss-tx-unblock
 84:   45969783          0          0          0     GIC-0 412 Edge      nss_queue0
 85:          0     422929          0          0     GIC-0 413 Edge      nss_queue1
 86:          0          0     832809          0     GIC-0 414 Edge      nss_queue2
 87:          0          0          0     313232     GIC-0 415 Edge      nss_queue3
 88:          0          0          0          0     GIC-0 416 Edge      nss_coredump_complete
 89:          0          0          0          0     GIC-0 417 Edge      nss_paged_empty_buf_sos
 90:          0          0          0          0     GIC-0 418 Edge      nss_profile_dma
 91:     400126          0          0          0     GIC-0 422 Edge      nss_empty_buf_sos
 92:          0          0          0          0     GIC-0 423 Edge      nss_empty_buf_queue
 93:          0          0          0          0     GIC-0 424 Edge      nss-tx-unblock
 94:   41207537          0          0          0     GIC-0 425 Edge      nss_queue0
 95:          0          0          0          0     GIC-0 426 Edge      nss_queue1
 96:          0          0          0          0     GIC-0 427 Edge      nss_queue2
 97:          0          0          0          0     GIC-0 428 Edge      nss_queue3
 98:          0          0          0          0     GIC-0 429 Edge      nss_coredump_complete
 99:          0          0          0          0     GIC-0 430 Edge      nss_paged_empty_buf_sos
2 Likes

@robimarko

Interrupts during wireless LAN transfer with RX decap offload patch included and ath11k frame_mode=2 enabled.

root@AX3600:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
 11:    1902816    1959169    1899301    1895806     GIC-0  20 Level     arch_timer
 13:          0          0          0          0     GIC-0  39 Level     arch_mem_timer
 16:          2          0          0          0     GIC-0 354 Edge      smp2p
 17:          0          0          0          0     GIC-0 216 Level     4a9000.thermal-sensor
 18:          0          0          0          0     GIC-0 239 Level     bam_dma
 21:          0          0          0          0     GIC-0 270 Level     bam_dma
 22:          6          0          0          0     GIC-0 340 Level     msm_serial0
 23:      41976          0          0          0     GIC-0 178 Level     bam_dma
 24:          0          0          0          0     GIC-0  35 Edge      wdt_bark
 26:          0          0          0          0     GIC-0 357 Edge      q6v5 wdog
 30:          5          0          0          0     GIC-0 348 Edge      ce0
 31:     339109          0          0          0     GIC-0 347 Edge      ce1
 32:      81081          0          0          0     GIC-0 346 Edge      ce2
 33:      11412          0          0          0     GIC-0 343 Edge      ce3
 35:          6          0          0          0     GIC-0 443 Edge      ce5
 37:      25275          0          0          0     GIC-0  72 Edge      ce7
 39:          0          0          0          0     GIC-0 334 Edge      ce9
 40:          0          0          0          0     GIC-0 333 Edge      ce10
 41:          0          0          0          0     GIC-0  69 Edge      ce11
 48:          0          0          0          0     GIC-0 323 Edge      reo2ost-exception
 49:       2493          0          0          0     GIC-0 322 Edge      wbm2host-rx-release
 50:        341          0          0          0     GIC-0 321 Edge      reo2host-status
 51:      10518          0          0          0     GIC-0 320 Edge      reo2host-destination-ring4
 52:      28779          0          0          0     GIC-0 271 Edge      reo2host-destination-ring3
 53:       8595          0          0          0     GIC-0 268 Edge      reo2host-destination-ring2
 54:       5418          0          0          0     GIC-0 267 Edge      reo2host-destination-ring1
 58:      91516          0          0          0     GIC-0 263 Edge      ppdu-end-interrupts-mac3
 59:          0          0          0          0     GIC-0 262 Edge      ppdu-end-interrupts-mac2
 60:      88764          0          0          0     GIC-0 261 Edge      ppdu-end-interrupts-mac1
 61:          1          0          0          0     GIC-0 260 Edge      rxdma2host-monitor-status-ring-mac3
 62:          0          0          0          0     GIC-0 256 Edge      rxdma2host-monitor-status-ring-mac2
 63:          1          0          0          0     GIC-0 255 Edge      rxdma2host-monitor-status-ring-mac1
 64:          4          0          0          0     GIC-0 235 Edge      host2rxdma-host-buf-ring-mac3
 65:          0          0          0          0     GIC-0 215 Edge      host2rxdma-host-buf-ring-mac2
 66:          1          0          0          0     GIC-0 212 Edge      host2rxdma-host-buf-ring-mac1
 67:          0          0          0          0     GIC-0 211 Edge      rxdma2host-destination-ring-mac3
 68:          0          0          0          0     GIC-0 210 Edge      rxdma2host-destination-ring-mac2
 69:          0          0          0          0     GIC-0 209 Edge      rxdma2host-destination-ring-mac1
 74:         74          0          0          0     GIC-0 191 Edge      wbm2host-tx-completions-ring3
 75:        131          0          0          0     GIC-0 190 Edge      wbm2host-tx-completions-ring2
 76:     177694          0          0          0     GIC-0 189 Edge      wbm2host-tx-completions-ring1
 82:      70083          0          0          0     GIC-0 409 Edge      nss_empty_buf_sos
 83:        568          0          0          0     GIC-0 410 Edge      nss_empty_buf_queue
 84:          0          0          0          0     GIC-0 411 Edge      nss-tx-unblock
 85:    3975875          0          0          0     GIC-0 412 Edge      nss_queue0
 86:       5347        112          0          0     GIC-0 413 Edge      nss_queue1
 87:       5460          0         76          0     GIC-0 414 Edge      nss_queue2
 88:     103141          0          0         60     GIC-0 415 Edge      nss_queue3
 89:          0          0          0          0     GIC-0 416 Edge      nss_coredump_complete
 90:          0          0          0          0     GIC-0 417 Edge      nss_paged_empty_buf_sos
 91:          0          0          0          0     GIC-0 418 Edge      nss_profile_dma
 92:       8283          0          0          0     GIC-0 422 Edge      nss_empty_buf_sos
 93:          0          0          0          0     GIC-0 423 Edge      nss_empty_buf_queue
 94:          0          0          0          0     GIC-0 424 Edge      nss-tx-unblock
 95:     574138          0          0          0     GIC-0 425 Edge      nss_queue0
 96:          0          0          0          0     GIC-0 426 Edge      nss_queue1
 97:          0          0          0          0     GIC-0 427 Edge      nss_queue2
 98:          0          0          0          0     GIC-0 428 Edge      nss_queue3
 99:          0          0          0          0     GIC-0 429 Edge      nss_coredump_complete
100:          0          0          0          0     GIC-0 430 Edge      nss_paged_empty_buf_sos
102:          0          0          0          0   PCI-MSI   0 Edge      PCIe PME
103:          0          0          0          0     smp2p   0 Edge      q6v5 fatal
104:          1          0          0          0     smp2p   1 Edge      q6v5 ready
105:          0          0          0          0     smp2p   2 Edge      q6v5 handover
106:          0          0          0          0     smp2p   3 Edge      q6v5 stop
107:          0          0          0          0   msmgpio  34 Edge      keys
108:         30          0          0          0   PCI-MSI 524288 Edge      ath10k_pci
109:         65          0          0          0     GIC-0 353 Edge      glink-native
IPI0:     36844      71982      61085      46948       Rescheduling interrupts
IPI1:       651        781        739        786       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:     75646     163863     174379     181525       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

Repeated the same transfer but with @castiel652 smp_affinity_settings applied

#assign 4 rx interrupts to each cores
echo 8 > /proc/irq/51/smp_affinity
echo 4 > /proc/irq/52/smp_affinity
echo 2 > /proc/irq/53/smp_affinity
echo 1 > /proc/irq/54/smp_affinity

#assign 3 tcl completions to 3 CPUs
echo 4 > /proc/irq/74/smp_affinity
echo 2 > /proc/irq/75/smp_affinity
echo 1 > /proc/irq/76/smp_affinity

root@AX3600:~#  cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
 11:     186246     192499     186058     185684     GIC-0  20 Level     arch_timer
 13:          0          0          0          0     GIC-0  39 Level     arch_mem_timer
 16:          2          0          0          0     GIC-0 354 Edge      smp2p
 17:          0          0          0          0     GIC-0 216 Level     4a9000.thermal-sensor
 18:          0          0          0          0     GIC-0 239 Level     bam_dma
 21:          0          0          0          0     GIC-0 270 Level     bam_dma
 22:          4          0          0          0     GIC-0 340 Level     msm_serial0
 23:      42369          0          0          0     GIC-0 178 Level     bam_dma
 24:          0          0          0          0     GIC-0  35 Edge      wdt_bark
 26:          0          0          0          0     GIC-0 357 Edge      q6v5 wdog
 30:          5          0          0          0     GIC-0 348 Edge      ce0
 31:     228181          0          0          0     GIC-0 347 Edge      ce1
 32:       1220          0          0          0     GIC-0 346 Edge      ce2
 33:       1158          0          0          0     GIC-0 343 Edge      ce3
 35:          1          0          0          0     GIC-0 443 Edge      ce5
 37:       2489          0          0          0     GIC-0  72 Edge      ce7
 39:          0          0          0          0     GIC-0 334 Edge      ce9
 40:          0          0          0          0     GIC-0 333 Edge      ce10
 41:          0          0          0          0     GIC-0  69 Edge      ce11
 48:          0          0          0          0     GIC-0 323 Edge      reo2ost-exception
 49:        174          0          0          0     GIC-0 322 Edge      wbm2host-rx-release
 50:          0          0          0         12     GIC-0 321 Edge      reo2host-status
 51:          0          0       1261      17669     GIC-0 320 Edge      reo2host-destination-ring4
 52:          0        507        497          0     GIC-0 271 Edge      reo2host-destination-ring3
 53:        339        151          0          0     GIC-0 268 Edge      reo2host-destination-ring2
 54:        599          0          0          0     GIC-0 267 Edge      reo2host-destination-ring1
 58:        559          0          0          0     GIC-0 263 Edge      ppdu-end-interrupts-mac3
 59:          0          0          0          0     GIC-0 262 Edge      ppdu-end-interrupts-mac2
 60:      32463          0          0          0     GIC-0 261 Edge      ppdu-end-interrupts-mac1
 61:          1          0          0          0     GIC-0 260 Edge      rxdma2host-monitor-status-ring-mac3
 62:          0          0          0          0     GIC-0 256 Edge      rxdma2host-monitor-status-ring-mac2
 63:          1          0          0          0     GIC-0 255 Edge      rxdma2host-monitor-status-ring-mac1
 64:          4          0          0          0     GIC-0 235 Edge      host2rxdma-host-buf-ring-mac3
 65:          0          0          0          0     GIC-0 215 Edge      host2rxdma-host-buf-ring-mac2
 66:          1          0          0          0     GIC-0 212 Edge      host2rxdma-host-buf-ring-mac1
 67:          0          0          0          0     GIC-0 211 Edge      rxdma2host-destination-ring-mac3
 68:          0          0          0          0     GIC-0 210 Edge      rxdma2host-destination-ring-mac2
 69:          0          0          0          0     GIC-0 209 Edge      rxdma2host-destination-ring-mac1
 74:          0        372         98          0     GIC-0 191 Edge      wbm2host-tx-completions-ring3
 75:        460      34606          0          0     GIC-0 190 Edge      wbm2host-tx-completions-ring2
 76:      10849          0          0          0     GIC-0 189 Edge      wbm2host-tx-completions-ring1
 82:      75660          0          0          0     GIC-0 409 Edge      nss_empty_buf_sos
 83:        272          0          0          0     GIC-0 410 Edge      nss_empty_buf_queue
 84:          0          0          0          0     GIC-0 411 Edge      nss-tx-unblock
 85:     415280          0          0          0     GIC-0 412 Edge      nss_queue0
 86:          0      62814          0          0     GIC-0 413 Edge      nss_queue1
 87:          0          0        314          0     GIC-0 414 Edge      nss_queue2
 88:          0          0          0       8947     GIC-0 415 Edge      nss_queue3
 89:          0          0          0          0     GIC-0 416 Edge      nss_coredump_complete
 90:          0          0          0          0     GIC-0 417 Edge      nss_paged_empty_buf_sos
 91:          0          0          0          0     GIC-0 418 Edge      nss_profile_dma
 92:       1276          0          0          0     GIC-0 422 Edge      nss_empty_buf_sos
 93:          0          0          0          0     GIC-0 423 Edge      nss_empty_buf_queue
 94:          0          0          0          0     GIC-0 424 Edge      nss-tx-unblock
 95:     106218          0          0          0     GIC-0 425 Edge      nss_queue0
 96:          0          0          0          0     GIC-0 426 Edge      nss_queue1
 97:          0          0          0          0     GIC-0 427 Edge      nss_queue2
 98:          0          0          0          0     GIC-0 428 Edge      nss_queue3
 99:          0          0          0          0     GIC-0 429 Edge      nss_coredump_complete
100:          0          0          0          0     GIC-0 430 Edge      nss_paged_empty_buf_sos
102:          0          0          0          0   PCI-MSI   0 Edge      PCIe PME
103:          0          0          0          0     smp2p   0 Edge      q6v5 fatal
104:          1          0          0          0     smp2p   1 Edge      q6v5 ready
105:          0          0          0          0     smp2p   2 Edge      q6v5 handover
106:          0          0          0          0     smp2p   3 Edge      q6v5 stop
107:          0          0          0          0   msmgpio  34 Edge      keys
108:         31          0          0          0   PCI-MSI 524288 Edge      ath10k_pci
109:         63          0          0          0     GIC-0 353 Edge      glink-native
IPI0:      9861      11204      10650       9659       Rescheduling interrupts
IPI1:       244        214        289        267       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:      5550      20546      16918      15927       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

Load seems to be better spread over cores but the actual transfer speed didn't improve.

image

Well, how would it improve if you are not CPU bottlenecked?

@dchard They released the OEM integration QSDK11.5 today, it uses FW:
WLAN.HK.2.6-02385-QCAHKSWPL_SILICONZ-1

Is this public? Let me take a look.

I think you know the answer to that, you are gonna have to wait until they make a SPF release and then it will all get pushed out

Hi,

You cannot benchmark WiFi speed by copying files between NAS and Windows laptop. First, there is CIFS overhead. Second, NAS might not be able to push data faster. Please use iperf3 between two devices.

Without the smp_affinity_settings the transfer was loaded only on core0 and that more or less maxed out.

I was just doing some tests as some users asked.

Sure, I was reffering to the case where no core was maxed out

You are right but as a quick test I usually do it this way.

BTW the exact same wireless transfer maxes out gigabit on stock firmware.

Stock interrupts just for the interested

root@XiaoQiang:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  3:     566786      98541      90428    3341065       GIC  20 Edge      arch_timer
 40:          1          0          0          0   msmgpio  34 Edge      soc:gpio_keys
 76:          0          0          0          0       GIC 270 Level     bam_dma
 77:        396          0          0          0       GIC 340 Level     msm_serial0
 79:          2          0          0          0       GIC 127 Level     78b5000.spi
 80:          0          0          0          0       GIC 357 Edge      q6_wdog_interrupt
 81:          0          0          0          0       GIC 344 Edge      7803000.sdcc1ice
 82:          3          0          0          0       GIC 354 Edge      smp2p
 83:          0          0          0          0       GIC 276 Edge      tzerror
 88:     134610          0          0          0       GIC 409 Edge      nss_empty_buf_sos
 89:      49247          0          0          0       GIC 410 Edge      nss_empty_buf_queue
 90:          0          0          0          0       GIC 411 Edge      nss-tx-unblock
 91:    7046628          0          0          0       GIC 412 Edge      nss_queue0
 92:          0       4603          0          0       GIC 413 Edge      nss_queue1
 93:          0          0       4503          0       GIC 414 Edge      nss_queue2
 94:          0          0          0      61867       GIC 415 Edge      nss_queue3
 95:          0          0          0          0       GIC 416 Edge      nss_coredump_complete
 96:          0          0          0          0       GIC 417 Edge      nss_paged_empty_buf_sos
 97:      62821          0          0          0       GIC 422 Edge      nss_empty_buf_sos
 98:          0          0          0          0       GIC 423 Edge      nss_empty_buf_queue
 99:          0          0          0          0       GIC 424 Edge      nss-tx-unblock
100:    6544252          0          0          0       GIC 425 Edge      nss_queue0
101:          0          0          0          0       GIC 426 Edge      nss_queue1
102:          0          0          0          0       GIC 427 Edge      nss_queue2
103:          0          0          0          0       GIC 428 Edge      nss_queue3
104:          0          0          0          0       GIC 429 Edge      nss_coredump_complete
105:          0          0          0          0       GIC 430 Edge      nss_paged_empty_buf_sos
106:          0          0          0          0       GIC  35 Level     watchdog bark
107:         88          0          0          0       GIC 353 Edge      qcom,glink-smem-native-xprt-modem
108:          0          0          0          0       GIC 239 Level     bam_dma
109:     100429          0          0          0       GIC 178 Level     bam_dma
111:          0          0          0          0       GIC  84 Edge      qcom-pcie-msi
132:          4          0          0          0       GIC 348 Edge      ce0
133:     622897          0          0          0       GIC 347 Edge      ce1
134:     135587          0          0          0       GIC 346 Edge      ce2
135:       7790          0          0          0       GIC 343 Edge      ce3
137:         60          0          0          0       GIC 443 Edge      ce5
139:       8323          0          0          0       GIC  72 Edge      ce7
140:   20467709          0          0          0       GIC  71 Edge      ce8
141:          0          0          0          0       GIC 334 Edge      ce9
142:          0          0          0          0       GIC 333 Edge      ce10
143:          0          0          0          0       GIC  69 Edge      ce11
147:          8          0          0          0       GIC 326 Edge      host2rxdma-monitor-ring3
148:          0          0          0          0       GIC 325 Edge      host2rxdma-monitor-ring2
149:          8          0          0          0       GIC 324 Edge      host2rxdma-monitor-ring1
150:          0          0          0          0       GIC 323 Edge      reo2ost-exception
151:          0          0          0          0       GIC 322 Edge      wbm2host-rx-release
152:        160          0          0          0       GIC 321 Edge      reo2host-status
153:          0          0          0          0       GIC 320 Edge      reo2host-destination-ring4
154:          0          0          0          0       GIC 271 Edge      reo2host-destination-ring3
160:          1      87577          0          0       GIC 263 Edge      ppdu-end-interrupts-mac3
161:          0          0          0          0       GIC 262 Edge      ppdu-end-interrupts-mac2
162:          1          0          0      73383       GIC 261 Edge      ppdu-end-interrupts-mac1
163:          3          0          0          0       GIC 260 Edge      rxdma2host-monitor-status-ring-mac3
164:          0          0          0          0       GIC 256 Edge      rxdma2host-monitor-status-ring-mac2
165:          3          0          0          0       GIC 255 Edge      rxdma2host-monitor-status-ring-mac1
167:          0          0          0          0       GIC 215 Edge      host2rxdma-host-buf-ring-mac2
169:          0          0          0          0       GIC 211 Edge      rxdma2host-destination-ring-mac3
170:          0          0          0          0       GIC 210 Edge      rxdma2host-destination-ring-mac2
171:          0          0          0          0       GIC 209 Edge      rxdma2host-destination-ring-mac1
176:          0          0          0          0       GIC 191 Edge      wbm2host-tx-completions-ring3
182:          0          0          0          0       GIC 216 Edge      tsens_interrupt
183:          4          0          0          0       GIC  47 Edge      cpr3
185:        995          0      37160          0       GIC 107 Level     wlan_pci
186:          3          0          0          0  pmic_arb 3211277 Edge      spmi-vadc
187:          0          0          1          0     smp2p   1 Edge      error_ready_interrupt
188:          0          0          0          0     smp2p   0 Edge      err_fatal_interrupt
189:          0          0          0          0     smp2p   3 Edge      stop_ack_interrupt
190:          0          0          0          0       GIC 172 Edge      xhci-hcd:usb1
IPI0:     23349      61300      76753      59157       Rescheduling interrupts
IPI1:         6         11         11          9       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       Timer broadcast interrupts
IPI4:        62          0          0          2       IRQ work interrupts
IPI5:         0          0          0          0       CPU wakeup interrupts
Err:          0

Ahh... We need to wait a bit more then.

With the affinity patches, the loads are spread more evenly. Hopefully I can get my AX capable card within a week and we can do more proper tests to see where the limits are.

On the RX decap offload: are we sure that it actually works? mac80211 hooks are present? (refering to the discussion from last night)

To actually see whether router running OpenWRT is CPU-limited we would need someone to run iperf3 between wired computer and WiFi client using a full fledged (at least) 2x2 802.11ax card. If you can push 1Gbit (which is limited by gbit ethernet) without maxing CPU then you are fine.

I finally received a QCA6391 card, so I can test the PCI outside of AX9000 as well.
Thanks to @sumo for the donation.

I think that everything needed for RX decap is here as the mac80211 support for it was merged in early 2021, so it was in 5.12 as well.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/net/mac80211?h=v5.15.2&id=80a915ec4427f0083829f7e6518ee9f21521ee1e

1 Like

Just noticed that these two lines get inserted into /etc/sysctl.d/qca-nss-ecm.conf on every reboot even if they already exist?

net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1

Had maybe 10 duplicate entries.

I can confirm that. I have way more than 10, and based on the last write timestamp, it happens at every sysupgrade or reboot. (likely the first one).

MOD: no, every reboot the extra two lines are added...

1 Like