I have been doing some performance benchmarking of various wireless NICs and discovered the Ath9k/10k PCI driver does not seem to be able to spread the IRQ handling across multiple CPUs. Does anyone know why there is this limitation or the source of this limitation?
You have to move the pci interrupt, not the ath10k one
The following is the output of /proc/interrupts on system running the latest LEDE-17-01. As you can see all the "ath10k_pci" interrupts are being serviced only by CPU0. I would expect all CPUs to be servicing interrupts from the device, not just one of them.
CPU0 CPU1 CPU2 CPU3
16: 2528004 1448450 1088778 1224041 GIC 29 Edge twd
17: 0 0 0 0 GPC 55 Level i.MX Timer Tick
18: 81652 0 0 0 GPC 13 Level mxs-dma
19: 18564 0 0 0 GPC 15 Level bch
22: 9758349 0 0 0 GPC 120 Level ath10k_pci
268: 0 0 0 0 GPC 49 Level imx_thermal
273: 0 0 0 0 GPC 19 Level rtc alarm
279: 0 0 0 0 GPC 2 Level sdma
280: 29 0 0 0 GPC 40 Level 2184200.usb
281: 431 0 0 0 GPC 36 Level 21a0000.i2c
282: 0 0 0 0 GPC 37 Level 21a4000.i2c
283: 0 0 0 0 GPC 38 Level 21a8000.i2c
285: 2179 0 0 0 GPC 27 Level 21e8000.serial
286: 0 0 0 0 GPC 28 Level 21ec000.serial
287: 0 0 0 0 GPC 30 Level 21f4000.serial
290: 0 0 0 0 GPC 39 Level 2200000.sata
294: 8549328 0 0 0 GPC 122 Level ath10k_pci, eth0, eth1
295: 2 0 0 0 GIC 137 Level 2101000.jr0
296: 0 0 0 0 GIC 138 Level 2102000.jr1
IPI0: 0 0 0 0 CPU wakeup interrupts
IPI1: 0 0 0 0 Timer broadcast interrupts
IPI2: 622523 1568338 1477204 1513818 Rescheduling interrupts
IPI3: 12 14 18 16 Function call interrupts
IPI4: 113 1162187 2612314 1199799 Single function call interrupts
IPI5: 0 0 0 0 CPU stop interrupts
IPI6: 3 0 2 2 IRQ work interrupts
IPI7: 0 0 0 0 completion interrupts
Err: 0
This is the output from /proc/irq/<irq#>
root@OpenWrt:~# cat /proc/irq/22/affinity_hint
0
root@OpenWrt:~# cat /proc/irq/22/smp_affinity
f
root@OpenWrt:~# cat /proc/irq/22/smp_affinity_list
0-3
Then as an experiment, I modified the "smp_affinity" to try to move the interrupts off CPU0:
root@OpenWrt:~# echo "e" > /proc/irq/22/smp_affinity
root@OpenWrt:~# cat /proc/irq/22/affinity_hint
0
root@OpenWrt:~# cat /proc/irq/22/smp_affinity
e
root@OpenWrt:~# cat /proc/irq/22/smp_affinity_list
1-3
Now you can see from this output, the interrupts have moved from CPU0 to CPU1.
CPU0 CPU1 CPU2 CPU3
16: 2543967 1457680 1094105 1228926 GIC 29 Edge twd
17: 0 0 0 0 GPC 55 Level i.MX Timer Tick
18: 81652 0 0 0 GPC 13 Level mxs-dma
19: 18564 0 0 0 GPC 15 Level bch
22: 9823560 2649 0 0 GPC 120 Level ath10k_pci
268: 0 0 0 0 GPC 49 Level imx_thermal
273: 0 0 0 0 GPC 19 Level rtc alarm
279: 0 0 0 0 GPC 2 Level sdma
280: 29 0 0 0 GPC 40 Level 2184200.usb
281: 431 0 0 0 GPC 36 Level 21a0000.i2c
282: 0 0 0 0 GPC 37 Level 21a4000.i2c
283: 0 0 0 0 GPC 38 Level 21a8000.i2c
285: 2179 0 0 0 GPC 27 Level 21e8000.serial
286: 0 0 0 0 GPC 28 Level 21ec000.serial
287: 0 0 0 0 GPC 30 Level 21f4000.serial
290: 0 0 0 0 GPC 39 Level 2200000.sata
294: 8578743 0 0 0 GPC 122 Level ath10k_pci, eth0, eth1
295: 2 0 0 0 GIC 137 Level 2101000.jr0
296: 0 0 0 0 GIC 138 Level 2102000.jr1
IPI0: 0 0 0 0 CPU wakeup interrupts
IPI1: 0 0 0 0 Timer broadcast interrupts
IPI2: 625847 1569255 1492187 1524607 Rescheduling interrupts
IPI3: 12 14 18 16 Function call interrupts
IPI4: 131 1164228 2613568 1202284 Single function call interrupts
IPI5: 0 0 0 0 CPU stop interrupts
IPI6: 3 0 2 2 IRQ work interrupts
IPI7: 0 0 0 0 completion interrupts
Err: 0