Interupt allocation for drivers/devices on multi-core plattforms

Installing and Using OpenWrt
1

On a GL.iNet B1300 (snapshot 22/08/19, K4.19.66) all interrupts for all devices are pinned to CPU 0. See below:

       CPU0      CPU1   CPU2   CPU3       
18:     355502 258085 407547 288219   GIC-0  20 Level arch_timer
22:      11676      0      0      0   GIC-0 270 Level bam_dma
23:      31759      0      0      0   GIC-0 127 Level 78b5000.spi
24:          0      0      0      0   GIC-0 239 Level bam_dma
25:          5      0      0      0   GIC-0 139 Level msm_serial0
42:     835925      0      0      0   GIC-0 200 Level ath10k_ahb
59:     347540      0      0      0   GIC-0 201 Level ath10k_ahb
60:      33970      0      0      0   GIC-0  97 Edge  edma_eth_tx0
61:        286      0      0      0   GIC-0  98 Edge  edma_eth_tx1
62:      44181      0      0      0   GIC-0  99 Edge  edma_eth_tx2
63:        275      0      0      0   GIC-0 100 Edge  edma_eth_tx3
64:      24613      0      0      0   GIC-0 101 Edge  edma_eth_tx4
65:        333      0      0      0   GIC-0 102 Edge  edma_eth_tx5
66:      17112      0      0      0   GIC-0 103 Edge  edma_eth_tx6
67:         31      0      0      0   GIC-0 104 Edge  edma_eth_tx7
68:      52955      0      0      0   GIC-0 105 Edge  edma_eth_tx8
69:        145      0      0      0   GIC-0 106 Edge  edma_eth_tx9
70:      18104      0      0      0   GIC-0 107 Edge  edma_eth_tx10
71:        106      0      0      0   GIC-0 108 Edge  edma_eth_tx11
72:       2021      0      0      0   GIC-0 109 Edge  edma_eth_tx12
73:         73      0      0      0   GIC-0 110 Edge  edma_eth_tx13
74:        171      0      0      0   GIC-0 111 Edge  edma_eth_tx14
75:         10      0      0      0   GIC-0 112 Edge  edma_eth_tx15
76:     153699      0      0      0   GIC-0 272 Edge  edma_eth_rx0
78:      23386      0      0      0   GIC-0 274 Edge  edma_eth_rx2
80:      38812      0      0      0   GIC-0 276 Edge  edma_eth_rx4
82:      14953      0      0      0   GIC-0 278 Edge  edma_eth_rx6
92:          1      0      0      0 msmgpio   5 Edge  keys
93:          1      0      0      0 msmgpio  63 Edge  keys
94:       1217      0      0      0   GIC-0 164 Level xhci-hcd:usb1
95:          0      0      0      0   GIC-0 168 Level xhci-hcd:usb3

This is for a device just booted a few minutes prior.

The worst offenders seem to be:

      CPU0 CPU1 CPU2 CPU3       
42: 835925    0    0    0   GIC-0 200 Level ath10k_ahb
59: 347540    0    0    0   GIC-0 201 Level ath10k_ahb
60:  33970    0    0    0   GIC-0  97 Edge  edma_eth_tx0
62:  44181    0    0    0   GIC-0  99 Edge  edma_eth_tx2
64:  24613    0    0    0   GIC-0 101 Edge  edma_eth_tx4
66:  17112    0    0    0   GIC-0 103 Edge  edma_eth_tx6
68:  52955    0    0    0   GIC-0 105 Edge  edma_eth_tx8
76: 153699    0    0    0   GIC-0 272 Edge  edma_eth_rx0
78:  23386    0    0    0   GIC-0 274 Edge  edma_eth_rx2
80:  38812    0    0    0   GIC-0 276 Edge  edma_eth_rx4
82:  14953    0    0    0   GIC-0 278 Edge  edma_eth_rx6

And they make sense. WIFI & Ethernet. How could I re-distribute them?

E.g.:

CPU0         CPU1          CPU2          CPU3
ath10k_AHB
             ath10k_AHB
                           edma_eth_rx0
                                         edma_eth_rx2
edma_eth_rx4
             edma_eth_rx6
                           edma_eth_tx0
...

You see, where I'm going with this.

Can other owners of multi-core devices check on their devices (Openwrt/Stock/WE), and report, if they:

a) have the same situation on the_same/a_different kernel/architecture
b) also don't get the full speed out of their devices.
c) know how to balance the IRQs.

I think It should be possible, but can't put my cursor on a good description right now. Thanks.

2

irqbalance?

3 Likes
3

I found

echo 2 > /proc/irq/59/smp_affinity_list

etc ...

Found the solution shortly after send.

Oh, irq-balance does all for you. Cool, thanks :slight_smile:

1 Like
4

There is also a quite popular script named set_irq_affinity.sh originally from Intel.

[1] https://github.com/yousong/brtest/blob/master/set_irq_affinity.sh

5

After installing irq-balance , do you have to do any config?

6

Needs to be enabled in its config file, not just as a service.

1 Like
7

To enable IRQ balancing from command line:

sed -i s-0-1- /etc/config/irqbalance

And then reboot. Or launch manually this once.

(This works (currently), because the enabled bit is the only integer in the file)

8

OK thanks. I tryed to change the line in /etc/config/irqbalance from
option enabled '0'
To option enabled '1'
Will this make it start from bootup?

9

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.