The irqbalance does not seem to do anything on 22.03 at least on a Belkin RT3200. On a snapshot image the irqs for eth get hits on both cores. I installed the package, enabled it in the config and started it. Is something more required still?
cat /etc/config/irqbalance
config irqbalance 'irqbalance'
option enabled '1'
# The default value is 10 seconds
#option interval '10'
# List of IRQ's to ignore
#list banirq '36'
#list banirq '69'
ps | grep irqbalance 8097 root 1172 S /usr/sbin/irqbalance -f -t 10
grep eth /proc/interrupts
38: 375369 0 MT_SYSIRQ 224 Level 1b100000.ethernet
39: 659350 0 MT_SYSIRQ 225 Level 1b100000.ethernet
1 Like
Ping @hnyman
Maybe you have some insight into this? I noticed you are the maintainer of irqbalance package. Can you think of what's different between master and 22.03 so that irqbalance works in the earlier but not in the latter?
hnyman
September 14, 2022, 3:46pm
4
kernel 5.10 vs 5.15, also causing DTS changes from upstream.
In master they are 142 and 143 in my RT3200 (instead of 38 and 39 like yours).
142: 332 699685 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 1079024 0 MT_SYSIRQ 225 Level 1b100000.ethernet
I am only running the development master, so no comments regarding the older 22.03.
But to to honest, I have no recollection that my RT3200 in 2021 running master of that time (from which 22.03 has been branched) would have not balanced the IRQs.
Reviving this thread
It seems that irqbalance on RT3200 is not working:
/etc/config$ grep eth /proc/interrupts
38: 272797753 0 MT_SYSIRQ 224 Level 1b100000.ethernet
39: 191383697 0 MT_SYSIRQ 225 Level 1b100000.ethernet
I believe this was known from the above.
But snapshot always worked (for me at least). The latest snapshot (R21070 on router/switch) does not:
/etc/config$ grep eth /proc/interrupts
142: 3912987 0 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 2967516 0 MT_SYSIRQ 225 Level 1b100000.ethernet
On both devices is hardware and software offloading enabled.
I tried bisecting this but building a fresh takes so long that I gave up and simply went for master. Also, I wasn't sure if packages feed needed to match any specific commit revision. Now I am facing other issues on master build so I am consider moving back to 22.03 and maybe have another look at this. As per your experience the master seems like a no-go at the moment in this regard.
OpenWrt 22.03.3 is out and irqbalance still not fixed
hnyman
February 15, 2023, 9:16pm
8
I added a PR to master for arm64 (aarch64) routers like RT3200 etc.
openwrt:master ← hnyman:irqb
opened 09:13PM - 15 Feb 23 UTC
Add upstream fix for AARCH64 irq name parsing.
https://github.com/Irqbalance/… irqbalance/pull/253
> On arm64 SoCs like TI's K3 SoC and few other SoCs,
> IRQ names don't get parsed correct due to which they
> end up being classified into wrong class. Fix this by
> considering last token to contain IRQ name always.
The fix seems to enable e.g. RT3200 to notice a few more interrupts and start balancing them.
Maintainer: me
Run tested: mt7622/RT3200
... PR for CI testing ...
If it works, I will backport it to 22.03
7 Likes
@hnyman Is mvebu (Cortex A72) affected as well or do you have no way to test? I can check my A72 device if you'd like. Let me know how.
hnyman
February 16, 2023, 10:09am
10
Based on irqbalance upstream, it is about AARCH64 CPUs, meaning these, I think:
aarch64_cortex-a53 and aarch64_cortex-a72
But I think that the old mvebu ( arm_cortex-a9 or arm_cortex-a9_vfpv3-d16) itself is not affected.
You can test by running "irqbalance -d" from ssh console, and see if all interrupts (e.g. ethernet interfaces) are classified to no-op 0, instead of 5 or so.
IRQ (134) guessed as class 0
IRQ (224) guessed as class 0
vs.
8000000.ethernet-tx0(134) guessed as class 5
IRQ 8000000.ethernet(224) guessed as class 5
IRQ 8000000.ethernet(257) guessed as class 5
2 Likes
Anteus
February 16, 2023, 8:54pm
11
Thanks it looks good here on rt3200.
cat /proc/interrupts
CPU0 CPU1
10: 320136 357359 GICv2 30 Level arch_timer
15: 1 0 MT_SYSIRQ 163 Level mt-pmic-pwrap
22: 0 0 mt-eint 0 Edge gpio-keys
75: 3 0 mt-eint 53 Level mt7530
124: 0 0 mt-eint 102 Edge gpio-keys
125: 16 0 MT_SYSIRQ 91 Level ttyS0
128: 0 0 MT_SYSIRQ 118 Level 1100a000.spi
131: 8550 0 MT_SYSIRQ 96 Level mtk-snand
132: 29692 0 MT_SYSIRQ 95 Level mtk-ecc
133: 0 0 MT_SYSIRQ 122 Level 11016000.spi
134: 38951 0 MT_SYSIRQ 211 Level mt7615e
135: 0 0 MT_SYSIRQ 232 Level xhci-hcd:usb1
138: 0 0 MT_SYSIRQ 219 Level 1b007000.dma-controller
139: 74925 0 MT_SYSIRQ 214 Level mt7915e
142: 442056 0 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 1491 171784 MT_SYSIRQ 225 Level 1b100000.ethernet
146: 0 0 dummy 0 Edge PCIe PME
147: 0 0 mt7530 0 Edge mt7530-0:00
148: 0 0 mt7530 1 Edge mt7530-0:01
149: 1 0 mt7530 2 Edge mt7530-0:02
150: 1 0 mt7530 3 Edge mt7530-0:03
151: 1 0 mt7530 4 Edge mt7530-0:04
IPI0: 19981 10636 Rescheduling interrupts
IPI1: 163228 304930 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 12679 12566 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
1 Like
Jack007
February 17, 2023, 1:43pm
12
@Anteus : What version are you using?
I see this on my RT3200:
OpenWrt SNAPSHOT, r22063-f490295bf2
-----------------------------------------------------
root@RT3200_Router:~# cat /proc/interrupts
CPU0 CPU1
10: 13016827 16130574 GICv2 30 Level arch_timer
15: 1 0 MT_SYSIRQ 163 Level mt-pmic-pwrap
22: 0 0 mt-eint 0 Edge gpio-keys
75: 8 0 mt-eint 53 Level mt7530
124: 0 0 mt-eint 102 Edge gpio-keys
125: 14 0 MT_SYSIRQ 91 Level ttyS0
128: 0 0 MT_SYSIRQ 118 Level 1100a000.spi
131: 10579 0 MT_SYSIRQ 96 Level mtk-snand
132: 37720 0 MT_SYSIRQ 95 Level mtk-ecc
133: 0 0 MT_SYSIRQ 122 Level 11016000.spi
134: 48889009 0 MT_SYSIRQ 211 Level mt7615e
135: 0 0 MT_SYSIRQ 232 Level xhci-hcd:usb1
138: 0 0 MT_SYSIRQ 219 Level 1b007000.dma-controller
142: 21593550 0 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 19400904 0 MT_SYSIRQ 225 Level 1b100000.ethernet
146: 0 0 dummy 0 Edge PCIe PME
147: 0 5 mt7530 0 Edge mt7530-0:00
148: 0 0 mt7530 1 Edge mt7530-0:01
149: 0 1 mt7530 2 Edge mt7530-0:02
150: 0 1 mt7530 3 Edge mt7530-0:03
151: 0 1 mt7530 4 Edge mt7530-0:04
152: 15864675 0 MTK PCIe MSI 524288 Edge mt7915e
IPI0: 1716103 1690759 Rescheduling interrupts
IPI1: 39535520 84401097 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 3571736 8822281 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
As reported earlier it doesn't seem to work on snapshot.
hnyman
February 17, 2023, 2:05pm
13
Jack007:
this on my RT3200:
This on my RT3200 master snapshot with irqbalance 1.9.2-2
OpenWrt SNAPSHOT, r22071-3c66ac7e22
-----------------------------------------------------
root@router4:~# opkg list-installed | grep irq
irqbalance - 1.9.2-2
root@router4:~# cat /proc/interrupts
CPU0 CPU1
10: 2035197 2043057 GICv2 30 Level arch_timer
15: 1 0 MT_SYSIRQ 163 Level mt-pmic-pwrap
22: 0 0 mt-eint 0 Edge gpio-keys
75: 1 0 mt-eint 53 Level mt7530
124: 0 0 mt-eint 102 Edge gpio-keys
125: 14 0 MT_SYSIRQ 91 Level ttyS0
128: 0 0 MT_SYSIRQ 118 Level 1100a000.spi
131: 26151 0 MT_SYSIRQ 96 Level mtk-snand
132: 94838 0 MT_SYSIRQ 95 Level mtk-ecc
133: 0 0 MT_SYSIRQ 122 Level 11016000.spi
134: 17 0 MT_SYSIRQ 211 Level mt7615e
135: 0 0 MT_SYSIRQ 232 Level xhci-hcd:usb1
138: 0 0 MT_SYSIRQ 219 Level 1b007000.dma-controller
142: 102993 0 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 32270 194565 MT_SYSIRQ 225 Level 1b100000.ethernet
146: 0 0 dummy 0 Edge PCIe PME
147: 0 0 mt7530 0 Edge mt7530-0:00
148: 0 0 mt7530 1 Edge mt7530-0:01
149: 0 0 mt7530 2 Edge mt7530-0:02
150: 1 0 mt7530 3 Edge mt7530-0:03
151: 0 0 mt7530 4 Edge mt7530-0:04
152: 3299337 0 MTK PCIe MSI 524288 Edge mt7915e
IPI0: 18474 20989 Rescheduling interrupts
IPI1: 1307530 2080112 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 1296805 1313652 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
Jack007
February 17, 2023, 2:32pm
14
hmmmmm.....
OpenWrt SNAPSHOT, r22063-f490295bf2
-----------------------------------------------------
root@RT3200_Router:~# opkg list-installed | grep irq
irqbalance - 1.9.2-1
Any idea how to debug this discrepancy?
Only things I've installed extra are:
Strange thing is that I have a second, almost identical RT3200 (as dumbap) which has the same behavior.
hnyman
February 17, 2023, 2:45pm
15
Jack007:
My install is a fraction older but should not be a problem, I suppose?
OpenWrt SNAPSHOT, r22063-f490295bf2
-----------------------------------------------------
root@RT3200_Router:~# opkg list-installed | grep irq
irqbalance - 1.9.2-1
"Fraction older" is the problem:
My PR that is discussed above (two and four messages before your first message...) contains the fix and bump of our local version to "-2".
I added a PR to master for arm64 (aarch64) routers like RT3200 etc.
If it works, I will backport it to 22.03
1 Like
Jack007
February 17, 2023, 3:20pm
16
hnyman:
You need 1.9.2-2
Sorry, overlooked that.
Jack007
February 18, 2023, 9:32am
17
Yes, works as expected now. Thanks.
OpenWrt SNAPSHOT, r22084-c0683ec458
-----------------------------------------------------
root@RT3200_Router:~# opkg list-installed | grep irq
irqbalance - 1.9.2-2
root@RT3200_Router:~# cat /proc/interrupts
CPU0 CPU1
10: 20415 35511 GICv2 30 Level arch_timer
15: 1 0 MT_SYSIRQ 163 Level mt-pmic-pwrap
22: 0 0 mt-eint 0 Edge gpio-keys
75: 6 0 mt-eint 53 Level mt7530
124: 0 0 mt-eint 102 Edge gpio-keys
125: 14 0 MT_SYSIRQ 91 Level ttyS0
128: 0 0 MT_SYSIRQ 118 Level 1100a000.spi
131: 10144 0 MT_SYSIRQ 96 Level mtk-snand
132: 36067 0 MT_SYSIRQ 95 Level mtk-ecc
133: 0 0 MT_SYSIRQ 122 Level 11016000.spi
134: 74023 0 MT_SYSIRQ 211 Level mt7615e
135: 0 0 MT_SYSIRQ 232 Level xhci-hcd:usb1
138: 0 0 MT_SYSIRQ 219 Level 1b007000.dma-controller
142: 34203 0 MT_SYSIRQ 224 Level 1b100000.ethernet
143: 526 28113 MT_SYSIRQ 225 Level 1b100000.ethernet
146: 0 0 dummy 0 Edge PCIe PME
147: 3 0 mt7530 0 Edge mt7530-0:00
148: 0 0 mt7530 1 Edge mt7530-0:01
149: 1 0 mt7530 2 Edge mt7530-0:02
150: 1 0 mt7530 3 Edge mt7530-0:03
151: 1 0 mt7530 4 Edge mt7530-0:04
152: 17461 0 MTK PCIe MSI 524288 Edge mt7915e
IPI0: 3692 4438 Rescheduling interrupts
IPI1: 45293 113983 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 3505 14596 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
root@RT3200_Router:~#
2 Likes
@hnyman Thanks. This is on a RB5009UG (Cortex A72) with 22.03, patch not applied. Seems interrupts are more or less spread our am I misreading?
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
11: 68923587 63940691 63528772 116012516 GICv2 30 Level arch_timer
17: 16 0 0 0 GICv2 51 Level ttyS0
27: 0 0 0 0 pMSI 2048 Edge f0400000.xor
28: 0 0 0 0 pMSI 4096 Edge f0420000.xor
29: 0 0 0 0 pMSI 6144 Edge f0440000.xor
30: 0 0 0 0 pMSI 8192 Edge f0460000.xor
31: 0 0 0 0 pMSI 10240 Edge f26a0000.xor
32: 0 0 0 0 pMSI 12288 Edge f26c0000.xor
33: 0 0 0 0 ICU-NSR 95 Level f2760000.trng
34: 138 0 0 0 ICU-NSR 115 Level marvell-nfc
35: 44262041 0 0 0 ICU-NSR 39 Level eth0
36: 0 5179587 0 0 ICU-NSR 43 Level eth0
37: 0 0 4584063 0 ICU-NSR 47 Level eth0
38: 0 0 0 6257681 ICU-NSR 51 Level eth0
39: 4 0 0 0 ICU-NSR 129 Level eth0
40: 0 0 0 0 ICU-NSR 105 Level xhci-hcd:usb1
41: 0 0 0 0 ICU-NSR 77 Level f2284000.rtc
42: 0 0 0 0 ICU-NSR 120 Level mv64xxx_i2c
43: 0 0 0 0 AP SEI 18 Level f06f8000.system-controller:thermal-sensor@80
47: 0 0 0 0 mv88e6xxx-g1 3 Edge mv88e6xxx-f212a200.mdio-mii:00-g1-atu-prob
49: 0 0 0 0 mv88e6xxx-g1 5 Edge mv88e6xxx-f212a200.mdio-mii:00-g1-vtu-prob
51: 4 3 0 3 mv88e6xxx-g1 7 Edge mv88e6xxx-f212a200.mdio-mii:00-g2
54: 1 1 0 0 mv88e6xxx-g2 0 Edge mv88e6xxx-f212a200.mdio-mii:00-serdes-0
55: 1 1 0 1 mv88e6xxx-g2 1 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:01
56: 0 0 0 0 mv88e6xxx-g2 2 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:02
57: 0 0 0 0 mv88e6xxx-g2 3 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:03
58: 1 0 0 0 mv88e6xxx-g2 4 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:04
59: 0 0 0 0 mv88e6xxx-g2 5 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:05
60: 0 0 0 0 mv88e6xxx-g2 6 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:06
61: 1 1 0 2 mv88e6xxx-g2 7 Edge !cp0!config-space@f2000000!mdio@12a200!switch@0!mdio:07
63: 0 0 0 0 mv88e6xxx-g2 9 Edge mv88e6xxx-f212a200.mdio-mii:00-serdes-9
64: 0 0 0 0 mv88e6xxx-g2 10 Edge mv88e6xxx-f212a200.mdio-mii:00-serdes-10
69: 0 0 0 0 mv88e6xxx-g2 15 Edge mv88e6xxx-f212a200.mdio-mii:00-watchdog
70: 0 0 0 0 ICU-SEI 116 Edge f2400000.system-controller:thermal-sensor@70
71: 0 0 0 0 f2440000.system-controller:gpio@100 28 Edge keys
72: 0 0 0 0 f2440000.system-controller:gpio@100 11 Edge sfp-mod-def0
73: 0 0 0 0 f2440000.system-controller:gpio@100 2 Edge sfp-los
74: 0 0 0 0 f2440000.system-controller:gpio@100 6 Edge sfp-tx-fault
IPI0: 36114 37297 36505 35228 Rescheduling interrupts
IPI1: 1215291 10033944 9732221 9913557 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 0 0 Timer broadcast interrupts
IPI5: 37019 60663 54355 59918 IRQ work interrupts
IPI6: 0 0 0 0 CPU wake-up interrupts
I tested on 22.03 branch it works there now too. Thanks!
cat /proc/interrupts
CPU0 CPU1
10: 680631 868362 GIC-0 30 Level arch_timer
15: 1 0 MT_SYSIRQ 163 Level mt-pmic-pwrap
22: 32 0 MT_SYSIRQ 91 Level ttyS0
25: 0 0 MT_SYSIRQ 118 Level 1100a000.spi
28: 13489 0 MT_SYSIRQ 96 Level mtk-snand
29: 0 0 MT_SYSIRQ 122 Level 11016000.spi
30: 246634 0 MT_SYSIRQ 211 Level mt7615e
31: 0 0 MT_SYSIRQ 232 Level xhci-hcd:usb1
34: 0 0 MT_SYSIRQ 219 Level 1b007000.dma-controller
38: 320957 0 MT_SYSIRQ 224 Level 1b100000.ethernet
39: 4618 521838 MT_SYSIRQ 225 Level 1b100000.ethernet
40: 0 0 mt-eint 0 Edge gpio-keys
93: 6 0 mt-eint 53 Level mt7530
142: 0 0 mt-eint 102 Edge gpio-keys
145: 0 0 mt7530 0 Edge mt7530-0:00
146: 0 0 mt7530 1 Edge mt7530-0:01
147: 4 0 mt7530 2 Edge mt7530-0:02
148: 1 0 mt7530 3 Edge mt7530-0:03
149: 1 0 mt7530 4 Edge mt7530-0:04
151: 505865 0 MTK PCIe MSI 524288 Edge mt7915e
IPI0: 7873 7476 Rescheduling interrupts
IPI1: 1284537 486942 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 154815 22241 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
AndrewZ
February 22, 2023, 7:20pm
20
I guess ERR: at the bottom is not good?irqbalance
root@RT1800:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
8: 16828 16812 16804 16795 MIPS GIC Local 1 timer
9: 138723 0 0 0 MIPS GIC 63 IPI call
10: 0 81312 0 0 MIPS GIC 64 IPI call
11: 0 0 133223 0 MIPS GIC 65 IPI call
12: 0 0 0 117289 MIPS GIC 66 IPI call
13: 1484 0 0 0 MIPS GIC 67 IPI resched
14: 0 1018 0 0 MIPS GIC 68 IPI resched
15: 0 0 689 0 MIPS GIC 69 IPI resched
16: 0 0 0 706 MIPS GIC 70 IPI resched
17: 0 0 0 0 MIPS GIC 19 1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
19: 12 0 0 0 MIPS GIC 33 ttyS0
20: 0 0 0 0 MIPS GIC 29 xhci-hcd:usb1
21: 151089 0 0 0 MIPS GIC 10 1e100000.ethernet
22: 8 0 0 0 MIPS GIC 30 mt7530
23: 3 0 0 0 mt7530 1 mt7530-0:01
24: 0 0 0 0 mt7530 2 mt7530-0:02
25: 0 0 0 0 mt7530 3 mt7530-0:03
26: 5 0 0 0 mt7530 4 mt7530-0:04
27: 0 0 0 0 1e000600.gpio 6 keys
28: 0 0 0 0 1e000600.gpio 18 keys
29: 432 0 0 0 MIPS GIC 11 mt7915e-hif
30: 56612 0 0 0 MIPS GIC 31 mt7915e
ERR: 2001