Multi-CPU DSA works on Rango, but additional patches are needed. Will create a PR when I have them all ready and packed. Obviously, the original multi-CPU DSA support PR must be upstreamed beforehand.
nitroshift
4 Likes
To see the CPU usage:
cat /proc/interrupts
To see which port is assigned to a CPU:
ip link
1 Like
That performance test by pkgadd on your link shows the huge performance hit moving to DSA did, at leaast on ipq806x target. We know it hits our targets as well, just less so. DSA is clearly a big performance problem unless it improves, and maybe not working out like they expected it to even though using upstream kernel features is generally preferred.
Using the Divested 15th Sept build (kernel v5.10.65)
I take it that as I am only seeing eth0 that DSA Multi-CPU is not yet available
cat /proc/interrupts
root@OpenWrt:~# cat /proc/interrupts
CPU0 CPU1
25: 0 0 GIC-0 27 Edge gt
26: 43714 36775 GIC-0 29 Edge twd
27: 0 0 MPIC 5 Level armada_370_xp_per_cpu_tick
29: 12900 0 GIC-0 34 Level mv64xxx_i2c
30: 22 0 GIC-0 44 Level ttyS0
40: 0 0 GIC-0 41 Level f1020300.watchdog
44: 0 0 GIC-0 96 Level f1020300.watchdog
45: 13620 0 MPIC 8 Level eth0
46: 0 0 GIC-0 50 Level ehci_hcd:usb1
47: 0 0 GIC-0 51 Level f1090000.crypto
48: 0 0 GIC-0 52 Level f1090000.crypto
49: 0 0 GIC-0 58 Level ahci-mvebu[f10a8000.sata]
50: 370 0 GIC-0 116 Level marvell-nfc
51: 0 0 GIC-0 49 Level xhci-hcd:usb2
52: 2 0 GIC-0 54 Level f1060800.xor
53: 2 0 GIC-0 97 Level f1060900.xor
Irqbalance doesn't do much on mvebu (way less than other targets unfortunately). Mostly irqs just stay on CPU0. But you should at least see mwlwifi driver running on CPU1. Check that 'enabled' line is set to '1' in /etc/config/irqbalance. Multi-CPU DSA is likely a ways off if it'll ever work on our target.
This is how it looks on Rango with DSA multi-CPU enabled:
CPU0 CPU1
25: 3780301 3541376 GIC-0 29 Edge twd
26: 0 0 MPIC 5 Level armada_370_xp_per_cpu_tick
29: 3265764 0 GIC-0 34 Level mv64xxx_i2c
30: 22 0 GIC-0 44 Level ttyS0
40: 0 0 GIC-0 41 Level f1020300.watchdog
44: 0 0 GIC-0 96 Level f1020300.watchdog
45: 2 1 MPIC 8 Level eth0
46: 62450239 24852293 MPIC 12 Level eth1
47: 0 0 GIC-0 50 Level ehci_hcd:usb1
48: 0 0 GIC-0 51 Level f1090000.crypto
49: 0 0 GIC-0 52 Level f1090000.crypto
50: 0 0 GIC-0 58 Level ahci-mvebu[f10a8000.sata]
51: 1400 0 GIC-0 116 Level marvell-nfc
52: 70388 0 GIC-0 57 Level mmc0
53: 11633326 0 GIC-0 49 Level xhci-hcd:usb2
54: 2 0 GIC-0 54 Level f1060800.xor
55: 2 0 GIC-0 97 Level f1060900.xor
59: 0 0 mv88e6xxx-g1 3 Edge mv88e6xxx-f1072004.mdio-mii:00-g1-atu-prob
61: 0 0 mv88e6xxx-g1 5 Edge mv88e6xxx-f1072004.mdio-mii:00-g1-vtu-prob
63: 5 2 mv88e6xxx-g1 7 Edge mv88e6xxx-f1072004.mdio-mii:00-g2
65: 0 0 mv88e6xxx-g2 0 Edge mv88e6xxx-1:00
66: 0 0 mv88e6xxx-g2 1 Edge mv88e6xxx-1:01
67: 0 0 mv88e6xxx-g2 2 Edge mv88e6xxx-1:02
68: 4 0 mv88e6xxx-g2 3 Edge mv88e6xxx-1:03
69: 1 1 mv88e6xxx-g2 4 Edge mv88e6xxx-1:04
76: 0 1 mv88e6xxx-g2 11 Edge mv88e6xxx-f1072004.mdio-mii:00-serdes-5
80: 0 0 mv88e6xxx-g2 15 Edge mv88e6xxx-f1072004.mdio-mii:00-watchdog
81: 0 0 f1018100.gpio 24 Edge gpio-keys
82: 0 0 f1018100.gpio 29 Edge gpio-keys
83: 5 185313 GIC-0 61 Level mwlwifi
84: 1918622 0 GIC-0 65 Level mwlwifi
IPI0: 0 1 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 1836 1655 Rescheduling interrupts
IPI3: 1626436 25525494 Function call interrupts
IPI4: 0 0 CPU stop interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 completion interrupts
Err: 0
Note the presence of eth1.
nitroshift
3 Likes
This is great news for mvebu which is already a very performant target. How are you running multi CPU? I tried the latest master snapshot and added irqbalance and it's not doing that on my WRT32X. EDIT: oh i see you applied the patches yourself. Oh well, hopefully someday soon it'll be added.
It is also possible for wrt1900acs v2 this?
-----------------------------------------------------
OpenWrt 21.02-SNAPSHOT, r16282-601864c09e
-----------------------------------------------------
root@swrt:~# irqbalance -df
This machine seems not NUMA capable.
Isolated CPUs: 00000000
Adaptive-ticks CPUs: 00000000
Banned CPUs: 00000000
Package 0: numa_node -1 cpu mask is 00000003 (load 0)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 0)
CPU number 1 numa_node is -1 (load 0)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 0)
CPU number 0 numa_node is -1 (load 0)
Adding IRQ 74 to database
Adding IRQ 73 to database
Adding IRQ 17 to database
Adding IRQ 18 to database
Adding IRQ 19 to database
Adding IRQ 21 to database
Adding IRQ 22 to database
Adding IRQ 32 to database
Adding IRQ 36 to database
Adding IRQ 37 to database
Adding IRQ 38 to database
Adding IRQ 39 to database
Adding IRQ 40 to database
Adding IRQ 41 to database
Adding IRQ 42 to database
Adding IRQ 43 to database
Adding IRQ 44 to database
Adding IRQ 45 to database
Adding IRQ 49 to database
Adding IRQ 51 to database
Adding IRQ 53 to database
Adding IRQ 55 to database
Adding IRQ 56 to database
Adding IRQ 57 to database
Adding IRQ 58 to database
Adding IRQ 59 to database
Adding IRQ 70 to database
Adding IRQ 71 to database
Adding IRQ 72 to database
NUMA NODE NUMBER: -1
LOCAL CPU MASK: ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,f fffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff, ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff ,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,fffffff f,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffff ff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,fffff fff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffff ffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,fff fffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ff ffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,f fffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff, ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff ,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,fffffff f,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffff ff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
-----------------------------------------------------------------------------
Package 0: numa_node -1 cpu mask is 00000003 (load 0)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 0)
CPU number 1 numa_node is -1 (load 0)
Interrupt 73 node_num is -1 (ethernet/0:20253)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 0)
CPU number 0 numa_node is -1 (load 0)
Interrupt 74 node_num is -1 (ethernet/0:20403)
Interrupt 72 node_num is -1 (other/0:0)
Interrupt 71 node_num is -1 (other/0:0)
Interrupt 70 node_num is -1 (other/0:0)
Interrupt 59 node_num is -1 (other/0:0)
Interrupt 58 node_num is -1 (other/0:0)
Interrupt 57 node_num is -1 (other/0:0)
Interrupt 56 node_num is -1 (other/0:0)
Interrupt 55 node_num is -1 (other/0:0)
Interrupt 53 node_num is -1 (other/0:0)
Interrupt 51 node_num is -1 (other/0:0)
Interrupt 49 node_num is -1 (other/0:0)
Interrupt 45 node_num is -1 (other/0:0)
Interrupt 44 node_num is -1 (other/0:0)
Interrupt 43 node_num is -1 (other/0:22)
Interrupt 42 node_num is -1 (other/0:0)
Interrupt 41 node_num is -1 (other/0:0)
Interrupt 40 node_num is -1 (other/0:0)
Interrupt 39 node_num is -1 (other/0:0)
Interrupt 38 node_num is -1 (other/0:0)
Interrupt 37 node_num is -1 (other/0:76)
Interrupt 36 node_num is -1 (other/0:0)
Interrupt 32 node_num is -1 (other/0:0)
Interrupt 22 node_num is -1 (other/0:0)
Interrupt 21 node_num is -1 (other/0:280)
Interrupt 19 node_num is -1 (other/0:0)
Interrupt 18 node_num is -1 (other/0:2342)
Interrupt 17 node_num is -1 (other/0:0)
-----------------------------------------------------------------------------
Package 0: numa_node -1 cpu mask is 00000003 (load 10000000)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 0)
CPU number 1 numa_node is -1 (load 0)
Interrupt 73 node_num is -1 (ethernet/1:19844)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 10000000)
CPU number 0 numa_node is -1 (load 10000000)
Interrupt 74 node_num is -1 (ethernet/9287910:19974)
Interrupt 72 node_num is -1 (other/1:0)
Interrupt 71 node_num is -1 (other/1:0)
Interrupt 70 node_num is -1 (other/1:0)
Interrupt 59 node_num is -1 (other/1:0)
Interrupt 58 node_num is -1 (other/1:0)
Interrupt 57 node_num is -1 (other/1:0)
Interrupt 56 node_num is -1 (other/1:0)
Interrupt 55 node_num is -1 (other/1:0)
Interrupt 53 node_num is -1 (other/1:0)
Interrupt 51 node_num is -1 (other/1:0)
Interrupt 49 node_num is -1 (other/1:0)
Interrupt 45 node_num is -1 (other/1:0)
Interrupt 44 node_num is -1 (other/1:0)
Interrupt 43 node_num is -1 (other/1864:8)
Interrupt 42 node_num is -1 (other/1:0)
Interrupt 41 node_num is -1 (other/1:0)
Interrupt 40 node_num is -1 (other/1:0)
Interrupt 39 node_num is -1 (other/1:0)
Interrupt 38 node_num is -1 (other/1:0)
Interrupt 37 node_num is -1 (other/43804:188)
Interrupt 36 node_num is -1 (other/1:0)
Interrupt 32 node_num is -1 (other/1:0)
Interrupt 22 node_num is -1 (other/1:0)
Interrupt 21 node_num is -1 (other/116500:500)
Interrupt 19 node_num is -1 (other/1:0)
Interrupt 18 node_num is -1 (other/537298:2306)
Interrupt 17 node_num is -1 (other/1:0)
-----------------------------------------------------------------------------
Package 0: numa_node -1 cpu mask is 00000003 (load 10000000)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 10000000)
CPU number 1 numa_node is -1 (load 10000000)
Interrupt 73 node_num is -1 (ethernet/9296820:19865)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 0)
CPU number 0 numa_node is -1 (load 0)
Interrupt 74 node_num is -1 (ethernet/1:20016)
Interrupt 72 node_num is -1 (other/1:0)
Interrupt 71 node_num is -1 (other/1:0)
Interrupt 70 node_num is -1 (other/1:0)
Interrupt 59 node_num is -1 (other/1:0)
Interrupt 58 node_num is -1 (other/1:0)
Interrupt 57 node_num is -1 (other/1:0)
Interrupt 56 node_num is -1 (other/1:0)
Interrupt 55 node_num is -1 (other/1:0)
Interrupt 53 node_num is -1 (other/1:0)
Interrupt 51 node_num is -1 (other/1:0)
Interrupt 49 node_num is -1 (other/1:0)
Interrupt 45 node_num is -1 (other/1:0)
Interrupt 44 node_num is -1 (other/1:0)
Interrupt 43 node_num is -1 (other/32387:139)
Interrupt 42 node_num is -1 (other/1:0)
Interrupt 41 node_num is -1 (other/1:0)
Interrupt 40 node_num is -1 (other/1:0)
Interrupt 39 node_num is -1 (other/1:0)
Interrupt 38 node_num is -1 (other/1:0)
Interrupt 37 node_num is -1 (other/20737:89)
Interrupt 36 node_num is -1 (other/1:0)
Interrupt 32 node_num is -1 (other/1:0)
Interrupt 22 node_num is -1 (other/1:0)
Interrupt 21 node_num is -1 (other/88540:380)
Interrupt 19 node_num is -1 (other/1:0)
Interrupt 18 node_num is -1 (other/537997:2309)
Interrupt 17 node_num is -1 (other/1:0)
-----------------------------------------------------------------------------
Package 0: numa_node -1 cpu mask is 00000003 (load 10000000)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 0)
CPU number 1 numa_node is -1 (load 0)
Interrupt 73 node_num is -1 (ethernet/1:19856)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 10000000)
CPU number 0 numa_node is -1 (load 10000000)
Interrupt 74 node_num is -1 (ethernet/9270380:20153)
Interrupt 72 node_num is -1 (other/1:0)
Interrupt 71 node_num is -1 (other/1:0)
Interrupt 70 node_num is -1 (other/1:0)
Interrupt 59 node_num is -1 (other/1:0)
Interrupt 58 node_num is -1 (other/1:0)
Interrupt 57 node_num is -1 (other/1:0)
Interrupt 56 node_num is -1 (other/1:0)
Interrupt 55 node_num is -1 (other/1:0)
Interrupt 53 node_num is -1 (other/1:0)
Interrupt 51 node_num is -1 (other/1:0)
Interrupt 49 node_num is -1 (other/1:0)
Interrupt 45 node_num is -1 (other/1:0)
Interrupt 44 node_num is -1 (other/1:0)
Interrupt 43 node_num is -1 (other/5082:22)
Interrupt 42 node_num is -1 (other/1:0)
Interrupt 41 node_num is -1 (other/1:0)
Interrupt 40 node_num is -1 (other/1:0)
Interrupt 39 node_num is -1 (other/1:0)
Interrupt 38 node_num is -1 (other/1:0)
Interrupt 37 node_num is -1 (other/32571:141)
Interrupt 36 node_num is -1 (other/1:0)
Interrupt 32 node_num is -1 (other/1:0)
Interrupt 22 node_num is -1 (other/1:0)
Interrupt 21 node_num is -1 (other/152460:660)
Interrupt 19 node_num is -1 (other/1:0)
Interrupt 18 node_num is -1 (other/532455:2305)
Interrupt 17 node_num is -1 (other/1:0)
-----------------------------------------------------------------------------
Package 0: numa_node -1 cpu mask is 00000003 (load 0)
Cache domain 0: numa_node is -1 cpu mask is 00000002 (load 0)
CPU number 1 numa_node is -1 (load 0)
Interrupt 73 node_num is -1 (ethernet/1:19852)
Cache domain 1: numa_node is -1 cpu mask is 00000001 (load 0)
CPU number 0 numa_node is -1 (load 0)
Interrupt 74 node_num is -1 (ethernet/1:20031)
Interrupt 72 node_num is -1 (other/1:0)
Interrupt 71 node_num is -1 (other/1:0)
Interrupt 70 node_num is -1 (other/1:0)
Interrupt 59 node_num is -1 (other/1:0)
Interrupt 58 node_num is -1 (other/1:0)
Interrupt 57 node_num is -1 (other/1:0)
Interrupt 56 node_num is -1 (other/1:0)
Interrupt 55 node_num is -1 (other/1:0)
Interrupt 53 node_num is -1 (other/1:0)
Interrupt 51 node_num is -1 (other/1:0)
Interrupt 49 node_num is -1 (other/1:0)
Interrupt 45 node_num is -1 (other/1:0)
Interrupt 44 node_num is -1 (other/1:0)
Interrupt 43 node_num is -1 (other/1:10)
Interrupt 42 node_num is -1 (other/1:0)
Interrupt 41 node_num is -1 (other/1:0)
Interrupt 40 node_num is -1 (other/1:0)
Interrupt 39 node_num is -1 (other/1:0)
Interrupt 38 node_num is -1 (other/1:0)
Interrupt 37 node_num is -1 (other/1:63)
Interrupt 36 node_num is -1 (other/1:0)
Interrupt 32 node_num is -1 (other/1:0)
Interrupt 22 node_num is -1 (other/1:0)
Interrupt 21 node_num is -1 (other/1:360)
Interrupt 19 node_num is -1 (other/1:0)
Interrupt 18 node_num is -1 (other/1:2305)
Interrupt 17 node_num is -1 (other/1:0)
^Croot@swrt:~# cat /proc/interrupts
CPU0 CPU1
17: 0 0 GIC-0 27 Edge gt
18: 37482 42627 GIC-0 29 Edge twd
19: 0 0 MPIC 5 Level armada_370_xp_per_cpu_tick
21: 21174 0 GIC-0 34 Level mv64xxx_i2c
22: 23 0 GIC-0 44 Level ttyS0
32: 0 0 GIC-0 41 Level f1020300.watchdog
36: 0 0 GIC-0 96 Level f1020300.watchdog
37: 1841677 0 MPIC 8 Level eth0
38: 0 0 GIC-0 50 Level ehci_hcd:usb1
39: 0 0 GIC-0 51 Level f1090000.crypto
40: 0 0 GIC-0 52 Level f1090000.crypto
41: 0 0 GIC-0 58 Level ahci-mvebu[f10a8000.sata]
42: 171 0 GIC-0 116 Level marvell-nfc
43: 1842 0 GIC-0 49 Level xhci-hcd:usb2
44: 2 0 GIC-0 54 Level f1060800.xor
45: 2 0 GIC-0 97 Level f1060900.xor
49: 0 0 mv88e6xxx-g1 3 Edge mv88e6xxx-g1-atu-prob
51: 0 0 mv88e6xxx-g1 5 Edge mv88e6xxx-g1-vtu-prob
53: 4 3 mv88e6xxx-g1 7 Edge mv88e6xxx-g2
55: 0 0 mv88e6xxx-g2 0 Edge mv88e6xxx-1:00
56: 0 0 mv88e6xxx-g2 1 Edge mv88e6xxx-1:01
57: 0 0 mv88e6xxx-g2 2 Edge mv88e6xxx-1:02
58: 1 2 mv88e6xxx-g2 3 Edge mv88e6xxx-1:03
59: 3 1 mv88e6xxx-g2 4 Edge mv88e6xxx-1:04
70: 0 0 mv88e6xxx-g2 15 Edge mv88e6xxx-watchdog
71: 0 0 f1018100.gpio 24 Edge gpio-keys
72: 0 0 f1018100.gpio 29 Edge gpio-keys
73: 23792 566009 GIC-0 61 Level mwlwifi
74: 598901 0 GIC-0 65 Level mwlwifi
IPI0: 0 1 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 14965 12640 Rescheduling interrupts
IPI3: 105 208888 Function call interrupts
IPI4: 0 0 CPU stop interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 completion interrupts
Err: 0
root@swrt:~# cat /etc/config/irqbalance
config irqbalance 'irqbalance'
option enabled '1'
# The default value is 10 seconds
#option interval '10'
# List of IRQ's to ignore
#list banirq '36'
#list banirq '69'
root@swrt:~#
I'm waiting for the generic PR to be merged. Making it work on Armada 385 family is pretty trivial but requires changes to the device tree file (armada-385-linksys.dtsi) since there's only one CPU port defined. I don't have any Armada XP or Armada 375 devices to test, but it's safe to assume it can be done for those SOC families too by changing their respective device tree files, the logic being the same.
nitroshift
3 Likes
Is this with RPS enabled also?
Why I don't have ~50% speed decrease on my device (WRT1200) with DSA enabled?
The highest wan <-> lan speed I have measured was ~900Mbit/s on a 1Gbit docsis connection.
Maybe more is possible. (Docsis and Internet servers are not consistent)
With cake enabled my speed indeed drops below ~50%.
AFAICT there's no RPS involved. Also, same as youself, I didn't see any speed degradation going from swconfig to DSA and I made the switch very early in the adoption process.
nitroshift
With mvebu moving to GCC 11 please let us know if anyone notices any difference in builds for mvebu since it's a much newer compiler version than GCC 8 from 21.02 release. I'll try a new build this weekend, my WRT32X has been rock solid the last couple weeks though.
1 Like
I'm running it for at least a week now in my own builds, no regression noticed. For more info you can find me on irc with the same handle as here.
nitroshift
1 Like
No issues here with the last two builds using 11.03. Plug and Play.
1 Like
Linksys WRT3200ACM here, was running r16723, just did a clean upgrade to r17571 and have been having stability problems since. Seemingly at random from a couple hours to 24hrs I'll have total network degradation, and luci ui will load and login but the status pages etc won't actually load until the unit is rebooted then everything works again. SSHing in and checking top doesn't show anything chewing the cpu or mem so no idea what could be going on.
I built r17581 9/24 for 1900ACS and it’s been stable since it booted. Bleeding edge snapshots sometimes don’t produce stable builds. Try a new build.
Yea I flashed to the previous build, [r17543], but it seems like dnscrypt-proxy2 package is incompatible with this version. Throws an error when you run it
Error relocating dnscrypt-proxy: __nanosleep_time64: symbol not found
Hmmm! Can’t speculate about that. I’ve used dnscrypt-proxy2 since Davidc’s build and continued building/using it here with no issues at all.
Thinking back, iirc and fwiw, the .toml config was in /etc/config in that build, but the app wanted it in /etc/dnscrypt-proxy2 .
There should be a -check or -config switch on invoking the init.d script that might give a clue if it will even return anything given the fail return message. I’m out of my league even trying to determine what the error response means - sorry.
I build Divested snapshots with customized configuration for my Linksys 32X and it has been running OK. Today I flashed it with the newest build and cannot use Luci nor ssh. With Luci it says something about main is missing in configuration, and with ssh the connection is reset during key exchange.
Anyone know what is the best way to access the router to fix it? Or we have to rely on factory reset? Or can I boot to the alternative partition that has an older build?
TIA.
Go back to the previous partition. Use 3X power cycle if you can’t invoke it from LuCI or SSH.
1 Like