5.10 kernel dsa driver issues on armada 370 devices

For Developers
1

I have an armada 370 box with 88e6171r switch chip, and my box is runing the latest snapshot version that built by myself.
First is the performance issue:

With the patch by @danitool from MVEBU: Low performance on Armada 370, Level 2 cache disabled?, my box got the full speed with mvsw driver:

[    2.984565] mvsw61xx 10.mvsw61xx: Found MV88E6171 at f1072004.mdio-mii:10
[    2.991394] mvsw61xx 10.mvsw61xx: Using direct addressing


root@OpenWrt:/# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 35744
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 35746
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   100 MBytes   841 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   929 Mbits/sec                  
[  5]   2.00-3.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   927 Mbits/sec                  
[  5]   4.00-5.00   sec   110 MBytes   924 Mbits/sec                  
[  5]   5.00-6.00   sec  99.2 MBytes   833 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   7.00-8.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.06 GBytes   910 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 35756
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 35758
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   110 MBytes   918 Mbits/sec                  
[  5]   1.00-2.00   sec   110 MBytes   924 Mbits/sec                  
[  5]   2.00-3.00   sec   102 MBytes   853 Mbits/sec                  
[  5]   3.00-4.00   sec   110 MBytes   925 Mbits/sec                  
[  5]   4.00-5.00   sec   110 MBytes   927 Mbits/sec                  
[  5]   5.00-6.00   sec   110 MBytes   924 Mbits/sec                  
[  5]   6.00-7.00   sec   110 MBytes   927 Mbits/sec                  
[  5]   7.00-8.00   sec   102 MBytes   853 Mbits/sec                  
[  5]   8.00-9.00   sec   110 MBytes   924 Mbits/sec                  
[  5]   9.00-10.00  sec   110 MBytes   924 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.06 GBytes   910 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 35764
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 35766
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   926 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec                  
[  5]   2.00-3.00   sec   101 MBytes   844 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   931 Mbits/sec                  
[  5]   4.00-5.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   930 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   931 Mbits/sec                  
[  5]   7.00-8.00   sec  98.5 MBytes   826 Mbits/sec                  
[  5]   8.00-9.00   sec   111 MBytes   930 Mbits/sec                  
[  5]   9.00-10.00  sec   110 MBytes   927 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

But with the dsa driver the performance was low:

[    1.775485] mv88e6085 f1072004.mdio-mii:00 lan1 (uninitialized): PHY [!soc!internal-regs!mdio@72004!switch0@0!mdio:00] driver [Generic PHY] (irq=POLL)
[    1.797108] mv88e6085 f1072004.mdio-mii:00 lan2 (uninitialized): PHY [!soc!internal-regs!mdio@72004!switch0@0!mdio:01] driver [Generic PHY] (irq=POLL)
[    1.818021] mv88e6085 f1072004.mdio-mii:00 lan3 (uninitialized): PHY [!soc!internal-regs!mdio@72004!switch0@0!mdio:02] driver [Generic PHY] (irq=POLL)
[    1.839594] mv88e6085 f1072004.mdio-mii:00 lan4 (uninitialized): PHY [!soc!internal-regs!mdio@72004!switch0@0!mdio:03] driver [Generic PHY] (irq=POLL)
[    1.858969] mv88e6085 f1072004.mdio-mii:00 wan (uninitialized): PHY [!soc!internal-regs!mdio@72004!switch0@0!mdio:04] driver [Generic PHY] (irq=POLL)



root@OpenWrt:/# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 42600
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 42602
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  80.4 MBytes   673 Mbits/sec                  
[  5]   1.00-2.00   sec  79.8 MBytes   670 Mbits/sec                  
[  5]   2.00-3.00   sec  81.0 MBytes   680 Mbits/sec                  
[  5]   3.00-4.00   sec  80.9 MBytes   678 Mbits/sec                  
[  5]   4.00-5.00   sec  80.8 MBytes   678 Mbits/sec                  
[  5]   5.00-6.00   sec  79.5 MBytes   666 Mbits/sec                  
[  5]   6.00-7.01   sec  80.4 MBytes   672 Mbits/sec                  
[  5]   7.01-8.00   sec  79.0 MBytes   666 Mbits/sec                  
[  5]   8.00-9.00   sec  81.1 MBytes   681 Mbits/sec                  
[  5]   9.00-10.00  sec  79.8 MBytes   670 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   803 MBytes   674 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 42604
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 42606
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  81.8 MBytes   680 Mbits/sec                  
[  5]   1.01-2.00   sec  81.9 MBytes   693 Mbits/sec                  
[  5]   2.00-3.00   sec  82.6 MBytes   693 Mbits/sec                  
[  5]   3.00-4.01   sec  82.8 MBytes   690 Mbits/sec                  
[  5]   4.01-5.00   sec  83.4 MBytes   704 Mbits/sec                  
[  5]   5.00-6.01   sec  83.0 MBytes   690 Mbits/sec                  
[  5]   6.01-7.00   sec  81.1 MBytes   686 Mbits/sec                  
[  5]   7.00-8.00   sec  82.3 MBytes   688 Mbits/sec                  
[  5]   8.00-9.01   sec  83.1 MBytes   694 Mbits/sec                  
[  5]   9.01-10.00  sec  82.3 MBytes   697 Mbits/sec                  
[  5]  10.00-10.01  sec   452 KBytes   386 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   825 MBytes   691 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.2, port 42612
[  5] local 192.168.1.200 port 5201 connected to 192.168.1.2 port 42614
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  81.7 MBytes   680 Mbits/sec                  
[  5]   1.01-2.00   sec  81.7 MBytes   691 Mbits/sec                  
[  5]   2.00-3.00   sec  83.5 MBytes   701 Mbits/sec                  
[  5]   3.00-4.01   sec  80.9 MBytes   673 Mbits/sec                  
[  5]   4.01-5.01   sec  82.5 MBytes   692 Mbits/sec                  
[  5]   5.01-6.01   sec  82.1 MBytes   692 Mbits/sec                  
[  5]   6.01-7.01   sec  83.5 MBytes   700 Mbits/sec                  
[  5]   7.01-8.00   sec  82.3 MBytes   692 Mbits/sec                  
[  5]   8.00-9.00   sec  83.0 MBytes   699 Mbits/sec                  
[  5]   9.00-10.01  sec  80.9 MBytes   675 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   822 MBytes   689 Mbits/sec                  receiver
iperf3: the client has unexpectedly closed the connection
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

Another issue is all the file transfered from armada 370 box to another device got stuck:

luci interface got loading all the time,and scp file got stalled:
root@OpenWrt:/tmp# scp openwrt-mvebu-cortexa9-marvell_a370-rtnasv3-squashfs-sysu
pgrade.bin Wacke@192.168.1.2:/tmp
User@192.168.1.2's password: 
openwrt-mvebu-cortexa9-marvell_a370-rtnasv3-s   0%  140KB   0.0KB/s - stalled

Is there any idea about these issues? Thanks.

2

maybe the speed issue

3

I think it's not about the multi cpu port, cause my box have only one cpu port connect to the 88e6171r switch, actually on armada 370 devices, the dsa driver performance was always (from 4.9 to 5.10 I've tested) lower then the mvsw driver.

4

The file transfer issue can be fixed by the following patch (both 88e6171r and 88e6161 dsa driver was broken since kernel 5.9.0):

diff -uNr linux-5.10.23_orig/drivers/net/dsa/mv88e6xxx/chip.c linux-5.10.23/drivers/net/dsa/mv88e6xxx/chip.c
--- linux-5.10.23_orig/drivers/net/dsa/mv88e6xxx/chip.c	2021-03-11 21:17:30.000000000 +0800
+++ linux-5.10.23/drivers/net/dsa/mv88e6xxx/chip.c	2021-04-21 23:09:27.014708217 +0800
@@ -3451,7 +3451,6 @@
 	.port_set_frame_mode = mv88e6351_port_set_frame_mode,
 	.port_set_egress_floods = mv88e6352_port_set_egress_floods,
 	.port_set_ether_type = mv88e6351_port_set_ether_type,
-	.port_set_jumbo_size = mv88e6165_port_set_jumbo_size,
 	.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
 	.port_pause_limit = mv88e6097_port_pause_limit,
 	.port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
@@ -3476,6 +3475,7 @@
 	.avb_ops = &mv88e6165_avb_ops,
 	.ptp_ops = &mv88e6165_ptp_ops,
 	.phylink_validate = mv88e6185_phylink_validate,
+	.set_max_frame_size = mv88e6185_g1_set_max_frame_size,
 };
 
 static const struct mv88e6xxx_ops mv88e6165_ops = {
@@ -3527,7 +3527,6 @@
 	.port_set_frame_mode = mv88e6351_port_set_frame_mode,
 	.port_set_egress_floods = mv88e6352_port_set_egress_floods,
 	.port_set_ether_type = mv88e6351_port_set_ether_type,
-	.port_set_jumbo_size = mv88e6165_port_set_jumbo_size,
 	.port_egress_rate_limiting = mv88e6097_port_egress_rate_limiting,
 	.port_pause_limit = mv88e6097_port_pause_limit,
 	.port_disable_learn_limit = mv88e6xxx_port_disable_learn_limit,
@@ -3550,6 +3549,7 @@
 	.vtu_getnext = mv88e6352_g1_vtu_getnext,
 	.vtu_loadpurge = mv88e6352_g1_vtu_loadpurge,
 	.phylink_validate = mv88e6185_phylink_validate,
+	.set_max_frame_size = mv88e6185_g1_set_max_frame_size,
 };
 
 static const struct mv88e6xxx_ops mv88e6172_ops = {

More detail can be found on this post:
https://forum.doozan.com/read.php?2,119449

1 Like