Mvebu: mvneta testing MQPrio in place of MQ

For Developers
1

MQPrio has been pushed to master for mvebu target, and can be tested by replacing the default MQ.

replace and set a couple of priorities (example) 
echo "test MQPrio w clsact"
tc qdisc del dev eth0 root 2> /dev/null > /dev/null
/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
	num_tc 4 map 0 0 1 1 2 2 3 3 0 0 0 1 1 1 2 3\
	queues 2@0 2@2 2@4 2@6\
	hw 1 mode channel shaper bw_rlimit\
	min_rate 0Mbit 0Mbit 0Mbit 0Mbit\
	max_rate 1Gbit 1Gbit 1Gbit 1Gbit
tc qdisc del dev eth0 handle ffff: clsact 2> /dev/null > /dev/null
tc qdisc add dev eth0 clsact
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 80 0xffff action skbedit priority 4
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 443 0xffff action skbedit priority 5
Summary 
root@mamba:/etc# tc -s -d -g qdisc show dev eth0 handle 1:
qdisc mqprio 1: root tc 4 map 0 0 1 1 2 2 3 3 0 0 0 1 1 1 2 3 
             queues:(0:1) (2:3) (4:5) (6:7) 
             mode:channel
             shaper:bw_rlimit	min_rate:0bit 0bit 0bit 0bit 	max_rate:1Gbit 1Gbit 1Gbit 1Gbit 
 Sent 2362539 bytes 15789 pkt (dropped 0, overlimits 0 requeues 2) 
 backlog 0b 0p requeues 2
root@mamba:/etc# tc -s -d -g class show dev eth0
+---(1:ffe3) mqprio 
|    |          Sent 1003348 bytes 8716 pkt (dropped 0, overlimits 0 requeues 0) 
|    |          backlog 0b 0p requeues 0
|    |
|    +---(1:7) mqprio 
|    |            Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
|    |            backlog 0b 0p requeues 0
|    |     
|    +---(1:8) mqprio 
|                 Sent 1003348 bytes 8716 pkt (dropped 0, overlimits 0 requeues 0) 
|                 backlog 0b 0p requeues 0
|    
+---(1:ffe2) mqprio 
|    |          Sent 304955 bytes 1454 pkt (dropped 0, overlimits 0 requeues 0) 
|    |          backlog 0b 0p requeues 0
|    |
|    +---(1:5) mqprio 
|    |            Sent 199287 bytes 856 pkt (dropped 0, overlimits 0 requeues 0) 
|    |            backlog 0b 0p requeues 0
|    |     
|    +---(1:6) mqprio 
|                 Sent 105668 bytes 598 pkt (dropped 0, overlimits 0 requeues 0) 
|                 backlog 0b 0p requeues 0
|    
+---(1:ffe1) mqprio 
|    |          Sent 1862 bytes 19 pkt (dropped 0, overlimits 0 requeues 0) 
|    |          backlog 0b 0p requeues 0
|    |
|    +---(1:3) mqprio 
|    |            Sent 980 bytes 10 pkt (dropped 0, overlimits 0 requeues 0) 
|    |            backlog 0b 0p requeues 0
|    |     
|    +---(1:4) mqprio 
|                 Sent 882 bytes 9 pkt (dropped 0, overlimits 0 requeues 0) 
|                 backlog 0b 0p requeues 0
|    
+---(1:ffe0) mqprio 
     |          Sent 1053646 bytes 5612 pkt (dropped 0, overlimits 0 requeues 2) 
     |          backlog 0b 0p requeues 2
     |
     +---(1:1) mqprio 
     |            Sent 571615 bytes 2934 pkt (dropped 0, overlimits 0 requeues 2) 
     |            backlog 0b 0p requeues 2
     |     
     +---(1:2) mqprio 
                  Sent 482031 bytes 2678 pkt (dropped 0, overlimits 0 requeues 0) 
                  backlog 0b 0p requeues 0

Tested as indicated and other config parameters with no other QOS. Played with cake using qosify, sqm setup (including ldir interesting ctinfo* sqmqosnfa version).

You will have to drop

  • target/linux/mvebu/patches-5.10/700-mvneta-tx-queue-workaround.patch

As concerns the pachyderm:

  • Does the issue that drove the 700-* patch still exist? Do not recall how to force and test the issue, but saw no manifestation of same.
  • Is it worthwhile. Things seem to test consistently better on the usual latency sites, but I certainly did not do anything scientific, probably just seeing what I had hoped to see.
2

Looks interesting. Did you try with different rate limits for different queues (priorities)?

It would be interesting to see if MQPRIO with different rates and FQ_CODEL attached to those could still manage bufferbloat but get better performance since the rate limiting is done by hardware.

If that is really the case (maybe wishful thinking on my part), we could try to add this type of functionality to other targets as well (with less CPU power).

3

Not yet. I was intending to try and set up something inline with cake diffserv4 values to see as to back-pressure from the driver, differing egress / ingress...

4

I assume @dtaht will be interested in this.

5

hello @anomeome, i build my own image and use qosify. I wonder if adding mqprio will gaine some extra performance on my wrt32x. If so ..what steps do i need to take?

6

Generating an image to test is simple enough, a minimum make it work:

enable in menuconfig
CONFIG_PACKAGE_kmod-sched-mqprio=y

apply

rm700.patch 
diff --git a/target/linux/mvebu/patches-5.10/700-mvneta-tx-queue-workaround.patch b/target/linux/mvebu/patches-5.10/700-mvneta-tx-queue-workaround.patch
deleted file mode 100644
index a0f15681f5..0000000000
--- a/target/linux/mvebu/patches-5.10/700-mvneta-tx-queue-workaround.patch
+++ /dev/null
@@ -1,38 +0,0 @@
-The hardware queue scheduling is apparently configured with fixed
-priorities, which creates a nasty fairness issue where traffic from one
-CPU can starve traffic from all other CPUs.
-
-Work around this issue by forcing all tx packets to go through one CPU,
-until this issue is fixed properly.
-
-Signed-off-by: Felix Fietkau <nbd@nbd.name>
----
---- a/drivers/net/ethernet/marvell/mvneta.c
-+++ b/drivers/net/ethernet/marvell/mvneta.c
-@@ -4903,6 +4903,16 @@ static int mvneta_ethtool_set_eee(struct
- 	return phylink_ethtool_set_eee(pp->phylink, eee);
- }
- 
-+#ifndef CONFIG_ARM64
-+static u16 mvneta_select_queue(struct net_device *dev, struct sk_buff *skb,
-+			       struct net_device *sb_dev)
-+{
-+	/* XXX: hardware queue scheduling is broken,
-+	 * use only one queue until it is fixed */
-+	return 0;
-+}
-+#endif
-+
- static const struct net_device_ops mvneta_netdev_ops = {
- 	.ndo_open            = mvneta_open,
- 	.ndo_stop            = mvneta_stop,
-@@ -4913,6 +4923,9 @@ static const struct net_device_ops mvnet
- 	.ndo_fix_features    = mvneta_fix_features,
- 	.ndo_get_stats64     = mvneta_get_stats64,
- 	.ndo_do_ioctl        = mvneta_ioctl,
-+#ifndef CONFIG_ARM64
-+	.ndo_select_queue    = mvneta_select_queue,
-+#endif
- 	.ndo_bpf	     = mvneta_xdp,
- 	.ndo_xdp_xmit        = mvneta_xdp_xmit,
- };

add

/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
	num_tc 8 map 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7\
	queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7\
	hw 1 mode channel shaper bw_rlimit\
	min_rate 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit\
	max_rate 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit

to device /etc/rc.local

Some things with which to play:

Summary 
###
#	num_tc 8 map 0 1 2 3 4 5 6 7 0 1 1 1 2 2 3 3\
#	num_tc 4 map 0 0 0 1 1 1 2 3 0 0 0 1 1 2 3 3\
#	num_tc 4 map 0 0 1 1 2 2 3 3 0 0 1 1 2 2 2 2\
#	num_tc 1 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
#	queues 2@0 2@2 2@4 2@6\
#	queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7\
#	hw 1 mode dcb shaper dcb
#	hw 1 mode channel shaper bw_rlimit\
#	min_rate 0Mbit 0Mbit 0Mbit 0Mbit\
#	max_rate 1Gbit 1Gbit 1Gbit 1Gbit
####
echo "del last root"
tc qdisc del dev eth0 root > /dev/null 2>&1
####
echo "set MQPrio at root(eth1), 8 TC->8(1x1) queues"
/sbin/tc qdisc add dev eth1 parent root handle 1 mqprio\
	num_tc 8 map 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7\
	queues 1@0 1@1 1@2 1@3 1@4 1@5 1@6 1@7\
	hw 1 mode channel shaper bw_rlimit\
	min_rate 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit 0Mbit\
	max_rate 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit 1Gbit
####
echo "set MQPrio at root(eth0), 4 TC->4(2x2) queues"
/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
	num_tc 4 map 0 0 1 1 2 2 3 3 0 0 0 1 1 1 2 3\
	queues 2@0 2@2 2@4 2@6\
	hw 1 mode channel shaper bw_rlimit\
	min_rate 0Mbit 0Mbit 0Mbit 0Mbit\
	max_rate 1Gbit 1Gbit 1Gbit 1Gbit
####
#echo "set MQPrio at root, 3 TC->2(3x3) 1(2x2) queues"
#/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
#	num_tc 3 map 0 0 0 1 1 1 2 2 1 1 1 1 1 2 2 2\
#	queues 3@0 3@3 2@6\
#	hw 1 mode channel shaper bw_rlimit\
#	min_rate 0Mbit 0Mbit 0Mbit\
#	max_rate 1Gbit 1Gbit 1Gbit
####
#echo "set MQPrio at root, 2 TC->2(4x4) queues"
#/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
#	num_tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0\
#	queues 4@0 4@4\
#	hw 1 mode channel shaper bw_rlimit\
#	min_rate 0Mbit 0Mbit\
#	max_rate 1Gbit 1Gbit
####
#echo "set MQPrio at root, 1 TC->1(1x8) queues"
#/sbin/tc qdisc add dev eth0 parent root handle 1 mqprio\
#	num_tc 1 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
#	queues 8@0\
#	hw 1 mode channel shaper bw_rlimit\
#	min_rate 0Mbit\
#	max_rate 1Gbit
####
#echo "set CAKE out"
#/sbin/tc qdisc add dev eth0 parent 1:1 handle 11:\
#	cake bandwidth 100Mbit diffserv4 dual-srchost\
#	nat nowash no-ack-filter split-gso rtt 100ms noatm\
#	overhead 18 mpu 64 fwmark 0xf
#echo "set CAKE in"
#ip link delete ifb-eth0 type ifb
#echo "test12"
#ip link add name ifb-eth0 type ifb
#echo "test2"
#tc qdisc del dev eth0 ingress > /dev/null 2>&1
#echo "test3"
#tc qdisc add dev eth0 handle ffff: ingress
#echo "test4"
#tc qdisc del dev ifb-eth0 root > /dev/null 2>&1
#echo "test5"
#tc qdisc add dev ifb-eth0 root\
#	cake bandwidth 400Mbit diffserv4 dual-dsthost\
#	nat nowash ingress no-ack-filter split-gso rtt 100ms noatm\
#	overhead 18 mpu 64 fwmark 0xf
#echo "test6"
#ip link set ifb-eth0 up
#echo "test7"
#tc filter add dev eth0 parent ffff: matchall action\
#	mirred egress redirect dev ifb-eth0
#tc filter add dev eth0 parent ffff: protocol all prio 10 u32 \
#	match u32 0 0 flowid 1:1 action connmark\
#	action mirred egress redirect dev ifb-eth0
#echo "test8"
####
#tc filter add dev eth0 parent ffff: protocol all prio 10 u32 \
#	match u32 0 0 flowid 1:1 action connmark\
#	action mirred egress redirect dev ifb1
####
echo "test PFIFO"
tc qdisc add dev eth1 parent 1:1 handle 11: pfifo limit 1020
tc qdisc add dev eth1 parent 1:2 handle 12: pfifo limit 1020
tc qdisc add dev eth1 parent 1:3 handle 13: pfifo limit 1020
tc qdisc add dev eth1 parent 1:4 handle 14: pfifo limit 1020
tc qdisc add dev eth1 parent 1:5 handle 15: pfifo limit 1020
tc qdisc add dev eth1 parent 1:6 handle 16: pfifo limit 1020
tc qdisc add dev eth1 parent 1:7 handle 17: pfifo limit 1020
tc qdisc add dev eth1 parent 1:8 handle 18: pfifo limit 1020
####
echo "test PFIFO"
tc qdisc add dev eth0 parent 1:1 handle 11: pfifo limit 1020
tc qdisc add dev eth0 parent 1:2 handle 12: pfifo limit 1020
tc qdisc add dev eth0 parent 1:3 handle 13: pfifo limit 1020
tc qdisc add dev eth0 parent 1:4 handle 14: pfifo limit 1020
tc qdisc add dev eth0 parent 1:5 handle 15: pfifo limit 1020
tc qdisc add dev eth0 parent 1:6 handle 16: pfifo limit 1020
tc qdisc add dev eth0 parent 1:7 handle 17: pfifo limit 1020
tc qdisc add dev eth0 parent 1:8 handle 18: pfifo limit 1020
####
#echo "test CODEL"
#tc qdisc add dev eth0 parent 1:1 handle 11: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:2 handle 12: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:3 handle 13: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:4 handle 14: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:5 handle 15: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:6 handle 16: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:7 handle 17: codel limit 10240 ecn
#tc qdisc add dev eth0 parent 1:8 handle 18: codel limit 10240 ecn
####
echo "test CLSACT"
tc qdisc del dev eth0 handle ffff: clsact > /dev/null 2>&1
tc qdisc add dev eth0 clsact
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 22 0xffff action skbedit priority 6
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 25 0xffff action skbedit priority 4
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 53 0xffff action skbedit priority 6
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 80 0xffff action skbedit priority 1
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 110 0xffff action skbedit priority 6
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 123 0xffff action skbedit priority 6
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 443 0xffff action skbedit priority 5
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 563 0xffff action skbedit priority 1
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 853 0xffff action skbedit priority 3
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 993 0xffff action skbedit priority 2
tc filter add dev eth0 egress prio 1 u32\
	match ip dport 5060 0xffff action skbedit priority 4
####
#tc filter add dev eth0 parent 1: protocol ip prio 1 u32\
#	match ip dst 192.168.0.3 \
#	action skbedit queue_mapping 3
####
#tc qdisc add dev eth0 parent 48:4 fq_codel
#tc qdisc add dev eth0 parent 48:3 fq_codel
#tc qdisc add dev eth0 parent 48:2 fq_codel
#tc qdisc add dev eth0 parent 48:1 fq_codel
#tc qdisc add dev eth0 parent 84:8 fq_codel
#tc qdisc add dev eth0 parent 84:7 fq_codel
#tc qdisc add dev eth0 parent 84:6 fq_codel
#tc qdisc add dev eth0 parent 84:5 fq_codel
# tc filter add dev eth0 egress prio 1\
#	u32 match ip dport 7776 0xffff\
#	action skbedit priority 0
### example
# tc qdisc add dev eth3 root mqprio\
#	num_tc 3 map 0 0 0 0 1 1 1 2 queues 2@0 2@2 1@4
# tc qdisc add dev eth3 parent 8002:4 fq_codel
# tc qdisc add dev eth3 parent 8002:5 fq_codel
# tc filter add dev eth3 parent 0: prio 1 protocol ip u32 \
#	match ip dport 118 0xffff action skbedit queue_mapping 4
###
tc qdisc show dev eth0
tc qdisc show dev eth1
#tc -s -d -g class show dev eth0

Notes:

  • Have not run the default fqcodel MQ with the 700 patch removed and have seen no indication in upstream codebase as to the issue that drove the patch as being resolved. Just have not seen any indication as to starvation when using MQPrio.
  • I have not played with any of the CLSACT filters in that last bit since the last qosify push which has the change to clsact commit
7

thnks for the info. I have a dsa on one partition and swconfig on the other partition. Both build from master.
See if i gain some more performance with mqprio.