WRT3200ACM and DSA

But the marvell 88f6281+ marvell 88e6171r device get the normal performance with DSA driver. Maybe it's a bug of mvneta driver or DSA driver?

So, how's everything going?
Can you guys tell me which patches you made?
@ParanoidZoid have you tried to make a pull request o merge your patches with OpenWRT?

Still working on it. I still need to understand DSA better. I find it really difficult to move away from the ease of use of swconfig right now, especially since it has so many easy to access and centralised documentation in the OpenWrt wiki/forums. I also need to stop playing around with the singular router in the household, lest I incur the greater wrath of my family.

I still don't have a transition script in the works and for reasons with regards to work, I'll be extremely busy for the following two weeks. I'll try and make some progress this weekend, but it's probably best if you (or someone else) work with what I have with the branch in my fork and tidying it up for submission as a pull request.

I believe what still needs to be done is:

  1. Fix assigning the MAC address properly instead of using a random one every single boot (I've already modified it in my branch but have not tested it yet).
  2. Make a transition script for all use cases esp. w.r.t. VLANs.
  3. Investigate what is going on with the 1900ACv1 Mamba's mv88e6171r + DSA.
  4. More tests required - iperf, netperf, flent. What happens with bufferbloat? What happens with throughput? Can we stress test & enable/disable various physical switch options and invoke a warning/failure?

With your current approach (using blogic's patch),
wan and lan1-4 will end up with the same mac address.
Can you actually confirm that the patch does work?

Since i can't get blogics patch working i used the following approach:

		mac=$(mtd_get_mac_ascii devinfo hw_mac_addr)
		mac_wan=$(macaddr_setbit_la "$mac")
		ip link set dev eth1 address $mac_wan 2>/dev/null
		ip link set dev eth0 address $mac 2>/dev/null
		ip link set dev lan1 address $mac 2>/dev/null
		ip link set dev lan2 address $mac 2>/dev/null
		ip link set dev lan3 address $mac 2>/dev/null
		ip link set dev lan4 address $mac 2>/dev/null
		ip link set dev wan address $mac_wan 2>/dev/null

Performance-wise. i can't see any difference, atleast on wrt1200.
But i can only test speeds up to ~400 Mbit/s.

Some other question i have...
dsa was developed by marvell and was first introduced like over 10y ago?
How is it possible that there is only support for one cpu port?

Yeah I've just tested it and it doesn't work. I guess your approach is good for now - until someone who knows more than us does a cleaner approach with an in-driver solution.

The multiple CPU port stuff was brought up by blogic two years ago - but upstream reviewers couldn't agree on whether to adopt it or not.


Tested on 4.19

Maybe this can help (?)
https://www.kernel.org/doc/Documentation/networking/dsa/dsa.txt

I really get you ahahaha

I'll try to work on it too, my teachers are really good at scripting so they can help me

That's a really good question, I was faced with that issue, so I had to improvise and made a bridge between the WAN/WAN6 interfaces and enabled SQM in there, and it worked ahahahah

I've made tests between my server and my laptop, the bridge is actually offloaded to the switch and the one CPU port isn't used at all, so it woeks perfectly!
Now, between the WAN and the LAN the CPU port is used as expected, but you can fully do downloads at gigabit speed without performance losses.

What I'm thinking is, this is supported upstream, so, supposedly this issue with the MAC's can't happen upstream right?

Your patch works on the latest trunk? And with kernel 4.19?

Also, on another not @ParanoidZoid your patch no longer works since changes were made to the mvebu tree(or whatever it's called) I tried to fix it but got nowhere. cries

1 Like

I've had a look at @LGA1150's branch and it should be easily adopted into 4.14. I'll rebase and merge his/her edits into mine. Still a bit busy - but I should be able to find the time to do it. However, I won't be able to compile/run-test it. And yeah, you'll still need to develop a transition script to run on sysupgrade.

Here's my take with @LGA1150's additional edits for MAC address preservation, adapted for the current master (0111b86), kernel 4.14. Not compile nor runtime tested. Patch version here: https://github.com/jeolives/openwrt/commit/e73361b7afb6ac68ebc4bd670a82aa67a93d2582.patch

Ok, I'll compile it to see if it works, I'll get back to you with some news.

1 Like

A sysupgrade whilst saving settings from a build will result in a soft-brick.

Not really. Even if the ethernet is unavailable after sysupgrade, one can use WiFi or UART to log in and change the network settings :upside_down_face:

1 Like

another way would be to manually add the interface before flash... (in the network config have in br-lan both eth0 and lan1 lan2 lan3 lan4)
The router won't complain about missing interface

OK, news here, I flashed Openwrt yesterday with your patch and it works like it did before, but every interface has the same MAC address, Is that supposed?
Other than that the patch works like a charm!

1 Like

@ParanoidZoid @neheb
FYI
https://patchwork.ozlabs.org/patch/441908/
https://patchwork.ozlabs.org/cover/820302/
Maybe we can use these 2 patches to bond two CPU ports for load balancing?

Maybe. I use a Turris Omnia where having two ports doesn't offer much benefit. Having said that, it may be useful for kernel 4.19. I still don't know if the OpenWrt team plans to release a new version with 4.14 given that it will stop getting updates next year.

Yeah, the interfaces are supposed to be like that. It's that way in the swconfig build. WiFi is +1 and +2 of the LAN MAC (i.e. if the switch's MAC address ends in 47, then radio0 will end in 48, and radio1 will end in 49).

@ParanoidZoid
I saw your post about mq on github.
Posting my answer here seems to be the better option cause more people are reading in the forums...
And maybe this post is useful to someone else..

According to the marvells spec sheet, the armada 38x has 8 tx/rx queues per cpu port.
I don't know how well this is actually implemented in the mvneta driver.

I'm not entirely sure how mq chooses the preferred queue.
But it is possible to setup tc filters to put traffic in different queues.
For example assign 4 queues for high prority traffic, 3 for "normal" traffic and 1 for bulk traffic.

You can also replace the root mq qdisc with your qdisc of choice, like fq_codel or cake.

tc qdisc replace dev eth0 handle 1 root fq_codel
tc qdisc replace dev eth1 handle 1 root fq_codel

(tc output will be "cleaner")

But fq_codel and cake are not multi queue aware.
pfifo_fast is multi queue aware but I haven't figured out how to assign the bands to the queues.
Also pfifo_fast is removed in openwrt. (by patch)

An other option is to replace the root mq qdisc with the multiq qdisc.
multiq qdisc will round-robin the packets through the queues.
But then you can't attach any other qdisc to each queue.
However when using dsa you can attach a qdisc to lan1-4 and wan. (and multiq on eth0/1)
That is what I'm currently using.
multiq on eth0/1 and cake configured with 1gbit bandwidth and lan rtt preset on lan1-4.
(and wan with sqm scripts)

tc qdisc replace dev eth0 handle 1 root multiq
tc qdisc replace dev eth1 handle 1 root multiq
tc qdisc replace dev lan1 handle 1 root cake bandwidth 1Gbit besteffort triple-isolate lan ethernet
tc qdisc replace dev lan2 handle 1 root cake bandwidth 1Gbit besteffort triple-isolate lan ethernet
tc qdisc replace dev lan3 handle 1 root cake bandwidth 1Gbit besteffort triple-isolate lan ethernet
tc qdisc replace dev lan4 handle 1 root cake bandwidth 1Gbit besteffort triple-isolate lan ethernet

Maybe it's better to disable cake's shaper to save some cpu cycles.
But in unlimited mode, cake will use a mem limit of 32 Mb. (add memlimit 4Mb to the command(s))
And replacing triple-isolate with flows?

I think it is also possible to replicate this setup with swconfig.
Attach multiq to eth0/1
Create a vlan for each lan (and wan) port (ethx.1-4) assign a qdisc to each interface.
Then bridge the interfaces together.

//edit
With this approach, traffic will go through (atleast?) 3 qdiscs (qdiscs on lanX and multiq),
when traffic is flowing between hosts that are connected to the switch.
Maybe its better to have fq_codel/cake directly attached to eth0/eth1.
But then the multi queue feature doesn't work.
Something like multiq + qdisc of choice (fq_codel/cake) would be nice.
So the flows get distributed over the hardware queues and
each hardware queue has fq_codel/cake qdisc.
But I haven't figure out how to accomplish this.

There is also the mqprio qdisc but i haven't played around with this yet.

Regarding the tx queue workaround patch.
I think this only effects traffic to and from the device itself.
I have it removed for some time now and the only thing I noticed is that sometimes the webui is painfully slow to load. But only on wifi, quite strange...
I also don't know what happens with multiq and the patch applied...

// and another edit..
regarding mq..
last time i tested this, all packets did go through one queue.
Now i can see packets going through all queues.
But the majority (like 90%?) of packets still going through one queue.
Seems like there is something like rps and xps.

RPS for receive queues and XPS for transmit queues.
Both are disabled by default.
RPS can be configured through /sys/class/net/ethX/queues/rx-/rps_cpus
rps_cpus is set to 2 on my system. (Why? //edit forced to cpu 1)
XPS can be configured as RPS with CPU mappings.
/sys/class/net/ethX/queues/tx-/xps_cpus
Or by mapping rx queues to tx queues.
/sys/class/net/eth0/queues/tx-/xps_rxqs
I have no clue how to configure the rx/tx mapping.
Does someone know? 00 seems to be disabled.
For example, if I want to map tx-0 to rx-0, I should use 11 as value?
And for tx-1 to rx-1, I should use 22?
But then there is still the question why most packets only go through one queue.
I can't find the website anymore but there was mentioned that mq does round robin.
So actually there should be an equal distribution of packets over the queues?

//edit again...
hmm...
I'm not sure if mq is even needed to make rps/xps work.
According to his presentation:

mq uses xps or jhash

queue_mapping strategy: xps, jhash

Why only xps and not rps, i don't know...
It is also quite hard to find any info on this because search engines are so bad these days...

// off topic
Remember those days when you actually typed in a search and you found the relevant info directly.
For example when I search for: linux mq rps or xps, 0 useful results. What?
Same for: linux mqprio example, 1 results that is actually related to that... x)
What happened...

// back to topic
I guess, rps and xps do work regardless of the used qdisc...
But I'm not sure. If some linux guru has more info that would be nice.
If this is the case, attaching fq_codel/cake directly to eth0/1 should be fine.
And then enable rps/xps.
I made this simple quick and dirty script:

#!/bin/sh

interfaces="eth0 eth1"
num_queues=7 # Queues start at 0
cpu_mask=3

for interface in $interfaces
do
	queue=0
	
	while [ $queue -le $num_queues ]
	do
		echo $cpu_mask > /sys/class/net/$interface/queues/rx-$queue/rps_cpus
		echo $cpu_mask > /sys/class/net/$interface/queues/tx-$queue/xps_cpus
		
		queue=`expr $queue + 1`
	done
done

And I think, having more queues then CPU's only make sense
if you actually use the queues to assign traffic classes to it.
Like I wrote above, having 4 queues for high prio, 3 for norma and 1 for bulk.
Because then there is a higher chance that the high prio traffic gets dequeued first.
WRT devices only have 2 CPU's that can process interrupts.
So having only 2 queues should be fine too?

For anyone working on a migration script to DSA, this might help:

It would need to be extended of course.

Hey everyone! I'm trying to make a transition script from the swconfig to DSA but I need some info first:

  • How does the script know the file has already been transitioned?
  • can someone with swconfig give me the link between the port I'd on swconfig and the real port?
    As soon as I get this info I'm going to try to make a script.

Thanks!

You can check the configuration if it does contains swconfig
On swconfig you have eth0 eth1
On DSA you have lan1 lan2 wan