The 1 CPU port thing is a bit misleading. Usually DSA drivers are faster as they sometimes offer some hardware acceleration. They also sit in kernel space whereas swconfig (I believe) is in user space.
So the slowdown isn't 2x in most cases.
Then there are devices with only 1 CPU port (my Archer C7v4 for example) where this is not an issue. Not that there's DSA for it yet... Although it would be a good idea to use qca8k for it when it gets ported to ath79.
That is all good, but could be getting off topic. Either way, DSA is a long term solution, but the problem exists now. As far as I am concerned, the latency problem is mostly solved: I am running a custom firmware that is very fast and also eliminates most latency spikes over wifi. The latency over wired is even better. It does take a number of patches/hacks, but it can be done now without having to wait. And it feels almost as fast as the stock.
The same here. I am also seeing great improvements to the ping latency over wifi.
Agree, and the first step could to have everyone interested to add comments to the PRs explaining the benefits of merging them. The more people are interested and benefited, the more likely they will be merged.
Here is a list of changes that I think makes this router great again.
Disable Export mac80211 internals in DebugFS followed by disabling DebugFS in kernel. This seems to have a positive impact on the wifi pings, but I do not know what it could have broken An independent verification is required. https://gist.github.com/fantom-x/78564320f785b977b6058f9c08ece49e
option cache size 50000 for DHCP as the default is 150 only: this thing has a lot of RAM to handle more entries.
If using SQM, then only move eth0 to CPU1. SQM is very CPU intensive (much more than the wifi interface) so let's let SQM use it all. I leave eth1, wifi0, and wifi1 on CPU0 for this reason.
Use performance scaling governor for both cores: helps with wifi latency too. In the absence of any hardware acceleration and considering that the cpu is taking 100us to switch frequencies, this is a good stop-gap solution.
as a follow up i've been running with the low-latency kernel for a couple of days now and it has spontaneously rebooted on me twice. the first time it was just master with the MIB counter fix and the low-latency kernel, this time r6628 with PR#669 and PR#632, the MIB counterfix and the low-latency kernel. i suspect the low-latency kernel as with a sample size of 2 it is one common factor. i'm going to try the same but with the default pre-emption model.
By low-latency kernel do you mean the "Preemptible Kernel (Low-Latency Desktop)"? If yes, then I would recommend switching to Voluntary Kernel Preemption (Desktop) since it appears to be stable.
Now that the latency over a wired link is more less understood, @nbd can you suggest where/how to start debugging the 300+ms spikes over a wireless connection?
I run this command while the router is either not in use or used lightly.
ping -c 9001 -i 0.2 8.8.8.8 | tee 8.8.8.8.wireless
The chart above is with a pre-emtable kernel and MIB counters disabled. With the default one it used to be much worse, but I will test again today with MIB counter disabled and with the commit you recommended.
Yes, from my laptop. I usually run two concurrent sessions: one over a wired connection and one over wireless. The wired ping chart is below and taken over the same 30 minute interval. Notice, they both were run at the same time. I get similar results if I ping the first several hops. I tried both tests with the stock and they were both much better.
@nbd, here is another latency chart over wifi with MIB Counters disabled: the spikes are reaching 300ms. The difference between this one and the one a few posts above that that this time I have not disabled Export mac80211 internals in DebugFS and Kernel Debug FS. A concurrent wired ping did not have these issues and topped up at <20ms a few times only.
In both cases a preemptive kernel is used, because it is much worse with the default one.
What is using the information from Export mac80211 internals in DebugFS ? Can it be safely and permanenlty disabled?
UPDATE: the wifi client is within 5..10 ft from the router and is using 5GHz band.
Let's take this step by step. First of all, did my changes resolve the switch related latency on wired connections? Or does that issue remain and wired latency still needs the preempt change?
Fair enough. Do you want me to test just your change without actually disabling the mib counters?
To be safe, I compile the firmware from scratch after every change and it takes a while. Is running make reliable enough to pick up any config and code changes if I keep reusing the same workspace?
Yes, please test my change without disabling MIB counters. Just make should be enough.
If you really want to make sure all changes are applied properly, you can run make target/linux/clean before running make again (the same will happen anyway when kernel changes are being picked up).