According to the original paper (https://www.usenix.org/system/files/conference/atc17/atc17-hoiland-jorgensen.pdf), it seems that the validation of the ATF scheduler implementation was done with the ath9k driver instead of the ath10k driver. Most of the reported issues so far seem to be related to ath10k. If that's the case, the peculiarities in the behavior of the ath10k driver/firmware may dictate more extensive modifications in the ATF implementation in order to accommodate them.
"We have implemented our proposed queueing scheme in
the Linux kernel, modifying the mac80211 subsystem to
include the queueing structure itself, and modifying the
ath9k and ath10k drivers for Qualcomm Atheros 802.11n
and 802.11ac chipsets to use the new queueing structure.
The airtime fairness scheduler implementation is limited
to the ath9k driver, as the ath10k driver lacks the required
scheduling hooks."
According to Felix' latest patch (after his numerous communications with Toke):
The virtual time scheduler code has a number of issues:
10 - queues slowed down by hardware/firmware powersave handling were not properly handled.
12 - on ath10k in push-pull mode, tx queues that the driver tries to pull from were starved, causing excessive latency
14 - delay between tx enqueue and reported airtime use were causing excessively bursty tx behavior
17 The bursty behavior may also be present on the round-robin scheduler, but there it is much easier to fix without introducing additional regressions
20 Signed-off-by: Felix Fietkau <nbd@nbd.name>
@sjpacket, do the patches make any difference in your test compared to leaving them out without any other changes to the tree? Does it make a difference if you only use the first patch (330-mac80211-switch-airtime-fairness-back-to-deficit-rou.patch)?
I haven't tested with applying one patch at a time. Will try that this week.
What I have noticed is that I do not see a crash when I switched to ath10k mainline. I see a crash a soon as I switch to ath10k-ct.
Results:
openwrt 22.03 + ath10k-ct + no patches = crash (OOM)
openwrt 22.03 + ath10k-ct + latest patches = crash (NULL Pointer deference)
openwrt 22.03 + ath10k + no patches = no crash
openwrt 22.03 + ath10k + latest patches = no crash (I see a quick dip around 8 mins or so but throughput recovers fine)
There are a lot of variables and patches to test and I also need to make sure I don't have any issues in my test setup.
Thanks, @sjpacket. The NULL pointer dereference crash that you pointed out also seems to be caused by OOM. It happens in a completely unrelated place, so I don't think it's directly related to my patch, except for the fact that my patch might slightly change memory usage patterns.
How is latency/throughput with my patch on mainline ath10k?
Hi @vochong,
regarding ATF and AQL: AQL is essential for making ATF work well. If AQL significantly reduces throughput, then that's an important issue which needs to be fixed. Please let me know how well the current version with my patches works for you with AQL and ATF enabled.
When using iperf3 for your tests, are you streaming data from the AP out to the clients, vise versa, or trying both? Both ways can be interesting, but it's my understanding that AQL/ATF provides the most benefit (it's used the most?) for streaming data out from the AP to the clients.
where is the iperf3 server(s) running (i.e. AP, client via wire to AP, etc)? If your running your iperf3 servers on the AP, please keep an eye on your AP cpu usage when observing throughput. I don't recall ever observing throughput drops when other clients connect with only one client streaming. I'll see if i can reproduce that on my device.
When using the latest set of patches, is ATF disabled in the ath10k-ct driver (this is more a question for @nbd i.e. should it be)?
You reported your device as "IPQ8064, QCA9984, QCA9980", given the rates you can get with your wifi clients, I'm assuming your testing on 5GHz and that 5GHz is (only?) the QCA9984 - is that correct? I ask as my ipq8064 device has QCA9980 for 2.4 and 5GHz.
I've built images but I can't get them on my AP to test quickly due to it being in use by family most of the time.
When I do multi client tests, I prefer to use a netperf server running on a client wired to the AP. A single netperf server will let you stream data both directions (server to clients, clients to server). The down side of netperf is there does not seem to be a client for windows (I think running it from wsl will work tho).
Given your reported wifi rates above, a single netperf server might saturate even a 1 gbps wired connection with 3+ clients. Also even if the iperf/netperf server(s) are on a separate device, keeping an eye on your AP cpu is a good idea if testing with 3+ clients.
In the past I've looked into the flent rtt_fair test (mentioned by dtaht above), but I'm uncertain it would show much when connecting to multiple servers outside your local network if you have a slow isp connection (like mine). Perhaps there is a way to setup ones own local servers for this test? (Again, not a question for you.)
Thank you for testing without the apple devices. I suggested doing that as its possible there are other issues with the ath10k-ct driver/firmware that results in latency symptoms. Crashing due to OOM is not one of those symptoms tho.
As soon as I get an opportunity to test, I'll report back. Likely at least a couple of days to do that.
If I understand this correctly, the server instances really are machines (symbolic dns names or numeric IP addresses) which have netserver running and a firewall that allows accessing netperf from the outside. So you could simply distribute multiple servers in your internal network? The bigger challenge is that most local nodes will have similar RTTs but the first question about RTT fairness is 'are rates roughly equal at equal RTT', so this might be a decent test bed.
What I am hoping for with more detailed testing is to see a detailed pattern of stalls that lead to an AHA! moment. I fear aircaps will also be needed. While we are establishing some groundwork for the crashes, what form of WPA is on or off for these tests?
(again, we have other problems, but in the "working" non -ct code, do you see a reasonable pattern of codel marks or drops?) This is what a single tcp upload looks like when I've locked the wifi to 6Mbits, using tcptrace -G and xplot.org on the resulting packet captures. Locking the wifi to a slow rate allows for testing of the wifi vs a vs an external server over a faster ISP.
A nice consistent set of drops and sack responses. I tend to ALSO use ecn as a debugging tool (is codel working?), sysctl -w net.ipv4.tcp_ecn=1 on both the server and client, and that ends up looking like this:
You can actually capture the CE marks if your capture point is after the router, but merely seeing CWR and no sacks is usually indicator enough. These plots are from a known working ath9k laptop to ath10k based turris, running an older version of openwrt.
Hi Felix @nbd, thank you for your work in fixing AQL and also maintaining mt76 driver. I know this thread is mainly about ath10k, but some people are reporting connectivity issues on mt76 driver that are supposedly fixed by disabling AQL as mentioned at Belkin RT3200/Linksys E8450 WiFi AX discussion - #2359 by jj86 .
Does mt76 use AQL and does it directly do ATF like ath9k? Does the Virtual TIme-Based AQL (that is yet to be reverted in OpenWrt 22.03) affect mt76 like it affects ath10k?
I am using a Belkin RT3200 (mt7622, mt7915) with 5GHz 802.11ax AP mode and sometimes I have an issue with my Google Pixel 6 phone that occurs randomly (and nothing related in openwrt system log), wherein wifi remains connected but no data loads in any browser or app. Toggling wifi off/on on the phone bring back connectivity. Not sure if that is related to AQL.
I can offer a step back re ath10k vs -ct firmware - with every release update of 21.02 (.1, .2, .3 ...) I've updated my Archer A7-us router, and every time had to uninstall ath10k-ct firmware and kmod-ct and use ath10k (non -ct flavors) -- due to blatant network stuttering.
Sorry no further details on that stuttering are available, except that the A7 had 4 wifi clients (2 5GHz, 2 2.4GHz) and was a simple home accesspoint network; and at that when I'd hit stuttering my first thought was ... 'yes, just updated to the next release of openwrt'.
I've since replaced the A7 with a Netgear Nighthawk X4S R7800 -- with some of the same issues and solution.
Suggestion: stick with -non -ct ath10k bits when digging into these bugs (plural use deliberate) or the multiple use case scenarios may be unworkable. I'm not trying to throw rocks here ... just that I've had issues with -ct on multiple devices over a significant span of time.
In Addition to reverting to ath10k firmware/kmod my R7800 runs much better with 21.02.1 rather than the 21.02.3 I initially tried. round robin.
Hi @ka2107,
mt76 uses AQL, but I suspect if disabling it fixes some issues, it's only hiding the real bug. Not sure how much the VTBA scheduler affects it other than producing very bursty tx behavior.
I'm also looking into a driver regression related to wifi-6 that seem to result in tx stalls in my tests. That one is unrelated to AQL or ATF though.
Regarding "wifi remains connected but no data loads in any browser or app".
Did you notice if someone might just exit the house (with their phone) prior to encountering such problem? I had this same problem as the bug described as follows. Please note that the AQL disabling workaround only helped reduce the frequency of the problem. It did not fully get rid of it.
Device: Netgear R7800 (IPQ8065, QCA9984)
WiFi driver + firmware: mainline (not CT)
Band: 5 GHz
When some devices such as phones leave the network, other devices left on the wifi network experience high latency and heavy packet loss for a few minutes.
This started happening after the update to 21.02.2, it was very stable before.
Others are reporting the same issue, see this thread:
https://forum.openwrt.org/t/ipq806x-nss-drivers/12613/2557
Workaround that appears to help:
echo 0 > /sys/kernel/debug/ieee80211/phy0/aql_enable
echo 0 > /sys/kernel/debug/ieee80211/phy1/aql_enable
Potentially related:
https://lists.infradead.org/pipermail/ath10k/2022-February/013341.html
I suspect it is some of the backports in hostapd/mac80211 that were introduced between 21.02.1 and 21.02.2
Not sure whether it affects only ath10k or other devices also.
+1 I have been banging on about this for some time.
I have seen many times iPhone connected to WiFi but no data. It reminds me of the mesh issues where reported Wi-Fi connection ul/dl bw would go down very low and the mesh node would become unresponsive even when it was shown as being still connected.
Whatever the cause I am hopeful that this will get resolved somehow to help improve the WAF with me using OpenWrt + RT3200. Didn't happen with broadcom-based Asus router.
Can you please post your image "openwrt 22.03 + ath10k + latest patches" somewhere so other people and I can use the same image to report any observations/issues to Felix. It's better to use the same image so we can prevent any other potential differences from affecting our observations. I think the tasks for us to validate Felix's fixes is to confirm whether the following main issues are still existing. Consistency in our observations is the key in helping Felix nail down the right culprit.
Disassociated WIFI clients (someone with phone leaving the house) don't cause very high latency or huge packet losses on currently associated clients (this is very easily observable during online meetings or games)
Multiple associated clients do not negatively the throughput.
No excessive memory consumption or crash over a few days.
Since this issue occurs randomly, I do not have a reliable way to test it. If the issue occurs again with this snapshot build, I will report here.
I look forward to seeing the mt76 updates in OpenWrt 22.03 branch as well so that the next RC or stable release contains the fixes. The only reason I am using MASTER SNAPSHOT instead of 22.03-SNAPSHOT or 22.03-RC build is because of updated mt76 version.
EDIT: I am hoping that this issue is fixed in ath10k as well as I have a R7800 (currently running 22.03-rc4 with ath10k non-ct firmware and ath10k-smallbuffers non-ct driver) being used in a different location. I can only connect to the R7800 from the WAN side and therefore cannot test the R7800 WiFi myself. The users at that location are not familiar with OpenWrt but they will let me know if they face any connectivity / lag issues with R7800's WiFi. However they haven't reported any issues till now.
and I also need to make sure I don't have any issues in my test setup.
