AQL and the ath10k is *lovely*

FWIW,

I get similar results independent of ct firmware. The results above are with
firmware-5-ct-full-htt-mgt-community-12.bin-lede.019
which is the latest in master.

I just tested again with
firmware-5-ct-htt-mgt-community-qcache.bin
dated 08/27/2020 (from here - I believe a beta qcache (re)enabled firmware) and get similar results so i don't think this is related to the "qcache" related observations reported on @greearb's github site.

EDIT: my wifi chipset is "9980" i.e. not the r7800...

EDIT 1: after about 6 hours up, the beta "qcache" firmware crashed. Moving on...

Tbh WiFi feels faster and latency no longer fluctuates erratically on fast.com test, haven't done extensive testing so

Have you tried the same tests on a non-CT build?

no

however, I was just waiting for that question... :grinning:

If you can suggest a non-ct firmware for the 9980 (newer than the kavalo one here) I could use with a recent ath10k driver with openwrt, please let me know.

Note:
the kavalo firmware for the 9980 is 5 years old - last time I tried I had no end of issues (but there might have been other contributing factors)

I've "binwalked" 3 recent stock firmwares and the only folder that has something ath10k firmware like is
/lib/firmware/AR900B/hw.2
which looks like:

athwlan.bin
athwlan.codeswap.bin
boarddata_0.bin
boarddata_1.bin
boardData_AR900B_CUS238_5GMipiHigh_v2_004.bin
... (bunch more of "boardData_* files)
otp.bin
utf.bin
utf.codeswap.bin
waltest.codeswap.bin

I'm seem to recall asking about this several years ago and I suspect I'd have to "stitch" some of the .bin's together to get a working firmware. Given my lack of knowledge about the firmware, this likely would be a lengthy trial an error process that may yield no working firmware.

BEGIN EDIT
OT but for anyone else who might find this post later...
I had to use the "wayback machine" to call up

https://wireless.kernel.org/en/users/Drivers/ath10k/firmware

which has this quote:

Firmware API 2

Embedding both firmware and otp images into same file firmware-2.bin. Firmware meta data provided through FW IE. Added in commit 1a222435a dated Sep 27 2013, for Linux 3.13.

END EDIT

Lastly, I did see a comment on the ddwrt forums about an updated firmware for the 9980 as recent as a few months ago - I haven't checked that yet mostly because I'd rather stick with firmware for which I can get support.

I'm not familiar with the 9980, personally. But do you have the option to just select Kernel modules > Wireless Drivers > kmod-ath10k along with Firmware > ath10k-firmware-qca99x0 (again, the non-CT variant) for your build? Since the issue I posted in the CT firmware GitHub, I have moved back to the non-CT ath10k and have been having great results with it at the present time (running master snapshot builds).

1 Like

Apologies for the long conversation but this issue is starting to be a problem for me and I appreciate the opportunity talk about it.

I'll look again but last time I checked that uses the "kavalo" firmware. As I have not tried it in some time, I can try it to see if it will work long enough to for me to see a difference.

Even if it works, I think it's possible the "issue" can originate outside the ath10k-ct driver/firmware (i.e. the issue might originate from code in mac80211 that only presents when it tries to use air time fairness with a firmware that supports it - like ct).

Running a non ct driver and firmware, do you see an output if you:

cat /sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/*/airtime
cat /sys/kernel/debug/ieee80211/phy1/netdev\:wlan1/stations/*/airtime

? ref. here

I do see that output because AQL is implemented in the mac80211 stack regardless of the firmware, right? (I could be wrong about that...)

root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/*/airtime
RX: 0 us
TX: 75713662 us
Weight: 256
Deficit: VO: 5 us VI: -215 us BE: 213 us BK: 127 us
RX: 0 us
TX: 21855523 us
Weight: 256
Deficit: VO: -1457 us VI: 256 us BE: -22 us BK: -92 us
RX: 0 us
TX: 1402354 us
Weight: 256
Deficit: VO: 134 us VI: -354 us BE: -63 us BK: -727 us
RX: 0 us
TX: 8775706 us
Weight: 256
Deficit: VO: 92 us VI: -55 us BE: -229 us BK: -90 us
RX: 0 us
TX: 65495711 us
Weight: 256
Deficit: VO: -15 us VI: 59 us BE: -345 us BK: -370 us
RX: 0 us
TX: 24782803 us
Weight: 256
Deficit: VO: -160 us VI: 129 us BE: -996 us BK: 48 us
RX: 0 us
TX: 62467683 us
Weight: 256
Deficit: VO: -18 us VI: 118 us BE: -63 us BK: 120 us
RX: 0 us
TX: 2454780 us
Weight: 256
Deficit: VO: 121 us VI: -58 us BE: 128 us BK: 69 us
RX: 0 us
TX: 247745338 us
Weight: 256
Deficit: VO: -202 us VI: -38 us BE: -354 us BK: -488 us
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy1/netdev\:wlan1/stations/*/airtime
RX: 0 us
TX: 1167 us
Weight: 256
Deficit: VO: -48 us VI: 256 us BE: -95 us BK: 256 us
1 Like

thanks for that.

Based on prior posts (most of which in this thread above) I got the impression support is needed in both the ath10k(-ct) driver/firmware and mac80211.

After a brief scan of the initial commit, most patches are to files in net/mac20811. There is only one line added to drivers/net/wireless/ath/ath10k/mac.c

+	wiphy_ext_feature_set(ar->hw->wiphy, NL80211_EXT_FEATURE_AQL);

which i think is the clue i needed to "disable AQL" for a test.

The firmware is a black box so I have no clue if it has/needs functionality to support AQL.

I have a few things to try and play with just not enough time to test do to "remote learning" starting up next week for my children. I may have to revert to something stable after that if this continues.

FWIW it looks like ath10k-ct 5.8 is comming (mailing list here) but if interpret @greearb's comment

If you are trying the 5.8 version, then it is almost completely untested and may be full of bugs.

then "master" may continue to be buggy for a while yet.

1 Like

I think you are indeed correct about -ct firmware + driver being necessary to support AQL.

I switched back to the -ct firmware + driver on my build. But I did some testing I had not done before and it appears the -ct-htt variant of the firmware is the one causing me issues at the moment. When I switch away from the -ct-htt variant to just the -ct variant, my odd latency issues that I was experiencing before seem to go away. I had always been using the -ct-htt variant prior to now.

I'll run with the -ct variant for a couple days and see how things look, but thankfully you got me thinking on this again. :slight_smile:

3 Likes

that's a great tip for me... I've always used the htt variant. I'll try the non htt and see if I get an improvement.

Based on what I've read, I think AQL should be functional with non ct driver/firmware (at least for the r7800), but I'm not the right person to confirm that.

EDIT: I just loaded the non htt variant and re-tested (like I have done above) and I still see the same undesirable (AQL) behaviour. irtt results seem little changed. I may have made some progress finding a non ct firmware to try but won't be able to test that for a bit. Regardless, I really appreciate the tip.

1 Like

Yeah, something just doesn’t feel right with the -ct firmware right now. I take a substantial throughput hit over the standard ath10k, even with the non HTT variant. I’ll probably be switching back to my non-CT build in a little while.

Sorry this non-HTT route didn’t work out better for the both of us :frowning:

So I built and loaded the ath10k driver commenting out the line
wiphy_ext_feature_set(ar->hw->wiphy, NL80211_EXT_FEATURE_AQL);
in mac.c

Running with that, there are no aql files in
/sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/*/aql when stations are connected to phy0; however,
cat /sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/*/airtime
still returns an output.

Repeating the tests above "sans aql" gives similar results so it seems my testing/observations are more airtime fairness related than aql.

What makes a bigger difference is client/AP location (AQL enabled in the ath10k driver this case). Airtime fairness seems to behave more like I'd expect when all clients have a line of sight to the AP. i.e. the faster client stay fast while slower clients take a hit. Even then, I'm still surprised at how much of a bandwidth hit the slow clients take when I start simultaneous netperf sessions from 2-3 clients. It still seems like some clients come to a complete stop briefly. This may happen more frequently with increased router uptime - I'll need to test over an extended period.

As an aside, most testing I've done is with 80 MHz width channels on the 5 GHz band (channel 36). I tried with 40 MHz width and generally see similar behavior just not as pronounced.

Lastly, I'd really like to adjust the airtime station weights with a command like

r7500v2 # iw dev wlan0 station set <MAC address> airtime_weight 26
command failed: Not supported (-95)

but I only get the output above. Perhaps this hostapd commit will enable that (but I'm likely just misunderstanding).

There hasn't been much video conferencing going on in my household since the lastest mac80211 updates so I'll need to give this some time to see if it's really improved now or not. That and I'd still like to try a non-ct firmware just to see if makes a difference.

If you'd like to compile ath10k without airtime fairness at all try commenting out the following:

NL80211_EXT_FEATURE_AIRTIME_FAIRNESS

I think it's lines 9260-9263 in mac.c in the ath10k driver for Linux/Backports version 5.8.

Thanks, I probably will. I assume doing so returns the airtime scheduler to what it was before (round robin) vs the new thing (virtual time-based).

It looks like there is a hostap-ct that has some form of "airtime config" option. I haven't looked into that yet - porting a package over to openwrt likely would take some effort.

It will completely disable the airtime fairness scheduler. The virtual time-based solution was never adopted and merged.

You will get this:

Rather than this:

Development of the virtual time solution stopped at v5 of the patch as seen here: https://patchwork.kernel.org/patch/11307545/

Looking into the author's history it was also the last thing they've submitted: https://patchwork.kernel.org/project/linux-wireless/list/?submitter=184429&state=*&archive=both

2 Likes

Was it effectively abandoned? Or will it be in the OpenWrt kernel for the 2020 release?

1 Like

hmm, I'm missing something...

after

        /* if (ath10k_peer_stats_enabled(ar) || */                                
        /*     test_bit(WMI_SERVICE_REPORT_AIRTIME, ar->wmi.svc_map)) */          
        /*      wiphy_ext_feature_set(ar->hw->wiphy, */                           
        /*                            NL80211_EXT_FEATURE_AIRTIME_FAIRNESS); */   

in ath10k-5.4/mac.c, compiling, moving the resulting ath10k_core.ko
to /lib/modules/5.4.63/ath10k_core.ko and rebooting I still see an output from
cat /sys/kernel/debug/ieee80211/phy0/netdev\:wlan0/stations/*/airtime
and I can still reproduce the symptoms above.

I think I need to start a new thread specific to my device as this does not seem "AQL" related. Also I may have made some progress finding more up to date non-ct ath10k firmware to try (but I'll likely have to use the ath10k-ct driver).

2 Likes

it looks like your search ended in 2019. It looks like work has continued into 2020 https://lore.kernel.org/linux-wireless/?q=Toke+Høiland-Jørgensen

Has anyone else observed this same issue (https://lists.bufferbloat.net/pipermail/make-wifi-fast/2020-September/002933.html) where Airtime Fairness doesn't play well with AQL?

2 Likes

I note that I kind of dropped out of openwrt dev for the past year. What's the status of the AQL patches in mainline presently? And this patch?

1 Like