How OpenWrt Vanquishes Bufferbloat

[richb-hanover-priv]

The OpenWrt Project is a Linux operating system targeting embedded devices. A common use of OpenWrt is to replace the manufacturer's firmware on a commercial, off-the-shelf router to produce a device with performance, features and security not available from the original vendor.

In addition, OpenWrt has been used as a research platform to investigate (and solve) the problem of bufferbloat: the undesirable latency that comes from a router or other network equipment buffering too much data. The first real solution came in 2012 with the introduction of the fq_codel queue discipline that made a "no knobs" way to eliminate bufferbloat. The CAKE algorithm (created in 2015, merged into OpenWrt in 2018) provided even better performance. These techniques have since been applied to Wi-Fi to decrease buffering delays by an order of magnitude (typically, down to 10-100 msec) from earlier drivers.

Bufferbloat occurs at a bottleneck — when fast links hit a slower link. This happens in a home router when Ethernet traffic reaches the slower ISP link, or when a fast laptop hits a slower Wi-Fi link. A router will queue up the data, to be doled out as it can. In a bad router, all the traffic goes into one queue, imposing a delay on the big flow as well as all other traffic. This is bufferbloat, and can reach several seconds, making the entire network "feel slow".

About Smart Queue Management (SQM)

OpenWrt minimizes bufferbloat using either the fq_codel or CAKE "Smart Queue Management" algorithms. These introduce a bottleneck queue within the router itself, generally, about 5% slower than the ISP link. This means that the traffic builds up in the SQM queues where they can be controlled.

Both algorithms work by separating all traffic flows into their own queues: this isolates low traffic flows from the high volume flows. SQM then round-robins between the queues, sequentially removing a "fair" portion of data from each queue and transmitting it. A large flow can't monopolize the bottleneck link, but will use the full capacity when there's no other traffic.

On ISP Connections

OpenWrt supports SQM to control all the traffic flowing through the ISP bottleneck connection to provide fairness to all flows, and minimize the latency for all packets sent through the link.

• What devices support this? All modern versions of OpenWrt support both CAKE and fq_codel from the Network -> SQM QoS settings. (You may need to install luci-app-sqm first.)

• How do I configure it? You'll need to enter download and upload settings into SQM. Enter values that are about 5% slower than your ISP's claimed speed. For more details, see SQM (Smart Queue Management) on the OpenWrt wiki.

On Wireless Connections

Since OpenWrt is an access point, it can manage all the traffic flowing toward the many devices ("stations") that it serves. The OpenWrt Wi-Fi driver uses the following features to reduce latency:

1. Individual transmit queues for each wireless station/client. Each queue uses fq_codel to control the amount of data queued for that station.

2. Airtime Fairness (ATF) to give each wireless client a fair share of the airtime as opposed to the number of bytes sent/received with ordinary Wi-Fi drivers. This ensures a low-rate (slow) station does not use up all the airtime (the so-called "WiFi anomaly"). ATF uses round-robin scheduling of all active stations to ensure they each use only their fair share of the available airtime.

3. Airtime Queue Limits (AQL) to prevent the chip hardware from queueing too much data for any one wireless station. Many wireless chipsets have significant (unnecessary) queueing. By default, AQL analyzes the actual bit rate for that station and ensures that no more than 12 msec of data is queued for it.

Modern OpenWrt supports the Wi-Fi driver for certain chipsets.

• What devices support this? All MediaTek MT76xx and MT79xx chipsets support the three features. We are working to crowd-source Wi-Fi chip info information about Qualcomm Atheros chip sets.

• How do you configure this? The default settings for the Wi-Fi drivers work well for virtually all situations. For more information about configuration, see this note.

On Variable Speed Connections

Many ISP data rates vary from hour to hour (or even second by second), for example, cable modems, Starlink, 4G and 5G cellular plans, etc. This is a problem for the CAKE algorithm because it uses fixed speed settings to control the buffering. This leads to sub-optimal settings: if you set the speed too high and the actual link speed drops, then the latency goes up. If you set it too low, CAKE artificially limits the potential transfer rate.

The cake-autorate algorithm addresses this. It continually measures the traffic load and latency to adjust the upload and download settings of the CAKE algorithm. See the README for more details.

• What devices support this? All modern OpenWrt devices can install the cake-autorate software.

• How do you configure this? The INSTALLATION.md instructions at the Github repo describe the process.

[richb-hanover-priv]

+1 for @zekica for summarizing well. With all the other comments, the article practically writes itself.

@zekica

• Linux mac80211 wifi stack, used by ath9k, ath10k, ath11k, ath12k?, mt76, brcmsmac, rt2x00, rtl8xxxu, rtw88, rtw89 and possibly others was reworked to support ... all those good things listed in his list...

One outstanding question: How would a newcomer know which chip sets to favor?

Specifically, I see the WLAN Hardware entry in the Table of Hardware. Is there a "secret decoder ring" that helps me go from the following entries to the list above? Many thanks!

Router	WLAN Hardware entry
WNDR3800	Atheros AR9220, Atheros AR9223
Archer C7	Qualcomm Atheros QCA9558, Qualcomm Atheros QCA9880-BR4A
Linksys WRT3200ACM	Marvell 88W8964
Netgear R7800	Qualcomm Atheros QCA9984
Linksys MX4200	Qualcomm QCN5024, 2x Qualcomm QCN5054
Xiaomi Redmi AX6000	MediaTek MT7976GN, MediaTek MT7976AN
GL.iNet GL-MT6000	2.4GHz: MT7976GN 4T4R, 5GHz: MT7976AN 4T4R
etc.	etc...

[zekica]

Here is a non-comprehensive list, linux wireless wiki also has a mostly complete list:

ath5k = atheros 802.11a/b/g SDIO/PCI
ath9k = qualcomm/atheros 802.11n PCI
ath9k_htc = atheros 802.11n USB
ath10k = qualcomm/atheros 802.11ac PCI/SDIO
ath11k = qualcomm/atheros 802.11ax PCI
ath12k = qualcomm/atheros 802.11be PCI

mt76 = mediatek 802.11n/ac/ax/be SoC/PCI/USB except mt7601u usb and mt7620 soc
rt2x00 = ralink/mediatek 802.11a/g/n (only mediatek chip supported by this driver is mt7620, is not very stable and has issues with external power amplifier used in some devices)

brcmsmac = some Broadcom 802.11n/ac that use SoftMAC - see the wiki linked

rtl8xxxxu = an attempt to write a single driver supporting a range of Realtek 8xxx USB 802.11n/ac/ax chips and is the most modern driver - all other Realtek USB drivers AFAIK don't use cfg80211 or were not modernized to include all the features listed here.
rtw88 = 802.11ac PCI driver for Realtek chips
rtw89 = 802.11ax PCI driver for Realtek chips

[slh]

Let's face it, the only relevant wifi chipsets in 2024 are:

• mt76
• ath11k
• with some caveats (as in 'only' 802.11ac) ath10k
  (there's little bad to say here, but you're not going to buy new devices for it, even less if you are keen perfect performance)

Even if they had perfect bufferbloat support, 802.11g (ath5k, ar5523), 802.11n (ath9k/ ath9k_htc/ carl9270, rt2x00) really don't have any place in terms of decent performance anymore. Bufferbloat may be an important aspect, but so is basic throughput as well - and with that in mind, anything below 802.11ac doesn't meet the bar (and yes, I do still own everything from 802.11b and upwards, for a simple client with modest needs that may be fine, for your main AP and an expectation of smooth performance, no, not at all).

mwlwifi/ brcmfmac are each plagued by their own issues, and -as fullmac devices- won't ever give you driver side abilities to tune bufferbloat, the binary firmware is what it is.

The various Realtek chipsets have barely any support in OpenWrt at all. While rtw88/ rtw89 is getting vendor support for modern chipsets and rtl8xxxu quite some community support for older chipsets, I doubt anyone has tested it on OpenWrt (with AP mode) in anger at all.

Forget about brcmsmac, Broadcom just threw that stuff over the fence to kill off b43 and immediately forgot about it, it doesn't support AP mode at all.

We don't have any ath12k devices so far either, it will become relevant with ipq957x and ipq53xx some time in the future, but expect a double digit amount of months (my guess would be at least two years, celebrations if it's only going to be one year, but I wouldn't wonder if it's going to be 3+ years).

[richb-hanover-priv]

More about Wireless: From the replies so far, I am beginning to piece together the information needed to give guidance so someone can tell which routers have wireless chips that support all the Wi-Fi latency fixes that are available ("the OpenWrt Wi-Fi Goodness", or OWFG )

Here's my summary so far - please forgive any misunderstandings.

1. The "linux mac80211 wifi stack" appears to support all the Wi-Fi fixes. Is this the same thing as described in the Wireless Wiki at kernel.org?

2. As a practical matter, it only makes sense to choose routers/Wi-Fi that supports 802.11ac or better (802.11ax and 802.11be, and ...?) Although the OWGF may be available for earlier chipsets, they're too slow to meet today's expectations.

3. From How OpenWrt Vanquishes Bufferbloat - #7 by zekica and How OpenWrt Vanquishes Bufferbloat - #10 by slh it looks as if these chip sets (families of chip sets?) qualify:

   • mt76 = mediatek 802.11n/ac/ax/be SoC/PCI/USB
     • except mt7601u usb and mt7620 soc
   • ath11k = qualcomm/atheros 802.11ax PCI
   • ath12k = qualcomm/atheros 802.11be PCI (but no devices so far)

4. I see the Device Pages from the Table of Hardware have a "WLAN Hardware" entry that lists the actual chip part numbers for the radios. I'm still not clear how to convert, say, "MT7976AN 4T4R" or "Qualcomm QCN5054" to one of the families above that provides OWFG.

PS I do not intend to use the term OWFG anywhere else except in the discussion in this thread...

[richb-hanover-priv]

@zekica @tohojo @slh @dlakelan @lynx @dtaht - this note is now good enough to criticize.

I'm fairly confident of my descriptions of Bufferbloat, and the mitigations from fq_codel/CAKE. But the Wi-Fi description feels pretty shaky.

And I still don't have a clue about how to use a device's WLAN Hardware entry in the Table of Hardware to know if it has the Wi-Fi fixes present.

Thanks in advance for any comments you can provide.

[slh]

@richb-hanover-priv

1. The "linux mac80211 wifi stack" appears to support all the Wi-Fi fixes. Is this the same thing as described in the Wireless Wiki at kernel.org?

Yes.
We can provide a more nuanced answer here (as in, what OpenWrt calls mac80211 is a backport (suite) of the mac80211 stack and its drivers to older kernels, in case of 23.05.x kernel v6.1 based kernel wireless drivers for the v5.15 kernel used by OpenWrt - and the first link conflates mac80211/ cfg80211 based drivers with legacy ones), but those are details - and none of that matters for the contemporary chipsets/ drivers you can (should, with OpenWrt in mind) buy today.

@richb-hanover-priv

2. As a practical matter, it only makes sense to choose routers/Wi-Fi that supports 802.11ac or better (802.11ax and 802.11be, and ...?) Although the OWGF may be available for earlier chipsets, they're too slow to meet today's expectations.

Yes, a good experience is only the combination of acceptable throughputs and decent latencies.
Right now, we don't have any supported wifi7/ 802.11be routers yet (filogic 880 and corresponding wireless cards for OpenWrt aren't that far away though; ipq53xx/ ipq957x is already on the market, but its path towards OpenWrt will be much longer).
802.11ac devices are fine if you already own them or can get them for a bargain on the used markets, there's no reason at all to replace them - but at the same time it does not really make sense to buy them (new) anymore either, as we have a solid OpenWrt supported selection of mt7621+mt7915DBDC, mt7622bv+mt7915, filogic 820/830 and ipq807x (and hopefully soon ipq60xx) devices for reasonable prices (often less than their wifi5/ 802.11ac predecessors) on the market. Yes, the more modern devices might need snapshot builds for now and may be harder to flash, but you get better hardware in return for your efforts.

@richb-hanover-priv

3. From How OpenWrt Vanquishes Bufferbloat - #7 by zekica and How OpenWrt Vanquishes Bufferbloat - #10 by slh it looks as if these chip sets (families of chip sets?) qualify:

• mt76 = mediatek 802.11n/ac/ax/be SoC/PCI/USB
  • except mt7601u usb and mt7620 soc
• ath11k = qualcomm/atheros 802.11ax PCI
• ath12k = qualcomm/atheros 802.11be PCI (but no devices so far)

Yes. mt76 might be slightly easier to profit from work on this.

@richb-hanover-priv

4. I see the Device Pages from the Table of Hardware have a "WLAN Hardware" entry that lists the actual chip part numbers for the radios. I'm still not clear how to convert, say, "MT7976AN 4T4R" or "Qualcomm QCN5054" to one of the families above that provides OWFG.

MT7976AN is the 5/ 6 GHz phy frontend of the Mediatek mt7916 (wifi6/ wifi6e, 802.11ax) chipset, supported by mt76 - supporting 4 antennas/ 4 rx/ tx chains.

Qualcomm QCN5054 is the 5 GHz wireless chipset (QC5024 would be the 2.4 GHz counterpart, QCN9074 the 6 GHz one) from Qualcomm (wifi6, 802.11ax), supported by ath11k - supporting 4 antennas/ 4 rx/ tx chains.

[dtaht]

I am a little testy sometimes, about backpressure at normal rates. There are a ton devices now running at native ethernet 1Gbit or 100Mbit rates, with cake or fq_codel + bql - and they just work.... people run this stuff natively and don't know they are and decry that they do not need QoS! but that's because it is already there.... and so far as I know (I would like more to check!!! BQL is universally implemented on the 2.5Gbit network chips also)

Anyway, a REALLY long and over-detailed thread about cake on mikrotik could perhaps be leveraged seomewhet here: https://forum.mikrotik.com/viewtopic.php?t=179307 s

And a really good thread about the history of this stuff off of another thread over tere

[mpenning]

@dtaht

There are a ton devices now running at native ethernet 1Gbit or 100Mbit rates, with cake or fq_codel + bql - and they just work.... people run this stuff natively and don't know they are and decry that they do not need QoS!...

BQL is universally implemented on the 2.5Gbit network chips also

I'm not sure if you implicitly had limits in mind when you say "BQL is universally implemented"... does that include commercial network infrastructure products from Cisco / Juniper / Arista (all of them)?

As for the ton of devices running CAKE / fq_codel... Cisco is the largest commercial network infrastructure vendor, and I am not sure they implement fq_codel or CAKE in Cisco IOS... if they did, there should be a configuration CLI knob to enable or disable those AQM features (to fall back to FIFO just for testing and comparison purposes). However, I don't think fq_codel / CAKE exist in Cisco IOS..

This point is huge... if I'm right, it's the reason people are using libreqos middleboxes at Comcast in the first place. If commercial network infrastructure vendors implemented non-RED AQM, we might have a very different internet these days.

[moeller0]

@mpenning

This point is huge... if I'm right, it's the reason people are using libreqos middleboxes at Comcast in the first place. If commercial network infrastructure vendors implemented non-RED AQM, we might have a very different internet these days.

Heck, if they deployed even RED AQMs we would be in a better place... but then we can certainly do better than RED

[mpenning]

@moeller0

if they deployed even RED AQMs we would be in a better place... but then we can certainly do better than RED

RED AQM is used in my Cisco deployments, but as you mention we can do much better, which is what I was saying. RED is based on average queue size, and often-enough microbursts won't increase average queue depth enough to random-drop in time to control queue latency.

[moeller0]

Well, here is the thing, to do fq in big iron routers somebody would need to front the R&D cost, traditionally that would be the buyers of that gear that need to badger the manufacturer offer such capabilities and be prepared to pay a bit extra for that. However both these customers and the manufacturers are currently pretending that L4S snakeoil will be enough.
Tl:dr: it will not, L4S is too little too late and is built essentially on some sort of cooperative scheduling, where it is not the scheduler, but the endpoints that are expected to assert some level of fairness...
This is going to fail sooner or later, it legacy might turn out to be to push the PIE AQM to replace RED... Now according to the RFC its as a gimped version of PIE made by folks who believe the remedy for traffic burst in a network is an RFC that says, end-points, please do not send traffic burstily... anyway, even that gimped PIE variant might turn out to be superior to RED... (however it likely will also be inferior to CoDel, let alone fq_codel)

Some folks say, fq in big iron is impossible, my take is big iron manufacturers so have never tried for real so the lack of such a product does not tell us much about its feasibility...

[richb-hanover-priv]

Hi folks,

This is a delightful discussion of many congestion control/adaptation processes, and various places they can be applied. But they're getting off-topic.

I would really like to get comments on the original post where I "run my mouth" about how the SQM techniques within OpenWrt help minimize latency.

My motivation is two-fold: it'll be good to have a concise article about this subject. If I can get you all to tell me what's true, I'll write it down. Also, I can never remember all the components of the Wi-Fi fixes, and how I can tell which chipsets work. Once I write that down, I'll be able to find the info again.

(I'm taking the same approach as Alan Kay had for the 128K Macintosh: it's the first computer that's good enough to criticize. Apple II, DOS, CP/M computers were all so dreadful, that it wasn't even fair to point out their flaws...)

Same rules for this article - please give me comments! Thanks.

[tohojo]

A few notes on your questions in the original post:

• Cake first appeared in 2015, according to the commit log and copyright notices in the out-of-tree repo. It was upstreamed in 2018.
• The driver can be queried for support through iw list - look for TXQS, AIRTIME_FAIRNESS and AQL in the "Supported extended features" list at the end of the output for a given phy device.
• The AQL limits can be tuned through debugfs:

 cat /sys/kernel/debug/ieee80211/phy0/aql_txq_limit 
AC	AQL limit low	AQL limit high
VO	5000		12000
VI	5000		12000
BE	5000		12000
BK	5000		12000

The values are in microseconds and can be written to using a "x y z" format, where x is the AC number (0-3), and y and z are the low and high limits. The high limit is used as a per-station limit if the total outstanding airtime for the interface is below the aql_threshold (tuned by the file of that name in the same directory, default 24ms), and the low limit is used once the limit goes over. So basically, the first two stations queueing can get 12ms each, and after that, each station is capped at 5ms (or some other combination that adds up to 24ms total).

[richb-hanover-priv]

This is great information (I knew I could inveigle people into telling me the answers ) I'll incorporate it as I go.

A few more thoughts/questions:

• What is a "correct but shorter" name for "full set of Wi-Fi fixes"? (ATF, Airtime Fairness, AirtimeFQ, TWFG, ...?)
• How is all this related to the mwlwifi driver? Are there any other Wi-Fi driver implementations common in modern OpenWrt?
• When did the Wi-Fi fixes hit OpenWrt mainline?
• I'm going to work to keep this particular document fairly descriptive, and provide links to detailed info. For example, I have said that defaults work great, and provided a link to Toke's note here in the Forum for info about configuration of the full set of Wi-Fi fixes).
• I would like to guidance for selecting Wi-Fi chipsets, but I really need help converting from the chipsets listed in ToH "WLAN Hardware" entries to those are supported by the Wi-Fi fixes.
• My eyes glaze over when I review the options on the Device Pages: is there a "Cliff Notes" that says "All Qualcomm that starts with "QCN3" or "MT123..."? What other clues are there?

Thanks as always.

[moeller0]

@tohojo

The values are in microseconds and can be written to using a "x y z" format, where x is the AC number (0-3), and y and z are the low and high limits. The high limit is used as a per-station limit if the total outstanding airtime for the interface is below the aql_threshold (tuned by the file of that name in the same directory, default 24ms), and the low limit is used once the limit goes over. So basically, the first two stations queueing can get 12ms each, and after that, each station is capped at 5ms (or some other combination that adds up to 24ms total).

Mmh, what is the rationale behind the 24ms threshold? (Just having to pick a value or some interaction with the aggregation logic?)

[richb-hanover-priv]

Here's the final outstanding question before I release version 1.0 of "How OpenWrt Vanquishes Bufferbloat"... (I apologize in advance for not grokking the advice I've received so far...)

I have seen the recommendations that "mt76" and "ath11k" (and sometimes "ath10k") are the only sensible choices today (others are obsolete, or so new as not to be available.) I've also seen the advice to look at iw list to see if TXQS and other options are present.

But how can I tell whether the hardware is supported by the "OpenWrt Wi-Fi Driver" without installing OpenWrt? How do I know what to purchase?

I know I can look at vendor specifications or the Table of Hardware to find the WLAN hardware. How can I go from that chip/SoC part number to making a purchasing decision?

Thanks.

[moeller0]

@richb-hanover-priv

I know I can look at vendor specifications or the Table of Hardware to find the WLAN hardware. How can I go from that chip/SoC part number to making a purchasing decision?

Getting from driver to supported chips/SoCs can be somewhat short circuited
This:
https://wireless.wiki.kernel.org/en/users/drivers/mediatek
this:
https://wireless.wiki.kernel.org/en/users/drivers/ath10k
and this:
https://wireless.wiki.kernel.org/en/users/drivers/ath11k

But that still leaves a hole where your question how to figure out which brand/model contains which chip expected an answer...

[richb-hanover-priv]

I was hopeful that the ToH could cross-reference a chip/SoC part number. However, the ToH has these "WLAN Hardware" entries for the first three "newcomer routers":

• GL-MT3000: MediaTek MT7976CN
• GL-MT6000: 2.4GHz: MT7976GN 4T4R, 5GHz: MT7976AN 4T4R
• R7800: Qualcomm Atheros QCA9984

The first two don't seem to appear in the Mediatek page at wireless.wiki.kernel.org. Does that mean they're not supported?

Thanks.

[Spacebar]

Is this list helpful?
https://openwrt.org/toh/views/toh_available_16128_ax-wifi