Backported "Linux TCP BBR patches for higher wifi throughput and lower queuing delays"

https://wiki.linuxfoundation.org/networking/tcpprobe

On a node with Master branch (kernel 4.19.85, ECN enabled and kmod-tcp-bbr installed) run a couple of iperf3 download (remote node to local node) tests against a public server and varying -C cubic | reno | bbr

Whilst the resulting differences seem marginal (on a 100 Mb/s d & 20 Mb/s up VDSL2 subscriber line with ds-lite and PPPoE) bandwidth throughput was best with:

  1. reno
  2. cubic
  3. bbr

BBRv1 does not evaluate/react to ECN's CE signal, it will mainly respond to its own bandwidth probes (which interpret increased latency during the probe as indicator for having reached/temporarily exceeded path capacity) and also to sustained packet losses. There is a detector inside BBR that tries to figure out whether the path is limited by an AQM though, as competent AQMs typically combine packet losses with relative low latency a combination that BBRv1 without this detector will not respond to well.

I read about this prior, my point was just sharing some benchmarking with that particular setup

1 Like

Fair enough, I guess my point is that BBR's main promise is not necessarily more bandwidth than other TCP's (although there are papers showing it dominating Reno and Cubic) but that it is bandwidth efficient while keeping buffering/latency somewhat better controlled. And unlike pure delay based TCPs like Vegas it is not dominated itself by loss based TCPs like Reno and Cubic.
But there is literature out there addressing this,
like https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/IFIP-Networking-2018-TCP-BBR.pdf.

1 Like

Insteresting read. Perhaps BBR makes a difference for WLan but on the wire I have not seen a difference in my real world nodes though.

And the potential neglect of ECN flags makes me wondering whether ECN is thus obsolete (dead) or why the developers choosen to neglect ECN.

Then there is ipv4 vs ipv6 and whether there is difference or ipv6 potentially handling congestion control better than ipv4?

Another question is which net.core.default_qdisc is best suited for BBR?

fq_codel or cake maybe not. Currently OpenWrt uses fq_codel if it's installed, but BBR ignoring the ECN (on both of which relies) may be a bit of a trouble.
That's why I'm asking about an alternative for CUBIC and BBR for people using QoS/SQM. I think that if Reno is fast, Westwood+ should be similarly fast.

1 Like

Well, as far as I can tell google sees improovements for their youtube work-load, but I would be amazed if for end-users like us the selection of a TCP congestion controll algorithm would be that noticeable.... But sure over slightly lossy media like wifi BBR should do better than Reno and Cubic...

That is simple, the BBR team is surprisingly small so all features need to justify the man-power invested quickly as the team can only do so much at the same time. I believe that for the youtube use-case google simply did not expect tp encounter too many rfc3168-compatible AQMs to justify the investment. BBRv2 is looking into the feasibility of using a different ECN signal than rfc3168 though.
Whether the obsolescence of ECN can be directly deduced from BBR using or not using it, seems unclear to me :wink:

Nope, congestion control is the job for the transport or higher layers, the IP layer is only involved in so far that it carries the 2 ECN bits.

No idea, but I am with @ldir at the moment, I am not google and my traffic is not youtube dash-style traffic, so I do not assume that BBR is ideal for me.

1 Like

Neither fq_codel nor cake depend on ECN signaling, both will happily & gently drop packets to signal a "slow down" command to non ECN-capable flows anyway. Both also try to keep the queue sojourn time short and drop gently, which makes BBR ignore their interventions/signals for too long, so unless BBR's AQM detector triggers BBR flows are not good codel/cake citizens, but the fall-out should be limited to themselves mostly.

Counter question, what in Cubic does not work as you desire? I know it is rather boring, but are you really TCP CC limited in your networking? :wink:

To everyone:
A TCP congestion control algorithm on a router does not affect TCP streams that are not terminated by the router. TCP packets that are being forwarded by the router go through L3 processing but not L7.
BBR will improve upload speed of TCP applications that run on the router such as OpenVPN (TCP mode) and yggdrasil.

2 Likes

The OP said as much in the first post:

The discussion diverged more into a rationale for or against using BBRv1 independent of whether on the router or on end-hosts, as these discussions do :wink:

1 Like

It's not about how good is CUBIC, it's about what's a better option. For example, Veno is designed for wireless links and will shine in those in comparison with CUBIC. That simple fact makes the administrator wonder what the heck is the best. People accepting the defaults have no problems with CUBIC but the question is... Can it be better? Bet the answer is an absolute "yes".
Also: some people care about ECN, particularly people like me with very asymmetrical links (DOCSIS) and with an ISP that doesn't mess up with the ECN and DSCP.

Either way, if we are using a qdisc, the only thing that BBR does is have an extra overhead to perform it's amazingly complex calculation that will be useless anyway because the qdisc is doing the same... And maybe better.

And yes, my network is constantly congested because my ISP sucks. Not all use cases are the same, and if there are 10+ algorithms in the Linux kernel, known that Linux is really pedantic, then we can be sure that we have at least 10 use cases worth for tuning.

So the question is not how bad is CUBIC, the question is what is better.

1 Like

Fun fact: OpenVPN works better when using UDP because nobody in all of it's mental capacity will want to do TCP over TCP. Still, application like Samba can benefit, particularly if the CPU or the memory begins to starve (and the subsequent latency changes because of that).

1 Like

+1.

I guess if you are CPU bound the TCP CC algo is not going to be your primary concern :wink: (unless the used CC caused the CPU overload)

That is a surprisingly hard question to answer, since as far as I can tell, none of the TCP CCs is superior to all others under all conditions. So selecting a better one first requires to be able to actually easily measure goodness. I can see the attraction in doing that, searching for better or an optimum though.

Yes and no because most of us can agree that Veno is better for wireless links where you don't want the random package loss (common in wireless links) to half the throughout every single time it happens.
And yes, the congestion control does makes sense under load because your connection is being congested, it doesn't actually matter to the client if it's the link or the server which is going nuts.
And it's particularly important because all of that stuff runs in real-time, which means that the server must tell the client to calm down even if the server is starving to death.
Also: it's important to remember that in the wireless link one doesn't use a qdisc (sometimes you want to limit bandwidth and it can be easier to setup than firewall rules), and there is no an ISP between your router and your laptop that messed up the DSCP marks.

1 Like

But here you seem to compare Veno with Reno, but it seems to do things similar to BBR (in that it treats drops as noise and not a strict AQM signal). But sure on a lossy link that makes sense.
But I am not fully convinced that my wifi links are lossy enough to merit either Veno or BBR. A matter of policy and taste, without a correct answer IMHO, so every admin needs to decide for the local network.

Erm, on ath9K OpenWrt defaults to air time fairness which works basically by including a clever fq_codel instance into the wifi stack...

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.