For those interested in running BBR on their routers running Samba/CIFS or VPN (tcp connections terminating at the router), I've attempted to backport the patches from this post: https://groups.google.com/forum/#!topic/bbr-dev/8pgyOyUavvY
This is compile-time, run-tested on mvebu: 1900ACS (Shelby). No numbers or comparative statistics yet - which is why I'm putting this out there ( to get more testing).
Well, BBR has two promises, one it will cyclically try to probe the current path bandwidth and adjust itself such that it will not cause excessive (own-) bufferbloat, and that it is reasonably tolerant to relative low rates of packets loss as encountered e.g. over wifi links. So if you use a VPN from a coffee shop's WiFi network back into your network, BBR might be a good thing (except that you should not run a VPN over a TCP connection in the first place, but that often is not optional unfortunately).
Also BBRv1 was reported as not being terribly fair to non-BBR flows and might not play too well with an ECN/loss based AQM as sqm-scripts (but that should depend of the actual BBR version, v2 is supposed to be better in that regards, and I believe without evidence that BBRv1 also improved in that regard)
Yep, it's not fair/doesn't co-exist with TCP Cubic/Reno or any loss-based TCP congestion control algorithms.
Theoretically it won't play well with Cake's BLUE or FQ_CoDel's Bulk Dropper. In any case, I agree in that it doesn't play well with ECN.
I don't think BBRv2 will play nicely with ECN still – it looks like they're adopting DCTCP/L4S style rather than SCE or RFC3168.
Though with my use case/ISP, I never receive ECN bits (even though I've enabled it everywhere). I'm pretty sure it's not a config error on my side anyways. I'm stuck with an ISP that uses old Cisco routers and Huawei/FiberHome ONTs/CPEs that probably screw up ECN as collateral due to their effort to nuke DSCP bits.
user@MacBook-Pro ~> netstat -sp TCP | grep ECN
0 client connection attempted to negotiate ECN
0 client connection successfully negotiated ECN
0 time graceful fallback to Non-ECN connection
0 time lost ECN negotiating SYN, followed by retransmission
0 server connection attempted to negotiate ECN
0 server connection successfully negotiated ECN
0 time lost ECN negotiating SYN-ACK, followed by retransmission
0 connection using ECN have seen packet loss but no CE
0 connection using ECN have seen packet loss and CE
0 connection using ECN received CE but no packet loss
0 connection fell back to non-ECN due to SYN-loss
0 connection fell back to non-ECN due to reordering
0 connection fell back to non-ECN due to excessive CE-markings
user@MacBook-Pro ~>
Hell, HTTP3 and HTTP2 is/was based off of standards developed by Google – I'm pretty sure we're probably going to go that way with BBR too given that they've got the manpower/funding/hours to throw their weight around in the IETF.
e.g. Cloudflare/Netflix/Amazon Cloudfront also doing BBR.
It may even be commonly used in Chinese sites (Alibaba/Baidu) as well now – a lot of the BBR review papers come out from Chinese universities. (Proliferation in the East & West)
BBR is actually developed by a surprisingly small team and with surprisingly low resources IIRC. The reason why BBRv1 does not support ECN is/was not lack of interest, but lack of available resources I have heard...
In the mean time, what's a same alternative to both BBR and the default (reno/cubic)?
I mean, most people want to try BBR as an alternative to the default, so what would be a better option to try? (I'm thinking about either Vegas or yeAH; but I don't know how they play with ECN)
Yes, BBR is not the best TCP congestion control algorithm for a path with an rfc3169-copliant ECN or droping AQM like fq_codel or cake. That said, it will most likely not be catastrophic, but keep in mind that BBR typically gets an edge over more traditional TCP CCs like Reno or Cubic. BBRv2 is designed to play nicer with AQMs on general, but might not be rfc3168 compliant.
The good news is that with cake's host fairness and integration of a BLUE like algorithm, persistent offenders will simply have more of their packets thrown away.
My Qnap NAS has the option of using BBR for remote backups - I've turned it off. My CAKE instance does amazing work in keeping latency under control (<3ms average increase on fully utilised link) with per IP fairness too - I cannot tell when my backups are running.
Until I see reports of BBR2's fairness and response to ECN, I simply won't use it or BBR if I have a choice.
The second paper does a much better job at explaining what's wrong with the non-coexistence of BBRv1 with other TCP CCAs.
So is this why when I look at YouTube's "Stats for nerds", the graph for network activity periodically changes to red/orange? A BBR flow being reined in by Cake? Interesting.
I remember back in my DD-WRT days I always selected Westwood as that gave me the lowest latencies when visiting websites. I guess things have changed now...
Ok, Westwood is not the faster. Probably it is now, it's been a long trip since Linux 3.0 to be honest. Anyway it doesn't matter.
I'm just thinking of what we can do as an alternative to CUBIC, given that BBR breaks SQM (hint: Westwood doesn't).
Well, I just mentioned my experience with old WRT54G running ancient 2.4 kernel. I’m sure a lot has changed since then, both on client and server side.
