BBR for VPN client on router

It looks like BBR might be better than cubic:
/etc/sysctl.conf net.ipv4.tcp_congestion_control=bbr
requires kmod-tcp-bbr to be added

when centralizing all internet requests through a single VPN client connection on the router, using wireguard to a VPN host - vs a more conventional setup of running the VPN service provider's software on every client on the network.

I've dove through numerous results on how to implement bbr correctly. Some of them point to luci-app-sqm, which is dead, as it requires legacy pre-nftables components that aren't available. Some mention 'fq' or 'fq_codel', but there is no step-by-step guide for openwrt on how to enable them (that is current). Some articles, forum posts, seem to suggest that bbr by itself, doesn't work well unless fq* is enabled. And the fq proceedure for openwrt points back to the luci-app-sqm articles.

fq_codel is default qdisc in OpenWrt
bbr would then apply to local sockets only, eg apache webserver running on the router, not routed traffic just passing.

1 Like

Right, but as traffic at the router passes from TCP -> UDP Wireguard, it would affect that, also, right?

Here's one of the guides I found online that I was using, in reference to the OP

Nope, brr applies to tcp sockets, not firewall states only.
you can try tc qdisc replace dev wg0 root fq_codel to shape traffic inside wireguard

1 Like

Do you recommend tc-tiny or tc-full package?

The build/hardware:

ARMv7 Processor rev 5 (v7l)
ipq40xx/generic
OpenWrt SNAPSHOT r29081-2ced0e59a4 / LuCI Master 25.080.50251~fc8ad20
Kernel: 6.6.83
Memory: 81% available (404MiB), 16% used (81MiB), 5% cached (27MiB)
Disk space: 50MiB available & 250MiB of temp space (both less than 1% used)

Using the debian image in WSL2 for a build environment.

Typically get 110Mb/s down and up to 30Mb/s up on a cell connection to a verizon tower about 8 miles away. Clear LoS. The VPN overrides the weird video throttling that verizon does, and when I run the VPN client (Proton) directly on the client device, I get good speeds, smooth loading. I don't get as good performance with the VPN moved to the router, wireguard interface.

Reading their documentation, they say they used BBR to improve the performance, along with other techniques, 'up to 400% more' than without it.

Sysupgrade to stable 24.10.0 about yesterday and post output of

ubus call system board

tc-bpf is the biggest

I'm not sure what you are telling me to do.
It's running a build newer than 24.10. And ubus call system board doesn't say anything I didn't already post. It's an Orbi LBR20.

Sysupgrade to this

install any of tc flavours and try to set fq_codel or fq on interfaces towards internet.
Test here after each change:
https://www.waveform.com/tools/bufferbloat

1 Like

If I understand you right, try tc qdisc replace dev wg0 root fq_codel vs tc qdisc replace dev wg0 root fq, with wg0 being substituted to the name of the wireguard interface.

Try each setting fq & fq_codel against https://www.waveform.com/tools/bufferbloat

For downgrading to 24.10 before testing, I'm not sure why I would do that.

Snapshots are for expert testers on spare routers.

What I'm trying to do right now is figure out how to tell 'tc' that I have an uplink 'wg0' that is limited to 100Mb/s down and 30Mb/s up.

Those are the numbers I get when doing bandwidth tests with the VPN up.

https://www.waveform.com/tools/bufferbloat?test-id=364e3d3d-6a70-4a12-b39c-e665dec5d998

BBR is of no benefit for WG, as it is only for TCP, and WG is udp based.

Where I've found benefit for BBR is on either the client or server side for TCP apps - it only has to be on one endpoint, not both ends - and there's a lot of servers that have gone over to BBR if one trusts what Google says...

FWIW - luci-app-sqm is not dead, it's very much alive - with both cake and fq_codel supported - and it works just fine, at least on snapshot.

If you're looking to get best performance on VPN's, consider the HFO/SFO options without SQM - but even there, perhaps limited use, as a VPN tunnel is considered a single flow through the NAT...

2 Likes

It's hard to trust Waveform from some networks - for some, it's been inconsistent as of late, wondering if they are not provisioning it for enough connections as it's rather popular :slight_smile:

Might consider Cloudflare's speed test - https://speed.cloudflare.com/

I like it as it supports IPv6, and it does give useful info across a few different metrics...

1 Like

Try sqm, 74ms base latency somewhat limits improvement

2 Likes

BBRv1 (the one in Linux) is old and should not be used. BBRv2 never made into kernel. Now they're talking about BBRv3 as the next big thing. But all of that has got little to do with VPN...

2 Likes

Nice presentation for BBRv3 - interesting to note that Google on their external facing services are already doing BBRv3, and since BBR only needs one end for it to work...

Google-external traffic:
○ BBRv3 is TCP CC for all Google.com and [new!] YouTube public Internet traffic
○ A/B experiments: BBRv3 vs v1 for small % of users for:
■ QUIC for google.com and YouTube

1 Like

The fact that Google's own BBR implementation is different from public one has been discussed quite a while ago. This is what is known as BBRv3 now.

Any CC, it's not what is unique for BBR.

YouTube was among the earliest adopters of new CC. But video delivery runs mostly over UDP (QUIC). To my knowledge both Chrome and FF use Cubic in their QUIC implementations. Even more, they don't include BBR at all (FF includes New Reno as an option). So, true, BBR adoption is one-sided.

1 Like

Nope, it needs an iptables binary, but will happily use iptables-nft that is an iptables binary that uses the nftables back-end. And if you install iptables-nft first that should solve the issue of sqm-scripts/luci-app-sqm wanting to drag in iptables.

3 Likes

No, net.ipv4.tcp_congestion_control only affects local TCP sockets, so TCP traffic terminating at the router it self. Note traffic from LAN machines is treated as "packets" by your router independent of their L4 protocol. So configuring your router to use BBR will do exactly nothing for your central wireguard-UDP VPN, with the exception of TCP traffic terminating at the router that also is routed via the VPN.

fq_codel is not a traffic shaper, in that it can not control the egress rate.

Maybe try cake-autorate?

4 Likes

I remember it took me quite a while to realize that. What's the reason it doesn't depend on nft version directly even in OpenWrt 24.10?