I am currently setting up a more complex OpenWRT routing device.
I have a 5G WAN that I need to shape with adaptive bandwidth (e.g. sqm-autorate or cake-autorate).
Unfortunately, I have three different LAN-Networks in there:
LAN_A: directly routed to WAN
LAN_B: via VPN (that goes via WAN)
LAN_C: via VPN (same as B) but should have higher priority (achieved via assigning different DSCP class and letting diffserv8 do the magic)
I have experience with this setup when it is not using a VPN, so I can just shape on the WAN device and mark the packets beforehand.
When a part of my traffic now passes via VPN, I cannot do the shaping on the WAN-Device because I cannot differentiate the traffic there (e.g. shaping would randomly drop some VPN tunnel packages which cannot be a good idea, it would lose all connection tracking etc.)
So basically I would like to shape both my WAN-Device (... but ignore VPN packets) and the VPN-Device but attribute all traffic together to a single queue.
If I would only shape the VPN traffic, I cannot account for the traffic leaving WAN directly.
Maybe I am thinking to complicated. Can anyone help me with some ideas on how to get this done or suggest another way how to implement this?
Also very nice to know that upload shaping wireguard can work out of the box for besteffort. I guess I 'll use the more sophisticated version here, that script sounds almost perfect already.
Yeah should be ideal. Feel free to post on this thread with any implementation issues.
Also we can work on optimizing cake-autorate in the cake with adaptive bandwidth thread. 5G connections seem to benefit from slightly more aggressive delay thresholds.
By the way what device are you doing the shaping on? We will want to ensure there are sufficient CPU cycles.
it is a banana pi r4 lite - quad core 2 ghz a53 with 2 gigs of ram. i am planning 2000 clients (mostly idle) with 500/200mbps capabilities for the uplink. There should be plenty of resources available:-)
First: I use vlans on the bridge interfaces, so the default mechanism of catching lan traffic in cake-dual-ifb-bridge didn't work for me - I added ibrname br-lan vlan id {1, 101, 102, 103} mark set 1. Actually I don't understand why you work at bridge level here at all - only for the IOT device filter? I would think that handling everything in the cake-dual-ifb-inet forward chain should be fine?
Second, since the refactoring in october 2022 none of the rules mark any wireguard traffic - it is assumed to have fwmark 3, but no rule sets it.
I got the problem of duplicated traffic for everything that went through VPN (my vlans 101 and 103). I tried fixing with manually fwmarking 3 traffic outgoing from wan to the VPN providers IP as well as adding a new input rule for that traffic as well. I am not sure if this is how you imagined the flow to work.
Sorry for asking these basic questions, I feel I am missing some thought about how the traffic flow should have worked.
Looking at the early commits, you apply different fwmarks already to the incoming traffic depending on the source interface - should we apply the wireguard fwmark there already? It seems wrong in the way that we'd like to shape data early, e.g. before it becomes encrypted VPN traffic
Besides the described changes, I added rules to mark my traffic regarding my wished priorities in the dual-ifb-bridge block:
It's been a long time since I worked on this because I stopped using Wireguard a few years ago.
The idea is that at the bridge level all the traffic is unencrypted. So that can be pulled off and mirrored to the appropriate ifb interfaces. That data is associated with fwmark 1.
Then we have OpenWrt generated traffic that is sent out through WAN that is not WireGuard traffic. That is fwmark 2.
We also restore DSCPs (stored in conntrack from upload) on download (this is really useful because it facilitates use of tinning in cake - setting DSCPs on upload will mean they get applied back on download for the tracked connections) and simultaneously pull out download data that is for OpenWrt itself by filtering on bit 256.
So this way we pull off all the unencrypted data from a) the bridge interfaces and b) OpenWrt itself and place it in appropriate IFBs. And we set up cake interfaces on those IFBs.
I think I had WireGuard fwmarks applied using a standard OpenWrt WireGuard option to apply those marks.
I'm not familiar enough with VLANs to advise how to make that work properly, albeit I'm confident it's entirely possible and it sounds like you are half way there already. If you get stuck @dave14305 is a wizard on nftables and tc matters and can likely offer more help than I can in that respect.
Ohh thanks for the wireguard fwmark hint! (Also let me state, cake-autorate is really great, I don't really know why I used sqm-autorate on my last deployments, this looks so much more stable and easier!)
I have all the DSCP restore and filtering working properly, yes. Only the wireguard exclusion part is missing.
Regarding bridge traffic:
In inet forward chain, everything is unencrypted as well. So if you don't remember another reason why you mark on the bridge level, i will try to port that to just using the inet layer. It is simpler because here I can use the proper interfaces br-lan.101 etc. instead of having to manually filter for vlan. But as I said, this part works already, it just looks nicer to not have the bridge stuff in there when it's not needed.
Thank you so much for your support!
I think initially I was also unsure about the exact flow of wireguard traffic. But it is actually just "simple" from the outside, forwarded to the wireguard interface and leaving on out again as a new flow, so nothing special to keep in mind here.
Thanks - yes it seems to be in a very stable and mature state now. It generally works well but again for your 5G connection some of the delay thresholds probably should be tweaked a bit. It is naturally a little on the greedy side when it comes to CPU cycles since it is written in bash, but we tried to make it as efficient as possible within the constraints of what is possible in bash and so long as it is verified that the CPU cores have sufficient breathing room then all is fine.
Yes the points where marks are applied can probably be altered. But take care to ensure everything works as expected. I seem to remember this being fairly sensitive. I personally used tcpdump extensively to make sure everything worked as expected.
@moeller0 is an expert at using tcpdump to check for DSCPs and other weird and wonderful filters - see posts like this one:
Feel free to submit any PRs if you can see a way to generalise anything out or other improvements.
By the way, I think you should also cross reference:
I use that rather than sqm-scripts to set up by cake instances (and also apply the DSCP restoration stuff). And I just remembered from cake-qos-simple (see here) that the conntrack stuff can be simplified in newer OpenWrt versions:
chain store-dscp-in-conntrack {
ct mark set ip dscp or 128
ct mark set ip6 dscp or 128
# For older versions of nftables (OpenWrt versions prior to 23.05), replace the two lines above with:
# meta nfproto ipv4 ct mark set (@nh,8,8 & 252) >> 2
# meta nfproto ipv6 ct mark set (@nh,0,16 & 4032) >> 6
# ct mark set ct mark or 128
}
I think the conntrack-DSCP solution is fantastic since you can then set DSCPs either by clients or using router on upload and you get automatic restoration on download. You can verify all is well using the cake statistics options. So this means you can ensure bulk download stuff is dropped on saturation before e.g. more sensitive voice stuff.