I have setup CAKE on my WireGuard interface. This seems to work well for the case that all traffic is sent through the WireGuard interface. But I need to enable PBR to allow Netflix and Amazon Prime traffic to bypass WireGuard through the WAN interface.
So I suppose I need to also set up SQM on the WAN interface. In LuCi what should I set? I imagine I should simply set bandwidth and prioritise all WireGuard traffic. If so, how would I do that?
Any ideas?
@moeller0 I have been reading your posts on similar threads on this topic. The problem I face is that CAKE on WireGuard works perfectly. But as soon as I have non VPN traffic go through WAN the SQM gets messed up. I tried CAKE on just WAN, but this doesn't work well for the WireGuard traffic.
It seems there are three options:
1) set up a virtual interface that combines the WireGuard and WAN traffic. And shape on that.
2) try to preserve DSCP markings prior to and post WireGuard encapsulation. This seems very promising and perhaps the best solution in theory, but very difficult to implement at present, right?
3) apply CAKE to both the WireGuard and WAN interfaces.
Might the best option here be 1)? Would that be an IFB interface set up in a similar way to how the IFB interface is set up in the SQM code? So matchall VPN traffic from WireGuard but match only non-VPN traffic from WAN, and combine both to the new interface? And then CAKE on that?
In any case, I would be extremely grateful if you could help in terms of giving any hints as to how to implement a good approach? I know there are others out there trying to work out how to deal with SQM when a VPN is used and how to handle a mixture of VPN and non-VPN traffic.
Well, for egress there is exists a solution, in that wireguard can/will set the flow-id in the fwmarks and cake can actually consume this, so that it will recognize the VPNd flows individually (but IIRC, this only works in besteffort mode, so not in conjunction with dual-dshost or dual-src-host or triple-isolate, and it requires that wireguard-encryption/decryption runs on the same router, and finally this will only work for egress traffic).
Regarding your proposals:
ad 1) why bother with an virtual interface, just instantiate SQM/cake on the wan device (preferably the L2-device as reported by ifstatus wan), the problem here is that even with dual-dsthost your VPN traffic will get at best 1/N of the capacity, with N-1 the number of concurrently bypassing internal IP-adresses... for ingress that should work better than besteffort, as all wireguard traffic should look like belonging to a single flow so pure per-flow fairness is going to throttle the VPN even more that per-IP fairness.
(For egress just try the flows nonat keywords instead of dual-dsthost so that you get at least per flow fairness).
ad 2) This by itself is problematic as in each priority tier cake will still apply its normal fairness mode, and all the VPN flows look like a single flow... (but it is possible to combine this approach with the flows nonat trick from above). And the typical problem is that for ingress traffic you can not really mark your VPN traffic well, since you have no working iptables/nftables available. If the external port of wireguard stays constant, you could try to use a tc filter to set DSCPs according to the port-number, but since anybody can send you packets with any port number that is not super robust.
ad 3) Doable, but unless the VPN sqm instance is set to considerably lower speed than the WAN sqm instance, the VPN sqm instance will rarely trigger (at least for ingress).
As I tried to illustrate for egress things are not that bad, but for ingress the picture is considerably bleaker (unless you are happy with only guaranteeing the combined VPN traffic as much capacity as as single IP address in per-IP fairness)
Hope that helps....
Thanks @moeller0. Concerning the virtual interface why wouldn't it be possible to present a virtual interface to CAKE that allows CAKE to see not only unencrypted traffic before it gets encrypted, but also the bypassed traffic. That way the fairness could surely be applied properly for all flows. Is there something preventing this from working? For example, at br-lan all the traffic is unencrypted and includes both (but I do not want to shape lan-lan, so just using br-lan wouldn't work). Sorry if I am missing something fundamental here.
I guess I did not understand this idea. Sure should work for egress, but for egress flows nonat should already solve most of the issue.
On a wired only router, this should be achievable more easily by instantiating SQM on the LAN interface (just make sure to flip download and upload speeds around, as the fields are interpreted from the perspective of the interface handling the traffic, and "downloads" from the view of a LAN interface are actually uploads if viewed from the perspective of the internet. You still need some filtering/redirection/routing logic to move the stuff that should hit the VPN through wireguard while bypassing it for the rest. Since this sees the traffic exclusively in your internal network you can forego the nat keywords and you can use the dual-dsthost/dual-src-host keywords...
And yes, you probably can build a set of veth interfaces and routing rules to achieve this even on a full wifi-router, but I have never done that so will be of no help.
OK thank you. I will try to do what I can with my limited skills and report back. Hopefully this might be helpful to someone. In the meantime, if anyone else has successfully implemented this, please do chip in!
Looks like plenty of helpful information at:
in terms of how to manage the virtual interface.
Before I plunge into this further @moeller0, mindful that I am using an LTE connection (with bandwidth always higher than 30Mbit/s, but fluctuating between perhaps 35 and 60 Mbit/s) could there be a case to switch from the UDP-based WireGuard to OpenVPN using TCP, and then forego SQM entirely and just rely on the TCP process to deal with bufferbloat? Or would saturating the connection still ruin Zoom and Teams calls I wonder?
Or, mindful that I get A+ bufferbloat on WireGuard with CAKE set to 30Mbit/s (save for the issue of how to handle the non-VPN traffic), do you think there is still merit in me trying to get a UDP-based VPN + CAKE solution to work?
Using a TCP tunnel for a VPN is always sub-optimal (albeit sometimes the only option) because if a packet gets lost both the outer and the inner TCP connection will try to retransmit the missing packet, resulting in repeated transmissions of the same packet/packets, that eats up capacity without helping. And if TCP would be well equi[ped to solve the bufferbloat problem, we would not need AQM/SQM at all, so I am a bit skeptical whether that would work well out pf the box.
Well, you need a traffic shaper so that your incoming and outgoing traffic does not clog up the buffers in the LTE-modem and LTE-basestation, so if the LTE system in use has little bufferbloat you might get away with that, but I seem to recall you where not that happy and hence tried SQM, no?
Well, I would try the SQM on the LAN interface of a wired-only router, and then use PBR to direct some traffic via the VPN and some directly via WAN bypassing the VPN tunnel; that is if I were in your situation.
For download, it seems you could hack things to have wireguard (outer) packets not tc mirred over to the IFB, but then set up an ingress filter on the wireguard interface itself and mirred the inner packets from there?
That might work, but honestly, why bother when SQM on the LAN side of a raspberry pi4B+USB3-Gbps-Ethernet dongle should get you basically the same with far less hassle? 
That part seems a bit tricky, no? Because at that point you are basically routing incoming packets by some header based information along two paths, in which case veth seems like a more natural approach, no?
But again, I would side step that whole issue, by instantiating SQM on the internal side of the router so before PBR/wireguard do their thing ...
Yes, it's fine as long as you don't have multiple VLANs on the internal side.
I have been trying to set up the veth. Am I right that the commands:
ip link show lanveth || ip link add lanveth type veth peer name lanbrport
LAN=lanveth
ip link set lanveth up
ip link set lanbrport up
ip link set lanbrport master $LANBR
ip route flush table 100
ip route add default dev $LAN table 100
ip -6 route add default dev $LAN table 100
ip rule add iif $WAN priority 100 table 100
ip -6 rule add iif $WAN priority 100 table 100
all have a counterpart in LuCi?
I have been able to set up the Virtual Ethernet device with peer name in LuCi:
And also the veth interface:
ip route add default dev $LAN table 100
ip -6 route add default dev $LAN table 100
ip rule add iif $WAN priority 100 table 100
ip -6 rule add iif $WAN priority 100 table 100
you can set up routing tables and rules in static routes page somewhere I think, but I'm not really sure. It's been a long time since I've looked at the luci settings.
Thank you for the very helpful contributions. So am I correct in summarizing that whilst I could apply SQM to br-lan, the problem is that this will not just capture packets going in and out of br-lan from WAN, but it will also capture packets going in and out of br-lan from Wi-Fi. And I have the added complication that I want to handle WireGuard traffic.
So I need to create a 'veth' interface through which both traffic destined for WAN and traffic destined for WireGuard must be routed? I would really appreciate a summary of what routing I would need to achieve this. Is it that:
traffic from LAN should go via 'veth' and then WAN, and
traffic from LAN should go via 'veth' and then WireGuard'?
Also, in terms of routing, something that confuses me is that I have not set any routes at the moment. But traffic is nevertheless apparently routed from LAN to WAN. Is this presently defined in the firewall configuration:
Excuse my ignorance here, but is the above not setting up a route between lan and wan/WireGuard?
And, if the firewall can be used to establish routes, can I not just set up the appropriate routes in the firewall?
Is the existing networking route options not sufficient to handle such routing using 'veth':
@vgaetera I think I've just about got everything figured out:
I created a Virtual Ethernet Device veth0 with peer veth1
I added veth1 to br-lan
I added an IPV4 rule for incoming wan on table 100:
But I am stuck with the LuCi counterpart to the command:
ip route add default dev veth0 table 100
Do you know how to achieve the above in LuCi? I tried:
But this results in:
default dev veth0 proto static scope link
The latter is not a routing rule, but a static route.
Sorry I updated my graphic. I tried static route, but that results in the following:
ip route show table 100:
default dev veth0 proto static scope link
But I think I need:
default dev veth0 scope link
There should be no functional difference regards to that extra parameter.
