Probably you want to create custom rules with LAN/veth0 interface as source.
In addition, it's best to assign both WAN and VPN interfaces to separate tables.
I think I am making progress thanks to your help. I have set up:
root@OpenWrt:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
128 prelocal
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
1 br-lan
2 veth-lan
3 veth-wan
4 WireGuard
5 wan
so I have table for each interface and will set default route to that interface on each table.
I am now trying to implement instead:
I think this may make more sense.
I set up (just testing wan part for now):
root@OpenWrt:~# ip rule
0: from all lookup local
10000: from 192.168.1.1 lookup br-lan
10000: from 10.10.75.253 lookup wan
10000: from 10.5.0.2 lookup WireGuard
15000: from all iif br-lan lookup veth-lan
15000: from all iif veth-wan lookup wan
15000: from all iif wan lookup veth-wan
15000: from all iif veth-lan lookup br-lan
20000: from all to 192.168.1.1/24 lookup br-lan
20000: from all to 10.10.75.253/8 lookup wan
20000: from all to 10.5.0.2 lookup WireGuard
32766: from all lookup main
32767: from all lookup default
90007: from all iif lo lookup wan
90025: from all iif lo lookup br-lan
90026: from all iif lo lookup veth-lan
90027: from all iif lo lookup veth-wan
90034: from all iif lo lookup WireGuard
20000: from all to 192.168.1.1/24 lookup br-lan
20000: from all to 10.10.75.253/8 lookup wan
20000: from all to 10.5.0.2 lookup WireGuard
32766: from all lookup main
32767: from all lookup default
90007: from all iif lo lookup wan
90025: from all iif lo lookup br-lan
90026: from all iif lo lookup veth-lan
90027: from all iif lo lookup veth-wan
90034: from all iif lo lookup WireGuard
Does that look about right (specifically the rules with priority 15000)? For some reason it's not working. I can do this with veth-wan + peer veth-lan, both unmanaged interfaces and not part of br-lan, right? I don't think I need to give veth-wan / veth-lan IP addresses?
So with:
root@OpenWrt:~# ip route show table wan
default via 10.0.0.1 dev wan src 10.36.202.233
10.0.0.0/8 dev wan scope link
root@OpenWrt:~# ip route show table veth-lan
default dev veth-lan scope link
root@OpenWrt:~# ip route show table veth-wan
default dev veth-wan scope link
It seems I am missing something. Why does this rule work:
15000: from all iif br-lan lookup wan
As in, traffic is routed from br-lan through to wan.
But:
15000: from all iif br-lan lookup veth-lan
15000: from all iif veth-wan lookup wan
fails? I would expect that to route traffic from br-lan, onto veth-lan, which then means it automatically goes to veth-wan because that is how veth works, and then get caught at exit from veth-wan to go through wan.
As in, I just don't see any traffic on the veth pair with tcpdump on either end when I ping, but I would at least expect outgoing traffic now to go through veth.
Really appreciate your help so far. I would have thought that creating intermediate interface through which incoming and outgoing wan VPN/non-VPN traffic flows as single interface should be easy. But I am failing.
1 Like
If I remember correctly, "bare" veth pairs work weirdly. One end really wants to be either in a bridge, or in another network namespace. I can't remember why that is, but I think for example that ARP or NDP or something breaks for a "bare" veth pair.
here's an example article:
What is it you're trying to accomplish here? It should be possible to put SQM on your WAN, and then use tc to mirror packets from the wireguard to the same IFB used by SQM in the download direction, I think this should work.
1 Like
Here's a diagram of how this would work I think:
SQM already sets up the "tc mirred" action on the receive side of WAN, you need to set up an additional tc mirred action that sends inbound packets on the WG (wireguard) interface over to the same IFB that SQM uses.
2 Likes
@dlakelan thank you for helpful insight into 'veth'. So I am going to give up with that now.
THE IFB APPROACH
So with your diagram does that rely upon this 'flows' keyword so that CAKE on upload can see the flows inside the encrypted packets? As in what @moeller0 wrote above here:
Well, for egress there is exists a solution, in that wireguard can/will set the flow-id in the fwmarks and cake can actually consume this, so that it will recognize the VPNd flows individually (but IIRC, this only works in besteffort mode, so not in conjunction with dual-dshost or dual-src-host or triple-isolate, and it requires that wireguard-encryption/decryption runs on the same router, and finally this will only work for egress traffic).
If so, I can see how that would work. Separate SQM script and give up on LuCi's SQM options. The complexity in that approach lies in setting up the tc filters / mirror rules. Could you provide a rough idea of how I would set those up?
THE REROUTING THROUGH INTERMEDIATE INTERFACE APPROACH
Alternatively, I have a different approach in mind that replaces that complexity with what ought to be a simple routing exercise.
I spent the whole of yesterday simply trying to set up the following:
That is, simply an intermediate software bridge interface 'lan-wan'. I gave it a static IP 192.168.2.1 with subnet 255.255.255.0.
If opting for re-routing, this feels like the simplest and most elegant solution here, namely an intermediate interface through which all VPN and bypassed WAN outgoing and incoming flows goes through. Makes sense, right? It's just a single interface that exposes all the WAN-related flows.
Despite its apparent simplicity, for the life of me I cannot determine the correct IP rules / routes to get this to work. Do you have a sense of what these might be? I would really appreciate a basic outline to work with.
Is it wrong to suppose that something like:
from all iif br-lan lookup lan-wan
from all iif lan-wan lookup wan
for upload to WAN would work, with 'lan-wan' table set with default device 'lan-wan', i.e.
default dev lan-wan
When I tried that, it just didn't work, i.e. I didn't see ougoing packet on LAN-WAN when I tried 'ping 8.8.8.8'.
WHICH APPROACH IS BETTER?
Which approach is better? Your approach requires tricky tc filter mirror / rules. My approach requires re-routing all packets on the router just for the sake of SQM. I am not sure which is better. Perhaps re-routing traffic just for sake of SQM is mad, which would point towards your solution being favourable.
FURTHER PROGRESS
I think to make further progress I am going to need either:
-
further input from you @dlakelan (on how to set up the tc filter / mirroring); or
-
further input from @vgaetera on how to solve this from a re-routing perspective.
Or of course further input from any other individual that can offer some insight here.
Surely there are others wanting CAKE to work with VPN PBR too.
It's just one line of tc, you should be able to see what SQM scripts are doing and then copy and paste. It should set up a "mirred" action on the ingress hook of the wireguard virtual interface... I actually don't know if it will work on a virtual interface... But worth a try
1 Like
won't the existing mirror of incoming wan (top left in your diagram) include both encrypted and unencrypted packets though?
Yes, you'll run the incoming packets through twice... I guess what is the goal? SQM on the WAN will already avoid bufferbloat. SQM on the wireguard interface would only be useful to do something like shape the VPN down to a lower speed and/or prioritize or spread the bandwidth evenly.
1 Like
I'm running with this.. almost there now!
@dlakelan I think I have successfully implemented your idea:
with this init.d script:
root@OpenWrt:/etc/init.d# cat sqm-vpn-pbr
#!/bin/sh /etc/rc.common
# Copyright (C) 2007 OpenWrt.org
exec &> /tmp/sqm.log
START=50
STOP=4
start() {
ip link add name ifb type ifb
tc qdisc add dev wan root cake bandwidth 30Mbit besteffort flows nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 60
tc qdisc add dev wan handle ffff: ingress
tc qdisc add dev vpn handle ffff: ingress
tc qdisc add dev ifb root cake bandwidth 30Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 60
ip link set ifb up
tc filter add dev wan parent ffff: protocol ip prio 1 u32 match ip sport 51820 0xffff action pass
tc filter add dev wan parent ffff: prio 2 matchall action mirred egress redirect dev ifb
tc filter add dev vpn parent ffff: matchall action mirred egress redirect dev ifb
}
stop() {
tc qdisc del dev wan ingress
tc qdisc del dev wan root
tc qdisc del dev vpn ingress
tc qdisc del dev ifb root
ip link set ifb down
ip link del ifb
}
So on upload outgoing flow I make use of the wireguard/CAKE compatibility by using the 'flows' command. On download I copy all incoming wireguard traffic onto the ifb and only copy the incoming wan traffic that does not have source port 51820 (the wireguard peer port). So that way I copy all the VPN + non VPN flow without overlap. Make sense?
@moeller0 does the above and below look reasonable to you:
root@OpenWrt:/etc/init.d# tc -s qdisc show
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
Sent 4567810913 bytes 7928135 pkt (dropped 0, overlimits 0 requeues 23)
backlog 0b 0p requeues 23
maxpacket 1518 drop_overlimit 0 new_flow_count 639 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan2 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan3 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan4 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8013: dev wan root refcnt 2 bandwidth 30Mbit besteffort flows nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 60
Sent 369839652 bytes 653901 pkt (dropped 457, overlimits 952750 requeues 0)
backlog 0b 0p requeues 0
memory used: 718080b of 4Mb
capacity estimate: 30Mbit
min/max network layer size: 28 / 1480
min/max overhead-adjusted size: 88 / 1540
average network hdr offset: 14
Tin 0
thresh 30Mbit
target 5ms
interval 100ms
pk_delay 2.73ms
av_delay 522us
sp_delay 10us
backlog 0b
pkts 654358
bytes 370514546
way_inds 3064
way_miss 1236
way_cols 0
drops 457
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 13740
quantum 915
qdisc ingress ffff: dev wan parent ffff:fff1 ----------------
Sent 972522093 bytes 786003 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev vpn root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc ingress ffff: dev vpn parent ffff:fff1 ----------------
Sent 902510809 bytes 772170 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8014: dev ifb root refcnt 2 bandwidth 30Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 60
Sent 856352931 bytes 737289 pkt (dropped 48701, overlimits 1136858 requeues 0)
backlog 0b 0p requeues 0
memory used: 1392104b of 4Mb
capacity estimate: 30Mbit
min/max network layer size: 40 / 1420
min/max overhead-adjusted size: 100 / 1480
average network hdr offset: 1
Tin 0
thresh 30Mbit
target 5ms
interval 100ms
pk_delay 2.24ms
av_delay 743us
sp_delay 11us
backlog 0b
pkts 785990
bytes 925453606
way_inds 2158
way_miss 1246
way_cols 0
drops 48701
marks 0
ack_drop 0
sp_flows 2
bk_flows 1
un_flows 0
max_len 29348
quantum 915
1 Like
Just curious about the filter rules stats. Output from:
tc -s filter show dev wan
tc -s filter show dev vpn
Have a feeling this doesn't work in busybox. Is there another way to show filter stats?
root@OpenWrt:/etc/init.d# tc -s -d qdisc show dev wan
qdisc cake 8013: root refcnt 2 bandwidth 30Mbit besteffort flows nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 60
Sent 383872679 bytes 701772 pkt (dropped 461, overlimits 1001515 requeues 0)
backlog 0b 0p requeues 0
memory used: 718080b of 4Mb
capacity estimate: 30Mbit
min/max network layer size: 28 / 1480
min/max overhead-adjusted size: 88 / 1540
average network hdr offset: 14
Tin 0
thresh 30Mbit
target 5ms
interval 100ms
pk_delay 4.19ms
av_delay 302us
sp_delay 95us
backlog 0b
pkts 702233
bytes 384552193
way_inds 3177
way_miss 2330
way_cols 0
drops 461
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 13740
quantum 915
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 1055133139 bytes 860198 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
root@OpenWrt:/etc/init.d# tc -s -d qdisc show dev vpn
qdisc noqueue 0: root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 981616406 bytes 846845 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
Oh, probably need to append parent ffff: to each command.
1 Like
root@OpenWrt:/etc/init.d# tc -s filter show dev wan parent ffff:
filter protocol ip pref 1 u32 chain 0
filter protocol ip pref 1 u32 chain 0 fh 800: ht divisor 1
filter protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? not_in_hw
match ca6c0000/ffff0000 at 20
action order 1: gact action pass
random type none pass val 0
index 1 ref 1 bind 1 installed 3172 sec used 0 sec firstused 3168 sec
Action statistics:
Sent 1045633232 bytes 850369 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
filter protocol all pref 2 matchall chain 0
filter protocol all pref 2 matchall chain 0 handle 0x1
not_in_hw (rule hit 7934)
action order 1: mirred (Egress Redirect to device ifb) stolen
index 1 ref 1 bind 1 installed 3172 sec used 0 sec firstused 3172 sec
Action statistics:
Sent 16059952 bytes 15294 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
root@OpenWrt:/etc/init.d# tc -s filter show dev vpn parent ffff:
filter protocol all pref 49152 matchall chain 0
filter protocol all pref 49152 matchall chain 0 handle 0x1
not_in_hw (rule hit 678792)
action order 1: mirred (Egress Redirect to device ifb) stolen
index 2 ref 1 bind 1 installed 3195 sec used 0 sec firstused 3191 sec
Action statistics:
Sent 987729752 bytes 852048 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
What does this indicate I wonder? Very interested to know..
1 Like
Just how well your filters work to (presumably) split VPN and non-VPN traffic on ingress.
Are they working well? I am not 100% but I think the script is working as it should be. At least 'tcpdump -vpni ifb' seems to show correct flows. Haven't tested very much yet but so far so good.
I think this:
tc filter add dev wan parent ffff: protocol ip prio 1 u32 match ip sport 51820 0xffff action pass
tc filter add dev wan parent ffff: prio 2 matchall action mirred egress redirect dev ifb
tc filter add dev vpn parent ffff: matchall action mirred egress redirect dev ifb
should take out the incoming wireguard from wan, copy the rest, and copy all from wireguard. So ifb ends up with VPN + bypassed VPN without overlap.
Try to generate a bunch of VPN flows and see what CAKE does with them. I personally like to watch flows come and go by repeating this command (adapted for your config):
tc -s class show dev ifb
I’m not sure yet what benefit you get from mirroring the ingress from vpn since they haven’t been unencrypted yet. Or have they?
Will do. In the meantime @dave14305 I thought of an alternative solution. Can an IFB be set up in both directions? If so it would be possible to set up ifb-ul and ifb-dl based on br-lan (br0 in Asus-Wrt) and then just apply the tc filters for both to pass on all 192.168.1.0/24 traffic, right? Then apply SQM on 'ifb-ul' and 'ifb-dl' and that way circumvent applying SQM on the router-lan and lan-router traffic.
root@OpenWrt:/etc/init.d# tcpdump -vpni vpn |grep -i icmp
tcpdump: listening on vpn, link-type RAW (Raw IP), capture size 262144 bytes
22:01:41.924880 IP (tos 0x0, ttl 127, id 31595, offset 0, flags [none], proto ICMP (1), length 60)
10.5.0.2 > 8.8.8.8: ICMP echo request, id 1, seq 1, length 40
22:01:41.979876 IP (tos 0x0, ttl 60, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 10.5.0.2: ICMP echo reply, id 1, seq 1, length 40
22:01:42.964614 IP (tos 0x0, ttl 127, id 31596, offset 0, flags [none], proto ICMP (1), length 60)
10.5.0.2 > 8.8.8.8: ICMP echo request, id 1, seq 2, length 40
22:01:43.010898 IP (tos 0x0, ttl 60, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 10.5.0.2: ICMP echo reply, id 1, seq 2, length 40
22:01:43.980593 IP (tos 0x0, ttl 127, id 31597, offset 0, flags [none], proto ICMP (1), length 60)
10.5.0.2 > 8.8.8.8: ICMP echo request, id 1, seq 3, length 40
22:01:44.029901 IP (tos 0x0, ttl 60, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 10.5.0.2: ICMP echo reply, id 1, seq 3, length 40
22:01:45.000377 IP (tos 0x0, ttl 127, id 31598, offset 0, flags [none], proto ICMP (1), length 60)
10.5.0.2 > 8.8.8.8: ICMP echo request, id 1, seq 4, length 40
22:01:45.060922 IP (tos 0x0, ttl 60, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 10.5.0.2: ICMP echo reply, id 1, seq 4, length 40
Should I be using 'nat' on IFB then? But 'nonat' on 'wan' for upload? Or 'nat' on both?