SQM with WireGuard and PBR?

Lynx · September 17, 2021, 6:53am

Thanks for your help @vgaetera. Last night I thought I saw it working, but it seems I am missing something. Can you see what I am missing? Right now I see only:

root@OpenWrt:~# tcpdump -vpni veth0
tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
06:49:26.634465 IP (tos 0x0, ttl 255, id 62024, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.119.49160 > 255.255.255.255.6666: UDP, length 175
06:49:27.247711 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.120 tell 192.168.1.120, length 28
06:49:28.019859 IP (tos 0x0, ttl 255, id 51000, offset 0, flags [none], proto UDP (17), length 216)
    192.168.1.179.49154 > 255.255.255.255.6667: UDP, length 188
06:49:28.425658 IP (tos 0x0, ttl 255, id 62416, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.156.49159 > 255.255.255.255.6666: UDP, length 175
06:49:28.969674 IP (tos 0x0, ttl 255, id 1181, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.192.49177 > 255.255.255.255.6666: UDP, length 175
06:49:29.202037 IP (tos 0x0, ttl 255, id 36749, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.191.49178 > 255.255.255.255.6666: UDP, length 175

I have set as follows:

root@OpenWrt:~# ip route show table 100
default dev veth0 proto static scope link
root@OpenWrt:~# ip rule show all
0:      from all lookup local
98:     from all fwmark 0x20000/0xff0000 lookup WireGuard
99:     from all fwmark 0x10000/0xff0000 lookup wan
100:    from all iif wan lookup 100
100:    from all iif WireGuard lookup 100
32766:  from all lookup main
32767:  from all lookup default
90064:  from all iif lo lookup 100

root@OpenWrt:~# ip route show table all
default dev veth0 table 100 proto static scope link
default via 10.0.0.1 dev wan table wan
192.168.1.0/24 dev br-lan table wan proto kernel scope link src 192.168.1.1
default via 10.5.0.2 dev WireGuard table WireGuard
192.168.1.0/24 dev br-lan table WireGuard proto kernel scope link src 192.168.1.1
default via 10.0.0.1 dev wan proto static src 10.14.39.177
10.0.0.0/8 dev wan proto kernel scope link src 10.14.39.177
81.92.203.202 via 10.0.0.1 dev wan proto static
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
broadcast 10.0.0.0 dev wan table local proto kernel scope link src 10.14.39.177
local 10.5.0.2 dev WireGuard table local proto kernel scope host src 10.5.0.2
local 10.14.39.177 dev wan table local proto kernel scope host src 10.14.39.177
broadcast 10.255.255.255 dev wan table local proto kernel scope link src 10.14.39.177
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev br-lan table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.1
default dev veth0 table 100 proto static metric 1024 pref medium
fd30:aae4:ef12:a800::/64 dev wan proto static metric 256 pref medium
unreachable fd30:aae4:ef12:a800::/64 dev lo proto static metric 2147483647 pref medium
fdfb:224c:75d0::/64 dev br-lan proto static metric 1024 pref medium
unreachable fdfb:224c:75d0::/48 dev lo proto static metric 2147483647 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev wan proto kernel metric 256 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev wlan0 proto kernel metric 256 pref medium
fe80::/64 dev wlan1 proto kernel metric 256 pref medium
fe80::/64 dev ifb4WireGuard proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fd30:aae4:ef12:a800:: dev wan table local proto kernel metric 0 pref medium
local fd30:aae4:ef12:a800:ea9f:80ff:fed5:d52e dev wan table local proto kernel metric 0 pref medium
anycast fdfb:224c:75d0:: dev br-lan table local proto kernel metric 0 pref medium
local fdfb:224c:75d0::1 dev br-lan table local proto kernel metric 0 pref medium
anycast fe80:: dev wan table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan table local proto kernel metric 0 pref medium
anycast fe80:: dev wlan0 table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb4WireGuard table local proto kernel metric 0 pref medium
anycast fe80:: dev wlan1 table local proto kernel metric 0 pref medium
anycast fe80:: dev veth0 table local proto kernel metric 0 pref medium
local fe80::64ce:9dff:fe62:7a87 dev veth0 table local proto kernel metric 0 pref medium
local fe80::88a2:b7ff:fe87:5f2 dev ifb4WireGuard table local proto kernel metric 0 pref medium
local fe80::ea9f:80ff:fed5:d52e dev wan table local proto kernel metric 0 pref medium
local fe80::ea9f:80ff:fed5:d52f dev eth0 table local proto kernel metric 0 pref medium
local fe80::ea9f:80ff:fed5:d52f dev br-lan table local proto kernel metric 0 pref medium
local fe80::ea9f:80ff:fed5:d530 dev wlan0 table local proto kernel metric 0 pref medium
local fe80::ea9f:80ff:fed5:d531 dev wlan1 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wan table local proto kernel metric 256 pref medium
multicast ff00::/8 dev br-lan table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wlan0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wlan1 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev WireGuard table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ifb4WireGuard table local proto kernel metric 256 pref medium
multicast ff00::/8 dev veth0 table local proto kernel metric 256 pref medium

veth0     Link encap:Ethernet  HWaddr 72:D9:56:56:4B:F2
          inet6 addr: fe80::70d9:56ff:fe56:4bf2/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:1193 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:217855 (212.7 KiB)  TX bytes:3951 (3.8 KiB)

veth1     Link encap:Ethernet  HWaddr 82:D8:4E:68:77:97
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:18 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1193 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3951 (3.8 KiB)  TX bytes:217855 (212.7 KiB)

vgaetera · September 17, 2021, 9:50am

Looks like the fwmark rules have higher priorities overriding the manually configured ones.

Lynx · September 17, 2021, 10:00am

@vgaetera yes you are right once again - setting priority to '50' fixed it. Now I see proper flows:

root@OpenWrt:~# tcpdump -vpni veth0 |grep 192.168.1.136
tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    52.97.202.66.443 > 192.168.1.136.58296: Flags [P.], cksum 0x2112 (correct), seq 3762234478:3762234521, ack 4171265608, win 2048, length 43
    152.199.22.144.443 > 192.168.1.136.63015: Flags [.], cksum 0x6950 (correct), ack 1781984207, win 137, options [nop,nop,sack 1 {0:1}], length 0
    152.199.21.118.443 > 192.168.1.136.63566: Flags [.], cksum 0x939f (correct), ack 96150410, win 164, options [nop,nop,sack 1 {0:1}], length 0

Is it correct that wireless is set to only 'lan' and not 'veth'?

All of this is extremely confusing to me. Would some kind soul - @moeller0 - perhaps? be able to provide an explanation as to what I have set up here? I have set up veth0 with veth1 peer, then attached veth1 to 'br-lan'. So I think this can be thought of like plugging veth1 end of an ethernet cable into 'br-lan' switch. And then I set rules to match 'wan' and 'WireGuard' traffic. So does that match up what is seen by 'veth0/veth1' with the corresponding 'wan' / 'WireGuard' traffic? And what is the significance of the table 100?

Would really value some kind of explanation here.

Lynx · September 17, 2021, 10:18am

Also, there is still some unmatched traffic (which I think is the stuff to 255.255.255.255.255) - any idea @vgaetera ?

root@OpenWrt:~# tcpdump -vpni veth0
tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:16:42.860734 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.4 tell 192.168.1.136, length 46
10:16:43.112599 IP (tos 0x0, ttl 255, id 54146, offset 0, flags [none], proto UDP (17), length 216)
    192.168.1.179.49154 > 255.255.255.255.6667: UDP, length 188
10:16:43.518890 IP (tos 0x0, ttl 240, id 48612, offset 0, flags [DF], proto TCP (6), length 52)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [S.], cksum 0x7c8f (correct), seq 432772054, ack 1538819099, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
10:16:43.577916 IP (tos 0x0, ttl 240, id 48616, offset 0, flags [DF], proto TCP (6), length 348)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [P.], cksum 0xf719 (correct), seq 4141:4449, ack 207, win 2047, length 308
10:16:43.577993 IP (tos 0x0, ttl 240, id 48613, offset 0, flags [DF], proto TCP (6), length 1420)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [.], cksum 0x0cc9 (correct), seq 1:1381, ack 207, win 2047, length 1380
10:16:43.578433 IP (tos 0x0, ttl 240, id 48614, offset 0, flags [DF], proto TCP (6), length 1420)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [.], cksum 0x8d63 (incorrect -> 0x90a2), seq 1381:2761, ack 207, win 2047, length 1380
10:16:43.578801 IP (tos 0x0, ttl 240, id 48615, offset 0, flags [DF], proto TCP (6), length 1420)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [.], cksum 0x8d63 (incorrect -> 0xfc0c), seq 2761:4141, ack 207, win 2047, length 1380
10:16:43.584822 IP (tos 0x0, ttl 112, id 54591, offset 0, flags [DF], proto TCP (6), length 87)
    52.114.77.99.443 > 192.168.1.136.55906: Flags [P.], cksum 0xa154 (correct), seq 909934191:909934238, ack 2735617736, win 2044, length 47
10:16:43.637840 IP (tos 0x0, ttl 240, id 48617, offset 0, flags [DF], proto TCP (6), length 366)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [P.], cksum 0x1add (correct), seq 4449:4775, ack 365, win 2047, length 326
10:16:43.679492 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.4 tell 192.168.1.136, length 46
10:16:43.913946 IP (tos 0x0, ttl 240, id 48618, offset 0, flags [DF], proto TCP (6), length 366)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [P.], cksum 0x1add (correct), seq 4449:4775, ack 365, win 2047, length 326
10:16:44.018803 IP (tos 0x0, ttl 255, id 5744, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.192.49177 > 255.255.255.255.6666: UDP, length 175
10:16:44.145886 IP (tos 0x0, ttl 240, id 48619, offset 0, flags [DF], proto TCP (6), length 40)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [.], cksum 0x9bec (correct), ack 1745, win 2048, length 0
10:16:44.205904 IP (tos 0x0, ttl 240, id 48620, offset 0, flags [DF], proto TCP (6), length 40)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [.], cksum 0x9124 (correct), ack 4505, win 2048, length 0
10:16:44.277184 IP (tos 0x0, ttl 255, id 1526, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.156.49159 > 255.255.255.255.6666: UDP, length 175
10:16:44.285779 IP (tos 0x0, ttl 255, id 41308, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.191.49178 > 255.255.255.255.6666: UDP, length 175
10:16:44.288928 IP (tos 0x0, ttl 240, id 48621, offset 0, flags [DF], proto TCP (6), length 1234)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [P.], cksum 0x4d76 (correct), seq 4775:5969, ack 4741, win 2047, length 1194
10:16:44.289268 IP (tos 0x0, ttl 240, id 48622, offset 0, flags [DF], proto TCP (6), length 74)
    52.98.145.82.443 > 192.168.1.136.56155: Flags [P.], cksum 0xf385 (correct), seq 5969:6003, ack 4741, win 2047, length 34
10:16:44.387022 IP (tos 0x0, ttl 52, id 61416, offset 0, flags [DF], proto TCP (6), length 80)
    139.59.210.197.443 > 192.168.1.136.63898: Flags [P.], cksum 0x5252 (correct), seq 156749949:156749989, ack 3271008670, win 3509, length 40
10:16:44.387087 IP (tos 0x0, ttl 52, id 61417, offset 0, flags [DF], proto TCP (6), length 83)
    139.59.210.197.443 > 192.168.1.136.63898: Flags [P.], cksum 0xe04d (correct), seq 40:83, ack 1, win 3509, length 43
10:16:44.387135 IP (tos 0x0, ttl 52, id 61418, offset 0, flags [DF], proto TCP (6), length 78)
    139.59.210.197.443 > 192.168.1.136.63898: Flags [P.], cksum 0x6898 (correct), seq 83:121, ack 1, win 3509, length 38
10:16:44.531978 IP (tos 0x0, ttl 255, id 40292, offset 0, flags [none], proto UDP (17), length 216)
    192.168.1.120.49154 > 255.255.255.255.6667: UDP, length 188
10:16:44.637001 IP (tos 0x0, ttl 255, id 1070, offset 0, flags [none], proto UDP (17), length 203)
    192.168.1.119.49160 > 255.255.255.255.6666: UDP, length 175
10:16:44.671777 IP (tos 0x0, ttl 226, id 47699, offset 0, flags [DF], proto UDP (17), length 72)
    34.249.116.47.87 > 192.168.1.223.49239: UDP, length 44
10:16:44.682742 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.4 tell 192.168.1.136, length 46
10:16:44.690781 IP (tos 0x0, ttl 240, id 56311, offset 0, flags [DF], proto TCP (6), length 83)
    52.97.202.66.443 > 192.168.1.136.54851: Flags [P.], cksum 0xf012 (correct), seq 1044231199:1044231242, ack 569635196, win 2048, length 43
10:16:45.500987 IP (tos 0x0, ttl 255, id 39940, offset 0, flags [none], proto UDP (17), length 216)
    192.168.1.220.49154 > 255.255.255.255.6667: UDP, length 188
10:16:45.535607 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.220 tell 192.168.1.220, length 28

See also this with running 'ping 8.8.8.8' from 192.168.1.136 machine:

root@OpenWrt:~# tcpdump -vpni veth0 |grep 192.168.1.136
tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 280, length 40
10:43:23.706314 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.4 tell 192.168.1.136, length 46
    139.59.210.197.443 > 192.168.1.136.58451: Flags [P.], cksum 0x9393 (correct), seq 1717289715:1717289755, ack 3327456974, win 501, length 40
    139.59.210.197.443 > 192.168.1.136.58451: Flags [P.], cksum 0xe5fd (correct), seq 40:83, ack 1, win 501, length 43
    139.59.210.197.443 > 192.168.1.136.58451: Flags [P.], cksum 0x1994 (correct), seq 83:121, ack 1, win 501, length 38
    8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 281, length 40
10:43:24.710640 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.4 tell 192.168.1.136, length 46
    52.97.133.162.443 > 192.168.1.136.63498: Flags [P.], cksum 0xc63c (correct), seq 864246506:864246549, ack 490710447, win 2048, length 43
    66.102.1.189.443 > 192.168.1.136.59144: UDP, length 25
    8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 282, length 40
    20.54.36.229.443 > 192.168.1.136.63688: Flags [P.], cksum 0x066e (correct), seq 2468990717:2468990891, ack 636567286, win 6816, length 174
    8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 283, length 40
    188.240.145.40.6667 > 192.168.1.136.61536: Flags [P.], cksum 0x0a95 (correct), seq 2510783949:2510784005, ack 1861430789, win 501, length 56
    8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 284, length 40
    52.114.77.99.443 > 192.168.1.136.50444: Flags [P.], cksum 0x18d0 (correct), seq 2703409065:2703409112, ack 449886654, win 2047, length 47
    172.217.23.234.443 > 192.168.1.136.64018: UDP, length 79
^C34 packets captured
50 packets received by filter
0 packets dropped by kernel

Seems like it is getting all the inbound traffic (source + dest), but not the outbound traffic (source + dest)?

Lynx · September 18, 2021, 5:29pm

@moeller0, @vgaetera and @dlakelan - I thought I found a really cool and easy solution, but I am not sure if it is working. Please can you let me know what you think?

Steps taken:

1. create 'veth0' with peer 'veth1'
2. add 'veth1' to 'br-lan'
3. change LAN device from 'br-lan' to 'veth0'
4. create new interface named 'br_lan' and assign device 'br-lan'

This seems to have worked and I see all the right flows on 'veth0'.

See e.g.:

tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
    192.168.1.168 > 8.8.8.8: ICMP echo request, id 1, seq 274, length 40
    8.8.8.8 > 192.168.1.168: ICMP echo reply, id 1, seq 274, length 40
    192.168.1.168 > 8.8.8.8: ICMP echo request, id 1, seq 275, length 40
    8.8.8.8 > 192.168.1.168: ICMP echo reply, id 1, seq 275, length 40
    192.168.1.168 > 8.8.8.8: ICMP echo request, id 1, seq 276, length 40
    8.8.8.8 > 192.168.1.168: ICMP echo reply, id 1, seq 276, length 40
    192.168.1.168 > 8.8.8.8: ICMP echo request, id 1, seq 277, length 40
    8.8.8.8 > 192.168.1.168: ICMP echo reply, id 1, seq 277, length 40
164 packets captured
170 packets received by filter
0 packets dropped by kernel

However to my surprise it seems uploads and downloads on my LAN network are getting throttled - see:

root@OpenWrt:~# tc qdisc ls
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc noqueue 0: dev wan root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev veth1 root refcnt 2
qdisc cake 8028: dev veth0 root refcnt 2 bandwidth 5Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 60
qdisc ingress ffff: dev veth0 parent ffff:fff1 ----------------
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev WireGuard root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 8029: dev ifb4veth0 root refcnt 2 bandwidth 30Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 60

What am I missing?

vgaetera · September 18, 2021, 5:40pm

You can ask me about PBR, but I have no practical experience with traffic shaping and veth interfaces.

moeller0 · September 18, 2021, 6:09pm

You can ask me about traffic shaping, but I have no first hand experience with PBR and veth interfaces.

Lynx · September 18, 2021, 6:13pm

@moeller0 OK. Just shaping on br-lan - how would you prevent throttling LAN to LAN flows?

Because I just tested and if br-lan is set as the interface it seems my LAN traffic is shaped... but I thought from the above only wlan <> lan traffic would get shaped?

moeller0 · September 18, 2021, 6:23pm

Well, that is a configuration i would only recommend for a wired only router with a single used LAN interface in the first place. But a shaper on br-lan will not magically shape all traffic going over the switch, (unless everything is piped through the CPU-port), but it will affect LAN to/from WiFi traffic, hence br-lan is not a good target for SQM on a full fledged WiFi router.

That should only be the case if all traffic is piped through the switch's CPU port...

Remind me again, what router make/model do you use and what OpenWrt version? Specifically is the switch already using the modern DSA model?

Lynx · September 18, 2021, 6:25pm

I really really value your thoughts here.

I am using the RT3200 with snapshot: r17517-46dec9952b:

What does that mean here?

Lynx · September 18, 2021, 6:39pm

@moeller0 does this help:

root@OpenWrt:~# cat /etc/board.json
{
        "model": {
                "id": "linksys,e8450-ubi",
                "name": "Linksys E8450 (UBI)"
        },
        "led": {
                "wan": {
                        "name": "WAN",
                        "sysfs": "inet:blue",
                        "type": "netdev",
                        "device": "wan",
                        "mode": "link tx rx"
                }
        },
        "network": {
                "lan": {
                        "ports": [
                                "lan1",
                                "lan2",
                                "lan3",
                                "lan4"
                        ],
                        "protocol": "static"
                },
                "wan": {
                        "device": "wan",
                        "protocol": "dhcp"
                }
        }
}

moeller0 · September 18, 2021, 7:41pm

Yes it does, the "ports" part indicates DSA is in use here, which might effect the scope of instantiating a shaper on the bridge... but my own device is still based on OpenWrt 19.07, so I have no first hand experience yet with the potential different scoping of a bridge.

In the past this was mostly solved by the CPU port having its own name, e.g. eth1, so that a shaper on eth1 only affected traffic between CPU and switch but not between the switch ports....

Lynx · September 18, 2021, 8:21pm

@moeller0 a clever individual on #OpenWrt gave me the idea to set up br-wlan and then attach that to br-lan using veth. So then wlan0 and wlan1 send traffic to br-wlan.

But again this is failing for me because CAKE is shaping at least the LAN flow to and from my router on br-lan.

I am sure there is a solution using some form of 'veth' / 'br-lan' combination here.

Lynx · September 19, 2021, 6:41am

@vgaetera OK I have a routing question for you. With 'veth1' attached to 'br-lan', I would like to take off the wan and WireGuard related ingress and egress from 'br-lan' and have it directed to and go from 'veth0'. So this way CAKE can be placed on 'veth0' and it sees all ingress and egress relating to wan and WireGuard.

For packet destined from WireGuard or WAN for 'br-lan' is redirected to 'veth0'. Packet from 'br-lan' to WireGuard or WAN is somehow made to be egress from 'veth0'?

Is something like that possible?

I am conscious of:

root@OpenWrt:~# ip rule
0:      from all lookup local
32764:  from all fwmark 0x20000/0xff0000 lookup WireGuard
32765:  from all fwmark 0x10000/0xff0000 lookup wan
32766:  from all lookup main
32767:  from all lookup default
root@OpenWrt:~# ip route show table WireGuard
default via 10.5.0.2 dev WireGuard
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
root@OpenWrt:~# ip route show table wan
default via 10.0.0.1 dev wan
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

I can see how by changing 'br-lan' to 'veth0' in the 'WireGuard' and 'wan' tables, I would deal with redirecting the inbound flow from 'br-lan' to 'veth0'.

But how to deal with the outbound flow I am less certain. Can I make it so that flow that would normally egress from 'br-lan' to 'wan' or 'WireGuard' instead goes to 'veth0' and egresses from there? Would that make sense?

Maybe there is an easy fix here I am not thinking of. Like an intercept or something.

vgaetera · September 19, 2021, 8:42am

It's problematic to add any custom routing rules while using the PBR app.
The app creates its own rules always on top of all existing ones.
You should consider switching to PBR with netifd.

Lynx · September 19, 2021, 9:52am

Thank you for this suggestion. I looked but perhaps a bit beyond my skill set for now.

@moeller0, I just found out via experimentation that if I set LAN to veth0 with veth1 part of br-lan then the only unwanted additional shaping is between any LAN device and my router and back. As a bonus if I shape on veth1 then the downloads and uploads are in the right direction.

veth1 sees all wan and WireGuard ingress/egress.

So this seems like a promising way to deal with VPN PBR and SQM for those like me that have a more limited skill set. Just create veth device, attach veth1 to br-lan, create new br_lan interface set to device br-lan, and set LAN to veth0 rather than br-lan. Then shape on veth1.

@vgaetera the only missing piece of the jigsaw is LAN-originating ingress/egress on LAN (e.g. between lan1 device and my router) on veth0. This I think would not be affected by VPN PBR because it's local traffic. Can you think of a way to redirect incoming and outbound LAN traffic to br-lan even though veth0 is set as the LAN interface? So packet from lan1 destined for router goes to br-lan and not veth0 and packet that would otherwise exit from veth0 instead exits from br-lan?

dlakelan · September 19, 2021, 3:48pm

you can do a custom routing rule saying traffic coming from your router's IP address goes via br-lan, this will bypass the shaper on veth0.

Lynx · September 19, 2021, 4:37pm

Excited to try this when I get home!

@dlakelan will that work for flow in both directions? And can this be set in LuCi? Would you be able to elaborate? Sorry for my ignorance.

root@OpenWrt:~# ip route
default via 10.0.0.1 dev wan proto static src 10.2.249.211
10.0.0.0/8 dev wan proto kernel scope link src 10.2.249.211
81.92.203.202 via 10.0.0.1 dev wan proto static
192.168.1.0/24 dev veth0 proto kernel scope link src 192.168.1.1

root@OpenWrt:~# ip route show table local
broadcast 10.0.0.0 dev wan proto kernel scope link src 10.2.249.211
local 10.2.249.211 dev wan proto kernel scope host src 10.2.249.211
local 10.5.0.2 dev WireGuard proto kernel scope host src 10.5.0.2
broadcast 10.255.255.255 dev wan proto kernel scope link src 10.2.249.211
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev veth0 proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev veth0 proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev veth0 proto kernel scope link src 192.168.1.1

Would the idea be to change 'veth0' to 'br-lan' on the 'main' and 'local' tables (last lines)?

Maybe this shouldn't be done in LuCi but in a config file or so? Think LuCi can only configure 'proto static' not 'proto kernel'.

vgaetera · September 19, 2021, 10:04pm

There're several examples utilizing custom routing tables and rules.
That should isolate each interface in a separate routing table.
You just need to add default and custom routing rules.

Lynx · September 25, 2021, 11:40am

@vgaetera OK I think I am pretty close now!

It's problematic to add any custom routing rules while using the PBR app.
The app creates its own rules always on top of all existing ones.
You should consider switching to PBR with netifd .

Agreed, and also see:

So I have adopted PBR with netifd.

I have added 'veth1' to 'br-lan' and am seeking to have all incoming/outgoing wan/wireguard traffic go via 'veth0':

I have the following rules:

root@OpenWrt:~# ip rule
0:      from all lookup local
100:    from all iif WireGuard lookup 100
100:    from all iif wan lookup 100
10000:  from 192.168.1.1 lookup 1
10000:  from 10.36.202.233 lookup 2
20000:  from all to 192.168.1.1/24 lookup 1
20000:  from all to 10.36.202.233/8 lookup 2
32766:  from all lookup main
32767:  from all lookup default
40000:  from all iif br-lan lookup 2
90007:  from all iif lo lookup 2
90033:  from all iif lo lookup 1

Here are the relevant tables:

root@OpenWrt:~# ip route show table 1
192.168.1.0/24 dev br-lan scope link
root@OpenWrt:~# ip route show table 2
default via 10.0.0.1 dev wan  src 10.36.202.233
10.0.0.0/8 dev wan scope link
192.168.8.1 dev wan scope link
root@OpenWrt:~# ip route show table 100
default dev veth0 scope link

Now all traffic from wan or wireguard is correctly routed through 'veth0':

root@OpenWrt:~# tcpdump -vpni veth0 |grep -i icmp
tcpdump: listening on veth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:39:05.858622 IP (tos 0x0, ttl 117, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 15414, length 40
11:39:06.867636 IP (tos 0x0, ttl 117, id 0, offset 0, flags [none], proto ICMP (1), length 60)
8.8.8.8 > 192.168.1.136: ICMP echo reply, id 1, seq 15415, length 40

So the missing link is as follows: how can I get all the outgoing traffic destined for wan or wireguard from 'br-lan' to go via 'veth0' so that it appears on 'veth0'?