I have a simple setup my OpenWrt router have full-tunnel connections to a tailscale exit node, and a wireguard VPN server. My goal is to load balance internet-bound traffic between these two tunnel.
I have configured the mwan3 to load balance the traffic like in the screenshots shown below:
Interface status:
interface wgclient0 is online 00h:50m:59s, uptime 20h:09m:46s and tracking is active
interface tailscale is error (16) and tracking is active
Current ipv4 policies:
balanced:
tailscale (50%)
wgclient0 (50%)
Current ipv6 policies:
balanced:
default
Directly connected ipv4 networks:
127.0.0.0
192.168.1.255
10.6.40.255
224.0.0.0/3
10.1.50.255
103.116.9.168
192.168.1.243
192.168.1.0
10.6.40.8
10.6.40.0/24
10.6.40.0
10.1.50.0/24
192.168.1.0/24
127.0.0.0/8
127.255.255.255
10.1.50.0
10.1.50.1
100.112.23.141
127.0.0.1
Directly connected ipv6 networks:
fde5:d094:8763::/64
fe80::/64
fd7a:115c:a1e0:ab12:4843:cd96:6270:178d
Active ipv4 user rules:
15486 1325K - balanced all -- * * 0.0.0.0/0 0.0.0.0/0
Active ipv6 user rules:
********************************************
Running 'ping -c 5 -W 1 1.0.0.1' with DEVICE=tailscale0 SRCIP=100.112.23.141 FWMARK=0x3f00 FAMILY=ipv4
PING 1.0.0.1 (1.0.0.1): 56 data bytes
64 bytes from 1.0.0.1: seq=0 ttl=57 time=361.101 ms
64 bytes from 1.0.0.1: seq=1 ttl=57 time=360.850 ms
64 bytes from 1.0.0.1: seq=2 ttl=57 time=360.994 ms
64 bytes from 1.0.0.1: seq=3 ttl=57 time=360.877 ms
64 bytes from 1.0.0.1: seq=4 ttl=57 time=360.638 ms
--- 1.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 360.638/360.892/361.101 ms
Running 'ping -c 5 -W 1 1.1.1.1' with DEVICE=tailscale0 SRCIP=100.112.23.141 FWMARK=0x3f00 FAMILY=ipv4
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=56 time=390.325 ms
64 bytes from 1.1.1.1: seq=1 ttl=56 time=390.035 ms
64 bytes from 1.1.1.1: seq=2 ttl=56 time=389.848 ms
64 bytes from 1.1.1.1: seq=3 ttl=56 time=389.807 ms
64 bytes from 1.1.1.1: seq=4 ttl=56 time=389.662 ms
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 389.662/389.935/390.325 ms
The problem is that during my testing it appears that mwan3 always uses the wireguard tunnel but not the tailscale one.
Question: What did I miss? Or is this even possible?
Below is the logs:
Software-Version
-------------------------------------------------
OpenWrt - 22.03.5
Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet 192.168.1.243/24 brd 192.168.1.255 scope global eth1
valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.1.50.1/24 brd 10.1.50.255 scope global br-lan
valid_lft forever preferred_lft forever
5: wgclient0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
inet 10.6.40.8/24 brd 10.6.40.255 scope global wgclient0
valid_lft forever preferred_lft forever
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
inet 100.112.23.141/32 scope global tailscale0
valid_lft forever preferred_lft forever
Output of "ip -4 route show"
-------------------------------------------------
default via 192.168.1.254 dev eth1 proto static src 192.168.1.243
default dev wgclient0 proto static scope link metric 2
10.1.50.0/24 dev br-lan proto kernel scope link src 10.1.50.1
10.6.40.0/24 dev wgclient0 proto static scope link metric 2
103.116.9.168 via 192.168.1.254 dev eth1 proto static
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.243
Output of "ip -4 rule show"
-------------------------------------------------
0: from all lookup local
1001: from all iif wgclient0 lookup 1
1002: from all iif tailscale0 lookup 2
1310: from all fwmark 0x80000/0xff0000 lookup main
1330: from all fwmark 0x80000/0xff0000 lookup default
1350: from all fwmark 0x80000/0xff0000 unreachable
1370: from all lookup 52
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3001: from all fwmark 0x100/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default dev wgclient0 proto static scope link metric 2
10.1.50.0/24 dev br-lan proto kernel scope link src 10.1.50.1
10.6.40.0/24 dev wgclient0 proto static scope link metric 2
103.116.9.168 via 192.168.1.254 dev eth1 proto static
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.243
Routing table 2:
10.1.50.0/24 dev br-lan proto kernel scope link src 10.1.50.1
103.116.9.168 via 192.168.1.254 dev eth1 proto static
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.243
Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
642K 482M mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
302K 41M mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain mwan3_connected_ipv4 (2 references)
pkts bytes target prot opt in out source destination
432K 257M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 dst MARK or 0x3f00
Chain mwan3_custom_ipv4 (2 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 dst MARK or 0x3f00
Chain mwan3_dynamic_ipv4 (2 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00
Chain mwan3_hook (2 references)
pkts bytes target prot opt in out source destination
922K 521M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
172K 40M mwan3_ifaces_in all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
172K 40M mwan3_custom_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
172K 40M mwan3_connected_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
15600 1334K mwan3_dynamic_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
15600 1334K mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
944K 524M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0x3f00
412K 237M mwan3_custom_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
412K 237M mwan3_connected_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
136K 18M mwan3_dynamic_ipv4 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
Chain mwan3_iface_in_tailscale (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
27 1404 MARK all -- tailscale0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* tailscale */ MARK xset 0x200/0x3f00
Chain mwan3_iface_in_wgclient0 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- wgclient0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- wgclient0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- wgclient0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
136 14637 MARK all -- wgclient0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* wgclient0 */ MARK xset 0x100/0x3f00
Chain mwan3_ifaces_in (1 references)
pkts bytes target prot opt in out source destination
172K 40M mwan3_iface_in_wgclient0 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
172K 40M mwan3_iface_in_tailscale all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
Chain mwan3_policy_balanced (1 references)
pkts bytes target prot opt in out source destination
354 26903 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 statistic mode random probability 0.50000000000 /* tailscale 3 6 */ MARK xset 0x200/0x3f00
323 24544 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* wgclient0 3 3 */ MARK xset 0x100/0x3f00
Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
15598 1334K mwan3_policy_balanced all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
To be honest, the Tailscale implementation in OpenWrt leaves much to be desired in terms of customization and this negatively affects its compatibility with PBR and MWAN.
Perhaps MWAN expects a default route for Tailscale with custom metric in the main table, or maybe it needs override the routing rule created by Tailscale.
