MWAN no Default Route for OpenVPN

Installing and Using OpenWrt
#1

I have to OpenVPN Interfaces, tun0 and tun1 which I try to fail over with mwan3 on OpenWRT 22.03. The interfaces have an error in mwan status

Interface status:
 interface wan is online 00h:01m:02s, uptime 121h:39m:20s and tracking is active
 interface wan2 is online 00h:01m:02s, uptime 154h:42m:27s and tracking is active
 interface tun0 is error (16) and tracking is active
 interface tun1 is error (16) and tracking is active
 interface wireguard is online 00h:01m:02s, uptime 121h:36m:42s and tracking is active

This is the troubleshooting output:

Software-Version
-------------------------------------------------
OpenWrt - 22.03.0

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.20/24 brd 192.168.1.255 scope global lan1
       valid_lft forever preferred_lft forever
7: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 213.47.14.55/24 brd 213.47.14.255 scope global wan
       valid_lft forever preferred_lft forever
10: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.123.1/24 brd 192.168.123.255 scope global br-lan
       valid_lft forever preferred_lft forever
14: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.22.0.8/16 scope global tun1
       valid_lft forever preferred_lft forever
19: wireguard: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.2.0.2/32 brd 255.255.255.255 scope global wireguard
       valid_lft forever preferred_lft forever
22: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.38.0.2/16 scope global tun0
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 213.47.14.1 dev wan proto static src 213.47.14.55 metric 10 
default via 192.168.1.1 dev lan1 proto static metric 20 
default dev wireguard proto static scope link metric 25 
default via 10.38.0.1 dev tun0 metric 55 
default via 10.22.0.1 dev tun1 metric 65 
10.2.0.2 dev wireguard proto static scope link metric 25 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
89.36.76.130 via 213.47.14.1 dev wan proto static metric 10 
89.36.76.130 via 192.168.1.1 dev lan1 proto static metric 20 
185.159.158.56 via 192.168.1.1 dev lan1 proto static metric 16 
185.159.158.106 via 192.168.1.1 dev lan1 proto static metric 15 
185.159.158.139 via 192.168.1.1 dev lan1 proto static metric 17 
192.168.1.0/24 dev lan1 proto static scope link metric 20 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 
213.47.14.0/24 dev wan proto static scope link metric 10 

Output of "ip -4 rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif wan lookup 1
1002:	from all iif lan1 lookup 2
1003:	from all iif wan lookup 3
1004:	from all iif lan1 lookup 4
1005:	from all iif wireguard lookup 5
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2003:	from all fwmark 0x300/0x3f00 lookup 3
2004:	from all fwmark 0x400/0x3f00 lookup 4
2005:	from all fwmark 0x500/0x3f00 lookup 5
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
3003:	from all fwmark 0x300/0x3f00 unreachable
3004:	from all fwmark 0x400/0x3f00 unreachable
3005:	from all fwmark 0x500/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 213.47.14.1 dev wan proto static src 213.47.14.55 metric 10 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
89.36.76.130 via 213.47.14.1 dev wan proto static metric 10 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 
213.47.14.0/24 dev wan proto static scope link metric 10 

Routing table 2:
default via 192.168.1.1 dev lan1 proto static metric 20 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
89.36.76.130 via 192.168.1.1 dev lan1 proto static metric 20 
185.159.158.56 via 192.168.1.1 dev lan1 proto static metric 16 
185.159.158.106 via 192.168.1.1 dev lan1 proto static metric 15 
185.159.158.139 via 192.168.1.1 dev lan1 proto static metric 17 
192.168.1.0/24 dev lan1 proto static scope link metric 20 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 3:
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 4:
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 5:
default dev wireguard proto static scope link metric 25 
10.2.0.2 dev wireguard proto static scope link metric 25 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
57213   50M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
13589 3491K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
23272   24M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 dst MARK or 0x3f00

Chain mwan3_custom_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 dst MARK or 0x3f00

Chain mwan3_dynamic_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
67296   54M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 1199  138K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1100  133K mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1100  133K mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  486 96178 mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  486 96178 mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
70803   54M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
30523   25M mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
30523   25M mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
 7865 1547K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_tun0 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* tun0 */ MARK xset 0x300/0x3f00

Chain mwan3_iface_in_tun1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    1    69 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    1    69 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* tun1 */ MARK xset 0x400/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   86  4538 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_in_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan2 */ MARK xset 0x200/0x3f00

Chain mwan3_iface_in_wireguard (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   11   790 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wireguard */ MARK xset 0x500/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1199  138K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1113  133K mwan3_iface_in_tun0  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1113  133K mwan3_iface_in_tun1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1111  133K mwan3_iface_in_wan2  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1109  133K mwan3_iface_in_wireguard  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_vpn_fo (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  287 63705 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wireguard 3 3 */ MARK xset 0x500/0x3f00

Chain mwan3_policy_wan_fo (5 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    4   128 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            80.64.136.37         mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            185.144.161.170      mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            185.242.177.6        mark match 0x0/0x3f00
    5   160 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            212.117.203.60       mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            193.22.104.2         mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            193.22.104.29        mark match 0x0/0x3f00
  453 92938 mwan3_policy_vpn_fo  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

If I try to ping the default gateway in mwan diagnostics it says

No gateway for interface tun0 found.

But apparently there is a default gateway

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.38.0.1       0.0.0.0         UG    55     0        0 tun0
default         10.22.0.1       0.0.0.0         UG    65     0        0 tun1

My OpenVPN config is a follows

client
dev tun0
proto udp
local 213.47.14.150
#route-nopull
#pull-filter ignore redirect-gateway
route-nopull
route 0.0.0.0 0.0.0.0 vpn_gateway 55

It used to work with 19.07.

Any help is appreciated!

#2

From a first glance the IP rules 1001-1004 are repeated and routing tables 3 and 4 don't have default gateway.

#3

Yes, thank you, I have fixed both. Now I get error 17 for tun0 and tun1.

Interface status:
 interface wan is online 00h:19m:05s, uptime 131h:25m:47s and tracking is active
 interface wan2 is online 00h:19m:05s, uptime 03h:18m:36s and tracking is active
 interface tun0 is error (17) and tracking is active
 interface tun1 is error (17) and tracking is active
 interface wireguard is online 00h:19m:05s, uptime 131h:23m:09s and tracking is active

Software-Version
-------------------------------------------------
OpenWrt - 22.03.0

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.20/24 brd 192.168.1.255 scope global lan1
       valid_lft forever preferred_lft forever
7: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 213.47.14.55/24 brd 213.47.14.255 scope global wan
       valid_lft forever preferred_lft forever
10: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.123.1/24 brd 192.168.123.255 scope global br-lan
       valid_lft forever preferred_lft forever
14: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.22.0.8/16 scope global tun1
       valid_lft forever preferred_lft forever
19: wireguard: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.2.0.2/32 brd 255.255.255.255 scope global wireguard
       valid_lft forever preferred_lft forever
22: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.38.0.2/16 scope global tun0
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 213.47.14.1 dev wan proto static src 213.47.14.55 metric 10 
default via 192.168.1.1 dev lan1 proto static metric 20 
default dev wireguard proto static scope link metric 25 
default via 10.38.0.1 dev tun0 metric 55 
default via 10.22.0.1 dev tun1 metric 65 
10.2.0.2 dev wireguard proto static scope link metric 25 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
89.36.76.130 via 213.47.14.1 dev wan proto static metric 10 
185.159.158.56 via 192.168.1.1 dev lan1 proto static metric 16 
185.159.158.106 via 192.168.1.1 dev lan1 proto static metric 15 
185.159.158.139 via 192.168.1.1 dev lan1 proto static metric 17 
192.168.1.0/24 dev lan1 proto static scope link metric 20 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 
213.47.14.0/24 dev wan proto static scope link metric 10 

Output of "ip -4 rule show"
-------------------------------------------------
0:	from all lookup local
999:	from all iif tun1 lookup 4
1000:	from all iif tun0 lookup 3
1001:	from all iif wan lookup 1
1002:	from all iif lan1 lookup 2
1005:	from all iif wireguard lookup 5
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2003:	from all fwmark 0x300/0x3f00 lookup 3
2004:	from all fwmark 0x400/0x3f00 lookup 4
2005:	from all fwmark 0x500/0x3f00 lookup 5
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
3003:	from all fwmark 0x300/0x3f00 unreachable
3004:	from all fwmark 0x400/0x3f00 unreachable
3005:	from all fwmark 0x500/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 213.47.14.1 dev wan proto static src 213.47.14.55 metric 10 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
89.36.76.130 via 213.47.14.1 dev wan proto static metric 10 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 
213.47.14.0/24 dev wan proto static scope link metric 10 

Routing table 2:
default via 192.168.1.1 dev lan1 proto static metric 20 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
185.159.158.56 via 192.168.1.1 dev lan1 proto static metric 16 
185.159.158.106 via 192.168.1.1 dev lan1 proto static metric 15 
185.159.158.139 via 192.168.1.1 dev lan1 proto static metric 17 
192.168.1.0/24 dev lan1 proto static scope link metric 20 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 3:
default via 10.38.0.1 dev tun0 metric 55 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 4:
default via 10.22.0.1 dev tun1 metric 65 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Routing table 5:
default dev wireguard proto static scope link metric 25 
10.2.0.2 dev wireguard proto static scope link metric 25 
10.22.0.0/16 dev tun1 proto kernel scope link src 10.22.0.8 
10.38.0.0/16 dev tun0 proto kernel scope link src 10.38.0.2 
192.168.123.0/24 dev br-lan proto kernel scope link src 192.168.123.1 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 106K  103M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
32713 7755K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
36294   48M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 dst MARK or 0x3f00

Chain mwan3_custom_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 dst MARK or 0x3f00

Chain mwan3_dynamic_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 134K  110M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 1785  206K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1622  196K mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1622  196K mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  865  150K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  865  150K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 138K  111M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
58446   51M mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
58446   51M mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
22909 2410K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_tun0 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* tun0 */ MARK xset 0x300/0x3f00

Chain mwan3_iface_in_tun1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* tun1 */ MARK xset 0x400/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    3   436 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   59  3414 MARK       all  --  wan    *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_in_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    2   138 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  lan1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan2 */ MARK xset 0x200/0x3f00

Chain mwan3_iface_in_wireguard (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   99  5660 MARK       all  --  wireguard *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wireguard */ MARK xset 0x500/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1785  206K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1723  202K mwan3_iface_in_wan2  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1721  202K mwan3_iface_in_tun0  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1721  202K mwan3_iface_in_tun1  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1721  202K mwan3_iface_in_wireguard  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_vpn_fo (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  235 38528 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wireguard 3 3 */ MARK xset 0x500/0x3f00

Chain mwan3_policy_wan_fo (5 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    32 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            80.64.136.37         mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            185.144.161.170      mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            185.242.177.6        mark match 0x0/0x3f00
    5   160 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            212.117.203.60       mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            193.22.104.2         mark match 0x0/0x3f00
    0     0 mwan3_policy_wan_fo  all  --  *      *       0.0.0.0/0            193.22.104.29        mark match 0x0/0x3f00
  834  147K mwan3_policy_vpn_fo  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

And if I restart tun0 or tun1 the old rules get added on top

0:	from all lookup local
999:	from all iif tun1 lookup 4
1000:	from all iif tun0 lookup 3
1001:	from all iif wan lookup 1
1002:	from all iif lan1 lookup 2
1003:	from all iif wan lookup 3
1004:	from all iif lan1 lookup 4
1005:	from all iif wireguard lookup 5
#4

A remark here is that 22.03 is working with nftables, so I am not sure the mix with iptables from mwan3 will work. What is the output of ubus call system board; uci export network; uci export mwan3 ?

#5

OK, here is the output

root@openwrt:~# ubus call system board
{
	"kernel": "5.10.138",
	"hostname": "openwrt",
	"system": "ARMv7 Processor rev 1 (v7l)",
	"model": "Linksys WRT1900ACS",
	"board_name": "linksys,wrt1900acs",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "22.03.0",
		"revision": "r19685-512e76967f",
		"target": "mvebu/cortexa9",
		"description": "OpenWrt 22.03.0 r19685-512e76967f"
	}
}

root@openwrt:~# uci export network
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd45:794c:23df::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	option ipv6 '0'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ipaddr '192.168.123.1'

config device
	option name 'wan'
	option macaddr '62:38:e0:da:40:ea'
	option ipv6 '0'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'
	option peerdns '0'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option metric '10'
	option delegate '0'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'lan1'
	option acceptlocal '1'
	option ipv6 '0'

config interface 'wan2'
	option proto 'static'
	option device 'lan1'
	option force_link '0'
	option ipaddr '192.168.1.20'
	option netmask '255.255.255.0'
	option gateway '192.168.1.1'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option metric '20'
	option delegate '0'

config interface 'wireguard'
	option proto 'wireguard'
	list addresses '10.2.0.2/32'
	option private_key 'XXX'
	option metric '25'
	option peerdns '0'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option delegate '0'

config wireguard_wireguard
	option description 'protonvpn'
	option public_key 'XXX'
	option endpoint_host '89.36.76.130'
	option endpoint_port '51820'
	option persistent_keepalive '25'
	list allowed_ips '0.0.0.0/0'
	option route_allowed_ips '1'

config interface 'tun0'
	option proto 'none'
	option device 'wan'
	option metric '30'
	option peerdns '0'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option delegate '0'
	option gateway '10.38.0.1'

config interface 'tun1'
	option proto 'none'
	option device 'lan1'
	option metric '40'
	option peerdns '0'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option delegate '0'
	option gateway '10.22.0.1'

config route
	option interface 'wan2'
	option gateway '192.168.1.1'
	option target '185.159.158.106/32'
	option metric '15'

config route
	option interface 'wan2'
	option gateway '192.168.1.1'
	option target '185.159.158.56/32'
	option metric '16'

config route
	option interface 'wan2'
	option gateway '192.168.1.1'
	option target '185.159.158.139/32'
	option metric '17'

config device
	option type '8021q'
	option ifname 'lan1'
	option vid '1'
	option name 'lan1.1'

config device
	option name 'wireguard'

config device
	option name 'tun0'

config device
	option name 'tun1'

config route
	option interface 'wan'
	option target '89.36.76.130/32'
	option gateway '213.47.14.1'

config route
	option interface 'wan'
	option target '194.126.177.8/0'
	option gateway '213.47.14.1'

config route
	option interface 'wan'
	option target '194.126.177.7/0'
	option gateway '213.47.14.1'


root@openwrt:~# uci export mwan3
package mwan3

config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wan'
	option enabled '1'
	option family 'ipv4'
	option initial_state 'online'
	option track_method 'ping'
	option reliability '1'
	option size '56'
	option max_ttl '60'
	option timeout '2'
	option down '3'
	option up '3'
	list track_ip '1.1.1.1'
	list track_ip '1.0.0.1'
	option count '3'
	option interval '3'
	option failure_interval '3'
	option recovery_interval '3'

config member 'wan_m2_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'

config policy 'wan_only'
	option last_resort 'unreachable'
	list use_member 'wan_m2_w3'

config interface 'wan2'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '2'
	option interval '1'
	option failure_interval '1'
	option recovery_interval '1'
	option down '3'
	option up '3'
	list track_ip '8.8.8.8'

config interface 'tun0'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '2'
	option interval '1'
	option failure_interval '1'
	option recovery_interval '1'
	option down '3'
	option up '3'
	list flush_conntrack 'ifup'
	list flush_conntrack 'ifdown'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'
	list track_ip '8.8.4.4'

config interface 'tun1'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '2'
	option interval '1'
	option failure_interval '1'
	option recovery_interval '1'
	option down '3'
	option up '3'
	list flush_conntrack 'ifup'
	list flush_conntrack 'ifdown'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'
	list track_ip 'univie.ac.at'

config interface 'wireguard'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '2'
	option interval '1'
	option failure_interval '1'
	option recovery_interval '1'
	option down '3'
	option up '3'
	list track_ip '8.8.8.8'
	list track_ip '8.8.4.4'

config member 'wan2_m5_w10'
	option interface 'wan2'
	option metric '5'
	option weight '10'

config member 'tun0_m5_w3'
	option interface 'tun0'
	option metric '5'
	option weight '3'

config member 'tun1_m10_w10'
	option interface 'tun1'
	option metric '10'
	option weight '10'

config member 'wg_m3_w3'
	option interface 'wireguard'
	option metric '3'
	option weight '3'

config policy 'wan_fo'
	list use_member 'wan_m2_w3'
	list use_member 'wan2_m5_w10'
	option last_resort 'unreachable'

config rule 'ntp1'
	option proto 'all'
	option dest_ip '80.64.136.37'
	option sticky '0'
	option use_policy 'wan_fo'
	option family 'ipv4'

config rule 'ntp2'
	option proto 'all'
	option sticky '0'
	option use_policy 'wan_fo'
	option family 'ipv4'
	option dest_ip '185.144.161.170'

config rule 'ntp3'
	option proto 'all'
	option dest_ip '185.242.177.6'
	option sticky '0'
	option use_policy 'wan_fo'
	option family 'ipv4'

config rule 'sip_wan'
	option proto 'all'
	option dest_ip '212.117.203.60'
	option sticky '0'
	option use_policy 'wan_only'
	option family 'ipv4'

config rule 'willhben_wan'
	option proto 'all'
	option dest_ip '193.22.104.2'
	option sticky '0'
	option use_policy 'wan_fo'
	option family 'ipv4'

config rule 'wlhbn_sso_wan'
	option proto 'all'
	option dest_ip '193.22.104.29'
	option sticky '0'
	option use_policy 'wan_fo'
	option family 'ipv4'

config rule 'vpn_fovr'
	option proto 'all'
	option dest_ip '0.0.0.0/0'
	option sticky '0'
	option family 'ipv4'
	option use_policy 'vpn_fo'

config policy 'vpn_fo'
	option last_resort 'unreachable'
	list use_member 'wg_m3_w3'
	list use_member 'tun0_m5_w3'
	list use_member 'tun1_m10_w10'

I get a message that there are iptabels-legacy rules present, although I have deleted the package and flushes the rules with

iptables -F
iptables -X

root@openwrt:~# mwan3 status
Interface status:
 interface wan is online 00h:05m:31s, uptime 00h:09m:28s and tracking is active
 interface wan2 is online 00h:05m:31s, uptime 00h:09m:30s and tracking is active
 interface tun0 is error (16) and tracking is active
 interface tun1 is error (16) and tracking is active
 interface wireguard is online 00h:05m:31s, uptime 00h:09m:30s and tracking is active

Current ipv4 policies:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
vpn_fo:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 wireguard (100%)
wan_fo:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 wan (100%)
wan_only:
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# Warning: iptables-legacy tables present, use iptables-legacy to see them
 wan (100%)

Current ipv6 policies:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
vpn_fo:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
 unreachable
wan_fo:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
 unreachable
wan_only:
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
 unreachable
#6

OK, I have decided to do away with the legacy OpenVPN interfaces and added a second wirguard interface which gets routed through wan2.

This works, but only if I add the default route manually, if not mwan gives error 16.

root@openwrt:~# ip route add default dev wireguardzwei proto static scope link metric 36
root@openwrt:~# ip route add table 3 default dev wireguardzwei proto static scope link metric 36

Why does the default route get added for the first wireguard interface but not for the second?

#7

The netmasks here are wrong.

Have you added the option route_allowed_ips '1' on the peer configuration for wg2?

1 Like
#8

True, I fixed that.

That seemed to be the problem. Both wireguard interfaces now track.

Thank you so much!

1 Like
#9

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.

#10

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.