Hi, I'm trying to setup and test a mwan3 failover to lte configuration. I'm using a EG25G modem and things seem to be mostly working but the last bit where wan fails and it then routes the traffic over the other interface doesn't quite work. Here is the troubleshooting output. The odd thing is if I do a "ping -I wwan0 8.8.8.8" the ping works fine. But it I do "ping 8.8.8.8" it says network unreachable.
Thanks,
Chris
Software-Version
-------------------------------------------------
OpenWrt - OpenWrt 21.02.2 r16495-bf0c965af0
LuCI - git-22.046.85957-59c3392
Output of "ip a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 94:83:c4:15:b9:94 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9683:c4ff:fe15:b994/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br-lan state DOWN group default qlen 1000
link/ether 94:83:c4:15:b9:93 brd ff:ff:ff:ff:ff:ff
4: wwan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
link/none
inet 10.169.184.100/29 brd 10.169.184.103 scope global wwan0
valid_lft forever preferred_lft forever
inet6 fe80::1d6c:89df:d961:198b/64 scope link stable-privacy
valid_lft forever preferred_lft forever
7: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
link/ether 94:83:c4:15:b9:93 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9683:c4ff:fe15:b993/64 scope link
valid_lft forever preferred_lft forever
8: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 94:83:c4:15:b9:93 brd ff:ff:ff:ff:ff:ff
inet 192.168.8.1/24 brd 192.168.8.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fde8:bff8:60b2:10::1/60 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::9683:c4ff:fe15:b993/64 scope link
valid_lft forever preferred_lft forever
Output of "ip route show"
-------------------------------------------------
default via 10.169.184.101 dev wwan0 proto static metric 50
10.169.184.96/29 dev wwan0 proto static scope link metric 20
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
Output of "ip rule show"
-------------------------------------------------
0: from all lookup local
1002: from all iif wwan0 lookup 2
2002: from all fwmark 0x200/0x3f00 lookup 2
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
Output of "ip route list table 1-250"
-------------------------------------------------
Table 1: 192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
Table 2: default via 10.169.184.101 dev wwan0 proto static metric 50
10.169.184.96/29 dev wwan0 proto static scope link metric 20
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
Output of "iptables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 3875 packets, 441K bytes)
pkts bytes target prot opt in out source destination
59353 6531K mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 3696 packets, 428K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 TCPMSS tcp -- * eth0 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- * usb0 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- usb0 * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- * wwan0 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- wwan0 * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
Chain OUTPUT (policy ACCEPT 4143 packets, 2038K bytes)
pkts bytes target prot opt in out source destination
64056 18M mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 3462 packets, 1984K bytes)
pkts bytes target prot opt in out source destination
Chain mwan3_connected (2 references)
pkts bytes target prot opt in out source destination
4839 653K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected dst MARK or 0x3f00
Chain mwan3_hook (2 references)
pkts bytes target prot opt in out source destination
114K 23M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
8475 936K mwan3_ifaces_in all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
7536 593K mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
4872 349K mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
123K 24M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0x3f00
7405 2438K mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
Chain mwan3_iface_in_cell2 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- wwan0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- wwan0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* cell2 */ MARK xset 0x200/0x3f00
Chain mwan3_iface_in_wan (1 references)
pkts bytes target prot opt in out source destination
939 343K MARK all -- eth0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- eth0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00
Chain mwan3_ifaces_in (1 references)
pkts bytes target prot opt in out source destination
8312 889K mwan3_iface_in_wan all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1711 135K mwan3_iface_in_cell2 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
Chain mwan3_policy_default_poli (1 references)
pkts bytes target prot opt in out source destination
856 66624 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* unreachable */ MARK xset 0x3e00/0x3f00
Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
4872 349K mwan3_policy_default_poli all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00