I'm trying to test mwan3 on a Tp-Link Archer A7 v5. My primary WAN is AT&T fiber where I'm using an Asus RT-AX86U router connected to the Fiber router. My secondary WAN is Xfinity and I'm using an Archer A7 with OpenWrt 22.03.5 r20134-5f15225c1e connected to a cable modem. The Archer A7 will eventually replace the RT-AX86U, but right now I don't want to disturb my current LAN so I only want to make minimal changes on the RT-AX86U and the fiber network so it continues to function normally.
To accomplish that, the only change on the RT-AX86U is to setup a DMZ with a static IP Address, 192.168.1.2. And then when testing Archer A7 configured with mwan3 I connect to Xfinity using the "wan" and configured as a DHCP client. I then created a new device, eth0.3 with a unique MAC address B4:B0:24:D1:28:52 and a new interface, wanb with the static IP address of 192.168.1.2 and it is connected to the RT-AX86U LAN. Thus it becomes the DMZ device. With these and some other changes, see my /etc/config/network below, mwan3 failover is working.
root@mwan3-2850:~# cat /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdf8:a2d5:f9df::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0.1'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option ipaddr '192.168.3.1'
config device
option name 'eth0.2'
option macaddr 'b4:b0:24:d1:28:51'
config interface 'wan'
option device 'eth0.2'
option proto 'dhcp'
option metric '10'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '0t 3 4 5'
option vid '1'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '0t 1'
option vid '2'
config switch_vlan
option device 'switch0'
option vlan '3'
option ports '0t 2'
option vid '3'
config interface 'wanb'
option device 'eth0.3'
option proto 'static'
option netmask '255.255.255.0'
option gateway '192.168.1.1'
option ipaddr '192.168.1.2'
option metric '20'
list dns '8.8.8.8'
list dns '8.8.4.4'
config device
option name 'eth0.3'
option type '8021q'
option ifname 'eth0'
option vid '3'
option macaddr 'B4:B0:24:D1:28:52'
The following test is "proof" to me it's working. The MultiWAN Manager overview shows both connections green and online. The wan connected to Xfinity and wanb connected to AT&T via the RT-AX86U. I now use traceroute to ucsc.edu and we see the first hop, 192.168.3.1, is the LAN connection on the Archer A7 and the second hop, 96.120.89.121, is the wan connection to Xfinity:
$ traceroute -n ucsc.edu
traceroute to ucsc.edu (128.114.119.88), 30 hops max, 60 byte packets
1 192.168.3.1 0.501 ms 0.751 ms 0.743 ms
2 96.120.89.121 9.179 ms 14.202 ms 14.195 ms
3 96.110.102.201 14.056 ms 14.184 ms 14.172 ms
4 68.85.154.137 14.166 ms 14.159 ms 14.156 ms
5 69.139.199.205 14.152 ms 14.164 ms 14.160 ms
6 96.108.99.153 19.030 ms 18.590 ms 18.348 ms
7 96.108.99.102 15.403 ms 12.442 ms 13.192 ms
8 50.233.65.106 14.835 ms 14.812 ms 13.443 ms
9 128.114.3.132 12.998 ms 12.983 ms 12.974 ms
10 128.114.102.217 13.196 ms 13.187 ms 13.173 ms
11 128.114.101.117 13.367 ms 12.187 ms 14.043 ms
12 128.114.102.228 14.012 ms 13.998 ms 11.423 ms
13 128.114.119.88 12.075 ms 12.040 ms 12.027 ms
If I now unplug the wan connection (Xfinity) and wait until the wan is red and offline, the traceroute output is different. The first hop is still 192.168.3.1, but the second hop, 192.168.1.1, is the RT-AX86U router, the third hop, 192.168.2.254, is the AT&T Router and the fourth hop, 107.199.76.1, is the internet side on AT&T:
$ traceroute -n ucsc.edu
traceroute to ucsc.edu (128.114.119.88), 30 hops max, 60 byte packets
1 192.168.3.1 0.571 ms 0.813 ms 0.798 ms
2 192.168.1.1 3.361 ms 3.348 ms 3.327 ms
3 192.168.2.254 3.313 ms 3.300 ms 3.288 ms
4 107.199.76.1 3.966 ms 4.931 ms 4.918 ms
5 99.161.44.92 5.789 ms 5.774 ms 5.760 ms
6 * * *
7 32.130.25.177 15.833 ms 14.096 ms 13.109 ms
8 12.122.149.133 8.189 ms 8.176 ms 9.495 ms
9 * * *
10 * 4.69.219.65 9.277 ms *
11 4.15.122.46 9.234 ms 9.113 ms 10.130 ms
12 137.164.11.81 10.137 ms 9.946 ms 9.909 ms
13 137.164.3.49 18.599 ms 18.570 ms 18.555 ms
14 128.114.3.132 17.499 ms 17.653 ms 17.845 ms
15 128.114.3.121 16.700 ms 16.660 ms 16.946 ms
16 128.114.101.201 16.908 ms 16.996 ms 16.935 ms
17 128.114.102.228 15.676 ms 15.638 ms 15.829 ms
18 128.114.119.88 16.851 ms 16.374 ms 16.335 ms
So that looks good but now to the "problem"
Why does the verify outbound traffic work via wan/eth0.2:
root@mwan3-2850:~# ping -c 1 -I eth0.2 ucsc.edu
PING ucsc.edu (128.114.119.88): 56 data bytes
64 bytes from 128.114.119.88: seq=0 ttl=54 time=11.570 ms
--- ucsc.edu ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 11.570/11.570/11.570 ms
But FAILS using wanb/eth0.3:
root@mwan3-2850:~# ping -c 1 -I eth0.3 ucsc.edu
PING ucsc.edu (128.114.119.88): 56 data bytes
--- ucsc.edu ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
Even though "failover" appears to be "Working As Intended" I don't feel comfortable replacing the RT-AX86U with the Archer A7 until I have some understanding of why the "verify outbound traffic" test fails.