All,
I recently upgraded my LinkSys WRT3200ACM from OpenWrt 19.07 to 21.02 (yes, I know, I am late to the party 
).
On 19.07 I had no issues with mwan3 at all, meaning if one of my both ISP links went down, conntrack was flushed and everything worked as usual but on the other link.
For some reason, I am unable to make this work again. I know that the WRT3200ACM migrated from swconfig to DSA, but I used as reference to set up mwan3 the documentation which specifies how to deal with devices with DSA 1.
Following my situation:
- 2 ISPs (Telekom, Vodafone, both have static external IPs [if that matters])
- Interface
wanis PPPoE and is Telekom (when onlinepppoe-wan) and is physically plugged into the WAN interface of the WRT3200ACM - PPPoE pass through is provided by my modem (DrayTek Vigor 165) which takes also care of applying the VLAN tag 7, which is required for Telekom (at least in Germany). The actual dial-in is performed from the WRT3200ACM
- Interface
wanbis a static address and is Vodafone. It is physically plugged into switch port 4. I removedlan4frombr-lancreated a seperate device configuration usingVLAN (802.1q)as device type. The VLAN tag is 109 (arbitrary chosen, not something that is required by the ISP) -
wanbis attached to a Fritzbox 6591 and is configured as an 'exposed host', thus does not know anything about the Fritzbox (the networking etc), but "directly" connects to the internet
So, to the actual problem. Once I unplug pppoe-wan, wanb stops working, although it seems to be working correctly. The other way round, however, works perfectly fine (pull wanb, then pppoe-wan remains working).
My default routing table looks like the following when both interfaces are up and running:
# both interfaces online
root@openwrt.example.com:~# ip r s
default via 192.168.244.18 dev pppoe-wan proto static metric 10
default via 192.168.217.229 dev lan4 proto static metric 20
192.168.244.18 dev pppoe-wan proto kernel scope link src 192.168.174.182
192.168.217.224/29 dev lan4 proto static scope link metric 20
172.31.2.0/24 dev br-lan proto kernel scope link src 172.31.2.1
172.31.3.0/24 via 172.31.2.3 dev br-lan proto static
172.31.4.0/24 via 172.31.2.4 dev br-lan proto static
172.31.5.0/24 via 172.31.2.5 dev br-lan proto static
172.31.6.0/24 via 172.31.2.6 dev br-lan proto static
172.31.10.0/24 via 172.31.2.3 dev br-lan proto static
172.31.100.0/24 via 172.31.2.3 dev br-lan proto static
root@openwrt.example.com:~#
Ping seems to work on both interfaces, but there the fun already starts. lan4 (wanb) loses for some reason the first three pings. This is not a one-off, but consistently happens whenever I start pinging any destination.
root@openwrt.example.com:~# ping -I lan4 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=4 ttl=57 time=36.693 ms
root@openwrt.example.com:~# ping -I pppoe-wan 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=118 time=20.916 ms
When pinging 1.1.1.1 and looking at the ICMP traffic with tcpdump -i lan4 icmp and dst host 1.1.1.1 I can only ever see outgoing the 4th and following pakets. Not the first three. What also confuses me are the different TTLs on the interfaces, but I don't know if that matters at all. pppoe-wan is a DSL connection and wanb is a cable connection (coax).
When I now unplug pppoe-wan the following can be observed:
# routing table looks good to me (other interface gone)
root@openwrt.example.com:~# ip r s
default via 192.168.217.229 dev lan4 proto static metric 20
192.168.217.224/29 dev lan4 proto static scope link metric 20
172.31.2.0/24 dev br-lan proto kernel scope link src 172.31.2.1
172.31.3.0/24 via 172.31.2.3 dev br-lan proto static
172.31.4.0/24 via 172.31.2.4 dev br-lan proto static
172.31.5.0/24 via 172.31.2.5 dev br-lan proto static
172.31.6.0/24 via 172.31.2.6 dev br-lan proto static
172.31.10.0/24 via 172.31.2.3 dev br-lan proto static
172.31.100.0/24 via 172.31.2.3 dev br-lan proto static
# pinging does not work
root@openwrt.example.com:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network unreachable
# route should be still good
root@openwrt.example.com:~# ip r g 8.8.8.8
8.8.8.8 via 192.168.217.229 dev lan4 src 192.168.217.230 uid 0
cache
Additionally, I get the very odd error message in the logs: br-lan: received packet on lan3 with own address as source address (addr:AA:BB:CC:DD:EE:FF, vlan:0)
I can confirm that AA:BB:CC:DD:EE:FF is definetively the MAC address of br-lan, but I have no idea what's happening there.
While trying to find a solution, I found some threads mentioning issues with the mvebu target, but these issue seem to happen starting from OpenWrt 22.03.
Only FYI, if it is not clear yet: throughout this post I replaced sensitive values with dummy values.
Following my complete /etc/network/config
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option ipv6 '0'
config globals 'globals'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option netmask '255.255.255.0'
option delegate '0'
option ipaddr '172.31.2.1'
option ipv6 '0'
config interface 'wan'
option device 'wan'
option metric '10'
option proto 'pppoe'
option username 'REDACTED'
option password 'REDACTED'
option ipv6 '0'
option peerdns '0'
option delegate '0'
config device
option name 'br-lan'
option type 'bridge'
list ports 'lan1'
list ports 'lan2'
list ports 'lan3'
config route
option target '172.31.3.0'
option gateway '172.31.2.3'
option netmask '255.255.255.0'
option interface 'lan'
config route
option target '172.31.4.0'
option gateway '172.31.2.4'
option netmask '255.255.255.0'
option interface 'lan'
config route
option target '172.31.5.0'
option gateway '172.31.2.5'
option netmask '255.255.255.0'
option interface 'lan'
config route
option target '172.31.6.0'
option gateway '172.31.2.6'
option netmask '255.255.255.0'
option interface 'lan'
config route
option target '172.31.10.0'
option gateway '172.31.2.3'
option netmask '255.255.255.0'
option interface 'lan'
config route
option target '172.31.100.0'
option gateway '172.31.2.3'
option netmask '255.255.255.0'
option interface 'lan'
config interface 'wanb'
option device 'lan4'
option metric '20'
option proto 'static'
option ipaddr '192.168.217.230'
option netmask '255.255.255.248'
option gateway '192.168.217.229'
option broadcast '192.168.217.231'
option delegate '0'
option force_link '0'
option macaddr 'C0:FF:EE:C0:FF:EE'
config device
option type '8021q'
option ifname 'lan4'
option vid '109'
option ipv6 '0'
option name 'lan4.109'
Complete iptables (yes, they are custom, firewall is disabled) - I left out the custom chains as they are merely logging and dropping/rejecting:
# Generated by iptables-save v1.8.7 on Fri Feb 24 23:41:45 2023
*nat
:PREROUTING ACCEPT [1959910:195634529]
:INPUT ACCEPT [145807:7100490]
:OUTPUT ACCEPT [596949:49268887]
:POSTROUTING ACCEPT [44258:2944309]
-A POSTROUTING -d 172.31.2.2/32 -j MASQUERADE
-A POSTROUTING -o pppoe-wan -j MASQUERADE
-A POSTROUTING -o lan4 -j MASQUERADE
COMMIT
# Completed on Fri Feb 24 23:41:45 2023
# Generated by iptables-save v1.8.7 on Fri Feb 24 23:41:45 2023
*raw
:PREROUTING ACCEPT [1103560812:1304656752507]
:OUTPUT ACCEPT [2233874:1043595569]
COMMIT
# Completed on Fri Feb 24 23:41:45 2023
# Generated by iptables-save v1.8.7 on Fri Feb 24 23:41:45 2023
*mangle
:PREROUTING ACCEPT [120851:47807189]
:INPUT ACCEPT [10862:2627089]
:FORWARD ACCEPT [109969:45136648]
:OUTPUT ACCEPT [10689:3497372]
:POSTROUTING ACCEPT [120650:48633126]
:mwan3_connected - [0:0]
:mwan3_hook - [0:0]
:mwan3_iface_in_wan - [0:0]
:mwan3_iface_in_wanb - [0:0]
:mwan3_ifaces_in - [0:0]
:mwan3_policy_balanced - [0:0]
:mwan3_policy_wan_only - [0:0]
:mwan3_policy_wan_wanb - [0:0]
:mwan3_policy_wanb_only - [0:0]
:mwan3_policy_wanb_wan - [0:0]
:mwan3_rules - [0:0]
-A PREROUTING -j mwan3_hook
-A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o lan4 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
-A OUTPUT -j mwan3_hook
-A mwan3_connected -m set --match-set mwan3_connected dst -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_ifaces_in
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark ! --mark 0x3f00/0x3f00 -j mwan3_connected
-A mwan3_iface_in_wan -i pppoe-wan -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wan -i pppoe-wan -m mark --mark 0x0/0x3f00 -m comment --comment wan -j MARK --set-xmark 0x100/0x3f00
-A mwan3_iface_in_wanb -i lan4 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wanb -i lan4 -m mark --mark 0x0/0x3f00 -m comment --comment wanb -j MARK --set-xmark 0x200/0x3f00
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wan
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wanb
-A mwan3_policy_balanced -m mark --mark 0x0/0x3f00 -m statistic --mode random --probability 0.50000000000 -m comment --comment "wanb 2 4" -j MARK --set-xmark 0x200/0x3f00
-A mwan3_policy_balanced -m mark --mark 0x0/0x3f00 -m comment --comment "wan 2 2" -j MARK --set-xmark 0x100/0x3f00
-A mwan3_policy_wan_only -m mark --mark 0x0/0x3f00 -m comment --comment "wan 3 3" -j MARK --set-xmark 0x100/0x3f00
-A mwan3_policy_wan_wanb -m mark --mark 0x0/0x3f00 -m comment --comment "wan 3 3" -j MARK --set-xmark 0x100/0x3f00
-A mwan3_policy_wanb_only -m mark --mark 0x0/0x3f00 -m comment --comment "wanb 2 2" -j MARK --set-xmark 0x200/0x3f00
-A mwan3_policy_wanb_wan -m mark --mark 0x0/0x3f00 -m comment --comment "wanb 2 2" -j MARK --set-xmark 0x200/0x3f00
-A mwan3_rules -s 172.31.4.9/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_wan
-A mwan3_rules -s 172.31.4.100/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_wan
-A mwan3_rules -s 172.31.4.20/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb
-A mwan3_rules -s 172.31.3.15/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb
-A mwan3_rules -s 172.31.4.101/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_wan
-A mwan3_rules -s 172.31.3.3/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_wan
-A mwan3_rules -s 172.31.3.33/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_wan
-A mwan3_rules -s 172.31.2.21/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_only
-A mwan3_rules -s 172.31.2.22/32 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_only
-A mwan3_rules -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb
COMMIT
# Completed on Fri Feb 24 23:41:45 2023
# Generated by iptables-save v1.8.7 on Fri Feb 24 23:41:45 2023
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [2206072:1036285101]
:chn_drop - [0:0]
:chn_log_forward_drop - [0:0]
:chn_log_forward_reject - [0:0]
:chn_log_input_drop - [0:0]
:chn_log_input_reject - [0:0]
:chn_log_output_drop - [0:0]
:chn_log_output_reject - [0:0]
:chn_reject - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i br-lan -j ACCEPT
-A INPUT -s 172.31.2.2/32 -d 255.255.255.255/32 -j chn_drop
-A INPUT -s 172.31.5.2/32 -d 255.255.255.255/32 -j chn_drop
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -s 172.31.2.0/24 -d 255.255.255.255/32 -j chn_drop
-A INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -j chn_drop
-A INPUT -s 172.31.2.0/24 -d 172.31.2.255/32 -j chn_drop
-A INPUT -s 172.31.2.0/24 -d 224.0.0.0/4 -j chn_drop
-A INPUT -j chn_log_input_reject
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
-A FORWARD -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -i br-lan -o pppoe-wan -j ACCEPT
-A FORWARD -i br-lan -o lan4 -j ACCEPT
-A FORWARD -i br-lan -o br-lan -j ACCEPT
-A FORWARD -j chn_log_forward_reject
-A chn_drop -j DROP
-A chn_log_forward_drop -m limit --limit 5/sec -j LOG --log-prefix "ipt_forward_drop: " --log-level 7
-A chn_log_forward_drop -j chn_drop
-A chn_log_forward_reject -m limit --limit 5/sec -j LOG --log-prefix "ipt_forward_reject: " --log-level 7
-A chn_log_forward_reject -j chn_reject
-A chn_log_input_drop -m limit --limit 3/sec -j LOG --log-prefix "ipt_input_drop: " --log-level 7
-A chn_log_input_drop -j chn_drop
-A chn_log_input_reject -m limit --limit 3/sec -j LOG --log-prefix "ipt_input_reject: " --log-level 7
-A chn_log_input_reject -j chn_reject
-A chn_log_output_drop -m limit --limit 5/sec -j LOG --log-prefix "ipt_output_drop: " --log-level 7
-A chn_log_output_drop -j chn_drop
-A chn_log_output_reject -m limit --limit 5/sec -j LOG --log-prefix "ipt_output_reject: " --log-level 7
-A chn_log_output_reject -j chn_reject
-A chn_reject -p tcp -j REJECT --reject-with tcp-reset
-A chn_reject -j REJECT --reject-with icmp-port-unreachable
COMMIT
Lastly my /etc/config/mwan3
config globals 'globals'
option mmx_mask '0x3F00'
option logging '1'
option loglevel 'notice'
option rtmon_interval '3'
config interface 'wan'
option enabled '1'
list family 'ipv4'
option track_method 'ping'
option size '56'
option max_ttl '60'
option check_quality '1'
option failure_latency '150'
option failure_loss '5'
option recovery_latency '75'
option recovery_loss '1'
option recovery_interval '3'
option down '3'
option failure_interval '5'
option timeout '3'
option interval '5'
option count '1'
option up '5'
list track_ip '8.8.8.8'
list track_ip '8.8.4.4'
list track_ip '4.2.2.2'
list track_ip '4.2.2.1'
option reliability '3'
option initial_state 'online'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
config interface 'wanb'
option enabled '1'
list family 'ipv4'
option track_method 'ping'
option size '56'
option max_ttl '60'
option check_quality '1'
option failure_latency '150'
option failure_loss '5'
option recovery_latency '75'
option recovery_loss '1'
option recovery_interval '3'
option down '3'
option failure_interval '5'
option timeout '3'
option interval '5'
option count '1'
option up '5'
option reliability '3'
list track_ip '8.8.8.8'
list track_ip '8.8.4.4'
list track_ip '4.2.2.2'
list track_ip '4.2.2.1'
option initial_state 'online'
list flush_conntrack 'connected'
list flush_conntrack 'disconnected'
config member 'wan_m1_w2'
option interface 'wan'
option metric '1'
option weight '2'
config member 'wan_m2_w2'
option interface 'wan'
option metric '1'
option weight '3'
config member 'wan_m1_w3'
option interface 'wan'
option metric '1'
option weight '3'
config member 'wan_m2_w3'
option interface 'wan'
option metric '2'
option weight '3'
config member 'wanb_m1_w2'
option interface 'wanb'
option metric '1'
option weight '2'
config member 'wanb_m2_w2'
option interface 'wan'
option metric '2'
option weight '2'
config rule 'rule01'
option src_ip '172.31.4.9/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_wan'
option family 'ipv4'
config rule 'rule02'
option src_ip '172.31.4.100/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_wan'
option family 'ipv4'
config rule 'rule03'
option src_ip '172.31.4.20/32'
option proto 'all'
option sticky '0'
option use_policy 'wan_wanb'
option family 'ipv4'
config rule 'rule04'
option src_ip '172.31.3.15/32'
option proto 'all'
option sticky '0'
option use_policy 'wan_wanb'
option family 'ipv4'
config rule 'rule05'
option src_ip '172.31.4.101/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_wan'
option family 'ipv4'
config rule 'rule06'
option src_ip '172.31.3.3/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_wan'
option family 'ipv4'
config rule 'rule07'
option src_ip '172.31.3.33/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_wan'
option family 'ipv4'
config rule 'rule08'
option src_ip '172.31.2.21/32'
option proto 'all'
option sticky '0'
option use_policy 'wan_only'
option family 'ipv4'
config rule 'rule09'
option src_ip '172.31.2.22/32'
option proto 'all'
option sticky '0'
option use_policy 'wanb_only'
option family 'ipv4'
config rule 'default'
option proto 'all'
option sticky '0'
option dest_ip '0.0.0.0/0'
option use_policy 'wan_wanb'
option family 'ipv4'
config policy 'wan_only'
list use_member 'wan_m1_w3'
option last_resort 'unreachable'
config policy 'wanb_only'
list use_member 'wanb_m1_w2'
option last_resort 'unreachable'
config policy 'balanced'
list use_member 'wan_m1_w2'
list use_member 'wanb_m1_w2'
option last_resort 'unreachable'
config policy 'wan_wanb'
list use_member 'wan_m1_w3'
list use_member 'wanb_m2_w2'
option last_resort 'unreachable'
config policy 'wanb_wan'
list use_member 'wan_m2_w3'
list use_member 'wanb_m1_w2'
option last_resort 'unreachable'
I really appreciate any hints or thoughts on this, as I am stuck and don't know how to proceed.
Thanks!