Mwan3 failover trouble in 21 or 22

Installing and Using OpenWrt
1

The setup is mwan3 with a main network interface, which fails over to the second backup interface only if the first goes down. Can't seem to get it to work properly in 21 or 22. I configure everything through the filesystem, no luci.

I've thrashed around for a couple hours on this, but no luck. I used to have it working in 19, and it worked good. I copied the same configuration to 22 and 21 for an upgrade, but when disconnecting the cable from one interface, the 2nd does not take over. The logs look good from mwan3's side. The interface is detected as being offline, and it reports mwan3-hotplug as disconnecting the interface, yet the routing tables don't change, and the second interface doesn't become the default route. I'll post some logs and configs, and then maybe someone will see something. Otherwise, I might have to downgrade to 19 and either run that or compare a working configuration to 21/22. Note that in 22 I did try installing iptables-nft. Any help is appreciated.

mwan3 from logread:

Tue Jun 13 09:04:48 2023 user.notice mwan3track[7043]: Interface wan (eth1) is connecting
Tue Jun 13 09:04:48 2023 user.notice mwan3track[7043]: Interface wan (eth1) is online
Tue Jun 13 09:10:33 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.0.0.1" on interface wan (eth1). Current score: 10
Tue Jun 13 09:10:37 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.1.1.1" on interface wan (eth1). Current score: 10
Tue Jun 13 09:10:41 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.222.222" on interface wan (eth1). Current score: 10
Tue Jun 13 09:10:45 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.220.220" on interface wan (eth1). Current score: 10
Tue Jun 13 09:10:45 2023 user.notice mwan3track[7043]: Interface wan (eth1) is disconnecting
Tue Jun 13 09:10:59 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.0.0.1" on interface wan (eth1). Current score: 9
Tue Jun 13 09:11:03 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.1.1.1" on interface wan (eth1). Current score: 9
Tue Jun 13 09:11:07 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.222.222" on interface wan (eth1). Current score: 9
Tue Jun 13 09:11:11 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.220.220" on interface wan (eth1). Current score: 9
Tue Jun 13 09:11:25 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.0.0.1" on interface wan (eth1). Current score: 8
Tue Jun 13 09:11:29 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.1.1.1" on interface wan (eth1). Current score: 8
Tue Jun 13 09:11:33 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.222.222" on interface wan (eth1). Current score: 8
Tue Jun 13 09:11:37 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.220.220" on interface wan (eth1). Current score: 8
Tue Jun 13 09:11:51 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.0.0.1" on interface wan (eth1). Current score: 7
Tue Jun 13 09:11:55 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.1.1.1" on interface wan (eth1). Current score: 7
Tue Jun 13 09:11:59 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.222.222" on interface wan (eth1). Current score: 7
Tue Jun 13 09:12:03 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.220.220" on interface wan (eth1). Current score: 7
Tue Jun 13 09:12:17 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.0.0.1" on interface wan (eth1). Current score: 6
Tue Jun 13 09:12:21 2023 user.info mwan3track[7043]: Check (ping) failed for target "1.1.1.1" on interface wan (eth1). Current score: 6
Tue Jun 13 09:12:25 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.222.222" on interface wan (eth1). Current score: 6
Tue Jun 13 09:12:29 2023 user.info mwan3track[7043]: Check (ping) failed for target "208.67.220.220" on interface wan (eth1). Current score: 6
Tue Jun 13 09:12:29 2023 user.notice mwan3track[7043]: Interface wan (eth1) is offline
Tue Jun 13 09:12:29 2023 user.notice mwan3-hotplug[17399]: Execute disconnected event on interface wan (eth1)

I'd be curious to know what are the exact commands mwan3 runs to disconnect the interface. I did see that there is mwan3.sh in lib and that seems to have some functions which run upon disconnection, but it's a little above my ability level to decipher them. Maybe I can look at it again.

The commands of interest are:

       disconnected)
                mwan3_set_iface_hotplug_state $INTERFACE "offline"
                mwan3_set_policies_iptables
                ;;
        ifdown)
                mwan3_set_iface_hotplug_state $INTERFACE "offline"
                mwan3_delete_iface_ipset_entries $INTERFACE
                mwan3_delete_iface_rules $INTERFACE
                mwan3_delete_iface_route $INTERFACE
                mwan3_delete_iface_iptables $INTERFACE
                procd_running mwan3 "track_$INTERFACE" && procd_send_signal mwan3 "track_$INTERFACE" USR1
                mwan3_set_policies_iptables

mwan3 config (from 19 which had been working)

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '2'
        option count '1'
        option timeout '2'
        option failure_latency '1000'
        option recovery_latency '500'
        option failure_loss '20'
        option recovery_loss '5'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wan6'
        option enabled '0'
        list track_ip '2001:4860:4860::8844'
        list track_ip '2001:4860:4860::8888'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wan2'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '1'
        option count '1'
        option timeout '2'
        option failure_latency '1000'
        option recovery_latency '500'
        option failure_loss '20'
        option recovery_loss '5'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wan26'
        option enabled '0'
        list track_ip '2001:4860:4860::8844'
        list track_ip '2001:4860:4860::8888'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'wan2_m1_w2'
        option interface 'wan2'
        option metric '1'
        option weight '2'

config member 'wan2_m2_w2'
        option interface 'wan2'
        option metric '2'
        option weight '2'
config member 'wan6_m1_w3'
        option interface 'wan6'
        option metric '1'
        option weight '3'

config member 'wan6_m2_w3'
        option interface 'wan6'
        option metric '2'
        option weight '3'

config member 'wan26_m1_w2'
        option interface 'wan26'
        option metric '1'
        option weight '2'

config member 'wan26_m2_w2'
        option interface 'wan26'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'
        list use_member 'wan6_m1_w3'

config policy 'wan2_only'
        list use_member 'wan2_m1_w2'
        list use_member 'wan26_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w2'
        list use_member 'wan6_m1_w3'
        list use_member 'wan26_m1_w2'

config policy 'wan_wan2'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m2_w2'
        list use_member 'wan6_m1_w3'
        list use_member 'wan26_m2_w2'

config policy 'wan2_wan'
        list use_member 'wan_m2_w3'
        list use_member 'wan2_m1_w2'
        list use_member 'wan6_m2_w3'
        list use_member 'wan26_m1_w2'

#double check these are correct. wan_wan2 etc
config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan_wan2'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'wan_wan2'
        option family 'ipv4'

config rule 'default_rule_v6'
        option dest_ip '::/0'
        option use_policy 'wan_wan2'
        option family 'ipv6'

mwan3 config try 2 (lightly edited example on wiki)

# For full documentation of mwan3 configuration:
# https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3#mwan3_configuration

config globals 'globals'
        option mmx_mask '0x3F00'
        option logging 1
        option loglevel info 

config interface 'wan'
        option enabled '1'
        list track_ip '1.0.0.1'
        list track_ip '1.1.1.1'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '2'

config interface 'wan6'
        option enabled '0'
        list track_ip '2606:4700:4700::1001'
        list track_ip '2606:4700:4700::1111'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '2'

config interface 'wan2'
        option enabled '1'
        list track_ip '1.0.0.1'
        list track_ip '1.1.1.1'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '1'

config interface 'wanb6'
        option enabled '0'
        list track_ip '2606:4700:4700::1001'
        list track_ip '2606:4700:4700::1111'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '1'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'wan2_m1_w2'
        option interface 'wan2'
        option metric '1'
        option weight '2'

config member 'wan2_m1_w3'
        option interface 'wan2'
        option metric '1'
        option weight '3'

config member 'wan2_m2_w2'
        option interface 'wan2'
        option metric '2'
        option weight '2'
config member 'wanb6_m1_w2'
        option interface 'wanb6'
        option metric '1'
        option weight '2'

config member 'wanb6_m1_w3'
        option interface 'wanb6'
        option metric '1'
        option weight '3'

config member 'wanb6_m2_w2'
        option interface 'wanb6'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'
        list use_member 'wan6_m1_w3'

config policy 'wan2_only'
        list use_member 'wan2_m1_w2'
        list use_member 'wanb6_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w3'
        list use_member 'wan6_m1_w3'
        list use_member 'wanb6_m1_w3'

config policy 'wan_wan2'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m2_w2'
        list use_member 'wan6_m1_w3'
        list use_member 'wanb6_m2_w2'

config policy 'wan2_wan'
        list use_member 'wan_m2_w3'
        list use_member 'wan2_m1_w2'
        list use_member 'wan6_m2_w3'
        list use_member 'wanb6_m1_w2'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan_wan2'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'wan_wan2'
        option family 'ipv4'

config rule 'default_rule_v6'
        option dest_ip '::/0'
        option use_policy 'wan_wan2'
        option family 'ipv6'

and the relevant parts of the network config, just for full disclosure (but there isn't much here)

config interface 'wan'
        option device 'eth1'
        option proto 'static'
        option ipaddr '##########'
        option netmask '255.255.255.0'
        option gateway '###########'
        option dns '1.1.1.1 4.2.2.2 8.8.8.8'
        option metric '10'

config interface 'wan2' 
        option ifname 'eth2'
        option proto 'static'
        option ipaddr '#############'
        option netmask '255.255.255.0'
        option gateway '###########'
        option dns '1.1.1.1 4.2.2.2 8.8.8.8'
        option metric '20'

Anything obvious I'm missing here? I see others are having mixed results with 22 due to the changeover to nft which makes me think I'll have to go back to 19. I'll also look at the other posts some more.

EDIT: and having written all that, I wonder if it's some error with the wan6 or wanb6. I leave the defaults in, which means there's only one wan6 (in /etc/config/network). Although the mwan3 status just says that the wan6 and wan6b interfaces are offline, which I assume it is ignoring them. They are disabled in mwan3's config...

EDIT 2: Ah, I looked at my old notes from mwan3 and I may have forgotten to include wan2 in the firewall zones (i.e. wan2 must be in /etc/config/firewall). That's probably why this wasn't working. I will double check (this has bitten me before...).

2

Without examination of your config I can add here, I have mwan3 working on 22.03.4, very well. However: Custom built image from src; no usage of the openwrt-firewall, but simple iptables rules instead; 3 interfaces for auto-switchover: wan, wwan, wifiwan; no ipv6.

3

Well the configs work OK. It turns out it was indeed the second network not being included in the firewall configuration. This is the 2nd time I've run into this, and I might add a FAQ on the existing page or make a new wiki page on configuration of mwan3 without luci. Most people use luci and follow the instructions on the wiki (which are for luci), but I do everything from the file system, so it's easy to miss that step.

The good news is that mwan3 should work with 21, and 22.

So it was a pebkac / arguably documentation error. I'll leave the thread in case anyone else runs into a similar problem.

Edit: Since this is not the first time this has bitten me I wrote up a page on the wiki here:

1 Like
4

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.