Link flapping on mwan3, using keepalived with VMAC

Installing and Using OpenWrt Network and Wireless Configuration
1

Ahoy friends.
Currently i'm bringing up, high availability, in my home network.
I got 2 routers, one master and one backup, and both have 2 internet connections available. The 2nd one is ONLY being used in case of failure of the frist one, not for load balancing.
In order to setup port forwardings properly, on the ISP router, i have to use use_vmac parameter on the WAN side of my OpenWrt routers.

Unfortunately, mwan3 is not able to handle this well, it brings the WAN links down and up again, more or less every 30 seconds. Without mwan3 it works well.
Is there a way to fix this, or workaround this issue? Maybe for explanation. When specifying the use_vmac parametre in keepalived, it creates a macvlan interface, with the name vrrp.51. Do i have to put this macvlan interface into mwan3 maybe, instead of the physical one? Or does someone have any suggestions?
Connectivity is given through the interface, for a certain amount of time, but mwan3 brings it down without any reason.

Shown below is my network topology.

WAN_Topology
WAN_Topology800×464 57.1 KB

That's how the logs look like:

Fri Oct 29 21:42:02 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:42:06 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:42:06 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is disconnecting
Fri Oct 29 21:42:15 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:42:19 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:42:23 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:42:27 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:42:36 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:42:37 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is connecting
Fri Oct 29 21:42:37 2021 user.info mwan3track[3637]: Lost 9 ping(s) on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:42:51 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:42:55 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:42:59 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:43:03 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:43:03 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is disconnecting
Fri Oct 29 21:43:12 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:43:16 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:43:20 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:43:24 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:43:33 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:43:34 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is connecting
Fri Oct 29 21:43:34 2021 user.info mwan3track[3637]: Lost 9 ping(s) on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:43:48 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:43:52 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:43:56 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:00 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:00 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is disconnecting
Fri Oct 29 21:44:09 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:44:13 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:44:17 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:44:21 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:44:30 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:44:32 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is connecting
Fri Oct 29 21:44:32 2021 user.info mwan3track[3637]: Lost 9 ping(s) on interface wanb (bonding-i0.1100). Current score: 8
Fri Oct 29 21:44:46 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:50 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:54 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:58 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 10
Fri Oct 29 21:44:58 2021 user.notice mwan3track[3637]: Interface wanb (bonding-i0.1100) is disconnecting
Fri Oct 29 21:45:07 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:45:11 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.8.8" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:45:15 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.222.222" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:45:19 2021 user.info mwan3track[3637]: Check (ping) failed for target "208.67.220.220" on interface wanb (bonding-i0.1100). Current score: 9
Fri Oct 29 21:45:19 2021 daemon.info dnsmasq-dhcp[25777]: DHCPREQUEST(bonding-i0.3) 192.168.3.157 8e:7d:94:79:83:f6
Fri Oct 29 21:45:19 2021 daemon.info dnsmasq-dhcp[25777]: DHCPACK(bonding-i0.3) 192.168.3.157 8e:7d:94:79:83:f6 Galaxy-A72
Fri Oct 29 21:45:28 2021 user.info mwan3track[3637]: Check (ping) failed for target "8.8.4.4" on interface wanb (bonding-i0.1100). Current score: 8

The specific part of the keepalived.conf

vrrp_instance E2 {
  state MASTER
  interface bonding-i0.1100
  virtual_router_id 51
  priority 100
  advert_int 1
  use_vmac # **That's causing an issue, while using mwan3, without mwan3 no issue. Creates a macvlan Interface**
  virtual_ipaddress {
    192.168.154.4/24
  }
  virtual_routes {
    src 192.168.154.4 to 0.0.0.0/0 via 192.168.154.1 dev bonding-i0.1100 metric 5
  }
  authentication {
    auth_type PASS
    auth_pass secret
  }
  preempt
}

Thanks in advance!

2

It's possible that the use of use_vmac parameter is conflicting with mwan3's configuration, causing the WAN links to go down and up again periodically. One way to work around this issue is to use a different VRRP implementation that does not require the use_vmac parameter.

One option is to use "keepalived" instead of "VRRP" to achieve high availability. Keepalived is a daemon that uses the VRRP protocol to provide a virtual IP address to a group of servers, and it does not require the use_vmac parameter. Keepalived is also supported by OpenWrt and can be easily installed using the package manager.

To configure keepalived, you would need to create a configuration file, similar to the VRRP configuration file you provided, but with some changes. Here's an example configuration file that you can modify to suit your needs:


vrrp_script chk_www {
    script "/usr/local/bin/check_apache.sh"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 100
    virtual_ipaddress {
        192.168.154.4/24 dev eth0
    }
    track_script {
        chk_www
    }
    authentication {
        auth_type PASS
        auth_pass secret
    }
    notify /etc/keepalived/notify.sh
}

In this example, we're using a script to check the availability of the web server, but you can modify it to check the availability of your WAN links. You would also need to modify the interface name and the virtual IP address to match your setup.

Once you have created the keepalived configuration file, you can start the keepalived daemon using the following command:

/etc/init.d/keepalived start

This should provide high availability without interfering with mwan3's configuration.