Edgerouter X for failover

My first post in this forum.
Trying to help a friend with an Edgerouter X for failover.
First time I'm configuring this with OpenWRT.

  • Two Wans (wan and wanb) with 10 and 20 metric.
  • Both get IPs from the "modem": wam: 192.168.15.X. wamb: 192.168.0.X
  • The "modems" get the valid IPs and both has DHCP servers.
  • I only edited the /etc/config/mwan3 changing the only instance of "balancing" to "failover"

But that seems to be not enough as if the wam cable is pulled the wanb do not assume the traffic...

Trying to follow this:
https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3_install_without_luci

Bellow is the troubleshooting report produced by Luci via Status-> MultiWAN Manager -> Troubleshooting:

Software-Version
-------------------------------------------------
OpenWrt - 23.05.0

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.15.71/24 brd 192.168.15.255 scope global eth0
       valid_lft forever preferred_lft forever
7: eth4@dsa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.0.102/24 brd 192.168.0.255 scope global eth4
       valid_lft forever preferred_lft forever
8: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 192.168.15.1 dev eth0 proto static src 192.168.15.71 metric 10 
default via 192.168.0.1 dev eth4 proto static src 192.168.0.102 metric 20 
192.168.0.0/24 dev eth4 proto static scope link metric 20 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.15.0/24 dev eth0 proto static scope link metric 10 

Output of "ip -4 rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif eth0 lookup 1
1002:	from all iif eth4 lookup 2
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 192.168.15.1 dev eth0 proto static src 192.168.15.71 metric 10 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.15.0/24 dev eth0 proto static scope link metric 10 

Routing table 2:
default via 192.168.0.1 dev eth4 proto static src 192.168.0.102 metric 20 
192.168.0.0/24 dev eth4 proto static scope link metric 20 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 24221 packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination         
24272   13M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 876 packets, 88519 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 23184 packets, 12M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 979 packets, 617K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1023  667K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 24161 packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
14590   11M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 dst MARK or 0x3f00

Chain mwan3_custom_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 dst MARK or 0x3f00

Chain mwan3_dynamic_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
25132   13M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
  313 76319 mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  119 13286 mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  119 13286 mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
   74  9453 mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
   74  9453 mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
25295   13M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
23320   12M mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
23320   12M mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
 8775 1308K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   86 33791 MARK       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    3   152 MARK       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_in_wanb (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  104 28874 MARK       all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    1   216 MARK       all  --  eth4   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb */ MARK xset 0x200/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  313 76319 mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  224 42376 mwan3_iface_in_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_failover (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 statistic mode random probability 0.50000000000 /* wanb 3 6 */ MARK xset 0x200/0x3f00
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_wanb (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   38  2434 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wanb_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb 2 2 */ MARK xset 0x200/0x3f00

Chain mwan3_policy_wanb_wan (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb 2 2 */ MARK xset 0x200/0x3f00

Chain mwan3_rule_https (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    8   416 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 MARK xset 0x100/0x3f00
    1    52 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x100/0x3f00 ! match-set mwan3_rule_ipv4_https src,src MARK and 0xffffc0ff
    1    52 mwan3_policy_wan_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
    8   416 SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 del-set mwan3_rule_ipv4_https src,src
    8   416 SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 add-set mwan3_rule_ipv4_https src,src

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    8   416 mwan3_rule_https  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 443 mark match 0x0/0x3f00
   37  2382 mwan3_policy_wan_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Powered by LuCI openwrt-23.05 branch (git-23.357.58018-024e7ab) / OpenWrt 23.05.0 (r23497-6637af95aa) 

Replying my own question:
After changed the: "Chain mwan3_rule_https (1 references)" to this:

Chain mwan3_rule_https (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  339 18247 mwan3_policy_wan_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
  339 18247 SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 del-set mwan3_rule_ipv4_https src,src
  339 18247 SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 add-set mwan3_rule_ipv4_https src,src

Now it is working !

But now, if I may, the question morph to:
My ISP plan is 800Mbps. This Edgerouter is allowing only about 93Mbps to reach the LAN clients ! It is a hardware limitation or there is some magic config to allow full speed ?

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.