1

I setup my Luxul with an Australian NBN (looks like a plain simple ethernet WAN to OpenWRT) and just connected a TP-Linix M7350 LTE modem via USB and both work ok without mwan3 - i.e. when the NBN is disconnected the USB interface takes over and when the NBN is reconnected it picks up the traffic again.

But I want mwan3 for the extra smarts:

  1. Use "ping" to verify that the NBN is really up
  2. prevent certain local hosts from using the much more limited bandwidth of the LTE link when the NBN is down.

I set a metric to both (10 for NBN, 20 for LTE, identified as usb0) but the mwan3 interface keeps saying that there is no metric set on the LTE interface.

Can someone please have a look at this and tell me what am I missing?

Here is the configuration:

root@OpenWrt:/etc/config# ubus call system board; \
> uci export network; uci export mwan3; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; mwan3 status
{
	"kernel": "5.15.167",
	"hostname": "OpenWrt",
	"system": "ARMv7 Processor rev 0 (v7l)",
	"model": "Luxul ABR-4500 V1",
	"board_name": "luxul,abr-4500-v1",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.5",
		"revision": "r24106-10cc5fcd00",
		"target": "bcm53xx/generic",
		"description": "OpenWrt 23.05.5 r24106-10cc5fcd00"
	}
}
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd6d:21d6:623a::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '172.16.69.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'wan'
	option macaddr 'A4:13:4E:62:50:b3'

config interface 'wan'
	option device 'wan'
	option proto 'pppoe'
	option username 'user@wba.nbnonline.com.au'
	option password 'pass'
	option ipv6 'auto'
	option metric '10'
	option peerdns '0'
	list track_ip '1.0.0.1'
	list track_ip '1.1.1.1'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	list dns '8.8.8.8'
	list dns '8.8.4.4'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'

config device
	option type '8021q'
	option ifname 'lan3'
	option vid '3'
	option name 'lan3.3'

config interface 'wanb'
	option proto 'dhcp'
	option device 'usb0'
	option metric '20'
	list track_ip '1.0.0.1'
	list track_ip '1.1.1.1'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'

package mwan3

config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wan'
	option enabled '1'
	list track_ip '1.0.0.1'
	list track_ip '1.1.1.1'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '2'

config interface 'wan6'
	option enabled '0'
	list track_ip '2606:4700:4700::1001'
	list track_ip '2606:4700:4700::1111'
	list track_ip '2620:0:ccd::2'
	list track_ip '2620:0:ccc::2'
	option family 'ipv6'
	option reliability '2'

config member 'wan_m1_w3'
	option interface 'wan'
	option metric '1'
	option weight '3'

config member 'wan_m2_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'

config member 'wan6_m2_w3'
	option interface 'wan6'
	option metric '2'
	option weight '3'

config policy 'wan_only'
	list use_member 'wan_m1_w3'
	list use_member 'wan6_m1_w3'

config policy 'balanced'
	list use_member 'wan_m1_w3'
	list use_member 'wan6_m1_w3'

config rule 'https'
	option sticky '1'
	option dest_port '443'
	option proto 'tcp'
	option use_policy 'balanced'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option use_policy 'balanced'
	option family 'ipv4'

config rule 'default_rule_v6'
	option dest_ip '::/0'
	option use_policy 'balanced'
	option family 'ipv6'

config interface 'usb0'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	list track_ip '8.8.8.8'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.0.154/24 brd 192.168.0.255 scope global usb0
       valid_lft forever preferred_lft forever
22: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 172.16.69.1/24 brd 172.16.69.255 scope global br-lan
       valid_lft forever preferred_lft forever
23: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    inet 100.71.121.60 peer 203.134.4.193/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
default via 203.134.4.193 dev pppoe-wan table 1 proto static metric 10
172.16.69.0/24 dev br-lan table 1 proto kernel scope link src 172.16.69.1
192.168.0.0/24 dev usb0 table 1 proto static scope link metric 20
203.134.4.193 dev pppoe-wan table 1 proto kernel scope link src 100.71.121.60
default via 203.134.4.193 dev pppoe-wan proto static metric 10
default via 192.168.0.1 dev usb0 proto static src 192.168.0.154 metric 20
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1
192.168.0.0/24 dev usb0 proto static scope link metric 20
203.134.4.193 dev pppoe-wan proto kernel scope link src 100.71.121.60
local 100.71.121.60 dev pppoe-wan table local proto kernel scope host src 100.71.121.60
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.16.69.1 dev br-lan table local proto kernel scope host src 172.16.69.1
broadcast 172.16.69.255 dev br-lan table local proto kernel scope link src 172.16.69.1
local 192.168.0.154 dev usb0 table local proto kernel scope host src 192.168.0.154
broadcast 192.168.0.255 dev usb0 table local proto kernel scope link src 192.168.0.154
0:	from all lookup local
1001:	from all iif pppoe-wan lookup 1
2001:	from all fwmark 0x100/0x3f00 lookup 1
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default
Interface status:
 interface wan is online 00h:06m:25s, uptime 00h:06m:27s and tracking is active
 interface wan6 is offline and tracking is down
 interface usb0 is offline and tracking is paused

Current ipv4 policies:
balanced:
 wan (100%)
wan_only:
 wan (100%)

Current ipv6 policies:
balanced:
 unreachable
wan_only:
 unreachable

Directly connected ipv4 networks:
172.16.69.0/24
224.0.0.0/3
192.168.0.255
172.16.69.255
127.0.0.0/8
127.255.255.255
203.134.4.193
172.16.69.1
192.168.0.0/24
192.168.0.154
127.0.0.1

Directly connected ipv6 networks:
fe80::8c8f:7632:f4aa:c6fa
2403:9e00:22c4:bc00::/64
fd6d:21d6:623a::/64
fe80::4a5a:dff:fe43:2213
fe80::/64

Active ipv4 user rules:
 1264 76811 S https  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 443
 2598  418K - balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Active ipv6 user rules:
  825 71892 S https  tcp      *      *       ::/0                 ::/0                 multiport dports 443
 1799  810K - balanced  all      *      *       ::/0                 ::/0
2

In the mwan3 configuration, it needs to be config interface wanb.

3

Thanks. Done.

I think mwan3 is more stable now, e.g. it shows the secondary wanb interface with the correct metric of 20.

However, I have MacOS "Display My IP" which keeps track of public IP changes and since I brought mwan3 up with the fix, it keeps flipping public IP addresses, I think on the same interface (they all come from the same /24 address range):

Screenshot 2025-03-23 at 10.45.56 am
Screenshot 2025-03-23 at 10.45.56 am355×923 176 KB

The frequency of change must be affected by the configuration of the app to check every minute, so it could be happening more often and just missed by the app.

The mwan3 status overview says that the secondary link is in no tracking state, but green, whatever that means:

Screenshot 2025-03-23 at 11.04.54 am
Screenshot 2025-03-23 at 11.04.54 am824×167 14.6 KB

Could it be because there is some redundant route configuration?
(the ip route show output in the troubleshooting output copied below)

Here is the output of the mwan3 troubleshooting page:

Software-Version
-------------------------------------------------
OpenWrt - 23.05.5

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.0.154/24 brd 192.168.0.255 scope global usb0
       valid_lft forever preferred_lft forever
22: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 172.16.69.1/24 brd 172.16.69.255 scope global br-lan
       valid_lft forever preferred_lft forever
23: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    inet 100.71.121.60 peer 203.134.4.193/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 203.134.4.193 dev pppoe-wan proto static metric 10 
default via 192.168.0.1 dev usb0 proto static src 192.168.0.154 metric 20 
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1 
192.168.0.0/24 dev usb0 proto static scope link metric 20 
203.134.4.193 dev pppoe-wan proto kernel scope link src 100.71.121.60 

Output of "ip -4 rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif pppoe-wan lookup 1
2001:	from all fwmark 0x100/0x3f00 lookup 1
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 203.134.4.193 dev pppoe-wan proto static metric 10 
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1 
192.168.0.0/24 dev usb0 proto static scope link metric 20 
203.134.4.193 dev pppoe-wan proto kernel scope link src 100.71.121.60 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 527K  470M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
11710 1328K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 372K  383M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 dst MARK or 0x3f00

Chain mwan3_custom_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 dst MARK or 0x3f00

Chain mwan3_dynamic_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 538K  472M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
13621 2071K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
12777 1992K mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
12777 1992K mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 9554 1534K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 9554 1534K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 539K  472M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
 528K  470M mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
 528K  470M mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
 160K   88M mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  844 78765 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
13619 2070K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_balanced (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_failover (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 7777 1426K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  1440 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_rule_https (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1760  106K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 MARK xset 0x100/0x3f00
   19  1265 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x100/0x3f00 ! match-set mwan3_rule_ipv4_https src,src MARK and 0xffffc0ff
   19  1265 mwan3_policy_failover  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1760  106K SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 del-set mwan3_rule_ipv4_https src,src
 1760  106K SET        all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xfc00/0xfc00 add-set mwan3_rule_ipv4_https src,src

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  1440 mwan3_policy_wan_only  all  --  *      *       0.0.0.0              0.0.0.0/0            mark match 0x0/0x3f00
 1760  106K mwan3_rule_https  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 443 mark match 0x0/0x3f00
 7758 1425K mwan3_policy_failover  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Thanks again.

4

The output, indeed, does not look correct. There is no table 2 for the USB modem and no corresponding rule.

Also, you use the "balanced" policy, which means literally that: half of the connections go out from the PPPoE interface and half from the modem.

Recommendations:

  • Disable IPv6 by removing all IPv6 config from /etc/config/network (we can reenable it later if needed, but this requires a non-standard script for mwan3 compatibility)
  • Delete all members
  • Add members for wan_m10_w1 and wanb_m20_w1 (metric 10 and 20 respectively, weight 1)
  • Add a failover policy that includes both members
  • Add a wan_only policy that only includes wan_m10_w1
  • Change the default rule to mention the failover policy.
1 Like
5

ok, I noticed some mess in my configuration:

  1. there were some mwan3 specific options in the network config file, I removed them.
  2. my fallback modem interface (TP-Link M7350), while able to accept an ipv6 from the ISP, doesn't present DHCPv6 to the LAN so it practically doesn't support ipv6. So I don't lose any functionality by excluding ipv6 from mwan3.
  3. Your explanation of the member naming finally dropped the penny for me so I could make it more consistent and sane
  4. I follow mwan ipv6 support to remove any mention of ipv6 in mwan3 and use the option last_resort 'default' in the policies to keep ipv6 routing out of mwan3 (I suspect that your link to the non-standard IPv6 script in your previous reply could be out of date, based on the current mwan3 page?)

What happens now is that:

  1. mwan3 will nicely fall over ipv4 when my primary link drops (I physically pulled the ethernet cable from the WAN modem) and fall back to the primary interface when it comes back online.
  2. I lost the external ipv6 address. That's something that would be nice to get back as it seems to be a bit more performant.
  3. The app which tracks external IP's still keeps notifying me about IP address change every minute.

BTW I think that when editing the configuration through the web Luci web interface, it rewrites the mwan3 config file out of order (e.g. config rule default_rule_v4 referencing a config policy failover before it is defined, I reordered the file to address this, does this matter to the software? It definitely makes it easier to eyeball the file when things are in order).

Thank you so much for your help.

Here is my current mwan3:


config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wan'
	option enabled '1'
	list track_ip '1.0.0.1'
	list track_ip '1.1.1.1'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '2'

config interface 'wanb'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	list track_ip '8.8.8.8'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'

config member 'wan_m10_w1'
	option interface 'wan'
	option metric '10'
	option weight '1'

config member 'wan_m20_w1'
	option interface 'wanb'
	option metric '20'
	option weight '1'

config policy 'wan_only'
	list use_member 'wan_m10_w1'
	option last_resort 'default'

config policy 'failover'
	list use_member 'wan_m10_w1'
	list use_member 'wan_m20_w1'
	option last_resort 'default'

config rule 'nuc_wan_only'
	option proto 'all'
	option src_ip '172.16.69.10'
	option family 'ipv4'
	option sticky '0'
	option use_policy 'wan_only'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option use_policy 'failover'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'

My current network file:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd6d:21d6:623a::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '172.16.69.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'wan'
	option macaddr 'A4:13:4E:62:50:b3'

config interface 'wan'
	option device 'wan'
	option proto 'pppoe'
	option username 'username@wba.nbnonline.com.au'
	option password 'password'
	option ipv6 'auto'
	option metric '10'
	option peerdns '0'
	option family 'ipv4'
	list dns '8.8.8.8'
	list dns '8.8.4.4'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'
	option metric '10'
	option family 'ipv6'

config device
	option type '8021q'
	option ifname 'lan3'
	option vid '3'
	option name 'lan3.3'

config interface 'wanb'
	option proto 'dhcp'
	option device 'usb0'
	option metric '20'
	list track_ip '1.0.0.1'
	list track_ip '1.1.1.1'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'

And the current mwan3 Troubleshooting output:

Software-Version
-------------------------------------------------
OpenWrt - 23.05.5

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
10: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.0.154/24 brd 192.168.0.255 scope global usb0
       valid_lft forever preferred_lft forever
21: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 172.16.69.1/24 brd 172.16.69.255 scope global br-lan
       valid_lft forever preferred_lft forever
23: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    inet 100.70.250.230 peer 203.134.4.115/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 203.134.4.115 dev pppoe-wan proto static metric 10 
default via 192.168.0.1 dev usb0 proto static src 192.168.0.154 metric 20 
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1 
192.168.0.0/24 dev usb0 proto static scope link metric 20 
203.134.4.115 dev pppoe-wan proto kernel scope link src 100.70.250.230 

Output of "ip -4 rule show"
-------------------------------------------------
0:	from all lookup local
1001:	from all iif pppoe-wan lookup 1
1002:	from all iif usb0 lookup 2
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 203.134.4.115 dev pppoe-wan proto static metric 10 
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1 
203.134.4.115 dev pppoe-wan proto kernel scope link src 100.70.250.230 

Routing table 2:
default via 192.168.0.1 dev usb0 proto static src 192.168.0.154 metric 20 
172.16.69.0/24 dev br-lan proto kernel scope link src 172.16.69.1 
192.168.0.0/24 dev usb0 proto static scope link metric 20 
203.134.4.115 dev pppoe-wan proto kernel scope link src 100.70.250.230 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 124K   74M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 6162  989K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
42268   22M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 dst MARK or 0x3f00

Chain mwan3_custom_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 dst MARK or 0x3f00

Chain mwan3_dynamic_ipv4 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 129K   75M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 6890 1218K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 6668 1184K mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 6668 1184K mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 4377  935K mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 4377  935K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 130K   75M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
 119K   73M mwan3_custom_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
 119K   73M mwan3_connected_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00
79206   51M mwan3_dynamic_ipv4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  121  8886 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_in_wanb (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  usb0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_custom_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
   40 19524 MARK       all  --  usb0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  usb0   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_dynamic_ipv4 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    7   571 MARK       all  --  usb0   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb */ MARK xset 0x200/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 6890 1218K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 6715 1205K mwan3_iface_in_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_failover (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1329  210K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 1 1 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    5   362 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 1 1 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   19  1884 mwan3_policy_wan_only  all  --  *      *       0.0.0.0              0.0.0.0/0            mark match 0x0/0x3f00
 4357  933K mwan3_policy_failover  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
6

The custom IPv6 script link is not really out of date, although a better version exists. While mwan3 supports IPv6, this comes with a hard-to-satisfy precondition:

  • the user must set up network prefix translation manually (which is a missing feature in OpenWrt, and that's what the script does)
  • the user must not allow non-ULA addresses in their LAN (i.e., must set list ip6class 'local' on their LAN interface in /etc/config/network).

And this is documented on the page that you linked to.

In any case, the IPv6 fail-over is not something that I would be ready to explain in a forum post - too many "if"s. Can you suggest a few timeslots for a video call in a private message?

1 Like
7

Thank you for your offer.
I think IPv6 fail over is not relevant for my situation for now anyway since the LTE modem doesn't seem to support IPv6.
My main concern right now is the constant address flapping that's being reported by my desktop app, which doesn't happen when mwan3 is disabled.

8

I think your LTE modem does support IPv6; you just need to change the PDP context type in its settings to IPv4v6.

As for the non-delegation of prefixes, please try the option extendprefix 1 line in the relevant interface configuration section.

9

That worked:

  1. The PDP option was already set on IPv4 & IPv6 and immutable
  2. I added the wanb6 interface back with the extra option you suggested and now it has an IPv6 address assigned to it.

Now back to the constantly changing public IP address - do I really need mwan3 if I don't want to do load balancing? I only want fail-over (LTE modem used ONLY when the wired WAN is down).

10

I think we still need a meeting to verify your configuration. mwan3, as long as its balancing functionality is not used (i.e., as long as there is no policy that includes two members of the same metric), should not change the public IP in the stable state. So something is misconfigured, this is not how mwan3 should behave.

Regarding the question of whether you need mwan3 for fail-over only: the answer is, it's complicated and I can't make choice for you. In my own network, I got rid of mwan3.

You tried to make sure that the NUC can only use WAN but not LTE. This functionality is misconfigured in your setup (it only works if the wan_only policy uses unreachable, not default, as a last resort) and so doesn't work. If you want to keep mwan3, you will need to fix it. If you want to get rid of mwan3, you will need to replace it with a firewall rule.

For IPv6 fail-over, both the mwan3 way and the other way require custom scripts, so pick your poison.

If you provide a few possible times for a video meeting, I will review your configuration and make a more concrete recommendation.