Mwan3 Help please

[openwrt20]

Hi hope everyone is well.
I recently setup mwan3 rules successfully with the much appreciated help of @pavelgl but I now seem to have run into an issue.

I have 2 WAN connections and 1 WG connection. By default LAN devices will connect via WG using the balanced policy of all 3 where WG has the lowest metric (this is to allow failover to WAN if WG interface goes down). The rules were created for some LAN devices and remote IPs to bypass WG and exclusively use WAN.

However, out of the blue last night all my default LAN devices were routing through WAN instead of WG (no changes made to settings). I rebooted the router which fixed that problem and now those devices are going via WG as per balanced policy. But I'm now finding that mwan3 is not enforcing the rules to bypass WG for those specfific LAN devices/remote IPs. Instead everything is being forced through WG. Any help would be greatly appreciated to explain why this is happening?

NB - Also when running the "ipset list netflix" command for example I get the error message "The set with the given name does not exist"

mwan3 config

config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wanb'
	option enabled '1'
	option family 'ipv4'
	option reliability '1'
	option initial_state 'online'
	option track_method 'ping'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'
	list track_ip '1.1.1.1'

config member 'wan_m1_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'

config policy 'balanced'
	option last_resort 'unreachable'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m1_w3'
	list use_member 'wg0_m1_w3'

config rule 'aws'
	option proto 'all'
	option sticky '1'
	option ipset 'aws'
	option use_policy 'wan_only'

config rule 'netflix'
	option family 'ipv4'
	option sticky '1'
	option ipset 'netflix'
	option proto 'all'
	option use_policy 'wan_only'

config rule 'xxx1'
	option proto 'all'
	option src_ip '192.168.x.x'
	option sticky '0'
	option use_policy 'wan_only'

config rule 'xxx2'
	option proto 'all'
	option src_ip '192.168.x.x'
	option sticky '0'
	option use_policy 'wan_only'

config rule 'xxx3'
	option proto 'all'
	option src_ip '192.168.x.x'
	option sticky '0'
	option use_policy 'balanced'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option use_policy 'balanced'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'

config rule 'default_rule_v6'
	option dest_ip '::/0'
	option use_policy 'balanced'
	option family 'ipv6'

config member 'wanb_m1_w3'
	option interface 'wanb'
	option metric '2'
	option weight '3'

config interface 'wg0'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list track_ip '1.1.1.2'

config member 'wg0_m1_w3'
	option interface 'wg0'
	option metric '1'
	option weight '3'

config policy 'wireguard'
	list use_member 'wg0_m1_w3'
	option last_resort 'unreachable'

config policy 'wan_only'
	option last_resort 'unreachable'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m1_w3'

config policy 'wanb_only'
	option last_resort 'unreachable'
	list use_member 'wanb_m1_w3'

config interface 'wan'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '30'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '5'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'
	list track_ip '1.0.0.1'

firewall config

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	list network 'wanb'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config include 'nss_ecm'
	option type 'script'
	option path '/etc/firewall.d/qca-nss-ecm'
	option family 'any'
	option reload '1'

config zone
	option name 'vpn'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'wg0'
	option masq '1'

config forwarding
	option src 'lan'
	option dest 'vpn'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'XXX'
	option src 'wan'
	option src_dport '4xxxx'
	option dest_ip '192.168.x.x'
	option dest_port '8xxx'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'XXX'
	list proto 'tcp'
	option src 'wan'
	option src_dport '3xxxx'
	option dest_ip '192.168.x.x'
	option dest_port '4xxx'

[pavelgl]

@openwrt20

when running the "ipset list netflix" command for example I get the error message "The set with the given name does not exist"

The ipset you created before is not persistent across reboots. Its absence results in an error when starting mwan3, and most likely you don't currently have any active rules.

Here is a possible (very inelegant) fix.

1. Disable the autostart of the service.

/etc/init.d/mwan3 disable

2. Insert the following into /etc/rc.local above exit 0:

ipset create netflix hash:net
ipset add netflix 108.175.32.0/20
ipset add netflix 185.2.220.0/22
ipset add netflix 185.9.188.0/22
ipset add netflix 192.173.100.0/24
ipset add netflix 192.173.101.0/24
ipset add netflix 192.173.102.0/24
ipset add netflix 192.173.104.0/24
ipset add netflix 192.173.105.0/24
ipset add netflix 192.173.108.0/24
ipset add netflix 192.173.109.0/24
/etc/init.d/mwan3 start

3. Reboot the device.

[openwrt20]

@pavelgl

Here is a possible (very inelegant) fix.

Thank you this worked!

@pavelgl

ipset add netflix 192.173.109.0/24
/etc/init.d/mwan3 start

And I take it I can create further ipsets by adding between these 2 lines?

[openwrt20]

Also, I've previously used watchcat to restart the wan interface. But instead I've added the following script in /etc/mwan3.user

if [ "${ACTION}" = "ifdown" ] && [ "${INTERFACE}" = "wan" ] ; then
   ifup wan
fi

However, I've found that the wan interface sometimes goes into a continuous reboot loop and I have to delete the above script to get it working. Is there a way to set a time delay like in watchcat before it restarts? Maybe with sleep?

[pavelgl]

@openwrt20

And I take it I can create further ipsets by adding between these 2 lines?

That's correct.

@openwrt20

the wan interface sometimes goes into a continuous reboot loop

ifup first invokes ifdown, hence the loop. To get the same result as with watchcat, the correct ACTION used in /etc/mwan3.user must be disconnected.

[openwrt20]

@pavelgl

ifup first invokes ifdown, hence the loop. To get the same result as with watchcat, the correct ACTION used in /etc/mwan3.user must be disconnected.

Thank you. So would the following be correct?

if [ "${ACTION}" = "disconnected" ] && [ "${INTERFACE}" = "wan" ] ; then
   ifup wan
fi

Also watchcat allows you to define a delay period (s,m,h,d) before the interface restarts. Is this possible here?

[pavelgl]

@openwrt20

Thank you. So would the following be correct?

Yes.

@openwrt20

Also watchcat allows you to define a delay period (s,m,h,d) before the interface restarts. Is this possible here?

When a ping failure via a particular interface is detected, the first action logged is disconnecting.

Before the interface enters the disconnected state, there is a timeout depending on the failure_interval and down options. You can set them according to your needs, or alternatively use the sleep command before running ifup.

[reinerotto]

@pavelgl

or alternatively use the sleep command before running ifup.

R U shure ? I think, this might interfere with the flow of control of mwan3 scripts. Safer should be a detached shell script, only containing the sleep and ifup.

[pavelgl]

@reinerotto

Safer should be a detached shell script, only containing the sleep and ifup.

Good point, I agree.

[openwrt20]

@pavelgl

Before the interface enters the disconnected state, there is a timeout depending on the failure_interval and down options

Thank you. I'll try and adjust those options first.

@reinerotto

Safer should be a detached shell script, only containing the sleep and ifup.

Thank you. Which directory/file would I need to input this please?

[openwrt20]

Also, this morning the wan and wg0 interface were offline and everything was routing through wanb. I restarted wan and both wan and wg0 came back online. Is there a way to force the wg0 interface to use wanb if wan were to go down?

[reinerotto]

@openwrt20

Which directory/file would I need to input this please?

Should not matter. As mwan3.usr is located in /etc, same can be used for this "helper" script. For a first test, you might consider to have a statement like "logger 'mwan3-helper executed"' at end of the script.

[openwrt20]

Thank you. Is this the relevant guide? https://openwrt.org/docs/guide-developer/write-shell-script

Although disconnected appears to be working well at the moment, thank you @pavelgl
My concern now is as my above post. If either wan or wanb were to go completely offline, is there a way to force wg0 via the wan that is still online?