Wireguard and MWAN output chain

[GrapeCent]

I'm having some trouble with a multi WAN load balancing solution I'v been working on and got sucessfully setup on OpenVPN because it's a userspace application, however I've found that the handshake for openvpn is not reliable enough and the MTU overhead is a bit much

So I've tried to set this up with wireguard. I have three ISP link with DHCP default gateways, each interface is set to a different metric

• WAN = 1
• WWAN = 5
• EasyTether = 10

I then configured MWAN rules to direct traffic based on a different port to the same IP to lock a stream to each ISP.

• to 130.44.56.28:45670 uses policy wan
• to 130.44.56.28:45671 uses policy wwan
• to 130.44.56.28:45672 uses policy easytether

However when I restart the interfaces or reboot the router, only the tunnel pinned to the interface that is up with the lowest metric is able to connect or pass traffic.
For example, I turn off the WAN and WWAN interfaces, the easytether pinned tunnel is able to connect. As soon as I turn WWAN or WAN back on, the connections in the easytether tunnel stop.
Also, tcpdump of easytether only shows an occational inbound packet from 130.44.56.28, but I don't see any RX traffic on the easytether tunnel

I can see the rules matching easytether tunnel on output of the lower metric interface in the logs if I enable loging

Is this an issue because wireguard is a kernel space tunnel, or am I missing some additional configuration?

[trendy]

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export mwan3; \
uci export dhcp; uci export firewall; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru

[GrapeCent]

Here's the requested config, thanks for your recent help. Note: I cleaned it up with a config redaction script I use. Also I had to reduce the character count, if you need this unredacted I can PM it to you.

x.x.x.x is the IP address in question that my policy is supposed to trigger on
the ports are 24680 24681 and 24682

At this moment, which is also in the config, I temporarily have a 2nd IP on the SDWAN router and I'm using a static route to pin the connections to each ISP uplink.

{
	"kernel": "5.10.161",
	"hostname": "npancwangw01.xxx.net",
	"system": "ARMv8 Processor rev 4",
	"model": "Linksys E8450 (UBI)",
	"board_name": "linksys,e8450-ubi",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "22.03.3",
		"revision": "r20028-43d71ad93e",
		"target": "mediatek/mt7622",
		"description": "OpenWrt 22.03.3 r20028-43d71ad93e"
	}
}
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'
	option hostname '*'
	option metric '1'
	option peerdns '0'
	option dns_metric '1'
	option force_link '1'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'

config bridge-vlan
	option device 'br-lan'
	option vlan '2'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '11'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:t'

config interface 'admin'
	option proto 'static'
	list ipaddr 'ipv4-scrubbed'
	option device 'br-admin'
	option defaultroute '0'
	option delegate '0'
	option ip6weight '100'
	list ip6addr 'FD00:x'

config interface 'devices'
	option proto 'static'
	list ipaddr 'ipv4-scrubbed'
	option device 'br-devices'
	option delegate '0'
	option defaultroute '0'
	list ip6addr 'FD00:x'

config interface 'dummy'
	option proto 'static'
	option device 'dummy0'
	list ipaddr '10.23.32.1/32'
	option delegate '0'
	option defaultroute '0'

config interface 'domain'
	option proto 'static'
	list ipaddr 'ipv4-scrubbed'
	option delegate '0'
	option defaultroute '0'
	list ip6addr 'FD00:x'
	option device 'dummy1'

config interface 'guest'
	option proto 'static'
	list ipaddr 'ipv4-scrubbed'
	option device 'br-guest'
	option delegate '0'
	option defaultroute '0'
	list ip6addr 'FD00:x'

config interface 'resident'
	option proto 'static'
	option device 'br-resident'
	list ipaddr 'ipv4-scrubbed'
	option delegate '0'
	option defaultroute '0'
	list ip6addr 'FD00:x'

config interface 'cs1'
	option proto 'wireguard'
	option delegate '0'
	option privatexx'
	option peerdns '0'
	option defaultroute '0'
	list addresses 'ipv4-scrubbed'
	list addresses 'ipv6-scrubbed'
	option nohostroute '1'
	option mtu '1392'

config wireguard_cs1
	option publicxx'
	option endpoint_port '24680'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option presharedxx'
	option endpoint_host 'y.y.y.y
	option persistent_keepalive '25'
	option description 'nvcs1wangw01'

config interface 'ovpn'
	option proto 'none'
	option defaultroute '0'
	option peerdns '0'
	option delegate '0'
	option device 'tun0'

config device
	option type 'bridge'
	option name 'br-admin'
	list ports 'br-lan.2'

config device
	option type 'bridge'
	option name 'br-devices'
	list ports 'br-lan.3'

config device
	option type 'bridge'
	option name 'br-resident'
	list ports 'br-lan.10'

config device
	option type 'bridge'
	option name 'br-guest'
	list ports 'br-lan.11'

config interface 'pv'
	option proto 'wireguard'
	option privatexx'
	option nohostroute '1'
	option defaultroute '0'
	option peerdns '0'
	list addresses 'ipv4-scrubbed'
	list addresses 'ipv6-scrubbed'
	option mtu '1392'

config wireguard_pv
	option description 'nppvwangw01'
	option publicxx'
	option presharedxx'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_host 'nppvwangw01.router.id'
	option persistent_keepalive '25'
	option endpoint_port '24681'

config interface 'wwan'
	option proto 'dhcp'
	option hostname '*'
	option type 'bridge'
	option peerdns '0'
	option device 'wwan1'
	option dns_metric '5'
	option metric '5'
	option force_link '1'

config interface 'easytether'
	option proto 'dhcp'
	option device 'tap-easytether'
	option hostname '*'
	option peerdns '0'
	option dns_metric '10'
	option metric '10'

config route
	option interface 'wwan'
	option gateway '192.168.1.1'
	option metric '1'
	option target '192.168.100.1/32'

config interface 'usb'
	option proto 'dhcp'
	option device 'usb0'
	option hostname '*'
	option peerdns '0'
	option delegate '0'
	option dns_metric '101'
	option metric '101'
	option auto '0'

config interface 'usb6'
	option proto 'dhcpv6'
	option device '@usb'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	option dns_metric '101'
	option metric '101'
	option auto '0'

config interface 'vbond0'
	option proto 'wireguard'
	option privatexx'
	option nohostroute '1'
	list addresses '10.33.23.2/30'
	option defaultroute '0'
	option peerdns '0'
	option listen_port '24680'

config wireguard_vbond0
	option description 'nvrnsdwangw01-wan'
	option publicxx'
	option presharedxx'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_host 'y.y.y.y
	option endpoint_port '24680'
	option persistent_keepalive '5'

config interface 'vbond1'
	option proto 'wireguard'
	option privatexx'
	list addresses '10.33.23.6/30'
	option nohostroute '1'
	option defaultroute '0'
	option peerdns '0'
	option listen_port '24681'

config wireguard_vbond1
	option description 'nvrnsdwangw01-wwan'
	option publicxx'
	option presharedxx'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_port '24681'
	option persistent_keepalive '5'
	option endpoint_host 'z.z.z.z'

config interface 'vbond2'
	option proto 'wireguard'
	option privatexx'
	list addresses '10.33.23.10/30'
	option nohostroute '1'
	option defaultroute '0'
	option peerdns '0'
	option listen_port '24682'

config wireguard_vbond2
	option publicxx'
	option presharedxx'
	option description 'nvrnsdwangw01'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option endpoint_port '24682'
	option persistent_keepalive '5'
	option endpoint_host 'y.y.y.y

config route
	option target '192.184.93.16/32'
	option gateway '192.168.1.1'
	option interface 'wwan'
	option metric '1'

config route
	option interface 'easytether'
	option target 'y.y.y.y
	option gateway '192.168.117.1'
	option metric '1'

package mwan3

config globals 'globals'
	option mmx_mask '0x3F00'
	option logging '1'
	option loglevel 'debug'

config interface 'wan'
	option enabled '1'
	option family 'ipv4'
	option keep_failure_interval '1'
	option interface 'wan'
	option metric '1'
	option weight '3'
	option reliability '1'
	option count '1'
	option initial_state 'online'
	option track_method 'ping'
	option size '56'
	option max_ttl '60'
	option timeout '1'
	option interval '1'
	option down '1'
	option up '1'

config rule 'sdwan_et_v4'
	option family 'ipv4'
	option proto 'udp'
	option sticky '0'
	option use_policy 'sdwan_et'
	option dest_port '24682'
	option dest_ip 'y.y.y.y
	option logging '1'

config rule 'sdwan_wan_v4'
	option proto 'udp'
	option dest_port '24680'
	option sticky '0'
	option use_policy 'sdwan_wan'
	option family 'ipv4'
	option dest_ip 'y.y.y.y

config rule 'sdwan_wwan_v4'
	option proto 'udp'
	option sticky '0'
	option use_policy 'sdwan_wwan'
	option family 'ipv4'
	option dest_port '24681'
	option dest_ip 'y.y.y.y

config rule 'StarlinkModem'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'
	option use_policy 'sdwan_wwan'
	option dest_ip '192.168.100.1'

config interface 'wwan'
	option family 'ipv4'
	option metric '2'
	option interface 'wwan'
	option weight '2'
	option count '1'
	option enabled '1'
	option reliability '1'
	option initial_state 'online'
	option track_method 'ping'
	option timeout '1'
	option interval '1'
	option down '1'
	option up '1'

config interface 'easytether'
	option enabled '1'
	option family 'ipv4'
	option interface 'easytether'
	option metric '3'
	option weight '1'
	option count '1'
	option up '1'
	option initial_state 'online'
	option reliability '1'
	option track_method 'ping'
	option size '56'
	option max_ttl '60'
	option timeout '1'
	option interval '1'
	option down '1'

config member 'wan_only'
	option interface 'wan'
	option metric '1'
	option weight '1'

config member 'wwan_only'
	option interface 'wwan'
	option metric '1'
	option weight '1'

config member 'easytether_only'
	option interface 'easytether'
	option metric '1'
	option weight '1'

config interface 'wan6'
	option family 'ipv6'
	option track_method 'ping'
	option size '56'
	option max_ttl '60'
	option enabled '1'
	option initial_state 'online'
	option reliability '1'
	option count '1'
	option timeout '1'
	option interval '1'
	option down '1'
	option up '1'

config interface 'usb'
	option family 'ipv4'
	option track_method 'ping'
	option size '56'
	option max_ttl '65'
	option up '1'
	option enabled '1'
	option initial_state 'online'
	option count '1'
	option timeout '1'
	option interval '1'
	option down '1'

config policy 'sdwan_wan'
	list use_member 'wan_only'
	option last_resort 'blackhole'

config policy 'sdwan_wwan'
	list use_member 'wwan_only'
	option last_resort 'blackhole'

config policy 'sdwan_et'
	list use_member 'easytether_only'
	option last_resort 'blackhole'

package dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option local '/lan/'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option ednspacket_max '1232'
	option rebind_protection '0'
	option domain 'lan'
	option localservice '0'
	list notinterface 'wan'
	list notinterface 'domain'
	list notinterface 'easytether'
	list interface 'guest'
	list interface 'admin'
	list interface 'devices'
	list interface 'resident'
	option confdir '/mnt/app/dnsmasq/'
	option cachesize 'ipv4-scrubbed'
	option filterwin2k '1'
	option leasefile '/mnt/app/dhcp/dhcp.leases'
	option allservers '1'
	option dnsforwardmax 'ipv4-scrubbed'
	list server '1.1.1.1'
	list server '1.0.0.1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'
	option start '100'
	option limit '150'
	option leasetime '12h'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'admin'
	option interface 'admin'
	option ra 'server'
	option ra_default '2'
	list domain 'xxx.net'
	option start '100'
	option leasetime '12h'
	option limit '150'
	list dhcp_option '15,xxx.net'
	list ra_flags 'none'

config dhcp 'domain'
	option interface 'domain'
	option dynamicdhcp '0'
	option ignore '1'
	option leasetime '12h'
	option limit '150'
	option start '100'

config dhcp 'guest'
	option start '100'
	option leasetime '12h'
	option limit '150'
	option interface 'guest'
	option ra 'server'
	option ra_management '1'
	list ra_flags 'none'
	option ra_default '2'
	list dns 'ipv6-scrubbed'
	list domain 'lan'

config dhcp 'resident'
	option interface 'resident'
	option ra 'server'
	list dns 'ipv6-scrubed'
	list dns 'ipv6-scrubed'
	list dns 'ipv6-scrubed'
	list domain 'xxx.net'
	option ra_default '2'
	option leasetime '12h'
	option start '100'
	option limit '150'
	list dhcp_option '15,xxx.net'
	list ra_flags 'none'

config dhcp 'devices'
	option interface 'devices'
	list dns 'ipv6-scrubed'
	list dns 'ipv6-scrubed'
	list dns 'ipv6-scrubed'
	list domain 'xxx.net'
	option ra_default '2'
	option start '100'
	option leasetime '12h'
	option limit '150'
	option ra 'server'
	list dhcp_option '6,addr-scrubbed'
	list dhcp_option '15,xxx.net'
	list ra_flags 'none'
	option force '1'

config isc_dhcpd 'isc_dhcpd'
	option authoritative '1'
	option default_lease_time '3600'
	option max_lease_time '86400'


package firewall

config defaults
	option output 'ACCEPT'
	option synflood_protect '1'
	option drop_invalid '1'
	option input 'DROP'
	option forward 'DROP'

config zone
	option output 'ACCEPT'
	option input 'DROP'
	option forward 'DROP'
	option name 'wan'
	option mtu_fix '1'
	list network 'wan'
	list network 'wwan'
	list network 'easytether'
	list network 'usb'
	list network 'usb6'

config zone
	option name 'sdwan_bond'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'REJECT'
	option log '1'
	option masq_allow_invalid '1'
	option mtu_fix '1'
	list network 'dummy'
	list network 'vbond0'
	list network 'vbond1'
	list network 'vbond2'

config zone
	option name 'admin'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'admin'

config zone
	option name 'ovpn'
	option output 'ACCEPT'
	option mtu_fix '1'
	option forward 'ACCEPT'
	option log '1'
	option input 'ACCEPT'
	list network 'ovpn'

config zone
	option output 'ACCEPT'
	option log '1'
	option input 'REJECT'
	option mtu_fix '1'
	option forward 'ACCEPT'
	option name 'sdwan_vpn'
	list network 'cs1'
	list network 'pv'

config zone
	option name 'domain'
	option output 'ACCEPT'
	option mtu_fix '1'
	option input 'REJECT'
	option forward 'ACCEPT'
	list network 'domain'

config rule
	option src '*'
	option name 'Allow DNS'
	list src_ip 'ipv4-scrubbed'
	list src_ip 'fd00::/8'
	option target 'ACCEPT'
	list dest_ip 'ipv4-scrubbed'
	list dest_ip 'ipv6-scrubed'
	option dest_port '53 123 547'

config rule
	option name 'DSCP DNS'
	option dest_port '53'
	option target 'DSCP'
	option set_dscp 'EF'
	option src 'domain'
	option dest 'wan'

config rule
	option name 'DSCP Voice'
	option ipset 'voice-traffic'
	option target 'DSCP'
	option set_dscp 'EF'
	list proto 'tcp'
	list proto 'udp'
	option enabled '0'
	option dest 'wan'

config rule
	option name 'DSCP Plex1'
	list proto 'tcp'
	option src '*'
	option dest_port '32400 20645'
	option target 'DSCP'
	option set_dscp 'AF31'
	option dest '*'

config rule
	option name 'DSCP TCP'
	list proto 'tcp'
	option src '*'
	option target 'DSCP'
	option set_dscp 'EF'
	option enabled '0'
	option dest 'wan'

config rule
	option name 'DSCP xxx'
	list proto 'tcp'
	option src '*'
	list dest_ip 'y.y.y.y
	option dest_port '443'
	option target 'DSCP'
	option set_dscp 'AF13'
	option dest 'wan'

config rule
	option target 'ACCEPT'
	option name 'Allow VPNGW'
	option src '*'
	list src_ip 'ipv4-scrubbed'
	list dest_ip 'ddb4:f0e0:b048::/64'
	list dest_ip '192.168.0.0/16'
	list proto 'all'
	option dest '*'

config rule
	option name 'Devices to zabbix'
	option src 'devices'
	option dest 'admin'
	option target 'ACCEPT'
	list dest_ip 'ipv4-scrubbed'
	list dest_ip 'ipv6-scrubbed'
	list proto 'all'

config rule
	option name 'Home assistant input'
	list proto 'tcp'
	option src 'devices'
	list src_ip 'ipv4-scrubbed'
	option target 'ACCEPT'
	option dest_port '443'

config rule
	list src_ip 'ipv6-scrubbed'
	option dest '*'
	list dest_ip 'fd00::/8'
	list dest_ip 'ipv4-scrubbed'
	list dest_ip '192.168.0.0/16'
	option target 'ACCEPT'
	option name 'VPN Admin'
	list proto 'all'
	option src 'sdwan_vpn'

config rule
	option name 'Allow-DHCP-Renew'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'
	option src 'wan'

config rule
	option name 'Allow-Ping'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'
	option src 'wan'

config rule
	option name 'Allow-IGMP'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'
	option src 'wan'

config rule
	option name 'Allow-DHCPv6'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'
	option src 'wan'

config rule
	option name 'Allow-MLD'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'
	option src 'wan'

config rule
	option name 'Allow-ICMPv6-Forward'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit 'ipv4-scrubbed'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled '0'
	option src 'wan'

config rule
	option name 'Support-UDP-Traceroute'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option src 'wan'

config include
	option path '/etc/firewall.user'

config include 'nat6'
	option path '/etc/firewall.nat6'
	option reload '1'

config rule 'glipv6_guest_icmp'
	option name 'Allow-ICMP-IPV6'
	option target 'ACCEPT'
	option family 'ipv6'
	option proto 'icmp'
	option src '*'

config zone
	option name 'resident'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'ACCEPT'
	list network 'resident'

config forwarding
	option src 'guest'
	option dest 'wan'

config forwarding
	option src 'resident'
	option dest 'wan'

config rule
	option src 'guest'
	option target 'ACCEPT'
	option name 'DNS in guest'
	option dest_port '53 123'

config forwarding
	option dest 'guest'
	option src 'admin'

config forwarding
	option dest 'resident'
	option src 'admin'

config forwarding
	option src 'admin'
	option dest 'sdwan_vpn'

config rule
	option dest_port '1194'
	list proto 'udp'
	option target 'ACCEPT'
	option src 'guest'
	option name 'Guest OpenVPN'

config rule
	option dest 'guest'
	option target 'DROP'
	option src_port '5353'

config rule
	option dest_port '5353'
	option src 'guest'
	option target 'DROP'

config rule
	option dest_port '5353'
	option src '*'
	option name 'mdns'
	option target 'ACCEPT'

config forwarding
	option dest 'ovpn'
	option src 'admin'

config forwarding
	option dest 'resident'
	option src 'ovpn'

config forwarding
	option src 'ovpn'
	option dest 'sdwan_vpn'

config rule
	option dest_port '67'
	option name 'Allow DHCP'
	option target 'ACCEPT'
	list proto 'udp'
	option src '*'

config rule
	option dest_port '1194'
	option src 'resident'
	option name 'Openvpn'
	option target 'ACCEPT'
	list proto 'udp'

config rule 'glipv6_guest_dhcp'
	option name 'Allow-DHCP-IPV6'
	option src 'guest'
	option target 'ACCEPT'
	option proto 'udp'
	option dest_port '547:548'
	option family 'ipv6'

config forwarding
	option src 'admin'
	option dest 'wan'

config zone
	option name 'devices'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'ACCEPT'
	list network 'devices'

config zone
	option name 'guest'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'ACCEPT'
	list network 'guest'

config forwarding
	option src 'domain'
	option dest 'wan'

config forwarding
	option src 'admin'
	option dest 'domain'

config forwarding
	option src 'devices'
	option dest 'wan'

config forwarding
	option src 'admin'
	option dest 'devices'

config forwarding
	option src 'resident'
	option dest 'devices'

config forwarding
	option src 'ovpn'
	option dest 'devices'

config forwarding
	option src 'devices'
	option dest 'sdwan_vpn'

config ipset
	option enabled '1'
	option name 'voice-traffic'
	option family 'ipv4'
	option match 'dst_ip'

config ipset
	option name 'bypassvpn'
	option enabled '1'
	option family 'ipv4'
	option match 'dst_ip'

config rule
	option name 'DSCP ALL'
	option src '*'
	option target 'DSCP'
	option set_dscp 'AF11'
	option dest 'wan'

config rule
	option name 'DSCP ALL'
	option dest '*'
	option target 'DSCP'
	option set_dscp 'AF11'
	option src '*'

config forwarding
	option src 'resident'
	option dest 'admin'

config rule
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv4-scrubbed'
	option dest '*'
	option target 'ACCEPT'
	option name 'Admin SDWAN'
	list proto 'all'
	option src 'sdwan_vpn'

config rule
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv4-scrubbed'
	option dest 'resident'
	option target 'ACCEPT'
	option name 'Resident SDWAN'
	list proto 'all'
	option src 'sdwan_vpn'

config forwarding
	option src 'resident'
	option dest 'sdwan_vpn'

config rule
	option src 'ovpn'
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv4-scrubbed'
	list src_ip 'ipv4-scrubbed'
	option dest '*'
	option target 'ACCEPT'
	option name 'Admin OVPN'
	list proto 'all'

config rule
	option src 'ovpn'
	option target 'ACCEPT'
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv4-scrubbed'
	list src_ip 'ipv4-scrubbed'
	option name 'Admin OVPN Input'
	list proto 'all'

config rule
	list proto 'tcp'
	option dest_port '179'
	option target 'ACCEPT'
	option name 'BGP SDWAN'
	option src 'sdwan_vpn'

config rule
	list src_ip 'ipv6-scrubbed'
	list src_ip 'ipv4-scrubbed'
	option target 'ACCEPT'
	option name 'ADMIN SDWAN'
	list proto 'all'
	option src 'sdwan_vpn'

config rule
	list src_ip 'ipv6-scrubbed'
	option target 'ACCEPT'
	option name 'ADMIN SDWAN Input'
	list proto 'all'
	option src 'sdwan_vpn'

config rule
	option name 'Deny VPNGW net'
	option src '*'
	option target 'REJECT'
	list proto 'all'
	option dest '*'
	list dest_ip 'ddb4:f0e0:b048::/64'
	list dest_ip '192.168.0.0/16'

config forwarding
	option src 'domain'
	option dest 'sdwan_vpn'

config rule
	option name 'OSPF WAN'
	list proto '89'
	option target 'ACCEPT'
	option src 'sdwan_bond'

config rule
	list src_ip 'ipv6-scrubed'
	list src_ip 'ipv4-scrubbed'
	option dest 'domain'
	option target 'ACCEPT'
	option name 'Domain'
	list proto 'all'
	option src 'sdwan_vpn'

config nat
	option name 'StarlinkModem'
	list proto 'all'
	option src 'wan'
	option dest_ip '192.168.100.1'
	option target 'MASQUERADE'

config forwarding
	option src 'admin'
	option dest 'sdwan_bond'

config forwarding
	option src 'devices'
	option dest 'sdwan_bond'

config forwarding
	option src 'domain'
	option dest 'sdwan_bond'

config forwarding
	option src 'guest'
	option dest 'sdwan_bond'

config forwarding
	option src 'ovpn'
	option dest 'sdwan_bond'

config forwarding
	option src 'resident'
	option dest 'sdwan_bond'

config rule
	option name 'et sdwan'
	option direction 'out'
	option device 'tap-easytether'
	list proto 'udp'
	option dest '*'
	option dest_port '24680 24681'
	option target 'REJECT'
	list dest_ip 'y.y.y.y
	list dest_ip '192.184.93.16'

config rule
	option name 'w sdwan'
	option direction 'out'
	option device 'wan'
	list proto 'udp'
	option dest_port '24681 24682'
	option target 'REJECT'
	option dest '*'
	list dest_ip 'y.y.y.y
	list dest_ip '192.184.93.16'

config rule
	option name 'ww sdwan'
	option direction 'out'
	option device 'wwan1'
	list proto 'udp'
	option dest_port '24680 24682'
	option target 'REJECT'
	option dest '*'
	list dest_ip 'y.y.y.y
	list dest_ip '192.184.93.16'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

config zone
	option name 'dummy'
	option input 'DROP'
	option output 'ACCEPT'
	option forward 'DROP'
	option mtu_fix '1'

config forwarding
	option src 'dummy'
	option dest 'wan'

config forwarding
	option src 'admin'
	option dest 'dummy'

config forwarding
	option src 'devices'
	option dest 'dummy'

config forwarding
	option src 'domain'
	option dest 'dummy'

config forwarding
	option src 'guest'
	option dest 'dummy'

config forwarding
	option src 'ovpn'
	option dest 'dummy'

config forwarding
	option src 'resident'
	option dest 'dummy'

config forwarding
	option src 'dummy'
	option dest 'sdwan_bond'

config rule
	option name 'et ww in'
	option direction 'in'
	option device 'tap-easytether'
	list proto 'udp'
	option src '*'
	option dest_port '24682'
	option target 'ACCEPT'
	list src_ip 'y.y.y.y
	option enabled '0'

config rule
	option name 'wwan sdwan in'
	option direction 'in'
	option device 'wwan1'
	list proto 'udp'
	option src '*'
	option dest_port '24681'
	option target 'ACCEPT'
	list src_ip 'y.y.y.y
	option enabled '0'

config rule
	list proto 'udp'
	list dest_ip 'y.y.y.y
	list dest_ip '192.184.93.16'
	option dest_port '24680-24682'
	option target 'REJECT'
	option direction 'out'
	option device 'vbond2'
	option dest '*'

config nat
	option name 'StarlinkRouter'
	list proto 'all'
	option src 'wan'
	option dest_ip '192.168.1.1'
	option target 'MASQUERADE'

config nat
	option name 'ATTPhone'
	list proto 'all'
	option src 'wan'
	option dest_ip '192.168.117.1'
	option target 'MASQUERADE'

config rule
	option name 'hassio dishy'
	list proto 'tcp'
	option src 'devices'
	list src_ip 'ipv4-scrubbed'
	option dest 'wan'
	list dest_ip '192.168.100.1'
	option dest_port '9200'
	option target 'ACCEPT'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
9: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.23.32.1/32 brd 255.255.255.255 scope global dummy0
       valid_lft forever preferred_lft forever
18: br-admin: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
22: br-devices: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
280: vbond1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.33.23.6/30 brd 10.33.23.7 scope global vbond1
       valid_lft forever preferred_lft forever
24: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
27: br-resident: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
52: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1378 qdisc fq_codel state UNKNOWN group default qlen 500
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
53: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
322: wwan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.163/24 brd 192.168.1.255 scope global wwan1
       valid_lft forever preferred_lft forever
103: cs1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1392 qdisc noqueue state UNKNOWN group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
104: pv: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1392 qdisc noqueue state UNKNOWN group default qlen 1000
    inet ipv4-scrubbed'
       valid_lft forever preferred_lft forever
180: tap-easytether: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.117.2/24 brd 192.168.117.255 scope global tap-easytether
       valid_lft forever preferred_lft forever
181: vbond2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.33.23.10/30 brd 10.33.23.11 scope global vbond2
       valid_lft forever preferred_lft forever
218: vbond0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.33.23.2/30 brd 10.33.23.3 scope global vbond0
       valid_lft forever preferred_lft forever
default via 192.168.1.1 dev wwan1 table 2 proto static src 192.168.1.163 metric 5 
ipv4-scrubbed'
10.33.23.0/30 dev vbond0 table 2 proto kernel scope link src 10.33.23.2 
10.33.23.4/30 dev vbond1 table 2 proto kernel scope link src 10.33.23.6 
10.33.23.8/30 dev vbond2 table 2 proto kernel scope link src 10.33.23.10 
ipv4-scrubbed'
192.168.1.0/24 dev wwan1 table 2 proto static scope link metric 5 
192.168.100.1 via 192.168.1.1 dev wwan1 table 2 proto static metric 1 
192.184.93.16 via 192.168.1.1 dev wwan1 table 2 proto static metric 1 
default via 192.168.117.1 dev tap-easytether table 3 proto static src 192.168.117.2 metric 10 
ipv4-scrubbed'
10.33.23.8/30 dev vbond2 table 3 proto kernel scope link src 10.33.23.10 
ipv4-scrubbed'
y.y.y.y
192.168.117.0/24 dev tap-easytether table 3 proto static scope link metric 10 
default nhid 12295 dev wwan1 table ipv4-scrubbed'
0.0.0.0/1 nhid 12362 via 10.33.23.9 dev vbond2 proto 196 metric 20 
default via 192.168.1.1 dev wwan1 proto static src 192.168.1.163 metric 5 
default via 192.168.117.1 dev tap-easytether proto static src 192.168.117.2 metric 10 
ipv4-scrubbed'
10.23.22.1 nhid 12362 via 10.33.23.9 dev vbond2 proto ospf metric 20 
10.33.23.0/30 dev vbond0 proto kernel scope link src 10.33.23.2 
10.33.23.4/30 dev vbond1 proto kernel scope link src 10.33.23.6 
10.33.23.8/30 dev vbond2 proto kernel scope link src 10.33.23.10 
ipv4-scrubbed'
ipv4-scrubbed'
10.255.0.0/22 nhid 1815 via ipv4-scrubbed'
128.0.0.0/1 nhid 12362 via 10.33.23.9 dev vbond2 proto 196 metric 20 
y.y.y.y
192.168.1.0/24 dev wwan1 proto static scope link metric 5 
192.168.2.0/24 nhid 1823 via ipv4-scrubbed'
192.168.100.1 via 192.168.1.1 dev wwan1 proto static metric 1 
192.168.117.0/24 dev tap-easytether proto static scope link metric 10 
192.184.93.16 via 192.168.1.1 dev wwan1 proto static metric 1 
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
local 10.23.32.1 dev dummy0 table local proto kernel scope host src 10.23.32.1 
broadcast 10.33.23.0 dev vbond0 table local proto kernel scope link src 10.33.23.2 
local 10.33.23.2 dev vbond0 table local proto kernel scope host src 10.33.23.2 
broadcast 10.33.23.3 dev vbond0 table local proto kernel scope link src 10.33.23.2 
broadcast 10.33.23.4 dev vbond1 table local proto kernel scope link src 10.33.23.6 
local 10.33.23.6 dev vbond1 table local proto kernel scope host src 10.33.23.6 
broadcast 10.33.23.7 dev vbond1 table local proto kernel scope link src 10.33.23.6 
broadcast 10.33.23.8 dev vbond2 table local proto kernel scope link src 10.33.23.10 
local 10.33.23.10 dev vbond2 table local proto kernel scope host src 10.33.23.10 
broadcast 10.33.23.11 dev vbond2 table local proto kernel scope link src 10.33.23.10 
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast ipv4-scrubbed'
local ipv4-scrubbed'
broadcast ipv4-scrubbed'
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
broadcast 192.168.1.0 dev wwan1 table local proto kernel scope link src 192.168.1.163 
local 192.168.1.163 dev wwan1 table local proto kernel scope host src 192.168.1.163 
broadcast 192.168.1.255 dev wwan1 table local proto kernel scope link src 192.168.1.163 
broadcast 192.168.117.0 dev tap-easytether table local proto kernel scope link src 192.168.117.2 
local 192.168.117.2 dev tap-easytether table local proto kernel scope host src 192.168.117.2 
broadcast 192.168.117.255 dev tap-easytether table local proto kernel scope link src 192.168.117.2 
0:	from all lookup local

[pavelgl]

@GrapeCent

only the tunnel pinned to the interface that is up with the lowest metric is able to connect or pass traffic.

This looks like a possible cause of the problem if it is still valid.
(Just trying to save you some time)

https://github.com/openwrt/packages/issues/9538#issuecomment-1047385310

When receiving connection, the original wireguard will reset the incoming fwmark (but all the incoming connection info is in kernel sk buffer) then use the 0.0.0.0 as responding source ip, use incoming source ip as dest ip to serching the route from main routing table, this cause the wireguard always use the ip of lowest metric as respoding ip.

https://lists.zx2c4.com/pipermail/wireguard/2021-September/007157.html

On ingress side, wg_reset_packet() resets skb->mark twice: with skb_scrub_packet() (xnet==true) and with memset() following it. But skb->mark does not have to be cleared at least when staying in the same net namespace, and other tunnels preserve it too similarly, especially vxlan.

EDIT:
On second thought, this should only be a problem for incoming connections.

The fwmark for outgoing/response packets should be added based on the protocol and destination port in the mangle output chain.

[trendy]

@GrapeCent

Also, tcpdump of easytether only shows an occational inbound packet from 130.44.56.28, but I don't see any RX traffic on the easytether tunnel

Could you show the tcpdump from this interface? (alter the IPs but not mask them)
Is it intentional that there is no masquerade in all the wan zones?

[GrapeCent]

I will get the dump posted but it is intentional because i route internet traffic over these tunnels via ospf, so theres no need to allow nat traffic over the isp links

[GrapeCent]

Here is the tunnel that should be locked to the easytether uplink matching on the wwan1 output chain. Currently it never connects becuase I have a firewall rule to reject tunnel connections
If I disable this firewall output rule, The tunnels all connect via the interface with the lowest metric, ignoring the

I do see MWAN rules mathing, below the DPT 24682 is supposed to be pinned to easytether, but here it's being output on wwan1

Sun Feb  5 00:21:57 2023 kern.debug kernel: [217091.550406] MWAN3(sdwan_et_v4)IN= OUT=wwan1 SRC=192.168.1.163 DST=167.88.x.x LEN=176 TOS=0x08 PREC=0x80 TTL=65 ID=40068 PROTO=UDP SPT=24682 DPT=24682 LEN=156
Sun Feb  5 00:21:57 2023 kern.debug kernel: [217091.579799] MWAN3(sdwan_et_v4)IN= OUT=wwan1 SRC=192.168.1.163 DST=167.88.x.x LEN=120 TOS=0x08 PREC=0x80 TTL=65 ID=40071 PROTO=UDP SPT=24682 DPT=24682 LEN=100
S

I also have seen tunnels try to use the fake 0.0.0.0/1 and 128.0.0.0/1 default gateway and match on the vbond interface output, so hopefully pavelgl's linked issue isn't actualy the issue I'm experiencing. I really would prefer to not return to openvpn's smaller MTU

[trendy]

Can you post again the ip -4 addr; ip -4 ro list table all; ip -4 ru; nft list ruleset
Try to scramble the IPs not completely mask them, so we can understand which one is which.

[GrapeCent]

Thanks for looking

Here is the output, the IP in question has been changed to 180.33.26.15
The 2nd ip on my VPS that I'm using to work around is 170.36.24.99

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
9: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.44.32.1/32 brd 255.255.255.255 scope global dummy0
       valid_lft forever preferred_lft forever
18: br-admin: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.44.18.1/24 brd 10.44.18.255 scope global br-admin
       valid_lft forever preferred_lft forever
22: br-devices: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.44.19.1/24 brd 10.44.19.255 scope global br-devices
       valid_lft forever preferred_lft forever
24: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.44.27.1/24 brd 10.44.27.255 scope global br-guest
       valid_lft forever preferred_lft forever
27: br-resident: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.44.26.1/24 brd 10.44.26.255 scope global br-resident
       valid_lft forever preferred_lft forever
53: dummy1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.44.53.2/29 brd 10.44.53.7 scope global dummy1
       valid_lft forever preferred_lft forever
336: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1378 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 10.44.3.1/24 scope global tun0
       valid_lft forever preferred_lft forever
338: pv: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1392 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.44.0.10/30 brd 10.44.0.11 scope global pv
       valid_lft forever preferred_lft forever
339: cs1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1392 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.44.0.1/30 brd 10.44.0.3 scope global cs1
       valid_lft forever preferred_lft forever
345: wwan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.163/24 brd 192.168.1.255 scope global wwan1
       valid_lft forever preferred_lft forever
351: vbond0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.44.23.2/30 brd 10.44.23.3 scope global vbond0
       valid_lft forever preferred_lft forever
376: vbond1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.44.23.6/30 brd 10.44.23.7 scope global vbond1
       valid_lft forever preferred_lft forever
393: vbond2: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc cake state UNKNOWN group default qlen 1000
    inet 10.44.23.10/30 brd 10.44.23.11 scope global vbond2
       valid_lft forever preferred_lft forever
180: tap-easytether: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    inet 192.168.117.2/24 brd 192.168.117.255 scope global tap-easytether
       valid_lft forever preferred_lft forever
default via 192.168.1.1 dev wwan1 table 2 proto static src 192.168.1.163 metric 5 
10.44.18.0/24 dev br-admin table 2 proto kernel scope link src 10.44.18.1 
10.44.19.0/24 dev br-devices table 2 proto kernel scope link src 10.44.19.1 
10.44.26.0/24 dev br-resident table 2 proto kernel scope link src 10.44.26.1 
10.44.27.0/24 dev br-guest table 2 proto kernel scope link src 10.44.27.1 
10.44.53.0/29 dev dummy1 table 2 proto kernel scope link src 10.44.53.2 
10.44.23.0/30 dev vbond0 table 2 proto kernel scope link src 10.44.23.2 
10.44.0.0/30 dev cs1 table 2 proto kernel scope link src 10.44.0.1 
10.44.0.8/30 dev pv table 2 proto kernel scope link src 10.44.0.10 
10.44.3.0/24 dev tun0 table 2 proto kernel scope link src 10.44.3.1 
192.168.1.0/24 dev wwan1 table 2 proto static scope link metric 5 
192.168.100.1 via 192.168.1.1 dev wwan1 table 2 proto static metric 1 
170.36.24.99 via 192.168.1.1 dev wwan1 table 2 proto static metric 1 
default via 192.168.117.1 dev tap-easytether table 3 proto static src 192.168.117.2 metric 10 
10.44.18.0/24 dev br-admin table 3 proto kernel scope link src 10.44.18.1 
10.44.19.0/24 dev br-devices table 3 proto kernel scope link src 10.44.19.1 
10.44.26.0/24 dev br-resident table 3 proto kernel scope link src 10.44.26.1 
10.44.27.0/24 dev br-guest table 3 proto kernel scope link src 10.44.27.1 
10.44.53.0/29 dev dummy1 table 3 proto kernel scope link src 10.44.53.2 
10.44.23.0/30 dev vbond0 table 3 proto kernel scope link src 10.44.23.2 
10.44.0.0/30 dev cs1 table 3 proto kernel scope link src 10.44.0.1 
10.44.0.8/30 dev pv table 3 proto kernel scope link src 10.44.0.10 
10.44.3.0/24 dev tun0 table 3 proto kernel scope link src 10.44.3.1 
180.33.26.15 via 192.168.117.1 dev tap-easytether table 3 proto static metric 1 
192.168.117.0/24 dev tap-easytether table 3 proto static scope link metric 10 
default nhid 21823 dev wwan1 table 10001 proto 195 metric 20 
0.0.0.0/1 nhid 22462 via 10.44.23.9 dev vbond2 proto 196 metric 20 
default via 192.168.1.1 dev wwan1 proto static src 192.168.1.163 metric 5 
default via 192.168.117.1 dev tap-easytether proto static src 192.168.117.2 metric 10 
10.44.18.0/24 dev br-admin proto kernel scope link src 10.44.18.1 
10.44.19.0/24 dev br-devices proto kernel scope link src 10.44.19.1 
10.44.26.0/24 dev br-resident proto kernel scope link src 10.44.26.1 
10.44.27.0/24 dev br-guest proto kernel scope link src 10.44.27.1 
10.44.53.0/29 dev dummy1 proto kernel scope link src 10.44.53.2 
10.44.53.16/30 nhid 22517 via 10.44.0.9 dev pv proto bgp metric 20 
10.44.22.1 nhid 22462 via 10.44.23.9 dev vbond2 proto ospf metric 20 
10.44.23.0/30 dev vbond0 proto kernel scope link src 10.44.23.2 
10.44.23.4/30 dev vbond1 proto kernel scope link src 10.44.23.6 
10.44.23.8/30 dev vbond2 proto kernel scope link src 10.44.23.10 
10.44.0.0/30 dev cs1 proto kernel scope link src 10.44.0.1 
10.44.0.4/30 nhid 22517 via 10.44.0.9 dev pv proto bgp metric 20 
10.44.0.8/30 dev pv proto kernel scope link src 10.44.0.10 
10.44.3.0/24 dev tun0 proto kernel scope link src 10.44.3.1 
128.0.0.0/1 nhid 22462 via 10.44.23.9 dev vbond2 proto 196 metric 20 
180.33.26.15 via 192.168.117.1 dev tap-easytether proto static metric 1 
192.168.1.0/24 dev wwan1 proto static scope link metric 5 
192.168.2.0/24 nhid 22517 via 10.44.0.9 dev pv proto bgp metric 20 
192.168.100.1 via 192.168.1.1 dev wwan1 proto static metric 1 
192.168.117.0/24 dev tap-easytether proto static scope link metric 10 
170.36.24.99 via 192.168.1.1 dev wwan1 proto static metric 1 
broadcast 10.44.18.0 dev br-admin table local proto kernel scope link src 10.44.18.1 
local 10.44.18.1 dev br-admin table local proto kernel scope host src 10.44.18.1 
broadcast 10.44.18.255 dev br-admin table local proto kernel scope link src 10.44.18.1 
broadcast 10.44.19.0 dev br-devices table local proto kernel scope link src 10.44.19.1 
local 10.44.19.1 dev br-devices table local proto kernel scope host src 10.44.19.1 
broadcast 10.44.19.255 dev br-devices table local proto kernel scope link src 10.44.19.1 
broadcast 10.44.26.0 dev br-resident table local proto kernel scope link src 10.44.26.1 
local 10.44.26.1 dev br-resident table local proto kernel scope host src 10.44.26.1 
broadcast 10.44.26.255 dev br-resident table local proto kernel scope link src 10.44.26.1 
broadcast 10.44.27.0 dev br-guest table local proto kernel scope link src 10.44.27.1 
local 10.44.27.1 dev br-guest table local proto kernel scope host src 10.44.27.1 
broadcast 10.44.27.255 dev br-guest table local proto kernel scope link src 10.44.27.1 
broadcast 10.44.53.0 dev dummy1 table local proto kernel scope link src 10.44.53.2 
local 10.44.53.2 dev dummy1 table local proto kernel scope host src 10.44.53.2 
broadcast 10.44.53.7 dev dummy1 table local proto kernel scope link src 10.44.53.2 
local 10.44.32.1 dev dummy0 table local proto kernel scope host src 10.44.32.1 
broadcast 10.44.23.0 dev vbond0 table local proto kernel scope link src 10.44.23.2 
local 10.44.23.2 dev vbond0 table local proto kernel scope host src 10.44.23.2 
broadcast 10.44.23.3 dev vbond0 table local proto kernel scope link src 10.44.23.2 
broadcast 10.44.23.4 dev vbond1 table local proto kernel scope link src 10.44.23.6 
local 10.44.23.6 dev vbond1 table local proto kernel scope host src 10.44.23.6 
broadcast 10.44.23.7 dev vbond1 table local proto kernel scope link src 10.44.23.6 
broadcast 10.44.23.8 dev vbond2 table local proto kernel scope link src 10.44.23.10 
local 10.44.23.10 dev vbond2 table local proto kernel scope host src 10.44.23.10 
broadcast 10.44.23.11 dev vbond2 table local proto kernel scope link src 10.44.23.10 
broadcast 10.44.0.0 dev cs1 table local proto kernel scope link src 10.44.0.1 
local 10.44.0.1 dev cs1 table local proto kernel scope host src 10.44.0.1 
broadcast 10.44.0.3 dev cs1 table local proto kernel scope link src 10.44.0.1 
broadcast 10.44.0.8 dev pv table local proto kernel scope link src 10.44.0.10 
local 10.44.0.10 dev pv table local proto kernel scope host src 10.44.0.10 
broadcast 10.44.0.11 dev pv table local proto kernel scope link src 10.44.0.10 
broadcast 10.44.3.0 dev tun0 table local proto kernel scope link src 10.44.3.1 
local 10.44.3.1 dev tun0 table local proto kernel scope host src 10.44.3.1 
broadcast 10.44.3.255 dev tun0 table local proto kernel scope link src 10.44.3.1 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
broadcast 192.168.1.0 dev wwan1 table local proto kernel scope link src 192.168.1.163 
local 192.168.1.163 dev wwan1 table local proto kernel scope host src 192.168.1.163 
broadcast 192.168.1.255 dev wwan1 table local proto kernel scope link src 192.168.1.163 
broadcast 192.168.117.0 dev tap-easytether table local proto kernel scope link src 192.168.117.2 
local 192.168.117.2 dev tap-easytether table local proto kernel scope host src 192.168.117.2 
broadcast 192.168.117.255 dev tap-easytether table local proto kernel scope link src 192.168.117.2 
0:	from all lookup local
1002:	from all iif wwan1 lookup 2
1003:	from all iif tap-easytether lookup 3
2002:	from all fwmark 0x200/0x3f00 lookup 2
2003:	from all fwmark 0x300/0x3f00 lookup 3
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
3003:	from all fwmark 0x300/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

[GrapeCent]

Character limit

table inet fw4 {
	set voice-traffic {
		type ipv4_addr
	}

	set bypassvpn {
		type ipv4_addr
	}

	chain mangle_ttl_out {
		type filter hook postrouting priority mangle; policy accept;
		oifname "usb0" ip ttl set 64
	}

	chain input {
		type filter hook input priority filter; policy drop;
		iifname "lo" accept comment "!fw4: Accept traffic from loopback"
		ct state established,related accept comment "!fw4: Allow inbound established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
		ip saddr 10.44.0.0/8 ip daddr 10.44.53.2 tcp dport { 53, 123, 547 } counter packets 0 bytes 0 accept comment "!fw4: Allow DNS"
		ip6 saddr fd00::/8 ip6 daddr hhhh:jjjj:53:1::2 tcp dport { 53, 123, 547 } counter packets 5 bytes 420 accept comment "!fw4: Allow DNS"
		ip saddr 10.44.0.0/8 ip daddr 10.44.53.2 udp dport { 53, 123, 547 } counter packets 13797 bytes 948463 accept comment "!fw4: Allow DNS"
		ip6 saddr fd00::/8 ip6 daddr hhhh:jjjj:53:1::2 udp dport { 53, 123, 547 } counter packets 18881 bytes 1616596 accept comment "!fw4: Allow DNS"
		tcp dport 5353 counter packets 0 bytes 0 accept comment "!fw4: mdns"
		udp dport 5353 counter packets 0 bytes 0 accept comment "!fw4: mdns"
		udp dport 67 counter packets 39 bytes 14908 accept comment "!fw4: Allow DHCP"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump input_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 input traffic"
		iifname "br-admin" jump input_admin comment "!fw4: Handle admin IPv4/IPv6 input traffic"
		iifname "tun0" jump input_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 input traffic"
		iifname { "pv", "cs1" } jump input_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 input traffic"
		iifname "dummy1" jump input_domain comment "!fw4: Handle domain IPv4/IPv6 input traffic"
		iifname "br-resident" jump input_resident comment "!fw4: Handle resident IPv4/IPv6 input traffic"
		iifname "br-devices" jump input_devices comment "!fw4: Handle devices IPv4/IPv6 input traffic"
		iifname "br-guest" jump input_guest comment "!fw4: Handle guest IPv4/IPv6 input traffic"
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		ip saddr { 10.44.2.0/24, 10.44.18.0/24, 10.44.1.0-10.44.3.255 } ip daddr 192.168.0.0/16 counter packets 1823 bytes 132348 accept comment "!fw4: Allow VPNGW"
		ip6 saddr { hhhh:jjjj:0:2::/64, hhhh:jjjj:0:42::/64, hhhh:jjjj:fffd::/64, hhhh:jjjj:fffe::/64, hhhh:jjjj:ffff::/64 } ip6 daddr ddb4:f0e0:b048::/64 counter packets 0 bytes 0 accept comment "!fw4: Allow VPNGW"
		ip daddr 192.168.0.0/16 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Deny VPNGW net"
		ip6 daddr ddb4:f0e0:b048::/64 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Deny VPNGW net"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump forward_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 forward traffic"
		iifname "br-admin" jump forward_admin comment "!fw4: Handle admin IPv4/IPv6 forward traffic"
		iifname "tun0" jump forward_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 forward traffic"
		iifname { "pv", "cs1" } jump forward_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 forward traffic"
		iifname "dummy1" jump forward_domain comment "!fw4: Handle domain IPv4/IPv6 forward traffic"
		iifname "br-resident" jump forward_resident comment "!fw4: Handle resident IPv4/IPv6 forward traffic"
		iifname "br-devices" jump forward_devices comment "!fw4: Handle devices IPv4/IPv6 forward traffic"
		iifname "br-guest" jump forward_guest comment "!fw4: Handle guest IPv4/IPv6 forward traffic"
	}

	chain output {
		type filter hook output priority filter; policy accept;
		oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		oifname "wan" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport { 24681, 24682 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: w sdwan"
		oifname "wwan1" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport { 24680, 24682 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: ww sdwan"
		oifname "tap-easytether" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport { 24680, 24681 } counter packets 12445 bytes 2083640 jump handle_reject comment "!fw4: et sdwan"
		oifname "vbond0" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport 24680-24682 counter packets 0 bytes 0 jump handle_reject comment "!fw4: @rule[43]"
		oifname "vbond1" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport 24680-24682 counter packets 0 bytes 0 jump handle_reject comment "!fw4: @rule[44]"
		oifname "vbond2" ip daddr { 180.33.26.15, 170.36.24.99 } udp dport 24680-24682 counter packets 22 bytes 3032 jump handle_reject comment "!fw4: @rule[45]"
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump output_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 output traffic"
		oifname "br-admin" jump output_admin comment "!fw4: Handle admin IPv4/IPv6 output traffic"
		oifname "tun0" jump output_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 output traffic"
		oifname { "pv", "cs1" } jump output_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 output traffic"
		oifname "dummy1" jump output_domain comment "!fw4: Handle domain IPv4/IPv6 output traffic"
		oifname "br-resident" jump output_resident comment "!fw4: Handle resident IPv4/IPv6 output traffic"
		oifname "br-devices" jump output_devices comment "!fw4: Handle devices IPv4/IPv6 output traffic"
		oifname "br-guest" jump output_guest comment "!fw4: Handle guest IPv4/IPv6 output traffic"
	}

	chain prerouting {
		type filter hook prerouting priority filter; policy accept;
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump helper_wan comment "!fw4: Handle wan IPv4/IPv6 helper assignment"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump helper_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 helper assignment"
		iifname "br-admin" jump helper_admin comment "!fw4: Handle admin IPv4/IPv6 helper assignment"
		iifname "tun0" jump helper_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 helper assignment"
		iifname { "pv", "cs1" } jump helper_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 helper assignment"
		iifname "dummy1" jump helper_domain comment "!fw4: Handle domain IPv4/IPv6 helper assignment"
		iifname "br-resident" jump helper_resident comment "!fw4: Handle resident IPv4/IPv6 helper assignment"
		iifname "br-devices" jump helper_devices comment "!fw4: Handle devices IPv4/IPv6 helper assignment"
		iifname "br-guest" jump helper_guest comment "!fw4: Handle guest IPv4/IPv6 helper assignment"
	}

	chain handle_reject {
		meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
		reject comment "!fw4: Reject any other traffic"
	}

	chain syn_flood {
		limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
		drop comment "!fw4: Drop excess packets"
	}

	chain input_wan {
		meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
		icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
		meta nfproto ipv4 meta l4proto igmp counter packets 260 bytes 9360 accept comment "!fw4: Allow-IGMP"
		meta nfproto ipv6 udp dport 546 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCPv6"
		ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 260 bytes 19760 accept comment "!fw4: Allow-MLD"
		meta nfproto ipv4 udp dport 33434-33689 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Support-UDP-Traceroute"
		jump drop_from_wan
	}

	chain output_wan {
		jump accept_to_wan
	}

	chain forward_wan {
		jump drop_to_wan
	}

	chain helper_wan {
	}

	chain accept_to_wan {
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 10317 bytes 11039288 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
	}

	chain drop_from_wan {
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 155 bytes 31128 drop comment "!fw4: drop wan IPv4/IPv6 traffic"
	}

	chain drop_to_wan {
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 drop comment "!fw4: drop wan IPv4/IPv6 traffic"
	}

	chain input_sdwan_bond {
		meta l4proto ospf counter packets 28804 bytes 2420180 accept comment "!fw4: OSPF WAN"
		jump reject_from_sdwan_bond
	}

	chain output_sdwan_bond {
		jump accept_to_sdwan_bond
	}

	chain forward_sdwan_bond {
		jump reject_to_sdwan_bond
		log prefix "drop sdwan_bond forward: "
	}

	chain helper_sdwan_bond {
	}

	chain accept_to_sdwan_bond {
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 228785 bytes 18204003 accept comment "!fw4: accept sdwan_bond IPv4/IPv6 traffic"
	}

	chain reject_from_sdwan_bond {
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 133 bytes 5708 log prefix "reject sdwan_bond in: " jump handle_reject comment "!fw4: reject sdwan_bond IPv4/IPv6 traffic"
	}

	chain reject_to_sdwan_bond {
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 0 bytes 0 log prefix "reject sdwan_bond out: " jump handle_reject comment "!fw4: reject sdwan_bond IPv4/IPv6 traffic"
	}

	chain input_admin {
		jump accept_from_admin
	}

	chain output_admin {
		jump accept_to_admin
	}

	chain forward_admin {
		jump accept_to_guest comment "!fw4: Accept admin to guest forwarding"
		jump accept_to_resident comment "!fw4: Accept admin to resident forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept admin to sdwan_vpn forwarding"
		jump accept_to_ovpn comment "!fw4: Accept admin to ovpn forwarding"
		jump accept_to_wan comment "!fw4: Accept admin to wan forwarding"
		jump accept_to_domain comment "!fw4: Accept admin to domain forwarding"
		jump accept_to_devices comment "!fw4: Accept admin to devices forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept admin to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept admin to dummy forwarding"
		jump accept_to_admin
	}

	chain helper_admin {
	}

	chain accept_from_admin {
		iifname "br-admin" counter packets 71470 bytes 7726524 accept comment "!fw4: accept admin IPv4/IPv6 traffic"
	}

	chain accept_to_admin {
		oifname "br-admin" counter packets 5361 bytes 421912 accept comment "!fw4: accept admin IPv4/IPv6 traffic"
	}

	chain input_ovpn {
		ip saddr { 10.44.3.200, 10.44.3.201 } counter packets 0 bytes 0 accept comment "!fw4: Admin OVPN Input"
		ip6 saddr { hhhh:jjjj:fffd::2000, hhhh:jjjj:fffd::2001 } counter packets 21 bytes 1680 accept comment "!fw4: Admin OVPN Input"
		jump accept_from_ovpn
	}

	chain output_ovpn {
		jump accept_to_ovpn
	}

	chain forward_ovpn {
		ip saddr { 10.44.3.200, 10.44.3.201 } counter packets 5073 bytes 312302 accept comment "!fw4: Admin OVPN"
		ip6 saddr { hhhh:jjjj:fffd::2000, hhhh:jjjj:fffd::2001 } counter packets 24 bytes 1954 accept comment "!fw4: Admin OVPN"
		jump accept_to_resident comment "!fw4: Accept ovpn to resident forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept ovpn to sdwan_vpn forwarding"
		jump accept_to_devices comment "!fw4: Accept ovpn to devices forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept ovpn to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept ovpn to dummy forwarding"
		jump accept_to_ovpn
		log prefix "drop ovpn forward: "
	}

	chain helper_ovpn {
	}

	chain accept_from_ovpn {
		iifname "tun0" counter packets 0 bytes 0 accept comment "!fw4: accept ovpn IPv4/IPv6 traffic"
	}

	chain accept_to_ovpn {
		oifname "tun0" counter packets 4 bytes 304 accept comment "!fw4: accept ovpn IPv4/IPv6 traffic"
	}

	chain input_sdwan_vpn {
		tcp dport 179 counter packets 125 bytes 9960 accept comment "!fw4: BGP SDWAN"
		ip saddr 10.44.2.0/24 counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN"
		ip6 saddr hhhh:jjjj:0:2::/64 counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN"
		ip saddr { 10.44.1.200, 10.44.1.201, 10.44.1.202, 10.44.1.203, 10.44.1.204, 10.44.1.205, 10.44.1.206, 10.44.1.207, 10.44.1.208, 10.44.1.209, 10.44.1.210, 10.44.2.200, 10.44.2.201, 10.44.2.202, 10.44.2.203, 10.44.2.204, 10.44.2.205, 10.44.2.206, 10.44.2.207, 10.44.2.208, 10.44.2.209, 10.44.2.210 } counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN Input"
		ip6 saddr { hhhh:jjjj:fffe::2000, hhhh:jjjj:fffe::2001, hhhh:jjjj:fffe::2002, hhhh:jjjj:fffe::2003, hhhh:jjjj:fffe::2004, hhhh:jjjj:fffe::2005, hhhh:jjjj:fffe::2006, hhhh:jjjj:fffe::2007, hhhh:jjjj:fffe::2008, hhhh:jjjj:fffe::2009, hhhh:jjjj:fffe::2010, hhhh:jjjj:ffff::2000, hhhh:jjjj:ffff::2001, hhhh:jjjj:ffff::2002, hhhh:jjjj:ffff::2003, hhhh:jjjj:ffff::2004, hhhh:jjjj:ffff::2005, hhhh:jjjj:ffff::2006, hhhh:jjjj:ffff::2007, hhhh:jjjj:ffff::2008, hhhh:jjjj:ffff::2009, hhhh:jjjj:ffff::2010 } counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN Input"
		jump reject_from_sdwan_vpn
	}

	chain output_sdwan_vpn {
		jump accept_to_sdwan_vpn
	}

	chain forward_sdwan_vpn {
		ip saddr { 10.44.1.200, 10.44.1.201, 10.44.1.202, 10.44.2.200, 10.44.2.201, 10.44.2.202 } ip daddr { 10.44.0.0/8, 192.168.0.0/16 } counter packets 0 bytes 0 accept comment "!fw4: VPN Admin"
		ip6 saddr { hhhh:jjjj:fffe::2000, hhhh:jjjj:fffe::2001, hhhh:jjjj:fffe::2002, hhhh:jjjj:ffff::2000, hhhh:jjjj:ffff::2001, hhhh:jjjj:ffff::2002 } ip6 daddr fd00::/8 counter packets 0 bytes 0 accept comment "!fw4: VPN Admin"
		ip saddr 10.44.2.0/24 counter packets 0 bytes 0 accept comment "!fw4: Admin SDWAN"
		ip6 saddr hhhh:jjjj:0:2::/64 counter packets 0 bytes 0 accept comment "!fw4: Admin SDWAN"
		ip saddr 10.44.10.0/24 counter packets 0 bytes 0 jump accept_to_resident comment "!fw4: Resident SDWAN"
		ip6 saddr hhhh:jjjj:0:10::/64 counter packets 0 bytes 0 jump accept_to_resident comment "!fw4: Resident SDWAN"
		ip saddr 10.44.53.0/27 counter packets 0 bytes 0 jump accept_to_domain comment "!fw4: Domain"
		ip6 saddr hhhh:jjjj:53::/62 counter packets 0 bytes 0 jump accept_to_domain comment "!fw4: Domain"
		jump accept_to_sdwan_vpn
		log prefix "drop sdwan_vpn forward: "
	}

	chain helper_sdwan_vpn {
	}

	chain accept_to_sdwan_vpn {
		oifname { "pv", "cs1" } counter packets 6890 bytes 545045 accept comment "!fw4: accept sdwan_vpn IPv4/IPv6 traffic"
	}

	chain reject_from_sdwan_vpn {
		iifname { "pv", "cs1" } counter packets 0 bytes 0 log prefix "reject sdwan_vpn in: " jump handle_reject comment "!fw4: reject sdwan_vpn IPv4/IPv6 traffic"
	}

	chain input_domain {
		jump reject_from_domain
	}

	chain output_domain {
		jump accept_to_domain
	}

	chain forward_domain {
		jump accept_to_wan comment "!fw4: Accept domain to wan forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept domain to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept domain to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept domain to dummy forwarding"
		jump accept_to_domain
	}

	chain helper_domain {
	}

	chain accept_to_domain {
		oifname "dummy1" counter packets 0 bytes 0 accept comment "!fw4: accept domain IPv4/IPv6 traffic"
	}

	chain reject_from_domain {
		iifname "dummy1" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject domain IPv4/IPv6 traffic"
	}

	chain input_resident {
		udp dport 1194 counter packets 0 bytes 0 accept comment "!fw4: Openvpn"
		jump reject_from_resident
	}

	chain output_resident {
		jump accept_to_resident
	}

	chain forward_resident {
		jump accept_to_wan comment "!fw4: Accept resident to wan forwarding"
		jump accept_to_devices comment "!fw4: Accept resident to devices forwarding"
		jump accept_to_admin comment "!fw4: Accept resident to admin forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept resident to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept resident to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept resident to dummy forwarding"
		jump accept_to_resident
	}

	chain helper_resident {
	}

	chain accept_to_resident {
		oifname "br-resident" counter packets 6607 bytes 541088 accept comment "!fw4: accept resident IPv4/IPv6 traffic"
	}

	chain reject_from_resident {
		iifname "br-resident" counter packets 2062 bytes 159412 jump handle_reject comment "!fw4: reject resident IPv4/IPv6 traffic"
	}

	chain input_devices {
		ip saddr 10.44.19.27 tcp dport 443 counter packets 48 bytes 2880 accept comment "!fw4: Home assistant input"
		jump reject_from_devices
	}

	chain output_devices {
		jump accept_to_devices
	}

	chain forward_devices {
		ip saddr 10.44.19.27 ip daddr 192.168.100.1 tcp dport 9200 counter packets 0 bytes 0 jump accept_to_wan comment "!fw4: hassio dishy"
		ip daddr 10.44.18.79 counter packets 0 bytes 0 jump accept_to_admin comment "!fw4: Devices to zabbix"
		ip6 daddr hhhh:jjjj:0:42:5054:ff:fecb:aedf counter packets 151 bytes 12080 jump accept_to_admin comment "!fw4: Devices to zabbix"
		jump accept_to_wan comment "!fw4: Accept devices to wan forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept devices to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept devices to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept devices to dummy forwarding"
		jump accept_to_devices
	}

	chain helper_devices {
	}

	chain accept_to_devices {
		oifname "br-devices" counter packets 25323 bytes 2103215 accept comment "!fw4: accept devices IPv4/IPv6 traffic"
	}

	chain reject_from_devices {
		iifname "br-devices" counter packets 42048 bytes 3413403 jump handle_reject comment "!fw4: reject devices IPv4/IPv6 traffic"
	}

	chain input_guest {
		tcp dport { 53, 123 } counter packets 0 bytes 0 accept comment "!fw4: DNS in guest"
		udp dport { 53, 123 } counter packets 1087 bytes 75052 accept comment "!fw4: DNS in guest"
		udp dport 1194 counter packets 0 bytes 0 accept comment "!fw4: Guest OpenVPN"
		tcp dport 5353 counter packets 0 bytes 0 drop comment "!fw4: @rule[21]"
		udp dport 5353 counter packets 0 bytes 0 drop comment "!fw4: @rule[21]"
		meta nfproto ipv6 udp dport 547-548 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-IPV6"
		jump reject_from_guest
	}

	chain output_guest {
		tcp sport 5353 counter packets 0 bytes 0 jump drop_to_guest comment "!fw4: @rule[20]"
		udp sport 5353 counter packets 0 bytes 0 jump drop_to_guest comment "!fw4: @rule[20]"
		jump accept_to_guest
	}

	chain forward_guest {
		jump accept_to_wan comment "!fw4: Accept guest to wan forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept guest to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept guest to dummy forwarding"
		jump accept_to_guest
	}

	chain helper_guest {
	}

	chain accept_to_guest {
		oifname "br-guest" counter packets 503 bytes 42728 accept comment "!fw4: accept guest IPv4/IPv6 traffic"
	}

	chain reject_from_guest {
		iifname "br-guest" counter packets 2304 bytes 165888 jump handle_reject comment "!fw4: reject guest IPv4/IPv6 traffic"
	}

	chain drop_to_guest {
		oifname "br-guest" counter packets 0 bytes 0 drop comment "!fw4: drop guest IPv4/IPv6 traffic"
	}

	chain input_dummy {
		jump drop_from_dummy
	}

	chain output_dummy {
		jump accept_to_dummy
	}

	chain forward_dummy {
		jump accept_to_wan comment "!fw4: Accept dummy to wan forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept dummy to sdwan_bond forwarding"
		jump drop_to_dummy
	}

	chain helper_dummy {
	}

	chain accept_to_dummy {
	}

	chain drop_from_dummy {
	}

	chain drop_to_dummy {
	}

	chain dstnat {
		type nat hook prerouting priority dstnat; policy accept;
	}

	chain srcnat {
		type nat hook postrouting priority srcnat; policy accept;
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
	}

	chain srcnat_wan {
		ip daddr 192.168.100.1 counter packets 1296 bytes 89208 masquerade comment "!fw4: StarlinkModem"
		ip daddr 192.168.1.1 counter packets 48 bytes 3148 masquerade comment "!fw4: StarlinkRouter"
		ip daddr 192.168.117.1 counter packets 478 bytes 40152 masquerade comment "!fw4: ATTPhone"
	}

	chain raw_prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain raw_output {
		type filter hook output priority raw; policy accept;
	}

	chain mangle_prerouting {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain mangle_postrouting {
		type filter hook postrouting priority mangle; policy accept;
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } ip daddr 130.5.77.2 tcp dport 443 counter packets 0 bytes 0 ip dscp set af13 comment "!fw4: DSCP 0xCBF"
		meta nfproto ipv4 meta l4proto tcp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 152924 bytes 24595274 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto tcp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv4 meta l4proto udp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 2493781 bytes 419859052 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto udp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 ip6 dscp set af11 comment "!fw4: DSCP ALL"
	}

	chain mangle_input {
		type filter hook input priority mangle; policy accept;
	}

	chain mangle_output {
		type route hook output priority mangle; policy accept;
	}

	chain mangle_forward {
		type filter hook forward priority mangle; policy accept;
		meta nfproto ipv4 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp dport 53 counter packets 0 bytes 0 ip dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv6 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp dport 53 counter packets 0 bytes 0 ip6 dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv4 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } udp dport 53 counter packets 0 bytes 0 ip dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv6 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } udp dport 53 counter packets 0 bytes 0 ip6 dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv4 tcp dport { 20645, 32400 } counter packets 7620 bytes 532860 ip dscp set af31 comment "!fw4: DSCP Plex1"
		meta nfproto ipv6 tcp dport { 20645, 32400 } counter packets 0 bytes 0 ip6 dscp set af31 comment "!fw4: DSCP Plex1"
		meta nfproto ipv4 meta l4proto tcp counter packets 5151104 bytes 819045955 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto tcp counter packets 4622 bytes 369696 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv4 meta l4proto udp counter packets 142249 bytes 122183377 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto udp counter packets 151 bytes 14208 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_bond IPv4/IPv6 ingress MTU fixing"
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_bond IPv4/IPv6 egress MTU fixing"
		iifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone ovpn IPv4/IPv6 ingress MTU fixing"
		oifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone ovpn IPv4/IPv6 egress MTU fixing"
		iifname { "pv", "cs1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_vpn IPv4/IPv6 ingress MTU fixing"
		oifname { "pv", "cs1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_vpn IPv4/IPv6 egress MTU fixing"
		iifname "dummy1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone domain IPv4/IPv6 ingress MTU fixing"
		oifname "dummy1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone domain IPv4/IPv6 egress MTU fixing"
	}
}

[trendy]

One thing is that there are the ip rules based on fwmarks (correct) but I don't see any rules applying fwmarks in the nftables output.

[GrapeCent]

So that would be an MWAN3 issue, not applying rules for the FWMark it's configured to. Can you give me a suggestion on how to kick start mwan3 to add these rules?

Edit,

Your observation got me in the right direction and I installed iptables-nft, now I see the mangle rules and xt_mark rules. But my excitement faded when I realized it's still not working, only vbond1 can connect with the interface wwan1 is up because of it's lower metric.

Edit 2How do i exclude routes from mwan3 directly connected networks? I see the fake 0.0.0.0/1 route listed as directly connected which seems likely to be the problem.

nft list ruleset
table inet fw4 {
	set voice-traffic {
		type ipv4_addr
	}

	set bypassvpn {
		type ipv4_addr
	}

	flowtable ft {
		hook ingress priority filter
		devices = { lan1, lan2, lan3, lan4, wan }
		flags offload
	}

	chain mangle_ttl_out {
		type filter hook postrouting priority mangle; policy accept;
		oifname "usb0" ip ttl set 64
	}

	chain input {
		type filter hook input priority filter; policy drop;
		iifname "lo" accept comment "!fw4: Accept traffic from loopback"
		ct state established,related accept comment "!fw4: Allow inbound established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
		ip saddr 10.0.0.0/8 ip daddr 10.0.53.2 tcp dport { 53, 123, 547 } counter packets 0 bytes 0 accept comment "!fw4: Allow DNS"
		ip6 saddr fd00::/8 ip6 daddr fd00:f9a8:53:1::2 tcp dport { 53, 123, 547 } counter packets 7 bytes 588 accept comment "!fw4: Allow DNS"
		ip saddr 10.0.0.0/8 ip daddr 10.0.53.2 udp dport { 53, 123, 547 } counter packets 4326 bytes 288578 accept comment "!fw4: Allow DNS"
		ip6 saddr fd00::/8 ip6 daddr fd00:f9a8:53:1::2 udp dport { 53, 123, 547 } counter packets 383 bytes 33771 accept comment "!fw4: Allow DNS"
		tcp dport 5353 counter packets 0 bytes 0 accept comment "!fw4: mdns"
		udp dport 5353 counter packets 0 bytes 0 accept comment "!fw4: mdns"
		udp dport 67 counter packets 20 bytes 6720 accept comment "!fw4: Allow DHCP"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump input_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 input traffic"
		iifname "br-admin" jump input_admin comment "!fw4: Handle admin IPv4/IPv6 input traffic"
		iifname "tun0" jump input_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 input traffic"
		iifname { "pv", "cs1" } jump input_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 input traffic"
		iifname "dummy1" jump input_domain comment "!fw4: Handle domain IPv4/IPv6 input traffic"
		iifname "br-resident" jump input_resident comment "!fw4: Handle resident IPv4/IPv6 input traffic"
		iifname "br-devices" jump input_devices comment "!fw4: Handle devices IPv4/IPv6 input traffic"
		iifname "br-guest" jump input_guest comment "!fw4: Handle guest IPv4/IPv6 input traffic"
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
		meta l4proto { tcp, udp } flow add @ft
		ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		ip saddr { 10.0.2.0/24, 10.0.18.0/24, 10.234.1.0-10.234.3.255 } ip daddr 192.168.0.0/16 counter packets 16 bytes 1152 accept comment "!fw4: Allow VPNGW"
		ip6 saddr { fd00:f9a8:0:2::/64, fd00:f9a8:0:42::/64, fd00:f9a8:fffd::/64, fd00:f9a8:fffe::/64, fd00:f9a8:ffff::/64 } ip6 daddr ddb4:f0e0:b048::/64 counter packets 0 bytes 0 accept comment "!fw4: Allow VPNGW"
		ip daddr 192.168.0.0/16 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Deny VPNGW net"
		ip6 daddr ddb4:f0e0:b048::/64 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Deny VPNGW net"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump forward_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 forward traffic"
		iifname "br-admin" jump forward_admin comment "!fw4: Handle admin IPv4/IPv6 forward traffic"
		iifname "tun0" jump forward_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 forward traffic"
		iifname { "pv", "cs1" } jump forward_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 forward traffic"
		iifname "dummy1" jump forward_domain comment "!fw4: Handle domain IPv4/IPv6 forward traffic"
		iifname "br-resident" jump forward_resident comment "!fw4: Handle resident IPv4/IPv6 forward traffic"
		iifname "br-devices" jump forward_devices comment "!fw4: Handle devices IPv4/IPv6 forward traffic"
		iifname "br-guest" jump forward_guest comment "!fw4: Handle guest IPv4/IPv6 forward traffic"
	}

	chain output {
		type filter hook output priority filter; policy accept;
		oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
		ct state established,related accept comment "!fw4: Allow outbound established and related flows"
		ct state invalid drop comment "!fw4: Drop flows with invalid conntrack state"
		oifname "wan" ip daddr { 180.33.26.15,  170.36.24.99 } udp dport { 53, 9469 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: w sdwan"
		oifname "wwan1" ip daddr { 180.33.26.156, 170.36.24.99 } udp dport { 9468, 9469 } counter packets 43 bytes 7568 jump handle_reject comment "!fw4: ww sdwan"
		oifname "tap-easytether" ip daddr { 180.33.26.15,  170.36.24.996 } udp dport { 53, 9468 } counter packets 6 bytes 1056 jump handle_reject comment "!fw4: et sdwan"
		oifname "vbond0" ip daddr { 180.33.26.15,  170.36.24.99 } udp dport { 53, 9468, 9469 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: @rule[43]"
		oifname "vbond1" ip daddr { 180.33.26.15,  170.36.24.99 } udp dport { 53, 9468, 9469 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: @rule[44]"
		oifname "vbond2" ip daddr { 180.33.26.15,  170.36.24.99 } udp dport { 53, 9468, 9469 } counter packets 0 bytes 0 jump handle_reject comment "!fw4: @rule[45]"
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump output_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 output traffic"
		oifname "br-admin" jump output_admin comment "!fw4: Handle admin IPv4/IPv6 output traffic"
		oifname "tun0" jump output_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 output traffic"
		oifname { "pv", "cs1" } jump output_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 output traffic"
		oifname "dummy1" jump output_domain comment "!fw4: Handle domain IPv4/IPv6 output traffic"
		oifname "br-resident" jump output_resident comment "!fw4: Handle resident IPv4/IPv6 output traffic"
		oifname "br-devices" jump output_devices comment "!fw4: Handle devices IPv4/IPv6 output traffic"
		oifname "br-guest" jump output_guest comment "!fw4: Handle guest IPv4/IPv6 output traffic"
	}

	chain prerouting {
		type filter hook prerouting priority filter; policy accept;
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } jump helper_wan comment "!fw4: Handle wan IPv4/IPv6 helper assignment"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } jump helper_sdwan_bond comment "!fw4: Handle sdwan_bond IPv4/IPv6 helper assignment"
		iifname "br-admin" jump helper_admin comment "!fw4: Handle admin IPv4/IPv6 helper assignment"
		iifname "tun0" jump helper_ovpn comment "!fw4: Handle ovpn IPv4/IPv6 helper assignment"
		iifname { "pv", "cs1" } jump helper_sdwan_vpn comment "!fw4: Handle sdwan_vpn IPv4/IPv6 helper assignment"
		iifname "dummy1" jump helper_domain comment "!fw4: Handle domain IPv4/IPv6 helper assignment"
		iifname "br-resident" jump helper_resident comment "!fw4: Handle resident IPv4/IPv6 helper assignment"
		iifname "br-devices" jump helper_devices comment "!fw4: Handle devices IPv4/IPv6 helper assignment"
		iifname "br-guest" jump helper_guest comment "!fw4: Handle guest IPv4/IPv6 helper assignment"
	}

	chain handle_reject {
		meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
		reject comment "!fw4: Reject any other traffic"
	}

	chain syn_flood {
		limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
		drop comment "!fw4: Drop excess packets"
	}

	chain input_wan {
		meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
		icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
		meta nfproto ipv4 meta l4proto igmp counter packets 2 bytes 72 accept comment "!fw4: Allow-IGMP"
		meta nfproto ipv6 udp dport 546 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCPv6"
		ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 2 bytes 152 accept comment "!fw4: Allow-MLD"
		meta nfproto ipv4 udp dport 33434-33689 counter packets 0 bytes 0 jump handle_reject comment "!fw4: Support-UDP-Traceroute"
		jump drop_from_wan
	}

	chain output_wan {
		jump accept_to_wan
	}

	chain forward_wan {
		jump drop_to_wan
	}

	chain helper_wan {
	}

	chain accept_to_wan {
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 16789 bytes 1313255 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
	}

	chain drop_from_wan {
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 drop comment "!fw4: drop wan IPv4/IPv6 traffic"
	}

	chain drop_to_wan {
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 drop comment "!fw4: drop wan IPv4/IPv6 traffic"
	}

	chain input_sdwan_bond {
		meta l4proto ospf counter packets 34 bytes 2856 accept comment "!fw4: OSPF WAN"
		jump reject_from_sdwan_bond
	}

	chain output_sdwan_bond {
		jump accept_to_sdwan_bond
	}

	chain forward_sdwan_bond {
		jump reject_to_sdwan_bond
		log prefix "drop sdwan_bond forward: "
	}

	chain helper_sdwan_bond {
	}

	chain accept_to_sdwan_bond {
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 976 bytes 78112 accept comment "!fw4: accept sdwan_bond IPv4/IPv6 traffic"
	}

	chain reject_from_sdwan_bond {
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 13 bytes 2288 log prefix "reject sdwan_bond in: " jump handle_reject comment "!fw4: reject sdwan_bond IPv4/IPv6 traffic"
	}

	chain reject_to_sdwan_bond {
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } counter packets 0 bytes 0 log prefix "reject sdwan_bond out: " jump handle_reject comment "!fw4: reject sdwan_bond IPv4/IPv6 traffic"
	}

	chain input_admin {
		jump accept_from_admin
	}

	chain output_admin {
		jump accept_to_admin
	}

	chain forward_admin {
		jump accept_to_guest comment "!fw4: Accept admin to guest forwarding"
		jump accept_to_resident comment "!fw4: Accept admin to resident forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept admin to sdwan_vpn forwarding"
		jump accept_to_ovpn comment "!fw4: Accept admin to ovpn forwarding"
		jump accept_to_wan comment "!fw4: Accept admin to wan forwarding"
		jump accept_to_domain comment "!fw4: Accept admin to domain forwarding"
		jump accept_to_devices comment "!fw4: Accept admin to devices forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept admin to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept admin to dummy forwarding"
		jump accept_to_admin
	}

	chain helper_admin {
	}

	chain accept_from_admin {
		iifname "br-admin" counter packets 603 bytes 65080 accept comment "!fw4: accept admin IPv4/IPv6 traffic"
	}

	chain accept_to_admin {
		oifname "br-admin" counter packets 41 bytes 3240 accept comment "!fw4: accept admin IPv4/IPv6 traffic"
	}

	chain input_ovpn {
		ip saddr {  } counter packets 0 bytes 0 accept comment "!fw4: Admin OVPN Input"
		ip6 saddr { fd00:f9a8:fffd::2000, fd00:f9a8:fffd::2001 } counter packets 56 bytes 4480 accept comment "!fw4: Admin OVPN Input"
		jump accept_from_ovpn
	}

	chain output_ovpn {
		jump accept_to_ovpn
	}

	chain forward_ovpn {
		ip saddr { 1 } counter packets 208 bytes 13837 accept comment "!fw4: Admin OVPN"
		ip6 saddr {  } counter packets 0 bytes 0 accept comment "!fw4: Admin OVPN"
		jump accept_to_resident comment "!fw4: Accept ovpn to resident forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept ovpn to sdwan_vpn forwarding"
		jump accept_to_devices comment "!fw4: Accept ovpn to devices forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept ovpn to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept ovpn to dummy forwarding"
		jump accept_to_ovpn
		log prefix "drop ovpn forward: "
	}

	chain helper_ovpn {
	}

	chain accept_from_ovpn {
		iifname "tun0" counter packets 0 bytes 0 accept comment "!fw4: accept ovpn IPv4/IPv6 traffic"
	}

	chain accept_to_ovpn {
		oifname "tun0" counter packets 2 bytes 152 accept comment "!fw4: accept ovpn IPv4/IPv6 traffic"
	}

	chain input_sdwan_vpn {
		tcp dport 179 counter packets 0 bytes 0 accept comment "!fw4: BGP SDWAN"
		ip saddr 10.0.2.0/24 counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN"
		ip6 saddr counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN"
		ip saddr { } counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN Input"
		ip6 saddr { fd00:f9a8:fffe::2000, fd00:f9a8:fffe::2001, fd00:f9a8:fffe::2002, fd00:f9a8:fffe::2003, fd00:f9a8:fffe::2004, fd00:f9a8:fffe::2005, fd00:f9a8:fffe::2006, fd00:f9a8:fffe::2007, fd00:f9a8:fffe::2008, fd00:f9a8:fffe::2009, fd00:f9a8:fffe::2010, fd00:f9a8:ffff::2000, fd00:f9a8:ffff::2001, fd00:f9a8:ffff::2002, fd00:f9a8:ffff::2003, fd00:f9a8:ffff::2004, fd00:f9a8:ffff::2005, fd00:f9a8:ffff::2006, fd00:f9a8:ffff::2007, fd00:f9a8:ffff::2008, fd00:f9a8:ffff::2009, fd00:f9a8:ffff::2010 } counter packets 0 bytes 0 accept comment "!fw4: ADMIN SDWAN Input"
		jump reject_from_sdwan_vpn
	}

	chain output_sdwan_vpn {
		jump accept_to_sdwan_vpn
	}

	chain forward_sdwan_vpn {
		ip saddr  } counter packets 0 bytes 0 accept comment "!fw4: VPN Admin"
		ip6 saddr { fd00:f9a8:fffe::2000, fd00:f9a8:fffe::2001, fd00:f9a8:fffe::2002, fd00:f9a8:ffff::2000, fd00:f9a8:ffff::2001, fd00:f9a8:ffff::2002 } ip6 daddr fd00::/8 counter packets 0 bytes 0 accept comment "!fw4: VPN Admin"
		ip saddr 10.0.2.0/24 counter packets 0 bytes 0 accept comment "!fw4: Admin SDWAN"
		ip6 saddr fd00:f9a8:0:2::/64 counter packets 0 bytes 0 accept comment "!fw4: Admin SDWAN"
		ip saddr 10.0.10.0/24 counter packets 0 bytes 0 jump accept_to_resident comment "!fw4: Resident SDWAN"
		ip6 saddr fd00:f9a8:0:10::/64 counter packets 0 bytes 0 jump accept_to_resident comment "!fw4: Resident SDWAN"
		ip saddr 10.0.53.0/27 counter packets 0 bytes 0 jump accept_to_domain comment "!fw4: Domain"
		ip6 saddr fd00:f9a8:53::/62 counter packets 0 bytes 0 jump accept_to_domain comment "!fw4: Domain"
		jump accept_to_sdwan_vpn
		log prefix "drop sdwan_vpn forward: "
	}

	chain helper_sdwan_vpn {
	}

	chain accept_to_sdwan_vpn {
		oifname { "pv", "cs1" } counter packets 87 bytes 7076 accept comment "!fw4: accept sdwan_vpn IPv4/IPv6 traffic"
	}

	chain reject_from_sdwan_vpn {
		iifname { "pv", "cs1" } counter packets 0 bytes 0 log prefix "reject sdwan_vpn in: " jump handle_reject comment "!fw4: reject sdwan_vpn IPv4/IPv6 traffic"
	}

	chain input_domain {
		jump reject_from_domain
	}

	chain output_domain {
		jump accept_to_domain
	}

	chain forward_domain {
		jump accept_to_wan comment "!fw4: Accept domain to wan forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept domain to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept domain to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept domain to dummy forwarding"
		jump accept_to_domain
	}

	chain helper_domain {
	}

	chain accept_to_domain {
		oifname "dummy1" counter packets 0 bytes 0 accept comment "!fw4: accept domain IPv4/IPv6 traffic"
	}

	chain reject_from_domain {
		iifname "dummy1" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject domain IPv4/IPv6 traffic"
	}

	chain input_resident {
		udp dport 1194 counter packets 0 bytes 0 accept comment "!fw4: Openvpn"
		jump reject_from_resident
	}

	chain output_resident {
		jump accept_to_resident
	}

	chain forward_resident {
		jump accept_to_wan comment "!fw4: Accept resident to wan forwarding"
		jump accept_to_devices comment "!fw4: Accept resident to devices forwarding"
		jump accept_to_admin comment "!fw4: Accept resident to admin forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept resident to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept resident to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept resident to dummy forwarding"
		jump accept_to_resident
	}

	chain helper_resident {
	}

	chain accept_to_resident {
		oifname "br-resident" counter packets 54 bytes 4464 accept comment "!fw4: accept resident IPv4/IPv6 traffic"
	}

	chain reject_from_resident {
		iifname "br-resident" counter packets 41 bytes 2922 jump handle_reject comment "!fw4: reject resident IPv4/IPv6 traffic"
	}

	chain input_devices {
		ip saddr 10.0.19.27 tcp dport 443 counter packets 0 bytes 0 accept comment "!fw4: Home assistant input"
		jump reject_from_devices
	}

	chain output_devices {
		jump accept_to_devices
	}

	chain forward_devices {
		ip saddr 10.0.19.27 ip daddr 192.168.100.1 tcp dport 9200 counter packets 0 bytes 0 jump accept_to_wan comment "!fw4: hassio dishy"
		ip daddr 10.0.18.79 counter packets 0 bytes 0 jump accept_to_admin comment "!fw4: Devices to zabbix"
		ip6 daddr fd00:f9a8:0:42:5054:ff:fecb:aedf counter packets 2 bytes 160 jump accept_to_admin comment "!fw4: Devices to zabbix"
		jump accept_to_wan comment "!fw4: Accept devices to wan forwarding"
		jump accept_to_sdwan_vpn comment "!fw4: Accept devices to sdwan_vpn forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept devices to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept devices to dummy forwarding"
		jump accept_to_devices
	}

	chain helper_devices {
	}

	chain accept_to_devices {
		oifname "br-devices" counter packets 258 bytes 26176 accept comment "!fw4: accept devices IPv4/IPv6 traffic"
	}

	chain reject_from_devices {
		iifname "br-devices" counter packets 383 bytes 28212 jump handle_reject comment "!fw4: reject devices IPv4/IPv6 traffic"
	}

	chain input_guest {
		tcp dport { 53, 123 } counter packets 0 bytes 0 accept comment "!fw4: DNS in guest"
		udp dport { 53, 123 } counter packets 506 bytes 36004 accept comment "!fw4: DNS in guest"
		udp dport 1194 counter packets 0 bytes 0 accept comment "!fw4: Guest OpenVPN"
		tcp dport 5353 counter packets 0 bytes 0 drop comment "!fw4: @rule[21]"
		udp dport 5353 counter packets 0 bytes 0 drop comment "!fw4: @rule[21]"
		meta nfproto ipv6 udp dport 547-548 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-IPV6"
		jump reject_from_guest
	}

	chain output_guest {
		tcp sport 5353 counter packets 0 bytes 0 jump drop_to_guest comment "!fw4: @rule[20]"
		udp sport 5353 counter packets 0 bytes 0 jump drop_to_guest comment "!fw4: @rule[20]"
		jump accept_to_guest
	}

	chain forward_guest {
		jump accept_to_wan comment "!fw4: Accept guest to wan forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept guest to sdwan_bond forwarding"
		jump accept_to_dummy comment "!fw4: Accept guest to dummy forwarding"
		jump accept_to_guest
	}

	chain helper_guest {
	}

	chain accept_to_guest {
		oifname "br-guest" counter packets 10 bytes 808 accept comment "!fw4: accept guest IPv4/IPv6 traffic"
	}

	chain reject_from_guest {
		iifname "br-guest" counter packets 34 bytes 2448 jump handle_reject comment "!fw4: reject guest IPv4/IPv6 traffic"
	}

	chain drop_to_guest {
		oifname "br-guest" counter packets 0 bytes 0 drop comment "!fw4: drop guest IPv4/IPv6 traffic"
	}

	chain input_dummy {
		jump drop_from_dummy
	}

	chain output_dummy {
		jump accept_to_dummy
	}

	chain forward_dummy {
		jump accept_to_wan comment "!fw4: Accept dummy to wan forwarding"
		jump accept_to_sdwan_bond comment "!fw4: Accept dummy to sdwan_bond forwarding"
		jump drop_to_dummy
	}

	chain helper_dummy {
	}

	chain accept_to_dummy {
	}

	chain drop_from_dummy {
	}

	chain drop_to_dummy {
	}

	chain dstnat {
		type nat hook prerouting priority dstnat; policy accept;
	}

	chain srcnat {
		type nat hook postrouting priority srcnat; policy accept;
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
	}

	chain srcnat_wan {
		ip daddr 192.168.100.1 counter packets 12 bytes 816 masquerade comment "!fw4: StarlinkModem"
		ip daddr 192.168.1.1 counter packets 0 bytes 0 masquerade comment "!fw4: StarlinkRouter"
		ip daddr 192.168.117.1 counter packets 4 bytes 336 masquerade comment "!fw4: ATTPhone"
	}

	chain raw_prerouting {
		type filter hook prerouting priority raw; policy accept;
	}

	chain raw_output {
		type filter hook output priority raw; policy accept;
	}

	chain mangle_prerouting {
		type filter hook prerouting priority mangle; policy accept;
	}

	chain mangle_postrouting {
		type filter hook postrouting priority mangle; policy accept;
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } ip daddr 167.88.49.11 tcp dport 443 counter packets 14 bytes 3088 ip dscp set af13 comment "!fw4: DSCP 0xCBF"
		meta nfproto ipv4 meta l4proto tcp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 8317 bytes 631086 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto tcp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv4 meta l4proto udp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 12606 bytes 1204115 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto udp oifname { "wan", "usb0", "wwan1", "tap-easytether" } counter packets 0 bytes 0 ip6 dscp set af11 comment "!fw4: DSCP ALL"
	}

	chain mangle_input {
		type filter hook input priority mangle; policy accept;
	}

	chain mangle_output {
		type route hook output priority mangle; policy accept;
	}

	chain mangle_forward {
		type filter hook forward priority mangle; policy accept;
		meta nfproto ipv4 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp dport 53 counter packets 0 bytes 0 ip dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv6 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp dport 53 counter packets 0 bytes 0 ip6 dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv4 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } udp dport 53 counter packets 0 bytes 0 ip dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv6 iifname "dummy1" oifname { "wan", "usb0", "wwan1", "tap-easytether" } udp dport 53 counter packets 0 bytes 0 ip6 dscp set ef comment "!fw4: DSCP DNS"
		meta nfproto ipv4 tcp dport { 20645, 32400 } counter packets 84 bytes 7852 ip dscp set af31 comment "!fw4: DSCP Plex1"
		meta nfproto ipv6 tcp dport { 20645, 32400 } counter packets 0 bytes 0 ip6 dscp set af31 comment "!fw4: DSCP Plex1"
		meta nfproto ipv4 meta l4proto tcp counter packets 14506 bytes 2601639 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto tcp counter packets 52 bytes 4160 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv4 meta l4proto udp counter packets 1059 bytes 72998 ip dscp set af11 comment "!fw4: DSCP ALL"
		meta nfproto ipv6 meta l4proto udp counter packets 0 bytes 0 ip6 dscp set af11 comment "!fw4: DSCP ALL"
		iifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
		oifname { "wan", "usb0", "wwan1", "tap-easytether" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
		iifname { "vbond0", "dummy0", "vbond1", "vbond2" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_bond IPv4/IPv6 ingress MTU fixing"
		oifname { "vbond0", "dummy0", "vbond1", "vbond2" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_bond IPv4/IPv6 egress MTU fixing"
		iifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone ovpn IPv4/IPv6 ingress MTU fixing"
		oifname "tun0" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone ovpn IPv4/IPv6 egress MTU fixing"
		iifname { "pv", "cs1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_vpn IPv4/IPv6 ingress MTU fixing"
		oifname { "pv", "cs1" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone sdwan_vpn IPv4/IPv6 egress MTU fixing"
		iifname "dummy1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone domain IPv4/IPv6 ingress MTU fixing"
		oifname "dummy1" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone domain IPv4/IPv6 egress MTU fixing"
	}
}
table ip mangle {
	chain PREROUTING {
		type filter hook prerouting priority mangle; policy accept;
		counter packets 5081 bytes 1467526 jump mwan3_hook
	}

	chain OUTPUT {
		type route hook output priority mangle; policy accept;
		counter packets 2850 bytes 836131 jump mwan3_hook
	}

	chain mwan3_ifaces_in {
		# xt_mark counter packets 1338 bytes 124284 jump mwan3_iface_in_easytether
		# xt_mark counter packets 1332 bytes 123935 jump mwan3_iface_in_wwan
	}

	chain mwan3_custom_ipv4 {
		# xt_set counter packets 0 bytes 0 # xt_MARK
	}

	chain mwan3_connected_ipv4 {
		# xt_set counter packets 304 bytes 37752 # xt_MARK
	}

	chain mwan3_dynamic_ipv4 {
		# xt_set counter packets 0 bytes 0 # xt_MARK
	}

	chain mwan3_rules {
		meta l4proto udp ip daddr 180.33.26.15 # xt_multiport # xt_mark counter packets 1 bytes 176 jump mwan3_policy_sdwan_wan
		meta l4proto udp ip daddr 180.33.26.15 # xt_multiport # xt_mark # xt_comment counter packets 0 bytes 0 # xt_LOG
		meta l4proto udp ip daddr 180.33.26.15 # xt_multiport # xt_mark counter packets 0 bytes 0 jump mwan3_policy_sdwan_wwan
		meta l4proto udp ip daddr 180.33.26.15 # xt_multiport # xt_mark # xt_comment counter packets 0 bytes 0 # xt_LOG
		meta l4proto udp ip daddr 180.33.26.15 # xt_multiport # xt_mark counter packets 0 bytes 0 jump mwan3_policy_sdwan_et
		ip daddr 192.168.100.1 # xt_mark counter packets 0 bytes 0 jump mwan3_policy_sdwan_wwan
	}

	chain mwan3_hook {
		# xt_mark counter packets 7779 bytes 2289933 # xt_CONNMARK
		# xt_mark counter packets 1390 bytes 132852 jump mwan3_ifaces_in
		# xt_mark counter packets 1390 bytes 132852 jump mwan3_custom_ipv4
		# xt_mark counter packets 1390 bytes 132852 jump mwan3_connected_ipv4
		# xt_mark counter packets 1086 bytes 95100 jump mwan3_dynamic_ipv4
		# xt_mark counter packets 1086 bytes 95100 jump mwan3_rules
		counter packets 7931 bytes 2303657 # xt_CONNMARK
		# xt_mark counter packets 1086 bytes 95100 jump mwan3_custom_ipv4
		# xt_mark counter packets 1086 bytes 95100 jump mwan3_connected_ipv4
		# xt_mark counter packets 1086 bytes 95100 jump mwan3_dynamic_ipv4
	}

	chain mwan3_iface_in_easytether {
		iifname "tap-easytether" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "tap-easytether" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "tap-easytether" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "tap-easytether" # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
	}

	chain mwan3_iface_in_wwan {
		iifname "wwan1" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "wwan1" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "wwan1" # xt_set # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
		iifname "wwan1" # xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
	}

	chain mwan3_policy_sdwan_wan {
		# xt_mark # xt_comment counter packets 1 bytes 176 # xt_MARK
	}

	chain mwan3_policy_sdwan_wwan {
		# xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
	}

	chain mwan3_policy_sdwan_et {
		# xt_mark # xt_comment counter packets 0 bytes 0 # xt_MARK
	}
}

[GrapeCent]

I think I may have gotten it working, the possible trick was creating a specific route to my SDWAN VPS IP for both the WWAN and Easytether interfaces with the same metric as the device's default gateway. I'm playing around with this remotely, so I won't be able to confirm until I'm home later (I have my starlink on my car, so the only connection at home right now is easytether.
I do see that with my firewall rules disabled, vbond1 (pinned to wwan) isn't connecting, which suggests that it's being directed to table 2

This excludes it from the fake 0.0.0.0/1 and 128.0.0.0/1 default routes in both routing table 2 and 3. However I suspect installing iptables-nft was also helpful since the mangle rules are appearing in the nft list ruleset now. It would be nice if iptables-nft was linked as a dependency for mwan3 in 22.03.x

i wonder what other packages I should have installed...

[GrapeCent]

the solution for wireguard source routing is to create a routing table for each interface, assign the physical and wireguard int to their dedicated routing table. this will force wireguard to use the default route in its routing table which is seperate from the other wireguard interfaces.

I also had to configure the firewall mark in the wireguard configuration, which marks every outbound encrypted packet with a unique mark. Finally I set an IP rule that matches on each interface's unique mark and looks up the matching routing table

i completely disabled mwan to achieve this.

[trendy]

Well done on achieving this! I have also reached a somewhat same conclusion that mwan3 is not able to achieve in its deployment. It won't work well in case of load balancing however, but this is not your use case.
@feckert @aaronjg for your information...

[GrapeCent]

You could load balance using any routing protocol that supports weighted, at least in my configuration.

My default route is a /32 IP at the other end of both of these wireguard tunnels, announced via OSPF and the recursive route is managed by frr. I just use this for failover, but some additional configuration could distribute network traffic across two links

The benefit is that TCP sessions are maintained by a VPS that's theoretically always running, so you don't have to worry about connections reset or sticking to a slower interface.

I have another post about this I'm going to fill in my configuration now that I solved this critical issue

wireguard just doesnt have a feature to bind to an iface that is required for mwan policies to work. but im happy with increased throughput and tunnel mtu over openvpn.

[trendy]

I was referring to the fact that one tunnel goes out of one wan interface only and cannot load balance or failover to another wan.

[GrapeCent]

I see now, but in this scenario, the tunnels individually are sacrificial. as long as one is up, traffic flows to the default gateway at the other end.

I also have site to site wireguard tunnels running with a smaller mtu (which is why its important to my usecase) to other sites, which traverse this default gateway tunnel, just like they would egress directly via ISP uplink.
so these tunnels do have failover, but its nearly instant with fast ospf dead timer and hello multiplier

[GrapeCent]

Firewall Mark
0x34ff
Optional. 32-bit mark for outgoing encrypted packets. Enter value in hex, starting with 0x.

This setting in the Wireguard interface advanced settings config was actually the final solution. If this were set to use the same mark that MWAN3 uses, then it would probably work with the mwan policies. I think that this is marking the packets as they are generated by the kernel, as opposed to on output chain when they're already about to hit the wire.