Can't get Wireguard to work with mwan3

[JustWilder]

Hello there!

So ive struggled with this problem for a week now and I cant get it to work.

I have a Wireguard interface connected to cloudflares warp vpn. it makes the handshake and i can traceroute any domain or ip thru the wireguard interface from the routers terminal with no issues. but my lan has no internet.

My wireguard interface has its own firewall zone and my lan zone is forwarded to it.

This did work initially without any issue. then i was doing some changes seemingly unrelated. I was adding an ipv6 connection to my multi wan setup and when done i had no wireguard thruput. So i undid what i had just done but internet thru wireguard was still dead. I hadn't have wireguard configured for long enough to rule out a total coincident or if my changes to mwan did something.

This was a little over a week now and i've spent countless hours trying to correct this. I've googled and followed all different setup and config regimes. tried to use my own head to adopt fixes and debugging based on problems similar to mine but not exact. Ive tried adding the interface to the same zone as lan, i've tried ip tables, firewall rules, static routes, vpn policy routing. Ive also googled all kinds of combinations based on other packages im running like "wireguard unbound", "wireguard multi wan" and so on. I haven't found any indication that these packages needs to be configured differently when together.

and yeah. i've even reverted the whole system to a backup done before my changes when i know the wireguard connection was working. but still nothing

Another little detail. Yesterday i could run opkg update without problem. today i cant. I get "failed to download ...wget returned 4.". Ive done nothing to the router between yesterday and today.

So Im swallowing my pride and ask for help. Im giving up on the guess work. I want to learn, I want to understand and most off all i want this working so I can return to normal life!

Update:
I have condenced this problem down to mwan3 configuration or the lack of it.

Im running OpenWrt 19.07.4 on Xiaomi Mi Router 3G.
Here's my configs.

/etc/config/network

config interface 'loopback'
   option ifname 'lo'
   option proto 'static'
   option ipaddr '127.0.0.1'
   option netmask '255.0.0.0'

config globals 'globals'
   option ula_prefix 'fd5f:0a0e:fe99::/48'

config interface 'lan'
   option type 'bridge'
   option proto 'static'
   option ip6assign '60'
   option ifname 'eth0.3'
   list ipaddr '192.168.15.1/24'

config device 'lan_eth0_1_dev'
   option name 'eth0.1'
   option macaddr 'ec:41:18:50:69:34'

config interface 'wan'
   option proto 'dhcp'
   option ifname 'eth0.1'
   option metric '110'

config interface 'wan6'
   option proto 'dhcpv6'
   option ifname 'eth0.1'
   option reqaddress 'try'
   option reqprefix 'auto'

config switch
   option name 'switch0'
   option reset '1'
   option enable_vlan '1'

config switch_vlan
   option device 'switch0'
   option vlan '1'
   option vid '1'
   option ports '3 6t'

config switch_vlan
   option device 'switch0'
   option vlan '2'
   option vid '2'
   option ports '2 6t'

config interface 'wwan'
   option proto 'dhcp'
   option metric '130'

config switch_vlan
   option device 'switch0'
   option vlan '3'
   option ports '1 6t'
   option vid '3'

config interface 'wanb'
   option ifname 'eth0.2'
   option proto 'dhcp'
   option metric '120'

config interface 'wanb6'
   option ifname 'eth0.2'
   option proto 'dhcp'

config interface 'wwanb'
   option proto 'dhcp'
   option metric '140'

config interface 'guest'
   option proto 'static'
   list ipaddr '10.0.0.1/24'

config interface 'usbwan'
   option ifname 'usb0'
   option proto 'dhcp'
   option metric '150'

config interface 'cf'
   option proto 'wireguard'
   option mtu '1280'
   list addresses '172.16.0.2/32'
   list addresses 'fd01:5ca1:ab1e:8418:d420:cd08:b1b3:88b0/128'
   option private_key '[redacted]'
   option metric '200'

config wireguard_cf
   option endpoint_port '2408'
   option public_key 'bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo='
   option endpoint_host 'engage.cloudflareclient.com'
   option persistent_keepalive '25'
   list allowed_ips '0.0.0.0/0'
   list allowed_ips '::/0'
   option route_allowed_ips '1'

config interface 'wwan6'
   option ifname 'wlan0'
   option proto 'dhcpv6'
   option reqaddress 'try'
   option reqprefix 'auto'

config interface 'usbwan6'
   option ifname 'usb0'
   option proto 'dhcpv6'
   option reqaddress 'try'
   option reqprefix 'auto'

/etc/config/firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option masq_allow_invalid '1'
	option network 'usbwan usbwan6 wan wan6 wanb wanb6 wwan wwan6 wwanb'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config include
	option path '/etc/firewall.user'

config include 'ss_rules'
	option path '/etc/firewall.ss-rules'
	option reload '1'

config zone
	option name 'guest'
	option output 'ACCEPT'
	option network 'guest'
	option input 'REJECT'
	option forward 'REJECT'

config rule
	option src 'guest'
	option name 'Guest DHCP'
	option target 'ACCEPT'
	option dest_port '67-68'
	list proto 'udp'

config rule
	option src 'guest'
	option name 'Guest DNS'
	option target 'ACCEPT'
	option dest_port '53'

config rule
	option src 'guest'
	option name 'Guest Isolate'
	option dest 'lan'
	list dest_ip '192.168.15.0/24'
	option target 'DROP'
	option enabled '0'

config zone
	option name 'wgcf'
	option forward 'ACCEPT'
	option output 'ACCEPT'
	option mtu_fix '1'
	option masq '1'
	option network 'cf'
	option input 'REJECT'

config forwarding
	option dest 'wgcf'
	option src 'lan'

config forwarding
	option dest 'wan'
	option src 'guest'

/etc/config/unbound

config unbound
	option dns64 '0'
	option domain 'lan'
	option edns_size '1280'
	option extended_stats '0'
	option hide_binddata '1'
	option interface_auto '1'
	option localservice '1'
	option manual_conf '0'
	option num_threads '2'
	option protocol 'default'
	option rate_limit '0'
	option rebind_localhost '0'
	option rebind_protection '1'
	option recursion 'default'
	option root_age '9'
	option ttl_min '120'
	option unbound_control '0'
	option verbosity '1'
	option enabled '1'
	option dhcp_link 'dnsmasq'
	option resource 'large'
	option validator '0'
	option listen_port '5553'
	list trigger_interface 'cf'
	list trigger_interface 'guest'
	list trigger_interface 'lan'
	list trigger_interface 'usbwan'
	list trigger_interface 'wan'
	list trigger_interface 'wanb'
	list trigger_interface 'wwan'
	list trigger_interface 'wwanb'

config zone 'forward'
	option enabled '1'
	option fallback '0'
	option zone_type 'forward_zone'
	option tls_upstream '1'
	list zone_name '.'
	option tls_index 'cloudflare-dns.com'
	list server '2606:4700:4700::1111'
	list server '2606:4700:4700::1001'
	list server '1.1.1.1'
	list server '1.0.0.1'

config zone
	option fallback '0'
	option zone_type 'forward_zone'
	list server '2001:4860:4860::8888'
	list server '2001:4860:4860::8844'
	list server '8.8.8.8'
	list server '8.8.4.4'
	option dns_assist 'none'
	option tls_upstream '1'
	option tls_index 'dns.google'
	list zone_name '.'
	option enabled '0'

/etc/config/mwan3

config globals 'globals'
	option mmx_mask '0x3F00'
	option rtmon_interval '5'

config interface 'wan'
	option enabled '1'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '2'
	option count '1'
	option timeout '2'
	option failure_latency '1000'
	option recovery_latency '500'
	option failure_loss '20'
	option recovery_loss '5'
	option interval '5'
	option down '3'
	option up '8'

config interface 'wan6'
	option enabled '0'
	list track_ip '2001:4860:4860::8844'
	list track_ip '2001:4860:4860::8888'
	list track_ip '2620:0:ccd::2'
	list track_ip '2620:0:ccc::2'
	option family 'ipv6'
	option reliability '2'
	option count '1'
	option timeout '2'
	option interval '5'
	option down '3'
	option up '8'

config interface 'wanb'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	list track_ip '208.67.222.222'
	list track_ip '208.67.220.220'
	option family 'ipv4'
	option reliability '1'
	option count '1'
	option timeout '2'
	option interval '5'
	option down '3'
	option up '8'
	option enabled '1'
	option initial_state 'online'
	option track_method 'ping'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option failure_interval '5'
	option recovery_interval '5'

config interface 'wanb6'
	option enabled '0'
	list track_ip '2001:4860:4860::8844'
	list track_ip '2001:4860:4860::8888'
	list track_ip '2620:0:ccd::2'
	list track_ip '2620:0:ccc::2'
	option family 'ipv6'
	option reliability '1'
	option count '1'
	option timeout '2'
	option interval '5'
	option down '3'
	option up '8'

config member 'wan_m1_w3'
	option interface 'wan'
	option metric '1'
	option weight '3'

config member 'wan_m2_w3'
	option interface 'wan'
	option metric '2'
	option weight '3'

config member 'wanb_m1_w2'
	option interface 'wanb'
	option metric '1'
	option weight '2'

config member 'wanb_m2_w2'
	option interface 'wanb'
	option metric '2'
	option weight '2'

config member 'wan6_m1_w3'
	option interface 'wan6'
	option metric '1'
	option weight '3'

config member 'wan6_m2_w3'
	option interface 'wan6'
	option metric '2'
	option weight '3'

config member 'wanb6_m1_w2'
	option interface 'wanb6'
	option metric '1'
	option weight '2'

config member 'wanb6_m2_w2'
	option interface 'wanb6'
	option metric '2'
	option weight '2'

config policy 'wan_only'
	list use_member 'wan_m1_w3'
	list use_member 'wan6_m1_w3'

config policy 'wanb_only'
	list use_member 'wanb_m1_w2'
	list use_member 'wanb6_m1_w2'

config policy 'balanced'
	option last_resort 'unreachable'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m1_w2'
	list use_member 'wan6_m1_w3'
	list use_member 'wanb6_m1_w2'
	list use_member 'wwan_m1_w3'
	list use_member 'usbwan_m1_w1'
	list use_member 'wwan6_m1_w3'
	list use_member 'usbwan6_m1_w1'

config policy 'wan_wanb'
	list use_member 'wan_m1_w3'
	list use_member 'wanb_m2_w2'
	list use_member 'wan6_m1_w3'
	list use_member 'wanb6_m2_w2'

config policy 'wanb_wan'
	list use_member 'wan_m2_w3'
	list use_member 'wanb_m1_w2'
	list use_member 'wan6_m2_w3'
	list use_member 'wanb6_m1_w2'

config rule 'https'
	option sticky '1'
	option dest_port '443'
	option proto 'tcp'
	option use_policy 'balanced'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option use_policy 'balanced'
	option family 'ipv4'

config rule 'default_rule_v6'
	option dest_ip '::/0'
	option use_policy 'balanced'
	option family 'ipv6'

config interface 'wwan'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '2'
	option interval '5'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '8'
	list track_ip '1.1.1.1'
	list track_ip '1.0.0.1'

config interface 'wwanb'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '2'
	option interval '5'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '8'
	list track_ip '8.8.8.8'
	list track_ip '8.8.4.4'

config member 'wwan_m1_w3'
	option interface 'wwan'
	option metric '1'
	option weight '3'

config member 'wwan_m2_w3'
	option interface 'wwan'
	option metric '2'
	option weight '3'

config policy 'wwan_only'
	list use_member 'wwan_m1_w3'
	option last_resort 'unreachable'

config interface 'usbwan'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	list track_ip '8.8.8.8'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '2'
	option interval '5'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '8'

config member 'wwanb_m1_w2'
	option interface 'wwanb'
	option metric '1'
	option weight '2'

config member 'wwanb_m2_w2'
	option interface 'wwanb'
	option metric '2'
	option weight '2'

config member 'usbwan_m1_w1'
	option interface 'usbwan'
	option metric '1'
	option weight '1'

config member 'usbwan_m2_w1'
	option interface 'usbwan'
	option metric '2'
	option weight '1'

config policy 'wwanb_only'
	list use_member 'wwanb_m1_w2'
	option last_resort 'unreachable'

config policy 'usbwan_only'
	list use_member 'usbwan_m1_w1'
	option last_resort 'unreachable'

config interface 'wwan6'
	option enabled '1'
	option initial_state 'online'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '2'
	option interval '5'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '8'
	option family 'ipv6'
	list track_ip '2606:4700:4700::1111'
	list track_ip '2606:4700:4700::1001'

config member 'wwan6_m1_w3'
	option interface 'wwan6'
	option metric '1'
	option weight '3'

config interface 'usbwan6'
	option enabled '0'
	option initial_state 'online'
	option family 'ipv6'
	list track_ip '2606:4700:4700::1111'
	list track_ip '2606:4700:4700::1001'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '2'
	option interval '5'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '8'

config member 'usbwan6_m1_w1'
	option interface 'usbwan6'
	option metric '1'
	option weight '1'

/etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option localservice '1'
	option confdir '/tmp/dnsmasq.d'
	list server '127.0.0.1#5553'
	list server '127.0.0.1#5053'
	option port '53'
	option noresolv '1'
	option doh_backup_noresolv '-1'
	list doh_backup_server '127.0.0.1#5553'
	list doh_backup_server '0:0:0:0:0:0:0:1#5553'
	list doh_backup_server '127.0.0.1#5053'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'
	list dhcp_option 'option:dns-server,192.168.15.1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'guest'
	option start '100'
	option limit '150'
	option interface 'guest'
	option leasetime '6h'
	option dhcpv6 'server'
	option ra_management '1'
	list dhcp_option 'option:dns-server,192.168.15.1'

iptables -S

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N forwarding_guest_rule
-N forwarding_lan_rule
-N forwarding_rule
-N forwarding_wan_rule
-N forwarding_wgcf_rule
-N input_guest_rule
-N input_lan_rule
-N input_rule
-N input_wan_rule
-N input_wgcf_rule
-N output_guest_rule
-N output_lan_rule
-N output_rule
-N output_wan_rule
-N output_wgcf_rule
-N reject
-N syn_flood
-N zone_guest_dest_ACCEPT
-N zone_guest_dest_REJECT
-N zone_guest_forward
-N zone_guest_input
-N zone_guest_output
-N zone_guest_src_REJECT
-N zone_lan_dest_ACCEPT
-N zone_lan_forward
-N zone_lan_input
-N zone_lan_output
-N zone_lan_src_ACCEPT
-N zone_wan_dest_ACCEPT
-N zone_wan_dest_REJECT
-N zone_wan_forward
-N zone_wan_input
-N zone_wan_output
-N zone_wan_src_REJECT
-N zone_wgcf_dest_ACCEPT
-N zone_wgcf_forward
-N zone_wgcf_input
-N zone_wgcf_output
-N zone_wgcf_src_REJECT
-A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
-A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
-A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
-A INPUT -i usb0 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i eth0.1 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i wlan0 -m comment --comment "!fw3" -j zone_wan_input
-A INPUT -i wlan0-2 -m comment --comment "!fw3" -j zone_guest_input
-A INPUT -i cf -m comment --comment "!fw3" -j zone_wgcf_input
-A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
-A FORWARD -i usb0 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i eth0.1 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i wlan0 -m comment --comment "!fw3" -j zone_wan_forward
-A FORWARD -i wlan0-2 -m comment --comment "!fw3" -j zone_guest_forward
-A FORWARD -i cf -m comment --comment "!fw3" -j zone_wgcf_forward
-A FORWARD -m comment --comment "!fw3" -j reject
-A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
-A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
-A OUTPUT -o usb0 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o eth0.1 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o wlan0 -m comment --comment "!fw3" -j zone_wan_output
-A OUTPUT -o wlan0-2 -m comment --comment "!fw3" -j zone_guest_output
-A OUTPUT -o cf -m comment --comment "!fw3" -j zone_wgcf_output
-A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
-A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
-A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3"
-j RETURN
-A syn_flood -m comment --comment "!fw3" -j DROP
-A zone_guest_dest_ACCEPT -o wlan0-2 -m comment --comment "!fw3" -j ACCEPT
-A zone_guest_dest_REJECT -o wlan0-2 -m comment --comment "!fw3" -j reject
-A zone_guest_forward -m comment --comment "!fw3: Custom guest forwarding rule chain" -j forwarding_guest_rule
-A zone_guest_forward -m comment --comment "!fw3: Zone guest to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_guest_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_guest_forward -m comment --comment "!fw3" -j zone_guest_dest_REJECT
-A zone_guest_input -m comment --comment "!fw3: Custom guest input rule chain" -j input_guest_rule
-A zone_guest_input -p udp -m udp --dport 67:68 -m comment --comment "!fw3: Guest DHCP" -j ACCEPT
-A zone_guest_input -p tcp -m tcp --dport 53 -m comment --comment "!fw3: Guest DNS" -j ACCEPT
-A zone_guest_input -p udp -m udp --dport 53 -m comment --comment "!fw3: Guest DNS" -j ACCEPT
-A zone_guest_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_guest_input -m comment --comment "!fw3" -j zone_guest_src_REJECT
-A zone_guest_output -m comment --comment "!fw3: Custom guest output rule chain" -j output_guest_rule
-A zone_guest_output -m comment --comment "!fw3" -j zone_guest_dest_ACCEPT
-A zone_guest_src_REJECT -i wlan0-2 -m comment --comment "!fw3" -j reject
-A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
-A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
-A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
-A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
-A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
-A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
-A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o usb0 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o eth0.1 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_ACCEPT -o wlan0 -m comment --comment "!fw3" -j ACCEPT
-A zone_wan_dest_REJECT -o usb0 -m comment --comment "!fw3" -j reject
-A zone_wan_dest_REJECT -o eth0.1 -m comment --comment "!fw3" -j reject
-A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject
-A zone_wan_dest_REJECT -o wlan0 -m comment --comment "!fw3" -j reject
-A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
-A zone_wan_forward -p esp -m comment --comment "!fw3: Allow-IPSec-ESP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -p udp -m udp --dport 500 -m comment --comment "!fw3: Allow-ISAKMP" -j zone_lan_dest_ACCEPT
-A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
-A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
-A zone_wan_input -p udp -m udp --dport 68 -m comment --comment "!fw3: Allow-DHCP-Renew" -j ACCEPT
-A zone_wan_input -p icmp -m icmp --icmp-type 8 -m comment --comment "!fw3: Allow-Ping" -j ACCEPT
-A zone_wan_input -p igmp -m comment --comment "!fw3: Allow-IGMP" -j ACCEPT
-A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
-A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
-A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
-A zone_wan_src_REJECT -i usb0 -m comment --comment "!fw3" -j reject
-A zone_wan_src_REJECT -i eth0.1 -m comment --comment "!fw3" -j reject
-A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject
-A zone_wan_src_REJECT -i wlan0 -m comment --comment "!fw3" -j reject
-A zone_wgcf_dest_ACCEPT -o cf -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wgcf_dest_ACCEPT -o cf -m comment --comment "!fw3" -j ACCEPT
-A zone_wgcf_forward -m comment --comment "!fw3: Custom wgcf forwarding rule chain" -j forwarding_wgcf_rule
-A zone_wgcf_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
-A zone_wgcf_forward -m comment --comment "!fw3" -j zone_wgcf_dest_ACCEPT
-A zone_wgcf_input -m comment --comment "!fw3: Custom wgcf input rule chain" -j input_wgcf_rule
-A zone_wgcf_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
-A zone_wgcf_input -m comment --comment "!fw3" -j zone_wgcf_src_REJECT
-A zone_wgcf_output -m comment --comment "!fw3: Custom wgcf output rule chain" -j output_wgcf_rule
-A zone_wgcf_output -m comment --comment "!fw3" -j zone_wgcf_dest_ACCEPT
-A zone_wgcf_src_REJECT -i cf -m comment --comment "!fw3" -j reject

[Baz76]

Hello,

I don't know if it helps, but I'm running a similar problem with an update to 19.07.4 from .3

In fact, it seems to me that all the https://openwrt.org/downloads repositories are down, as I was also looking for the image of the older version (which provide 5Ghz while the new one doesn't).

All the lists are impossible to update, so I'm basically stuck with a blank router for now.

Hope their servers will be rapidly back online.

Marco (from Italy)

[jamesmacwhite]

See the workaround to use a mirror instead of downloads.openwrt.org. Not being able to download Wireguard is unrelated, it just so happens the downloads.openwrt.org server went offline yesterday. If you change your opkg feed config to use an alternative mirror, you'll be able to install packages again.

Downloads link timing out

You can use OpenWrt mirrors: https://openwrt.org/downloads#mirrors A workaround: # Temporary umount /etc/opkg/distfeeds.conf 2> /dev/null cp -f /etc/opkg/distfeeds.conf /tmp/opkg-distfeeds.conf sed -i -e "s:downloads.openwrt.org:openwrt.freemirror.org:" \ /tmp/opkg-distfeeds.conf mount -o bind /tmp/opkg-distfeeds.conf /etc/opkg/distfeeds.conf # Permanent umount /etc/opkg/distfeeds.conf 2> /dev/null cp -f /etc/opkg/distfeeds.conf /etc/opkg/distfeeds.bak sed -i -e "s:downloads.openwrt.org:…

[trendy]

@JustWilder

config wireguard_cf

In peer config wireguard_cf you have not activated to route allowed IPs, hence there is no default route from the wg tunnel in the routing table and mwan3 will not use it.

[JustWilder]

Hey. Thanks for the input. It is on actually. In desperation I was trying to turn things on and of the last couple of days and I just happened to export my settings when it was off. I'll update my op.

But yeah mwan3 is the obstacle here. Ive done some more debugging and testing and I can conclude this.

I just cant wrap my head around this to figure out how to set it up correctly.

It seems like mwan3 is treating the wireguard interface like any other wan connection so if its not included in the policy then mwan3 wont send you thru it.

If i were to add wireguard to mwan3 wouldn't it become some kind of loop where my lan would go thru mwan3 twice?

[JustWilder]

Thanks. the repo is up again so no worries. Just the funny coincidence of everything happening at once messing with my debugging.

Ill consider adding mirrors after i solve my little wireguard issue. or mwan3 config issue to be more precise.

[trendy]

@JustWilder

It seems like mwan3 is treating the wireguard interface like any other wan connection so if its not included in the policy then mwan3 wont send you thru it.

That is correct.

@JustWilder

If i were to add wireguard to mwan3 wouldn't it become some kind of loop where my lan would go thru mwan3 twice?

I don't see why.
The ingress packets are classified according to the rules you have in mwan3. All your 3 rules use balanced policy. Balanced policy is not using the wg as member.

[JustWilder]

Hey and sorry for my slow response time. I've been in transit for a few days.

So i've tried including the wireguard interface in the balance policy amongst all the things i tried the week prior to writing this post. While I did get some thruput according to tcpdump that i have on monitoring the wireguard interface at all time during this process, I still had no internet. And frankly, I dont claim to understand all this but going down this route felt illogical to me.

Here's why and please tell me where i get this wrong.

FIG 1. Basic setup.

FIG 2. Wireguard setup. Wireguard should be treated as a WAN. Somewhere behind the scenes WG gets routed to whatever real WAN connection there is.

FIG 3. Mwan3 setup. Lets assume its an equal balanced setup for the sake of my point.

FIG 4. This is what I think in my head would happen if I add WG to the balanced policy in mwan3. In this scenario wouldn't just a third of my internet traffic go thru WG. And while the other two thirds would reach internet wouldn't WG just loop back to mwan3 for its own connection to internet?

FIG 5. This is how I think it should work and how I want it to work if its in anyway possible.

EDIT
I just tried to add WG to the mwan3 balanced policy again and while i do get connectivity I seem to have DNS leaks. So. dnsleaktest.com finds two servers. One is indeed the Hong Kong based DNS coming thru my VPN but the other is my typical Singapore DNS (Im stationed in Vietnam). So it seems like adding WG to the load balancing does just that. balance the load between VPN and nonVPN interfaces.

EDIT 2
I think i have come up with an elegant solution and added benefit of being able to run VPN failover if needed.
So I basically added another rule to mwan3 and moved it to the top of the rule list. This gives me the result im looking for. Please comment if you see any problem with this setup.

[trendy]

@JustWilder

FIG 3. Mwan3 setup. Lets assume its an equal balanced setup for the sake of my point.

It's like this, with an extra interface for wg on the right along with the wan.

@JustWilder

So it seems like adding WG to the load balancing does just that. balance the load between VPN and nonVPN interfaces.

As it should.
Is it fine now or do you have other issue?

[JustWilder]

So i was adding an edit to my previous post while you answered. can you please have a look and tell me if you se any issues with my end setup

And thank you for taking your time! Its appreciated

[trendy]

Nope, it is wrong.

1. Destination address cannot be 192.168.15.0/24 or any of your lan. It has to belong to the internet.
2. This arrow that goes from WG interfaces back to MWAN3 for another rule is wrong. Once mwan3 makes a routing decision then the packet leaves via the decided interface.

[JustWilder]

Alright. I was a little confused by it myself. but it did actually work exactly as i wished. My lan got connected thru the VPN and no dns leakage. while the Gust network got connected straight to wan.

However. Upon a reboot this stopped working so I assume there is some cache or whatever its called with iptables that got messed up and needed to get flushed.

BUT. I put the LAN IP (192.168.15.0/0) in the source address and 0.0.0.0/0 in the destination instead and everything works as intended again and it does so flawlessly even after reboot. So if this is wrong please explain.

And note. I had to specify 192.168.15.0/0 because 192.168.15.0/24 doesn't work. This feels wrong.

This solution has also given me another tiny problem that might or might not be any big deal but after adding the LAN IP to the source address i cant do traceroute -i cf from the routers terminal anymore.

Also. rebooting takes forever when you're in a flow flipping switches and digits. Is there any terminal command that can reproduce whatever happens during a reboot. I presume it would be something like flushing all the ip tables and dns cache.

Oh and the arrow from WG interface to the next mwan3 rule is my assumption of how WG connects to internet. It got to go thru the balanced wan connection like anything else i assume. but the above illustration is purely how i visualize this whole thing so its as flowed as myself.

[trendy]

@JustWilder

I put the LAN IP (192.168.15.0/0)

This is wrong. Either put the whole subnet 192.168.15.0/24 or a single host 192.168.15.23/32 or a subnet 192.168.15.16/28

@JustWilder

Is there any terminal command that can reproduce whatever happens during a reboot. I presume it would be something like flushing all the ip tables and dns cache.

A reboot will call the K scripts located in /etc/rc.d/
It will obviously flush the iptables and dns cache, but this is not related, as they will be recreated after the next boot.

@JustWilder

Oh and the arrow from WG interface to the next mwan3 rule is my assumption of how WG connects to internet. It got to go thru the balanced wan connection like anything else i assume. but the above illustration is purely how i visualize this whole thing so its as flowed as myself.

1. Packets are marked in mangle table of iptable according to your rules. iptables-save -c -t mangle
2. According to the mark, they are assigned to a routing table ip -4 ru
3. Packets are routed according to the routing table they belong ip -4 ro li tab all

[jamesmacwhite]

@JustWilder I think I'm having a similar issue with a Wireguard server. I also use mwan3, I have several Wireguard clients under mwan3 working fine however when coming to setup a wireguard server interface, I'm getting absolutely no traffic on the interface (No RX or TX). There is definitely a Wireguard server instance listening on a the port from netstat, but connecting to it doesn't show any traffic so I don't even think it's getting as far as negotiating the connection.

Did you have to make any specific configuration within mwan3 to get it to work? I'm not entirely sure what you did based on your posts.

[JustWilder]

Hey @jamesmacwhite. I wish i could help you but this was a little while ago now and I have since put this to the side. I cant recall much of it from the top of my head unfortunately. I Did get this to work tho. Maybe not the correct way since I dont fully understand ip tables and couldn't make sense of the guidance i got in this thread. But i achieved my desired result.

Now that was purely related to wireguard client. I used WG to connect to cloudflare's WARP.
Should perhaps be applicable for WG server as well but I got to little experience wit WG server to say anything about it.

have a look at my EDIT 2 higher up in this thread. If i don't miss remember that's how i got my client to work the way i wanted.

[jamesmacwhite]

No worries, thanks for letting me know. Adding the wg client interface does make sense to mwan3, I have done this already for the wg client interfaces I have. I do wonder if server interfaces are the same. I can ping the server but then I get:

root@linksys-wrt3200acm:~# ping -I wgserver 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No error information

So I'm not sure, equally I don't know if this is intended behaviour or not. I would assume not, but I can't be sure.