Hello,
I've got a setup on my main router where mwan3 has been doing a nice job balancing traffic across two VPNs. I am trying to setup another VLAN (192.168.3.0 network) that is coming from another access point (192.168.3.200, 192.168.2.200) which will bypass the VPN. If I set the mwan3 policy to use the balanced VPN set it works through it fine however when I try setting to wan_only I am only able to ping the access point and this main router.
mwan3 troubleshooting output:
Software-Version
-------------------------------------------------
OpenWrt - OpenWrt 18.06.1 r7258-5eb055306f
LuCI - git-19.020.41695-6f6641d
Output of "ip a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.1.2/32 scope global lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether b8:27:eb:2d:d1:64 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ba27:ebff:fe2d:d164/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
link/ether 00:e0:4c:36:01:9b brd ff:ff:ff:ff:ff:ff
4: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP group default qlen 1000
link/ether b8:27:eb:78:84:31 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ba27:ebff:fe78:8431/64 scope link
valid_lft forever preferred_lft forever
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:e0:4c:36:01:9b brd ff:ff:ff:ff:ff:ff
inet 192.168.2.1/24 brd 192.168.2.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fd37:d756:89fb::1/60 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::2e0:4cff:fe36:19b/64 scope link
valid_lft forever preferred_lft forever
6: br-lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:e0:4c:36:01:9b brd ff:ff:ff:ff:ff:ff
inet 192.168.3.1/24 brd 192.168.3.255 scope global br-lan2
valid_lft forever preferred_lft forever
inet6 fe80::2e0:4cff:fe36:19b/64 scope link
valid_lft forever preferred_lft forever
7: eth1.2@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan2 state UP group default qlen 1000
link/ether 00:e0:4c:36:01:9b brd ff:ff:ff:ff:ff:ff
8: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/none
inet 10.71.10.12 peer 10.71.10.25/32 scope global tun1
valid_lft forever preferred_lft forever
inet6 fe80::2901:8ce6:6854:162c/64 scope link stable-privacy
valid_lft forever preferred_lft forever
9: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
link/none
inet 10.74.10.12 peer 10.74.10.25/32 scope global tun0
valid_lft forever preferred_lft forever
inet6 fe80::a148:adca:d14b:80d9/64 scope link stable-privacy
valid_lft forever preferred_lft forever
Output of "ip route show"
-------------------------------------------------
default via 192.168.1.2 dev lo
default via 192.168.1.1 dev eth0 proto static metric 10
default via 10.74.10.25 dev tun0 metric 20
default via 10.71.10.25 dev tun1 metric 21
10.71.10.25 dev tun1 proto kernel scope link src 10.71.10.12
10.74.10.25 dev tun0 proto kernel scope link src 10.74.10.12
192.168.1.0/24 dev eth0 proto static scope link metric 10
192.168.2.0/24 dev br-lan proto kernel scope link src 192.168.2.1
192.168.3.0/24 dev br-lan2 proto kernel scope link src 192.168.3.1
Output of "ip rule show"
-------------------------------------------------
0: from all lookup local
1001: from all iif eth0 lookup main
1002: from all iif tun0 lookup main
1003: from all iif tun1 lookup main
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2003: from all fwmark 0x300/0x3f00 lookup 3
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
32766: from all lookup main
32767: from all lookup default
Output of "ip route list table 1-250"
-------------------------------------------------
Table 1: default via 192.168.1.1 dev eth0
Table 2: default dev tun0 scope link
Table 3: default dev tun1 scope link
Output of "iptables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 103K packets, 81M bytes)
pkts bytes target prot opt in out source destination
104K 81M mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 38079 packets, 40M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 65117 packets, 41M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 32461 packets, 7298K bytes)
pkts bytes target prot opt in out source destination
32648 7333K mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 97574 packets, 48M bytes)
pkts bytes target prot opt in out source destination
Chain mwan3_connected (2 references)
pkts bytes target prot opt in out source destination
73429 77M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected dst MARK or 0x3f00
Chain mwan3_hook (2 references)
pkts bytes target prot opt in out source destination
136K 89M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore mask 0x3f00
2438 197K mwan3_ifaces_in all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
2420 196K mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1844 140K mwan3_ifaces_out all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1306 95197 mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
136K 89M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0x3f00
134K 88M mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x3f00/0x3f00
Chain mwan3_iface_in_PIA (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- tun0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
17 1208 MARK all -- tun0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA */ MARK xset 0x200/0x3f00
Chain mwan3_iface_in_PIA2 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- tun1 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
1 52 MARK all -- tun1 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA2 */ MARK xset 0x300/0x3f00
Chain mwan3_iface_in_WAN (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- eth0 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
0 0 MARK all -- eth0 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* WAN */ MARK xset 0x100/0x3f00
Chain mwan3_iface_out_PIA (1 references)
pkts bytes target prot opt in out source destination
351 29484 MARK all -- * tun0 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA */ MARK xset 0x200/0x3f00
Chain mwan3_iface_out_PIA2 (1 references)
pkts bytes target prot opt in out source destination
187 15708 MARK all -- * tun1 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA2 */ MARK xset 0x300/0x3f00
Chain mwan3_iface_out_WAN (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * eth0 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* WAN */ MARK xset 0x100/0x3f00
Chain mwan3_ifaces_in (1 references)
pkts bytes target prot opt in out source destination
2438 197K mwan3_iface_in_WAN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
2429 194K mwan3_iface_in_PIA all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
2104 170K mwan3_iface_in_PIA2 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
Chain mwan3_ifaces_out (1 references)
pkts bytes target prot opt in out source destination
1844 140K mwan3_iface_out_WAN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1841 138K mwan3_iface_out_PIA all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
1187 86429 mwan3_iface_out_PIA2 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
Chain mwan3_policy_PIA2_only (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA2 1 1 */ MARK xset 0x300/0x3f00
Chain mwan3_policy_PIA_balFAIL (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 statistic mode random probability 0.50000000000 /* PIA2 1 2 */ MARK xset 0x300/0x3f00
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA 1 1 */ MARK xset 0x200/0x3f00
Chain mwan3_policy_PIA_balanced (3 references)
pkts bytes target prot opt in out source destination
141 8588 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 statistic mode random probability 0.50000000000 /* PIA2 1 2 */ MARK xset 0x300/0x3f00
151 14569 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA 1 1 */ MARK xset 0x200/0x3f00
Chain mwan3_policy_PIA_only (0 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* PIA 1 1 */ MARK xset 0x200/0x3f00
Chain mwan3_policy_wan_only (1 references)
pkts bytes target prot opt in out source destination
379 26580 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* WAN 1 1 */ MARK xset 0x100/0x3f00
Chain mwan3_rule_https (1 references)
pkts bytes target prot opt in out source destination
333 21224 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 MARK xset 0x300/0x3f00
88 5556 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x300/0x3f00 ! match-set mwan3_sticky_https src,src MARK and 0xffffc0ff
88 5556 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 MARK xset 0x200/0x3f00
4 240 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x200/0x3f00 ! match-set mwan3_sticky_https src,src MARK and 0xffffc0ff
4 240 mwan3_policy_PIA_balanced all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00
333 21224 SET all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0xfc00/0xfc00 del-set mwan3_sticky_https src,src
333 21224 SET all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0xfc00/0xfc00 add-set mwan3_sticky_https src,src
Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
266 16032 mwan3_policy_PIA_balanced all -- * * 192.168.3.128/28 0.0.0.0/0 mark match 0x0/0x3f00 /* Stv */
333 21224 mwan3_rule_https tcp -- * * 192.168.2.128/25 0.0.0.0/0 multiport sports 0:65535 multiport dports 443 mark match 0x0/0x3f00 /* https */
22 6885 mwan3_policy_PIA_balanced all -- * * 192.168.2.128/25 0.0.0.0/0 mark match 0x0/0x3f00 /* VPNbalance */
379 26580 mwan3_policy_wan_only all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0x3f00 /* default_rule */
0 0 mwan3_policy_PIA_balFAIL tcp -- * * 192.168.1.2 0.0.0.0/0 multiport sports 0:65535 multiport dports 853 mark match 0x0/0x3f00 /* DNSoverTLS */
network config:
cat network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fc37:d251:26fc::/48'
config interface 'lan'
option type 'bridge'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option stp '1'
option _orig_ifname 'wlan0'
option _orig_bridge 'true'
option ifname 'eth1'
option ipaddr '192.168.2.1'
config interface 'lan2'
option type 'bridge'
option proto 'static'
option netmask '255.255.255.0'
option stp '1'
option ipaddr '192.168.3.1'
option delegate '0'
option broadcast '192.168.3.255'
option ifname 'eth1.2'
config interface 'PIA'
option ifname 'tun0'
option proto 'none'
option delegate '0'
config interface 'PIA2'
option ifname 'tun1'
option proto 'none'
option delegate '0'
config interface 'WAN'
option ifname 'eth0'
option _orig_ifname 'eth0'
option _orig_bridge 'false'
option proto 'static'
option ipaddr '192.168.1.2'
option netmask '255.255.255.0'
option gateway '192.168.1.1'
option broadcast '192.168.1.255'
option metric '10'
option dns '8.8.8.8'
firewall config:
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd37:d756:89fb::/48'
config interface 'lan'
option type 'bridge'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option stp '1'
option _orig_ifname 'wlan0'
option _orig_bridge 'true'
option ifname 'eth1'
option ipaddr '192.168.2.1'
config interface 'lan2'
option type 'bridge'
option proto 'static'
option netmask '255.255.255.0'
option stp '1'
option ipaddr '192.168.3.1'
option delegate '0'
option broadcast '192.168.3.255'
option ifname 'eth1.2'
config interface 'PIA'
option ifname 'tun0'
option proto 'none'
option delegate '0'
config interface 'PIA2'
option ifname 'tun1'
option proto 'none'
option delegate '0'
config interface 'WAN'
option ifname 'eth0'
option _orig_ifname 'eth0'
option _orig_bridge 'false'
option proto 'static'
option ipaddr '192.168.1.2'
option netmask '255.255.255.0'
option gateway '192.168.1.1'
option broadcast '192.168.1.255'
option metric '10'
option dns '8.8.8.8'
root@linksys:/etc/config# cat firewall
config defaults
option syn_flood '1'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
config zone
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option network 'lan'
option name 'lan'
config zone
option output 'ACCEPT'
option network 'WAN'
option name 'wan'
option input 'REJECT'
option forward 'REJECT'
config zone
option name 'vpn'
option forward 'REJECT'
option output 'ACCEPT'
option network 'PIA'
option masq '1'
option input 'REJECT'
config zone
option name 'vpn2'
option forward 'REJECT'
option output 'ACCEPT'
option network 'PIA2'
option masq '1'
option input 'REJECT'
config rule
option name 'Allow-DHCP-Renew'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
option src 'wan'
config rule
option name 'Allow-DHCPv6'
option proto 'udp'
option src_ip 'fc00::/6'
option src_port '547'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
option src 'wan'
config rule
option target 'ACCEPT'
option name 'Allow-Ping'
option family 'ipv4'
option proto 'icmp'
list icmp_type 'echo-request'
option src 'wan'
config include
option path '/etc/firewall.user'
config forwarding
option dest 'vpn'
option src 'lan'
config forwarding
option dest 'vpn2'
option src 'lan'
config zone
option name 'lan2'
option network 'lan2'
option output 'ACCEPT'
option input 'ACCEPT'
option family 'ipv4'
option conntrack '1'
option log '1'
option forward 'REJECT'
config forwarding
option dest 'wan'
option src 'lan2'
openvpn config:
cat openvpn
config openvpn 'custom_config'
option config '/etc/openvpn/my-vpn.conf'
config openvpn 'Strong'
option client '1'
option proto 'udp'
option port '1197'
option resolv_retry 'infinite'
option nobind '1'
option persist_key '1'
option persist_tun '1'
option tls_client '1'
option remote_cert_tls 'server'
option comp_lzo 'adaptive'
option verb '1'
option reneg_sec '0'
option dev 'tun0'
option enabled '1'
option fast_io '1'
option float '1'
option route_metric '20'
option route_nopull '1'
option route '0.0.0.0 0.0.0.0 vpn_gateway 20'
option auth 'SHA256'
option cipher 'AES-256-CBC'
list remote 'vpn2.com'
config openvpn 'Strong2'
option client '1'
option proto 'udp'
option port '1197'
option resolv_retry 'infinite'
option nobind '1'
option persist_key '1'
option persist_tun '1'
option tls_client '1'
option remote_cert_tls 'server'
option comp_lzo 'adaptive'
option verb '1'
option reneg_sec '0'
option dev 'tun1'
option enabled '1'
option fast_io '1'
option float '1'
option route_metric '21'
option route_nopull '1'
option route '0.0.0.0 0.0.0.0 vpn_gateway 21'
option auth 'SHA256'
option cipher 'AES-256-CBC'
list remote 'vpn1.com'
Any hints as to what I haven't set correct?