MWAN3 with Wireguard VPN: no internet access from VLAN

Hello,

I have a NordLynx Wireguard client VPN configured on my Netgear R7800 w/ openwrt. My network setup is divided in 3 separate VLANS:

  • LAN on 192.168.12.0/24: this is my main network
  • APP on 192.168.13.0/24: this is a guest VLAN
  • WAN on 192.168.10.0/24: this is the WAN connection to my fritzbox

I have added two additional Wireguard VPNs:

  • One VPN server so I can connect to LAN from the outside called "HomeVPN"
  • One Nordlynx VPN client

I would like to use the Nordlynx VPN for internet for the "APP" VLAN, with a fallback to the main WAN (without VPN), if the VPN goes down.
So I have installed mwan3 and configured it like this:
Members:
image

Policies:
image

Rules:
image

And as you can see both interfaces are Online (the router itself can ping 8.8.8.8 on the VPN interface):
image

This is the ouput for route -n:

root@Main-Router:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.10.1    0.0.0.0         UG    0      0        0 eth0.100
10.14.0.0       0.0.0.0         255.255.255.0   U     0      0        0 HomeVPN
10.14.1.0       0.0.0.0         255.255.255.0   U     0      0        0 HomeVPN
10.14.2.0       0.0.0.0         255.255.255.0   U     0      0        0 HomeVPN
37.120.201.171  192.168.10.1    255.255.255.255 UGH   0      0        0 eth0.100
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0.100
192.168.12.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan
192.168.13.0    0.0.0.0         255.255.255.0   U     0      0        0 br-app

If I connect to the LAN network I get internet access though WAN as expected.
If I connect to the APP network I do not get internet access. If I stop the VPN then I get internet access through WAN (as expected).

This is the firewall configuration:

The only way to make the VPN work is by setting a static route to 0.0.0.0 on the NordLynx interface, but of course in this way I do not get internet access anymore on LAN, and in general I don't think is the right way to do it.

I think I'm missing something pretty stupid here.
Any advice?

Thank you

I had problems recently with the downstream dns resolution. The solution was to set dns servers on the interfaces in the dns section. This is I believe a hurdle in mwan3.

1 Like

Have you applied metrics on the wan and vpn interfaces as mentioned in the wiki?
I don't see the second default route from the vpn tunnel in the routing table so I suppose you didn't.
Also route utility is deprecated, prefer ip route instead.

Thank you both.

I don't see the second default route from the vpn tunnel

I indeed missed that step.
I did it but my VPN is not set as "default" as in the example:

default via 192.168.10.1 dev eth0.100 proto static src 192.168.10.2 metric 10 
10.5.0.2 dev NordLynx proto static scope link metric 20

Use default gateway is enabled an "Use gateway metric" is set to 20:
image

In this configuration it was still not working (no internet access for the "App" interface through vpn).
So I tried to add two Static Routes pointing to 0.0.0.0 as follows:

The routing table seems correct now:

default dev eth0.100 proto static scope link metric 10 
default dev NordLynx proto static scope link metric 20 
10.5.0.2 dev NordLynx proto static scope link metric 20

After a reboot everything seems to be working as it should.
Is my solution correct or is there a better way to do it?

Thank you

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export mwan3; \
uci export dhcp; uci export firewall; \
head -n -0 /etc/firewall.user; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
root@Main-Router:~# ubus call system board; \
> uci export network; uci export mwan3; \
> uci export dhcp; uci export firewall; \
> head -n -0 /etc/firewall.user; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
{
	"kernel": "5.4.154",
	"hostname": "Main-Router",
	"system": "ARMv7 Processor rev 0 (v7l)",
	"model": "Netgear Nighthawk X4S R7800",
	"board_name": "netgear,r7800",
	"release": {
		"distribution": "OpenWrt",
		"version": "21.02.1",
		"revision": "r16325-88151b8303",
		"target": "ipq806x/generic",
		"description": "OpenWrt 21.02.1 r16325-88151b8303"
	}
}
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fda3:8d3f:e838::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1.1'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ipaddr '192.168.12.1'

config interface 'wan'
	option proto 'dhcp'
	option device 'eth0.100'
	option metric '10'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option vid '1'
	option ports '2 3 4 6t'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '0t 5'
	option vid '100'

config interface 'app'
	option proto 'static'
	option device 'br-app'
	option ipaddr '192.168.13.1'
	option netmask '255.255.255.0'

config switch_vlan
	option device 'switch0'
	option vlan '3'
	option ports '1 6t'
	option vid '2'

config device
	option type 'bridge'
	option name 'br-app'
	list ports 'eth1.2'

config interface 'HomeVPN'
	option proto 'wireguard'
	option private_key 'xxx'
	option listen_port 'xxx'
	list addresses '10.14.0.1/24'
	list addresses '10.14.1.1/24'
	list addresses '10.14.2.1/24'

config wireguard_HomeVPN
	option description 'xxx'
	option public_key 'xxx'
	list allowed_ips '10.14.0.2/24'

config wireguard_HomeVPN
	option description 'xxx'
	option public_key 'xxx'
	list allowed_ips '10.14.1.2/24'

config wireguard_HomeVPN
	option description 'xxx'
	option public_key 'xxx'
	list allowed_ips '10.14.2.2/24'

config interface 'NordLynx'
	option proto 'wireguard'
	option private_key 'xxx'
	list addresses '10.5.0.2'
	option metric '20'

config wireguard_NordLynx
	option description 'Peer1'
	option public_key 'xxx'
	list allowed_ips '0.0.0.0/0'
	option endpoint_host 'xxx'
	option endpoint_port 'xxx'
	option persistent_keepalive '25'
	option route_allowed_ips '1'

package mwan3

config globals 'globals'
	option mmx_mask '0x3F00'

config interface 'wan'
	option enabled '1'
	option family 'ipv4'
	option initial_state 'online'
	list track_ip '8.8.8.8'
	list track_ip '1.1.1.1'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '4'
	option interval '20'
	option failure_interval '5'
	option recovery_interval '5'
	option down '4'
	option up '3'

config interface 'NordLynx'
	option enabled '1'
	option initial_state 'online'
	option family 'ipv4'
	list track_ip '8.8.8.8'
	list track_ip '1.1.1.1'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option check_quality '0'
	option timeout '4'
	option interval '20'
	option failure_interval '5'
	option recovery_interval '5'
	option down '3'
	option up '6'

config member 'nordlynx_policy'
	option interface 'NordLynx'
	option metric '10'

config member 'wan_policy'
	option interface 'wan'
	option metric '20'

config policy 'app_fallback'
	list use_member 'nordlynx_policy'
	list use_member 'wan_policy'
	option last_resort 'unreachable'

config rule 'app_rule'
	option src_ip '192.168.13.0/24'
	option proto 'all'
	option sticky '0'
	option use_policy 'app_fallback'

package dhcp

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option start '30'
	list ra_flags 'none'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'
	list ra_flags 'none'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'app'
	option interface 'app'
	option limit '150'
	option leasetime '12h'
	option start '30'
	list ra_flags 'none'

package firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan'

config zone
	option name 'app'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'app'

config zone
	option name 'nordvpn'
	option output 'ACCEPT'
	option forward 'REJECT'
	option input 'REJECT'
	option masq '1'
	list network 'NordLynx'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled 'false'

config include
	option path '/etc/firewall.user'

config forwarding
	option src 'app'
	option dest 'wan'

config zone
	option name 'homevpn'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	list network 'HomeVPN'

config forwarding
	option src 'homevpn'
	option dest 'lan'

config forwarding
	option src 'homevpn'
	option dest 'wan'

config redirect
	option target 'DNAT'
	option name 'HomeVPN'
	list proto 'udp'
	option src 'wan'
	option src_dport 'xxx'
	option dest 'lan'
	option dest_ip '192.168.12.1'
	option dest_port 'xxx'

config forwarding
	option src 'app'
	option dest 'nordvpn'

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
12: br-app: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake state UP group default qlen 1000
    inet 192.168.13.1/24 brd 192.168.13.255 scope global br-app
       valid_lft forever preferred_lft forever
14: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.12.1/24 brd 192.168.12.255 scope global br-lan
       valid_lft forever preferred_lft forever
16: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.10.2/24 brd 192.168.10.255 scope global eth0.100
       valid_lft forever preferred_lft forever
18: HomeVPN: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.14.0.1/24 brd 10.14.0.255 scope global HomeVPN
       valid_lft forever preferred_lft forever
    inet 10.14.1.1/24 brd 10.14.1.255 scope global HomeVPN
       valid_lft forever preferred_lft forever
    inet 10.14.2.1/24 brd 10.14.2.255 scope global HomeVPN
       valid_lft forever preferred_lft forever
21: NordLynx: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.5.0.2/32 brd 255.255.255.255 scope global NordLynx
       valid_lft forever preferred_lft forever
default via 192.168.10.1 dev eth0.100 table 1 proto static src 192.168.10.2 metric 10 
10.14.0.0/24 dev HomeVPN table 1 proto kernel scope link src 10.14.0.1 
10.14.1.0/24 dev HomeVPN table 1 proto kernel scope link src 10.14.1.1 
10.14.2.0/24 dev HomeVPN table 1 proto kernel scope link src 10.14.2.1 
xx.xxx.xxx.xxx via 192.168.10.1 dev eth0.100 table 1 proto static metric 10 
192.168.10.0/24 dev eth0.100 table 1 proto static scope link metric 10 
192.168.12.0/24 dev br-lan table 1 proto kernel scope link src 192.168.12.1 
192.168.13.0/24 dev br-app table 1 proto kernel scope link src 192.168.13.1 
default dev NordLynx table 2 proto static scope link metric 20 
10.5.0.2 dev NordLynx table 2 proto static scope link metric 20 
10.14.0.0/24 dev HomeVPN table 2 proto kernel scope link src 10.14.0.1 
10.14.1.0/24 dev HomeVPN table 2 proto kernel scope link src 10.14.1.1 
10.14.2.0/24 dev HomeVPN table 2 proto kernel scope link src 10.14.2.1 
192.168.12.0/24 dev br-lan table 2 proto kernel scope link src 192.168.12.1 
192.168.13.0/24 dev br-app table 2 proto kernel scope link src 192.168.13.1 
default via 192.168.10.1 dev eth0.100 proto static src 192.168.10.2 metric 10 
default dev NordLynx proto static scope link metric 20 
10.5.0.2 dev NordLynx proto static scope link metric 20 
10.14.0.0/24 dev HomeVPN proto kernel scope link src 10.14.0.1 
10.14.1.0/24 dev HomeVPN proto kernel scope link src 10.14.1.1 
10.14.2.0/24 dev HomeVPN proto kernel scope link src 10.14.2.1 
37.120.201.171 via 192.168.10.1 dev eth0.100 proto static metric 10 
192.168.10.0/24 dev eth0.100 proto static scope link metric 10 
192.168.12.0/24 dev br-lan proto kernel scope link src 192.168.12.1 
192.168.13.0/24 dev br-app proto kernel scope link src 192.168.13.1 
local 10.5.0.2 dev NordLynx table local proto kernel scope host src 10.5.0.2 
broadcast 10.14.0.0 dev HomeVPN table local proto kernel scope link src 10.14.0.1 
local 10.14.0.1 dev HomeVPN table local proto kernel scope host src 10.14.0.1 
broadcast 10.14.0.255 dev HomeVPN table local proto kernel scope link src 10.14.0.1 
broadcast 10.14.1.0 dev HomeVPN table local proto kernel scope link src 10.14.1.1 
local 10.14.1.1 dev HomeVPN table local proto kernel scope host src 10.14.1.1 
broadcast 10.14.1.255 dev HomeVPN table local proto kernel scope link src 10.14.1.1 
broadcast 10.14.2.0 dev HomeVPN table local proto kernel scope link src 10.14.2.1 
local 10.14.2.1 dev HomeVPN table local proto kernel scope host src 10.14.2.1 
broadcast 10.14.2.255 dev HomeVPN table local proto kernel scope link src 10.14.2.1 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
broadcast 192.168.10.0 dev eth0.100 table local proto kernel scope link src 192.168.10.2 
local 192.168.10.2 dev eth0.100 table local proto kernel scope host src 192.168.10.2 
broadcast 192.168.10.255 dev eth0.100 table local proto kernel scope link src 192.168.10.2 
broadcast 192.168.12.0 dev br-lan table local proto kernel scope link src 192.168.12.1 
local 192.168.12.1 dev br-lan table local proto kernel scope host src 192.168.12.1 
broadcast 192.168.12.255 dev br-lan table local proto kernel scope link src 192.168.12.1 
broadcast 192.168.13.0 dev br-app table local proto kernel scope link src 192.168.13.1 
local 192.168.13.1 dev br-app table local proto kernel scope host src 192.168.13.1 
broadcast 192.168.13.255 dev br-app table local proto kernel scope link src 192.168.13.1 
0:	from all lookup local
1001:	from all iif eth0.100 lookup 1
1002:	from all iif NordLynx lookup 2
2001:	from all fwmark 0x100/0x3f00 lookup 1
2002:	from all fwmark 0x200/0x3f00 lookup 2
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
3001:	from all fwmark 0x100/0x3f00 unreachable
3002:	from all fwmark 0x200/0x3f00 unreachable
32766:	from all lookup main
32767:	from all lookup default

Sorry, I made a typo. Please run it again and edit your post above.
One thing I noticed is that you don't have enabled to route allowed routes in WG peer configuration. That should fix the issue nor having the default route.

That was the issue! After enabling the "Route Allowed IPs" option, and deleting the Static Routes, it seems to be working!

default via 192.168.10.1 dev eth0.100 proto static src 192.168.10.2 metric 10 
default dev NordLynx proto static scope link metric 20 
10.5.0.2 dev NordLynx proto static scope link metric 20

I've edited the message above with the correct commands, now everything seems working.

Thank you for your help

1 Like

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.