VPN Policy-Based Routing + Web UI -- Discussion

Originally posted here.

TL;DR: I would like this configuration with Wireguard client and server interfaces instead of OpenVPN.

Long Version:
Desired Behavior: WG client device (phone, laptop, tablet, etc.) > WG0 server interface > LAN > WG1 client interface > Commercial VPN provider > Internet

In the desired setup, the WG client device has access to all LAN devices AND internet traffic is also tunneled to Commercial VPN provider.

What currently works: WG client device > WG0 server interface > LAN > WAN (ISP modem) > Internet

In this instance, the clients can connect to WG0 and have full access to LAN devices and internet (over non-VPN ISP WAN). This configuration only works if WG1 client interface is stopped. Once WG1 client interface is started, this tunnel brakes.

What also works: LAN > WG1 client interface > Commercial VPN provider > Internet

In this instance, all devices on the LAN have their internet traffic tunneled to the Commercial VPN provider. This configuration works regardless of whether WG0 server interface is running or stopped.

Here is the desired config:

Configs:
/etc/config/network

config interface 'loopback'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'
	option device 'lo'

config interface 'lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '10.10.10.1'
	option device 'eth0'
	option ipv6 '0'
	option delegate '0'

config interface 'wan'
	option proto 'dhcp'
	option device 'eth1'
	option ipv6 '0'
	option delegate '0'
	option peerdns '0'
	list dns '9.9.9.9'

config device
	option name 'eth1'

config device
	option name 'eth0'

config interface 'WireGuardClient1'
	option proto 'wireguard'
	option private_key 'PRIVATE-KEY'
	list addresses '10.5.0.2'
	option delegate '0'
	option peerdns '0'
	list dns '1.1.1.1'
	option auto '0'

config wireguard_WireGuardClient1
	option public_key 'PUBLIC-KEY'
	option persistent_keepalive '25'
	option route_allowed_ips '1'
	option endpoint_host 'X.X.X.X' #commercial VPN provider endpoint
	option endpoint_port '51820'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::0'

config interface 'WireGuardServer0'
	option proto 'wireguard'
	option private_key 'PRIVATE-KEY'
	option listen_port '51820'
	list addresses '10.14.0.1/24'

config wireguard_WireGuardServer0
	option description 'MyDevice'
	option public_key 'PUBLIC-KEY'
	list allowed_ips '10.14.0.3/32'
	option route_allowed_ips '1'
	option persistent_keepalive '25'

/etc/config/firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone 'lan'
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'
	list network 'LAN'
	list network 'WireGuardServer0'

config zone 'wan'
	option name 'wan'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option input 'REJECT'
	list network 'wan'
	list network 'WAN'
	list network 'WireGuardClient1'

config forwarding 'lan_wan'
	option src 'lan'
	option dest 'wan'

config redirect
	option target 'DNAT'
	option name 'Wireguard Server'
	list proto 'udp'
	option src 'wan'
	option src_dport '51820'
	option dest 'lan'
	option dest_ip '10.10.10.1'
	option dest_port '51820'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

/etc/config/vpn-policy-routing

config vpn-policy-routing 'config'
	option verbosity '2'
	option strict_enforcement '1'
	option src_ipset '0'
	option dest_ipset '0'
	option resolver_ipset 'dnsmasq.ipset'
	option ipv6_enabled '0'
	option boot_timeout '30'
	option iptables_rule_option 'append'
	option procd_reload_delay '1'
	option webui_enable_column '0'
	option webui_protocol_column '0'
	option webui_chain_column '0'
	option webui_show_ignore_target '0'
	option webui_sorting '1'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	list webui_supported_protocol 'all'
	option enabled '1'
	list ignored_interface 'WireGuardServer0'

config policy
	option name 'WireGuardServer'
	option proto 'udp'
	option chain 'OUTPUT'
	option interface 'wan'
	option dest_port '51820'

Support output (with public IP redacted)

root@OpenWrt:~#  /etc/init.d/vpn-policy-routing support 
vpn-policy-routing 0.3.4-8 running on OpenWrt 21.02.1.
============================================================
Dnsmasq version 2.85  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         *               0.0.0.0         U     0      0        0 WireGuardClient1

IPv4 Table 201: default via X.X.X.X dev eth1 
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.1 
10.14.0.0/24 dev WireGuardServer0 proto kernel scope link src 10.14.0.1 
10.14.0.3 dev WireGuardServer0 proto static scope link 
IPv4 Table 201 Rules:
32760:	from all fwmark 0x10000/0xff0000 lookup wan

IPv4 Table 202: default via 10.5.0.2 dev WireGuardClient1 
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.1 
10.14.0.0/24 dev WireGuardServer1 proto kernel scope link src 10.14.0.1 
10.14.0.3 dev WireGuardServer1 proto static scope link 
IPv4 Table 202 Rules:
32759:	from all fwmark 0x20000/0xff0000 lookup WireGuardClient1

IPv4 Table 203: 10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.1 
IPv4 Table 203 Rules:
32763:	from all fwmark 0x30000/0xff0000 lookup WireGuardServer0
============================================================
Mangle IP Table: PREROUTING
-N VPR_PREROUTING
============================================================
Mangle IP Table: FORWARD
-N VPR_FORWARD
-A VPR_FORWARD -p udp -m multiport --sports 51820 -m comment --comment WireGuardServer0 -c 0 0 -g VPR_MARK0x010000
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x010000
-N VPR_MARK0x010000
-A VPR_MARK0x010000 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_MARK0x010000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x020000
-N VPR_MARK0x020000
-A VPR_MARK0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_MARK0x020000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x030000
-N VPR_MARK0x030000
-A VPR_MARK0x030000 -c 0 0 -j MARK --set-xmark 0x30000/0xff0000
-A VPR_MARK0x030000 -c 0 0 -j RETURN
============================================================
Current ipsets
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. 

Edit: Consolidated all configs into single post. Also @stangri, I have VPN-Bypass running on OpenWRT to create a split tunnel for two clients on an ancient WNDR3400 being used as a wired switch. Uptime is >3 years. Amazing package!

There is asymmetric routing. Remove route_allowed_ips from wg client and create policy in pbr to route everything towards the wg client.

I've been meaning to update the README -- if the WG (client) is set as a default gateway, the single policy to allow option src_port '61820' is not going to work as intended (as @Smim0 has documented earlier) because the WG server will be replying from a different port. You can try a range of ports to see if it helps.

Thanks for the reply. I did not realize the random UDP port reply was a limitation of the WG protocol.

Do you think the README example is still valid if an OpenVPN server interface is used with the WG Client interface (i.e. hybrid approach with OpenVPN server and WG Client)? Making the WAN the default gateway as you suggest above is not practical for the other applications and services I have running.

the limitation is due to the UDP protocol and not necessarily WG, if you use Openvpn server in udp mode you will have the same exact issue.

But yes, the hibrid approach with Openvpn server and WGclient would work even with making the WGclient as a default gateway IF and only if the Openvpn server is running with TCP protocol.

2 Likes

Got this working but had to ultimately make the WAN the default gateway and set up a ton of rules to direct network traffic to the WireGuard Client.

Can you please share the rules to make the rules to direct the network traffic to the wgclient?

Thanks

Is anyone using VPR on snapshots?

I need help testing VPR/PBR on snapshots which include firewall4 as per https://github.com/openwrt/packages/issues/16818#issuecomment-1007269510

Thanks!

I'll bite. I've got a spare box to play with.

Not long on how to, but I'll assume I need to remove conflicting firewall and luci-app-firewall, add firewall4, and hope for the best with existing config?

1 Like

Well, so far this has seemed a pretty transparent roll-over. (20 minutes in).

Build-01-08-22-GCC11-FW4-r18523+6-8c501bf9fe-Kernel-5.10.90

make nconfig, select firewall4 save, build. luci-app-firewall appears as with FW3. I'll let it bake overnight and see what transpires.

1 Like

Thanks! As far as I understand, besides switching from fw3 to fw4 you also need to install iptables-nft instead of iptables.

Select:
Network / Firewall / Enable Nftables support

to enable iptables-nft

Can't find any way of deselecting iptables.

Selected by [y]
- PACKAGE_vpn-policy-routing [=y]
- PACKAGE_ip6tables [=y] && IPV6 [=y]                                                                            ā”‚
- PACKAGE_collectd-mod-iptables [=y] && PACKAGE_collectd [=y]

Running a new build overnight with Nftables support/iptables-nft

1 Like

next time, i'd happy to build something for you if you have a diffconfig if needed (20min turnaround if i'm awake :wink: )

These are the nconfig steps I took to run my latest build:

Select Network / Firewall / Enable Nftables support
to enable/select iptables-nft

Checked the Firewall4 Makefile and made sure all Firewall4 DEPENDS were included.

PACKAGE_kmod-nft-core [=y]
PACKAGE_kmod-nft-fib [=y]
PACKAGE_kmod-nft-nat [=y]
PACKAGE_kmod-nft-nat6 [=y]
PACKAGE_kmod-nft-offload [=y]
PACKAGE_nftables-json [=y]
PACKAGE_ucode [=y]
PACKAGE_ucode-mod-fs [=y]
PACKAGE_ucode-mod-ubus [=y]
PACKAGE_ucode-mod-uci [=y]

Let me know if you see anything else that should be included.

Thank you very kindly for that gesture. I'll put a diffconfig on my my google drive for you in that eventuality. IIRC though, we are probably both equidistant and polar opposites :grinning:

1 Like

@stangri

2 days up and nary a problem of any kind. Policies all working as intended, nothing out of the ordinary anywhere with the switch over from FW3 to FW4. YMMV, but I'm satisfied with performance.

are you running vpr in iptables parallel or using iptables-nft?

Not sure exactly how to answer that, but given a lot of this remains out of my depth, I suspect Iā€™m running it in parallel.

I did not disable/flush the ipset rules, and lsmod | grep ip_tables and lsmod | grep nf_tables show both iptables and nftables as active kernel modules.

nft list tables inet shows me the nft tables, and nft list ruleset references the same tables and the rules seem to be based on nftables rules format. If any of that is relevant.

1 Like

Well, that didn't work out well.

I decided to add a second VPN Client named vpn1 specifically for my Guest network. Added it to /etc/config/network thus:

config interface 'vpn1'
          option proto 'none'
          option device 'vpn1'
          option type 'bridge'

Added to /etc/config/openvpn thus:

config openvpn 'GuestVpn'
	option client '1'
	option enabled '1'
	option dev_type 'tun'
	option dev 'vpn1'

Restart /etc/init.d/network restart and checked LuCI Network-Firewall. Clicking Edit on the the vpn1 zone returned `TypeError -can't convert null to object', but I could toggle Input/Output/Forwarding and select Masquerading as well as Save/Apply those values.

So I manually edited the firewall to add the forwarding and restarted the firewall.

Bricked. Reboot - still Bricked.

I've restored the last build and it comes up fine but I get the same issue once I add the forwarding and restart firewall?????????????

Hi,
I have Linksys WRTX32 with OpenWrt installed (OpenWrt 19.07.7 r11306). I am trying to set up OpenVPN with VPN policy based routing to send traffic from predefined devices via different VPN. All remaining traffic should just go via wan bypassing VPN. Following the guide I tried to set up scenario with multiple openvpn clients. When starting OpenVPN I am getting the following error:

Fri Jan 14 20:25:58 2022 daemon.err openvpn(WarsawChopin)[7253]: Error: problem with tun vs. tap setting
Fri Jan 14 20:25:58 2022 daemon.notice openvpn(WarsawChopin)[7253]: Exiting due to fatal error

Software installed:

openvpn-openssl:2.4.11-1
luci-app-openvpn:git-21.357.58233-6f349c9-1
vpn-policy-routing:0.2.1-13
luci-app-vpn-policy-routing:0.3.2-20-1
$ cat /var/vpn-policy-routing-support
vpn-policy-routing 0.2.1-13 running on OpenWrt 19.07.7.
============================================================
Dnsmasq version 2.80  Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-nettlehash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         192.168.0.1     0.0.0.0         UG    0      0        0 br-wan

IPv4 Table 201: default via 192.168.0.1 dev br-wan
192.168.0.0/29 dev br-wan proto kernel scope link src 192.168.0.2
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
IPv4 Table 201 Rules:
32765:	from all fwmark 0x10000/0xff0000 lookup wan
============================================================
Mangle IP Table: PREROUTING
-N VPR_PREROUTING
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x010000
-N VPR_MARK0x010000
-A VPR_MARK0x010000 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_MARK0x010000 -c 0 0 -j RETURN
============================================================
Current ipsets
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [āœ“]
$ cat /etc/config/network
config interface 'lan'
	option type 'bridge'
	option ifname 'eth0.1'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wan'
	option ifname 'eth1.2'
	option proto 'static'
	option netmask '255.255.255.248'
	option ipaddr '192.168.0.2'
	option gateway '192.168.0.1'
	list dns '8.8.8.8'
	list dns '8.8.4.4'
	option type 'bridge'

config interface 'wan6'
	option ifname 'eth1.2'
	option proto 'dhcpv6'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '0 1 2 3 5t'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '4 6t'

config interface 'guest'
	option proto 'none'

config interface 'vpnpl'
  option proto 'none'
  option ifname 'ovpnpl'

config interface 'vpnus'
  option proto 'none'
  option ifname 'ovpnus'
$ cat /etc/config/firewall
config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config zone
	option name 'vpnpl'
	option network 'vpnpl'
	option input 'REJECT'
	option forward 'REJECT'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'

config zone
        option name 'vpnus'
        option network 'vpnus'
        option input 'REJECT'
        option forward 'REJECT'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config policy
  option name 'Google nest VPN PL'
  option interface 'vpnpl'
  option src_addr '192.168.1.124'
$ /etc/config/openvpn
config openvpn 'WarsawChopin'
	option client '1'
	option dev_type 'tun'
	option dev 'ovpnpl'
	option config '/etc/openvpn/WarsawChopin.ovpn'

config openvpn 'NewYork'
	option client '1'
	option dev_type 'tun'
	option dev 'ovpnus'
	option config '/etc/openvpn/NewYork.ovpn'
$ cat /etc/openvpn/WarsawChopin.ovpn
client
dev ovpnpl
proto tcp
remote waw-85.whiskergalaxy.com 443
verify-x509-name waw-85.windscribe.com name

nobind
auth-user-pass /etc/openvpn/WarsawChopin.auth

resolv-retry infinite

cipher AES-256-GCM
ncp-ciphers AES-256-GCM:AES-256-CBC:AES-128-GCM
auth SHA512

verb 2
mute-replay-warnings
remote-cert-tls server
persist-key
persist-tun

key-direction 1
$ cat /etc/openvpn/NewYork.ovpn
client
dev ovpnus
proto tcp
remote wfus-178.whiskergalaxy.com 443
verify-x509-name wfus-178.windscribe.com name

nobind
auth-user-pass /etc/openvpn/NewYork.auth

resolv-retry infinite

cipher AES-256-GCM
ncp-ciphers AES-256-GCM:AES-256-CBC:AES-128-GCM
auth SHA512

verb 2
mute-replay-warnings
remote-cert-tls server
persist-key
persist-tun

key-direction 1

Hi! I need help to NAT only UDP and local (all type of protocols) traffic to OpenVPN server side, i am waste few days and can't do it by myself. TCP traffic goes over tproxy with that iptable rules:

# Match on packet mark 1 using policy routing to have those packets delivered locally
ip rule add fwmark 1 table 100
ip route add local default dev lo table 100
# Identify packets with destination address matching a local socket, set the packet mark to 1
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
# Create new chain
iptables -t mangle -N V2RAY
# Routed devices won't be able to reach the server with this
#iptables -t mangle -A V2RAY -d $ip -j RETURN
# Ignore local IPv4 addresses to bypass the proxy
iptables -t mangle -A V2RAY -d 0.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A V2RAY -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A V2RAY -d 192.168.0.0/16 -j RETURN
iptables -t mangle -A V2RAY -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A V2RAY -d 240.0.0.0/4 -j RETURN
# Anything else should be redirected to Dokodemo-door's local port
iptables -t mangle -A V2RAY -p tcp -j TPROXY --on-ip 127.0.0.1 --on-port 12345 --tproxy-mark 1
# Transparent proxy for routed devices
iptables -t mangle -A PREROUTING -j V2RAY
# Create new chain for proxying the router
iptables -t mangle -N V2RAY_LOCAL
# Ignore v2ray server address
iptables -t mangle -A V2RAY_LOCAL -d $ip -j RETURN
# Ignore local IPv4 addresses to bypass the proxy
iptables -t mangle -A V2RAY_LOCAL -d 0.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 192.168.0.0/16 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 240.0.0.0/4 -j RETURN
# Mark anything else as 1
iptables -t mangle -A V2RAY_LOCAL -p tcp -j MARK --set-mark 1
# Transparent proxy for the router
iptables -t mangle -A OUTPUT -j V2RAY_LOCAL

$ip - proxy and OpenVPN server.

i am just need to add some rules or make another, that will compline with this rules (forward TCP traffic to tproxy and udp + local to OpenVPN server side). Can't find any good solution for this. :frowning:

If anybody helps, i will say very big thanks!