Hi. Thank you for this fundamental tool to use with VPNs.
It seems[1] that OpenWRT v19.07.8 branch is using vpn-policy-routing v0.3.2 instead of v0.2.1 and thus failing[2] to apply the same policies that work fine on OpenWRT v19.07.7 (which uses vpn-policy-routing v0.2.1). I downgraded to OpenWRT v19.07.7, which solved the issue. I might try to compile[3] OpenWRT v19.07.8 cherry-picking vpn-policy-routing's package files from OpenWRT v19.07.7 feeds in a nasty (but hopefully functional way).
Cheers!
It seems that the Remote Addresses/Domains needs to be able to resolve in order to use Remote Ports, so something like below is not working, is it possible to get it to work?
Hi, Stangri. Thank you for your suggestion, I followed[1] it and compiled OpenWRT v19.07.8 with vpn-policy-routing 0.3.5-2, but the issue[2][3] persists. After downgrading[4] to vpn-policy-routing 0.2.1-13 running on OpenWrt 19.07.8 the policies could be applied[5]. Hope this report helps us to solve the issue. Thank you for re-factoring the code and sorry for not being able to propose further code enhancements.
[1] git clone https://github.com/stangri/source.openwrt.melmac.net.git
cd source.openwrt.melmac.net.git
git checkout a6967fb0c69e5d52288ac5cad1a919a271bca7ab
rm -r ~/openwrt/feeds/packages/net/vpn-policy-routing
cp -r vpn-policy-routing/ ~/openwrt/feeds/packages/net
rm ~/openwrt/feeds/luci/applications/luci-app-vpn-policy-routing/ ~/openwrt/feeds/luci/applications/luci-app-vpn-policy-routing-js/
cp -r luci-app-vpn-policy-routing/ luci-app-vpn-policy-routing-js/ ~/openwrt/feeds/luci/applications
cd ~/openwrt/
make dirclean ; make download ; make
[2] # service vpn-policy-routing support
vpn-policy-routing 0.3.5-2 running on OpenWrt 19.07.8.
============================================================
Dnsmasq version 2.80 Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth nettlehash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default uplink1.soporte 0.0.0.0 UG 0 0 0 wlan0
IPv4 Table 201:
IPv4 Table 201 Rules:
IPv4 Table 202: default via 192.168.1.1 dev wlan0
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.2
192.168.12.0/24 dev br-guest proto kernel scope link src 192.168.12.254
192.168.14.0/24 dev br-lan proto kernel scope link src 192.168.14.1
IPv4 Table 202 Rules:
32765: from all fwmark 0x20000/0xff0000 lookup wwan
IPv6 Table 201: fe80::/64 dev eth0.2 proto kernel metric 256 pref medium
IPv6 Table 202: fe80::/64 dev wlan0 proto kernel metric 256 pref medium
============================================================
Mangle IP Table: PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -s 192.168.14.0/24 -m comment --comment lan -c 4435 404472 -g VPR_MARK0x30000
============================================================
Mangle IPv6 Table: PREROUTING
-N VPR_PREROUTING
============================================================
Current ipsets
create wan6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create wwan6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create wwan hash:net family inet hashsize 1024 maxelem 65536 comment
create Provider1VPN6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create Provider1VPN hash:net family inet hashsize 1024 maxelem 65536 comment
============================================================
[3] http://router/cgi-bin/luci/admin/vpn/vpn-policy-routing
VPN and WAN Policy-Based Routing
Service Status [vpn-policy-routing 0.3.5-2]
Service Status Running
Service Gateways wan/eth0.2/0.0.0.0/fe80::xxxx:xxxx:xxxx:xxxx/64 # modifed this line for privacy
The ✓ indicates default gateway. See the README for details.
Service Errors
Failed to set up 'wwan/wlan0/192.168.1.1/fe80::xxxx:cxfx:xxxx:xxxx/64 \033[0;32m[\xe2\x9c\x93]\033[0m' # modifed this line for privacy
Failed to set up 'Provider1VPN/tun0/xx.xx.xx.xx/xxxx::xxxx:xxxx:xxxx:xxxx/64' # modifed this line for privacy
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p tcp -s 192.168.14.0/24 -m multiport --sport 6881:6891 -m comment --comment torrent_in
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p udp -s 192.168.14.0/24 -m multiport --sport 6881:6891 -m comment --comment torrent_in
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p tcp -s 192.168.14.0/24 -m multiport --dport 6881:6891 -m comment --comment torrent_out
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p udp -s 192.168.14.0/24 -m multiport --dport 6881:6891 -m comment --comment torrent_out
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p tcp -s 192.168.14.0/24 -m multiport --sport 5500 -m comment --comment vnclisten
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p udp -s 192.168.14.0/24 -m multiport --sport 5500 -m comment --comment vnclisten
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p tcp -s 192.168.14.12 -m multiport --dport 22 -m comment --comment ent_22
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -p udp -s 192.168.14.12 -m multiport --dport 22 -m comment --comment ent_22
iptables -t mangle -A VPR_PREROUTING -g VPR_MARK0x10000 -s 192.168.15.0/24 -m comment --comment guest
[4] # opkg list-installed | grep vpn-policy
luci-app-vpn-policy-routing - 0.3.2-20-1
vpn-policy-routing - 0.2.1-13
# service vpn-policy-routing support
vpn-policy-routing 0.2.1-13 running on OpenWrt 19.07.8. WAN (IPv4): wwan/dev/192.168.1.1.
============================================================
Dnsmasq version 2.80 Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth nettlehash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default uplink1.soporte 0.0.0.0 UG 0 0 0 wlan0
IPv4 Table 201:
IPv4 Table 201 Rules:
IPv4 Table 202: default via 192.168.1.1 dev wlan0
192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.2
192.168.12.0/24 dev br-guest proto kernel scope link src 192.168.12.254
IPv4 Table 202 Rules:
IPv6 Table 201: fe80::/64 dev eth0.2 proto kernel metric 256 pref medium
IPv6 Table 202: fe80::/64 dev wlan0 proto kernel metric 256 pref medium
============================================================
IP Tables PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -s 192.168.15.0/24 -m comment --comment guest -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -m comment --comment lan -c 1355 99299 -j MARK --set-xmark 0x30000/0xff0000
-A VPR_PREROUTING -s 192.168.14.12/32 -p udp -m multiport --dports 22 -m comment --comment ent_22 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.12/32 -p tcp -m multiport --dports 22 -m comment --comment ent_22 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p udp -m multiport --sports 5500 -m comment --comment vnclisten -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p tcp -m multiport --sports 5500 -m comment --comment vnclisten -c 1 40 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p udp -m multiport --dports 6881:6891 -m comment --comment torrent_out -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p tcp -m multiport --dports 6881:6891 -m comment --comment torrent_out -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p udp -m multiport --sports 6881:6891 -m comment --comment torrent_in -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_PREROUTING -s 192.168.14.0/24 -p tcp -m multiport --sports 6881:6891 -m comment --comment torrent_in -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
IP6 Tables PREROUTING
-N VPR_PREROUTING
============================================================
IP Tables FORWARD
-N VPR_FORWARD
============================================================
IPv6 Tables FORWARD
-N VPR_FORWARD
============================================================
IP Tables INPUT
-N VPR_INPUT
============================================================
IPv6 Tables INPUT
-N VPR_INPUT
============================================================
IP Tables OUTPUT
-N VPR_OUTPUT
============================================================
IPv6 Tables OUTPUT
-N VPR_OUTPUT
============================================================
Current ipsets
============================================================
[5] Service Status [vpn-policy-routing 0.2.1-13]
Service Status Running
Service Gateways
wan/eth0.2/0.0.0.0/fe80::xxxx:xxxx:xxxx:xxxx/64 # modifed this line for privacy
wwan/wlan0/192.168.1.1/fe80::xxxx:xxxx:xxxx:xxxx/64 ✓ # modifed this line for privacy
Provider1VPN/tun0/xx.xx.xx.xx/fe80::xxxx:xxxx:xxxx:xxxx # modifed this line for privacy
I've two separate VPN client connections active on my device and want to provide separate Ethernet ports for each of the connections, as in port 3 using VPN 1 and port 4 using VPN2, and the rest of the ports using regular WAN connection (as described in this post, a reply led me to VPN Polic-Based Routing).
From what I understand, this doesn't do device-based policies, but I'm replying here hoping to be proved wrong, or maybe learn some hacky way that can let me do it anyway.
Still pretty new to all of this OpenWRT stuff, but pulling my way through. Alas, it seems I've hit a bit of a snag.
I'm trying to set up my network such that I'm able to run multiple OpenVPN clients without using them as the default gateway (thanks for the advice in the README by the way) and have them available as custom routes for this package.
At the moment though I've been getting stuck on a particular error after setting up both VPN clients and the Custom User Files basing myself on the Netflix example (I'll attach below).
It seems that PBR's getting stuck while setting up the gateway, and while I think it might be related to me using a different DHCP provider (odhcpd + unbound), I'm still not sure neither confident in how to reverse said decision if needed.
Hey, wulfy! Didn't expect to see you around this thread too.
I believe you're referring to what's covered in this section of the README, correct?
Multiple OpenVPN Clients
If you use multiple OpenVPN clients on your router, the order in which their devices are named (tun0, tun1, etc) is not guaranteed by OpenWrt. The following settings are recommended in this case.
If so, I think VPN-PBR is recognizing the new interface names, or at least seems to be doing so after following the instructions:
Advanced supported_interface list/string Allows to specify the space-separated list of interface names (in lower case) to be explicitly supported by the vpn-policy-routing service. Can be useful if your OpenVPN tunnels have dev option other than tun*.
note: dev = device = tun (etc)
you have
list supported_interface 'PIA_US_Texas'
list supported_interface 'PIA_UK_London'
list supported_interface 'PIA_SPA_Madrid'
= interface = logical = not dev
you have non-typical dev/ifname names
ovpncX
which are not tun... so in theory the advice above applies
(but due to the amount of recent changes to openwrt and probably vpn pbr... this observation is more of a question than advice... the simplest way to exclude issues here is to rename/recreate your 'ovpn' devices as 'tun' devices and update the network config accordingly)
Okay, I just tried changing the "PIA_X_Y" names to ovpncX names as you suggested. I think it might be a step in the right direction, though it seems that the problem might still be located somewhere else.
I'll attach the produced logs just to be sure.
Mon Aug 23 20:30:39 2021 user.notice vpn-policy-routing [18677]: Creating table 'wan/pppoe-wan/my-v4.addr/fc00:1020:1c:3940::1/64 fe80::1'
Mon Aug 23 20:30:39 2021 user.notice vpn-policy-routing [18677]: [✗]
Mon Aug 23 20:30:40 2021 daemon.err uhttpd[27348]: RTNETLINK answers: File exists
Mon Aug 23 20:30:40 2021 user.notice vpn-policy-routing [18677]: Creating table 'wg0/10.200.200.1/::/0' [✗]
Mon Aug 23 20:30:41 2021 daemon.err uhttpd[27348]: RTNETLINK answers: File exists
Mon Aug 23 20:30:41 2021 user.notice vpn-policy-routing [18677]: Creating table 'PIA_US_Texas/ovpnc0/10.41.112.200/::/0' [✗]
Mon Aug 23 20:30:41 2021 daemon.err uhttpd[27348]: RTNETLINK answers: File exists
Mon Aug 23 20:30:41 2021 user.notice vpn-policy-routing [18677]: Creating table 'PIA_UK_London/ovpnc1/10.20.112.75/::/0' [✗]
Mon Aug 23 20:30:42 2021 daemon.err uhttpd[27348]: RTNETLINK answers: File exists
Mon Aug 23 20:30:42 2021 user.notice vpn-policy-routing [18677]: Creating table 'PIA_SPA_Madrid/ovpnc2/10.4.112.247/::/0' [✗]
Mon Aug 23 20:30:43 2021 daemon.err uhttpd[27348]: ipset v7.6: Error in line 1: The set with the given name does not exist
Mon Aug 23 20:30:43 2021 user.notice vpn-policy-routing [18677]: Running /etc/vpn-policy-routing.vrv.user [✗]
Mon Aug 23 20:30:43 2021 daemon.err uhttpd[27348]: ipset v7.6: Error in line 1: The set with the given name does not exist
Mon Aug 23 20:30:43 2021 user.notice vpn-policy-routing [18677]: Running /etc/vpn-policy-routing.disney.user [✗]
Mon Aug 23 20:30:43 2021 user.notice vpn-policy-routing [18677]: service monitoring interfaces: wan wg0 PIA_US_Texas PIA_UK_London PIA_SPA_Madrid [✓]
Mon Aug 23 20:30:44 2021 user.notice vpn-policy-routing [18677]: ERROR: Failed to set up 'wan/pppoe-wan/my.v4.addr/fc00:1020:1c:3940::1/64 fe80::1 [✓]' ERROR: Failed to set up 'wg0/10.200.200.1/::/0' ERROR: Failed to set up 'PIA_US_Texas/ovpnc0/vpnaddr0/::/0' ERROR: Failed to set up 'PIA_UK_London/ovpnc1/vpnaddr1/::/0' ERROR: Failed to set up 'PIA_SPA_Madrid/ovpnc2/vpnaddr2/::/0' ERROR: Error running custom user file '/etc/vpn-policy-routing.vrv.user'! ERROR: Error running custom user file '/etc/vpn-policy-routing.disney.user'! ERROR: failed to set up any gateway!
Mon Aug 23 20:30:44 2021 daemon.err uhttpd[27348]: /usr/lib/lua/luci/dispatcher.lua:921: attempt to index local 'page' (a nil value)
Mon Aug 23 20:30:44 2021 daemon.err uhttpd[27348]: stack traceback:
Mon Aug 23 20:30:44 2021 daemon.err uhttpd[27348]: /usr/lib/lua/luci/dispatcher.lua:921: in function 'dispatch'
Mon Aug 23 20:30:44 2021 daemon.err uhttpd[27348]: /usr/lib/lua/luci/dispatcher.lua:479: in function </usr/lib/lua/luci/dispatcher.lua:478>
Oops, just saw your edit... I guess it's still useful to keep the thread posted?
note... you can (should) wait for stangri's input but you are also running a devel version of vpn-pbr... (my bad-ish)...
might be worth uninstalling and re-installing to see also of this makes much difference...
(I see that your network config is not migrated to 'config-network-device' format... so there is a likely chance that there is some mismatch somewhere related to this... I packed the newer version in expecting better support for anyone(everyone) who migrates to 'config-network-device' )... but as you haven't migrated... the older version/s should also be ok...
I'll keep the repo command in mind for when this is solved (you've certainly built a lot of custom solutions, it looks like you've thought of everything!), but for the moment I'll attach the results of the uninstall and reinstall (oddly enough, it seems stangri's repo is hosting an even newer version than expected)
root@OpenWRT-RPi4 /etc45# opkg remove luci-app-vpn-policy-routing
Removing package luci-app-vpn-policy-routing from root...
root@OpenWRT-RPi4 /etc46# opkg remove vpn-policy-routing
Removing package vpn-policy-routing from root...
Stopping service and removing rc.d symlink for vpn-policy-routing
vpn-policy-routing 0.3.5-3 stopped [✓]
Command failed: Not found
Not deleting modified conffile /etc/config/vpn-policy-routing.
root@OpenWRT-RPi4 /etc46# opkg install vpn-policy-routing
Installing vpn-policy-routing (0.3.5-3) to root...
Downloading https://repo.openwrt.melmac.net/vpn-policy-routing_0.3.5-3_all.ipk
Configuring vpn-policy-routing.
Creating table 'wan/pppoe-wan/my.v4.addr/fc00:1020:1c:3940::1/64
fe80::1' [✗]
Creating table 'wg0/10.200.200.1/::/0' RTNETLINK answers: File exists
[✗]
Creating table 'PIA_US_Texas/ovpnc0/vpnaddr0/::/0' RTNETLINK answers: File exists
[✗]
Creating table 'PIA_UK_London/ovpnc1/vpnaddr1/::/0' RTNETLINK answers: File exists
[✗]
Creating table 'PIA_SPA_Madrid/ovpnc2/vpnaddr2/::/0' RTNETLINK answers: File exists
[✗]
Running /etc/vpn-policy-routing.vrv.user ipset v7.6: Error in line 1: The set with the given name does not exist
[✗]
Running /etc/vpn-policy-routing.disney.user ipset v7.6: Error in line 1: The set with the given name does not exist
[✗]
vpn-policy-routing 0.3.5-3 monitoring interfaces: wan wg0 PIA_US_Texas PIA_UK_London PIA_SPA_Madrid [✓]
ERROR: Failed to set up 'wan/pppoe-wan/my.v4.addr/fc00:1020:1c:3940::1/64
fe80::1 [✓]'
ERROR: Failed to set up 'wg0/10.200.200.1/::/0'
ERROR: Failed to set up 'PIA_US_Texas/ovpnc0/vpnaddr0/::/0'
ERROR: Failed to set up 'PIA_UK_London/ovpnc1/vpnaddr1/::/0'
ERROR: Failed to set up 'PIA_SPA_Madrid/ovpnc2/vpnaddr2/::/0'
ERROR: Error running custom user file '/etc/vpn-policy-routing.vrv.user'!
ERROR: Error running custom user file '/etc/vpn-policy-routing.disney.user'!
ERROR: failed to set up any gateway!
Collected errors:
* resolve_conffiles: Existing conffile /etc/config/vpn-policy-routing is different from the conffile in the new package. The new conffile will be placed at /etc/config/vpn-policy-routing-opkg.
root@OpenWRT-RPi4 /etc48# opkg install luci-app-vpn-policy-routing
Installing luci-app-vpn-policy-routing (0.3.5-3) to root...
Downloading https://repo.openwrt.melmac.net/luci-app-vpn-policy-routing_0.3.5-3_all.ipk
Configuring luci-app-vpn-policy-routing.
uci: Parse error (invalid character in name field) at line 1, byte 37
uci: Parse error (option/list command found before the first section) at line 2, byte 1
uci: Parse error (invalid character in name field) at line 12, byte 36
uci: Parse error (too many arguments) at line 16, byte 245
Again, thanks for everything wulfy!
And in the meantime I'll wait for stangri's message as suggested.
