When the tunnel is up, everything that should be forced out the tunnel still just goes out the default gateway (ie; not the VPN).
When the tunnel is down, everything still goes out the default gateway. I'll grab those logs for the tunnel down condition. thanks!
EDIT: So roughly, VPR seems to have no effect in any circumstance.
With openvpn stopped via /etc/init.d/openvpn stop
root@lede:/etc/config# /etc/init.d/vpn-policy-routing reload
Creating table 'wan/192.168.1.1' [✓]
Creating table 'VPN_U/0.0.0.0' [✗]
vpn-policy-routing 0.0.2-20 started on wan/192.168.1.1 with errors [✗]
ERROR: Failed to set up 'VPN_U/0.0.0.0'
vpn-policy-routing 0.0.2-20 monitoring interfaces: wan VPN_U [✓]
root@lede:/etc/config# /etc/init.d/vpn-policy-routing status
vpn-policy-routing 0.0.2-20 running on LEDE 17.01.4. WAN (IPv4): wan/dev/192.168.1.1.
============================================================
Dnsmasq version 2.78 Copyright (c) 2000-2017 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify
============================================================
Routes/IP Rules
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
IPv4 Table 201: default via 192.168.1.1 dev eth0
IPv4 Table 201 Rules:
32743: from all fwmark 0x10000 lookup 201
IPv4 Table 202: unreachable default
IPv4 Table 202 Rules:
32742: from all fwmark 0x20000 lookup 202
============================================================
IP Tables PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -m set --match-set VPN_U dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -m dscp --dscp 0x10 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
Current ipsets
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create VPN_U hash:net family inet hashsize 1024 maxelem 65536 comment
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [✓]
root@lede:/etc/config# ps -w | grep openvpn
23981 root 1036 S grep openvpn
root@lede:/etc/config#
Thanks for your help
Is strict mode enabled in the config?
Regarding dscp-marked packets not being routed -- can you try another policy (ip/domain) to test if they would work?
Yes strict enforcement is on. I did try another standard policy, and that worked. Building on that success, I went on to get everything to work.
What it looks like to me right now, is that I had the default "dummy" rule deleted, and that was causing a problem? Could that be?
Weird, with strict enforcement enabled, VPR is not supposed to report errors when interface is down.
Which one is that?
Maybe I'm misunderstanding, but I think the usual behaviour in luci in VPR is that there is one (maybe it's blank, my memory fails me) rule listed by default. My config didn't have any rule at all listed, even a blank one that wasn't filled out. Am I wrong about this? If so then simply adding a rule seems to have fixed everything for me, or, coincidentlly everything just started working at that time and it's a placebo.
EDIT: missed your first line. Errors while strict is enabled combined with the "missing uci" x 4 that I got when reinstalling suggests to me there's some kind of corruption? I don't know enough about lede to imagine how that might be the case.
Just to confirm I understood you correctly -- as soon as you inserted any IP/domain/port policy the dscp-based policy started to work, but not before that?
I rewrote the whole interface setup function in 0.0.2-21, I haven't tested it yet, but it shouldn't error out if the tunnel is down with strict enforcement set to 1.
That's the way it seemed, but I'm hesitant to say that I know for certain that nothing else changed or affected the results. I'm sick right now and a little foggy but I'll try deleting the one rule later and see if I can recreate the problems I was having.
Is this protecting me from IP leak on my devices when OpenVPN connection dies by default or I need to setup something? I think i should be safe, but i'm not 100% sure.
It should. Before you apply it in any scenario where the negative outcome could be life-threatening, I'd test it in a non-critical environment first.
PS. Make sure strict enforcement is enabled.
Thank You! I made some basic testing and it's working properly so far.
Stangri, thanks for taking the time to help. I deleted the rule, and stopped openvpn, and could not replicate strict enforcing being violated. I think I have to chalk this up to a mistake on my part, sorry for the false alarm.
The one thing I didn't test was just a broken openvpn connection, but for my part I'm satisfied and don't want to mess with my configs there 
thanks again
Hi @stangri, still loving this piece of software, works really well! Also apologies if this issue has been posted before, I'm not entirely sure what to search for!
The exception for this is when my vpn provider doesn't do what I want them to do.
I have one vpn provider (NordVPN) and I run two tunnels to different endpoints, one local in my country and one in the US.
Mostly this works fine (and vpn-policy-routing is really cool when it does!) but it seems sometimes I get provided the same local IP address on both tunnels. This seems to cause an issue with vpn-policy-routing when defining the routes in the route tables.
Here are the relevant network interfaces from a show uci network for the nordvpn tun interfaces:
network.nordvpntun=interface
network.nordvpntun.proto='none'
network.nordvpntun.ifname='tun0'
network.nordvpntun1=interface
network.nordvpntun1.proto='none'
network.nordvpntun1.ifname='tun1'
Here is /etc/init.d/vpn-policy-routing status (with actual policy entries removed for brevity/tinfoil hat):
vpn-policy-routing 0.0.2-21 running on LEDE 17.01.4. WAN (IPv4): wan/dev/x.x.x.x.
============================================================
Dnsmasq version 2.78 Copyright (c) 2000-2017 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC no-ID loop-detect inotify
============================================================
Routes/IP Rules
default hostnameremoved 0.0.0.0 UG 0 0 0 eth1
IPv4 Table 201: default via x.x.x.x dev eth1
IPv4 Table 201 Rules:
32765: from all fwmark 0x10000 lookup 201
IPv4 Table 202: default via 10.1.0.1 dev tunsvr0
IPv4 Table 202 Rules:
32764: from all fwmark 0x20000 lookup 202
IPv4 Table 203: default via 10.8.8.16 dev tun0
IPv4 Table 203 Rules:
32763: from all fwmark 0x30000 lookup 203
IPv4 Table 204: default via 10.8.8.16 dev tun0
IPv4 Table 204 Rules:
32762: from all fwmark 0x40000 lookup 204
============================================================
IP Tables PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -m set --match-set nordvpntun1 dst -c 0 0 -j MARK --set-xmark 0x40000/0xff0000
-A VPR_PREROUTING -m set --match-set nordvpntun dst -c 0 0 -j MARK --set-xmark 0x30000/0xff0000
-A VPR_PREROUTING -m set --match-set vpnsvr0 dst -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_PREROUTING -m set --match-set wan dst -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
============================================================
Current ipsets
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create vpnsvr0 hash:net family inet hashsize 1024 maxelem 65536 comment
create nordvpntun hash:net family inet hashsize 1024 maxelem 65536 comment
create nordvpntun1 hash:net family inet hashsize 1024 maxelem 65536 comment
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [✓]
As you can see, the rules in the mangle table VPR_PREROUTING chain are correct, with the mark values, for 0x30000 to interface nordvpntun and 0x40000 to interface nordvpntun1.
The issue is that ip table 203 and 204 both specify device tun0, I assume because of the same client IP address of 10.8.8.16.
If I issue these commands, the issue is resolved (of course until the next time the router/vpn-policy-routing restarts):
# ip route show table 204
default via 10.8.8.16 dev tun0
# ip route del table 204 default
# ip route add table 204 default via 10.8.8.16 dev tun1
# ip route show table 204
default via 10.8.8.16 dev tun1
I have tried restarting openvpn, taking down one tun interface and bringing it back etc and I can't get a different client IP on the tunnel. Sometimes eventually I will, but it's not reliable.
Any suggestions on a setting I could change to get around this, other than manually running the commands I listed above?
In the OpenVPN config, do you have anything to do with dev for the clients?
I use OpenWrt's suggested way of obtaining device name for the interface: network_get_device dev "$iface". I'm guessing it sometimes returns tun0 for both of your OpenVPN clients. In my previous testing with multiple VPN clients on the same router the tun0 and tun1 names were sometimes swapped (I believe jow has confirmed that is the expected behaviour).
When that happens next time, could you please capture the output of:
source /lib/functions.sh
source /lib/functions/network.sh
config_load network
do_print() { local iface="$1" dev; network_get_device dev "$iface"; echo "$iface - $dev"; }
config_foreach do_print interface
As much as I'd like you to do that ^^^, if you want a permanent solution to your problem, you can do something like this:
/etc/config/network:
config interface 'nordvpntun'
option proto 'none'
option ifname 'ovpnc0'
config interface 'nordvpntun1'
option proto 'none'
option ifname 'ovpnc1'
and
/etc/config/openvpn:
config openvpn 'nordvpn'
option dev_type 'tun'
option dev 'ovpnc0'
config openvpn 'nordvpn1'
option dev_type 'tun'
option dev 'ovpnc1'
I'd recommend you then add both nordvpn and nordvpn1 to the list of supported interfaces, so that VPR properly processes them when they're down.
Awesome thanks for that, when I get home I will grab the output as you've requested, and test the solution you've suggested.
Would you want me to restart openvpn and/or vpn-policy-routing first? I do note the route table gets reset if i update one of the policies as well, assume the service also restarts when I do that.
If you can grab it only when VPR detects both tunnels having device tun0, that'd be great. Don't do anything special, just let it happen naturally if you can.
OK @stangri I have run your commands with my fix in place, and then after updating a VPR policy which resets table 204 to using tun0, both have the same output:
# ip route show table 204
default via 10.8.8.16 dev tun1
# source /lib/functions.sh
# source /lib/functions/network.sh
# config_load network
# do_print() { local iface="$1" dev; network_get_device dev "$iface"; echo "$iface - $dev"; }
# config_foreach do_print interface
loopback - lo
lan - br-lan
wan - eth1
wan6 -
vpnsvr0 - tunsvr0
nordvpntun - tun0
nordvpntun1 - tun1
#
<Policy in VPR changed here, logged out of ssh session and logged back in>
# ip route show table 204
default via 10.8.8.16 dev tun0
# source /lib/functions.sh
# source /lib/functions/network.sh
# config_load network
# do_print() { local iface="$1" dev; network_get_device dev "$iface"; echo "$iface - $dev"; }
# config_foreach do_print interface
loopback - lo
lan - br-lan
wan - eth1
wan6 -
vpnsvr0 - tunsvr0
nordvpntun - tun0
nordvpntun1 - tun1
#
I suppose what this alludes to is the config of the interface itself is correct at using tun1?
I will implement your suggested fix, which makes sense if there is an odd behaviour with using tun0/tun1. Each time I've seen this happen it's always set table 204 to use tun0. It happened some time ago, which did happen immediately after updating packages so I thought perhaps an update to openvpn/vpr caused it, until I figured out what was happening. Eventually I got different IPs from nord so the problem went away.
I have changed the two interfaces to ovpnc0 and opvnc1, and have added the interfaces names to the VPR supported interfaces config, and unfortunately seem to have the same behaviour, where table 204 now specifies the route via ovpnc0.
I also have an odd (openvpn not VPR) issue that I'm trying to work out, I can't have the two nord tunnels as uci config, I have to have one or both using an external config file (option config blah.ovpn) which uses the ovpn file from nord, with a couple of minor updates (auth-user-pass text file, route-nopull and the dev name). UCI config options are exactly as in the nord file, and the first client vpn will work fine so I'm stumped there.
As far as this issue goes I might have to just figure out if I can run a script each time VPR restarts to update route table 204.
Ah, sorry, my bad. Fixed in vpn-policy-routing 0.0.2-22.
Nice! Confirmed I still had the same client IP on both tunnels, tested changing some policies and table 204 remained on the correct interface.
Hopefully it was something simple, thanks!