OpenWrt Forum Archive

Topic: [SOLVED] Policy routing for OpenVPN server & client on the same router

The content of this topic has been archived on 27 Apr 2018. There are no obvious gaps in this topic, but there may still be some posts missing at the end.

PROBLEM
I've set up an OpenVPN server instance on an OpenWRT router that also runs an OpenVPN client instance: host "Lemmiwinks" (see SETUP) can successfully connect to the server instance, but only when the client instance is disabled.

SETUP

OPENWRT
---------------------------------------
| VPN client if: tun0, 10.28.10.6/32  |
| VPN server if: tun1, 10.255.0.1/24  |
| WAN if:        eth0, 192.168.1.2/24 |
---------------------------------------
              |
HOME GATEWAY  |
-------------------------------
| Internal if: 192.168.1.1/24 |
| External if: x.x.x.x/x      |
-------------------------------
              |
INTERNET      |      LEMMIWINKS
-----------------    --------------------------------------
|               |----| VPN client if: tun1, 10.255.0.6/24 |
-----------------    --------------------------------------
              |
VPN PROVIDER  |
--------------------------------    
| VPN server if: 10.28.10.1/32 |
--------------------------------
  • OpenWRT 15.05.1 on Linksys WRT1200ac

  • OpenWRT VPN server instance listens on port 1194

  • OpenWRT firewall accepts incoming connections to port 1194

  • Home Gateway firewall redirects to 192.168.1.2 incoming connections to port 1194

  • Home Gateway external IP x.x.x.x is dynamically mapped to a domain name

WHAT IS GOING ON
I'm pretty sure this has to do with the VPN client instance changing OpenWRT's main routing table, so that all its connections are tunneled through VPN Provider: Lemmiwinks tries to talk to OpenWRT but gets a response from VPN Provider instead, so he rejects it.

SOLUTION?
Lemmiwinks had a similar problem when trying to SSH into OpenWRT (with the VPN client instance enabled). I solved it by simply adding ip rule number 2:

0: from all lookup 128 (not sure what this is: there's no 128 table in /etc/iproute2/rt_table)
1: from all lookup local
2: from 192.168.1.0/24 lookup no_vpn_provider
32766: from all lookup main
32767: from all lookup default (default table is empty)

where no_vpn_provider is a routing table identical to the main table before it gets modified by the VPN client instance. Unfortunately this doesn't do the trick for incoming VPN connections. I tried adding another rule:

3: from 10.255.0.0/24 lookup no_vpn_provider

but it doesn't work either. I tried various combinations of some/all of the previous rules with some/all of the following rules:

w: from all iif tun1 lookup no_vpn_provider
x: from all oif tun1 lookup no_vpn_provider
y: from all to 192.168.0.0/24 lookup no_vpn_provider
z: from all to 10.255.0.0/24 lookup no_vpn_provider

but I had no success. What is the correct routing policy for this situation?

(Last edited by endvour on 11 Dec 2016, 17:13)

Well, I had a breakthrough! The connection to the OpenWRT VPN server instance is successful (even with the client instance enabled) when adding this ugly ip rule:

ip rule add from all to y.y.y.y/y table no_vpn_cli priority 2

the ugly part being that y.y.y.y/y is Lemmiwink's public network, which is not always the same! These are all the ip rules:

0: from all lookup 128
1: from all lookup local
2: from all to y.y.y.y/y lookup no_vpn_provider
32766: from all lookup main
32767: from all lookup default

where, again, "no_vpn_provider" is a copy of the "main" routing table before it gets modified by the VPN client instance, and "main" is the modified routing table that forces all connections through VPN Provider.
Is there a more flexible way to do this? For example, is it possible to do policy routing based on destination ports rather than on destination networks (preferably without iptables big_smile)? An ip rule like this one

ip rule add from all to port 1194 table no_vpn_cli priority 2

would probably solve my problem, but there's no "port" option in ip (also I'm not sure the entire VPN session uses port 1194).

endvour wrote:
ip rule add from all to port 1194 table no_vpn_cli priority 2

would probably solve my problem, but there's no "port" option in ip (also I'm not sure the entire VPN session uses port 1194).

Ip rule is not intended for complex equations.
In such cases iptables mark should be used. Iptables equations can be as complex as possible.
Also netifd can track rules and multiple routing table, no need in direct ip rule commands

Something like this :

config rule
        option mark '0x800/0x800'
        option priority '100'
        option lookup '100'

config route
        option interface 'tunvps'
        option target '0.0.0.0/0'
        option table '100'

It means netifd creates one more routing table with id=100 and puts there default route to interface tunvps, and all packets marked with bitmask 0x800/0x800 will go there

$ ip rule
0:      from all lookup prelocal 
1:      from all lookup local 
100:    from all fwmark 0x800/0x800 lookup 100 
32766:  from all lookup main 
32767:  from all lookup default 
$ ip r show table 100
default dev tun0  proto static  scope link 

To set mark use rules in /etc/config/firewall

config rule
 option target 'MARK'
 option proto 'udp'
 option dest_port '1194'
 option name 'vpn-1'
 option set_mark '0x800/0x800'

This way you're sure everything auto restores on any firewall reload, ifup/ifdown

(Last edited by bolvan on 3 Dec 2016, 10:54)

To clarify, what exactly are you try to do in terms of the VPN connections?

Is the OpenWrt VPN client supposed to be a replacement gateway that all normal traffic gets routed through? Or just some subnet of IPs?

And is the VPN server only supposed to provide access to local intranet, or work as an internet proxy as well? And if so, should that proxied traffic be routed through eth0, or out through the VPN client connection?

ExaltedVanguard wrote:

Is the OpenWrt VPN client supposed to be a replacement gateway that all normal traffic gets routed through? Or just some subnet of IPs?

Yes, the OpenWrt VPN client routes all traffic from the LAN and from the OpenWrt router itself to the VPN provider (except in special circumstances, e.g. to detect the home gateway's public IP for DDNS).

ExaltedVanguard wrote:

And is the VPN server only supposed to provide access to local intranet, or work as an internet proxy as well? And if so, should that proxied traffic be routed through eth0, or out through the VPN client connection?

The VPN server provides access both to the intranet and to the Internet... via the VPN client: the VPN server acts as a middleman to the VPN provider, which in turn acts as a middleman to the Internet.

ExaltedVanguard wrote:

To clarify, what exactly are you try to do in terms of the VPN connections?

Consider there's also an intercepting Squid proxy on the OpenWrt router and other neat things like DNSCrypt. The point of all this is the following: wherever he is, host "Lemmiwinks" should not only have secure access to its intranet, but also experience the Internet exactly as if he were doing it from its LAN. And to do this he doesn't need to configure a thing on its computer, he just needs to connect to the OpenWrt VPN server.

Okay then. Sounds like your connectivity is due to asymmetric routing.

Basically what's happening is that when you try to connect to your server, it's receiving the connection attempt but is trying to send the reply out through the VPN provider rather than through eth0.

Fixing this isn't too hard. I haven't done this in the OpenWrt environment, but I encountered the same issue when I was trying to run a headless torrent box (a raspberry pi) that used a VPN (my previous ISP throttled my connection if BT traffic was detected). When the VPN was up, I couldn't connect to the control interface. I solved the issue using this code, which you should be able to modify for your purposes (I've added comments to explain what's going on):

#Incoming new connections created on the eth0 are given the arbitrary identification mark 0x1
iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW -i eth0 -j CONNMARK --set-mark 0x1
#An arbitrarily, pre-created routing table (32) uses the eth0 (this was behind the router) as the default gateway
ip route add default via 192.168.1.1 table 32
#Connections we've marked are routed through our table, meaning the responses to those new connections are leaving through the same door they came in.
ip rule add fwmark 0x1 table 32
#Clean up our mark after we're done
iptables -t mangle -A OUTPUT -m connmark --mark 0x1 -j CONNMARK --restore-mark

Using this, any new connection being established on the eth0 interface will have its traffic properly routed through the eth0 interface. We don't have to worry about established or related connections, since networking will make sure those use the same routing table that the new connection was established on.

This should at least let clients connect to the sever. I haven't been in your exact situation, so I'm not certain, but I believe you'll have to do this for both the eth0 interface as well as the VPN server interface that clients are connecting to.

Note that none of this is persistent, and will disappear after reboot, so you'll need to us route-up scripts or rc.local to get things to stay. On the upside, this makes it really easy to fix things if you screw up while testing and can't connect to your router anymore (which may happen because that code was for a very simple device that doesn't have like 18 different firewall chains going on - does OpenWrt have any packet marking going on by default that could cause overlap?).

Also remember that the functions I've listed have nothing to do with whether the connection is actually accepted or not, just where to send it if it is. Firewalls gonna firewall.

(Last edited by ExaltedVanguard on 4 Dec 2016, 05:12)

ExaltedVanguard wrote:
#Incoming new connections created on the eth0 are given the arbitrary identification mark 0x1
iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW -i eth0 -j CONNMARK --set-mark 0x1
#An arbitrarily, pre-created routing table (32) uses the eth0 (this was behind the router) as the default gateway
ip route add default via 192.168.1.1 table 32
#Connections we've marked are routed through our table, meaning the responses to those new connections are leaving through the same door they came in.
ip rule add fwmark 0x1 table 32
#Clean up our mark after we're done
iptables -t mangle -A OUTPUT -m connmark --mark 0x1 -j CONNMARK --restore-mark

Thanks for the insight. What you suggest seems exactly what I need but for some reason it's not working (iptables-mod-ipopt is installed). To be safe, I also tried using as table 32 an entire copy of the main routing table (without the VPN client modifications). I did the same thing for the VPN server interface, tun1, with a 0x2 mark. Nothing. I'll try with a port-based approach. The fact that this works when targeting connections from the client's public IP gives me some hope.

(Last edited by endvour on 4 Dec 2016, 06:27)

Just to verify (I mentioned it in the comment but didn't think to modify the code itself), did you change 192.168.1.1 to your WAN gateway? (ISP related, your default gateway when not connected to the VPN provider).

If it's still not working, I'd suspect firewall issues.

No need to change it as it happens to be my WAN gateway as well. I also suspect firewall issues because bolvan's approach is not working either:
in /etc/config/firewall

config rule
        option name 'Mark packets to/from VPN clients'                            
        option target 'MARK'                    
        option proto 'udp'                   
        option dest_port '1194'                       
        option set_mark '0x800/0x800' 

and in /etc/config/network

config rule
        option mark '0x800/0x800'
        option priority '100'
        option lookup 'original_main_table'

To be clear, the VPN client instance uses another port, 1197. The "original_main_table" is fine, I tested it with the VPN client instance disabled.
Maybe "option dest_port '1194'" doesn't target all packets that I need to target? Either that or the firewall is not marking at all? LuCI misinterprets the firewall rule as a "Discard input" rule, but I believe that's just because there's no MARK support in LuCI. To use MARK I only need iptables-mod-ipopt, right?

There is a need to change it, unless the router you're configuring is behind another router. So if your external ISP ip was 234.654.987.45, the commands would probably look like:

iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW -i eth0 -j CONNMARK --set-mark 0x1
ip route add default via 234.654.987.1 table 32
ip rule add fwmark 0x1 table 32
iptables -t mangle -A OUTPUT -m connmark --mark 0x1 -j CONNMARK --restore-mark

Using 192.168.1.1 in my original example was a bad choice (due to the confusion it can cause).

You can find the correct gateway IP to use by looking at your default gateway in the main route table prior to connecting to the VPN provider (PIA?). I believe OpenVPN also passes this to a route-up script as an environmental variable for easy automation, as well.

(Last edited by ExaltedVanguard on 4 Dec 2016, 07:51)

As you can see in my lovely scheme in the first post, the OpenWrt router is indeed behind another router, that has precisely 192.168.1.1 as its internal IP address smile

Well then. I'm bad at reading.

Another thought. Try inserting those rules instead of appending them (use -I instead of -A). Not sure that it would make a difference, but worth a shot.

Just as a test, try adding route-nopull to your vpn provider config so that it doesn't add any routes. Is your client able to connect and operate as expected? Then you'll have at least narrowed the problem down to it being a routing issue and ruled out any process/port/firewall/etc conflicts.

Also, you should probably name (dev tun0 rather than just dev tun) your tunnels in their respective .conf files if you haven't already. Using automatically assigned tun0 and tun1 can cause issues if things aren't loaded in the correct order.

With a router behind a router... I hope you disabled all of this router's routing stuff. Double NAT (triple with a VPN?) and all that. Causes lots of headaches.

(Last edited by ExaltedVanguard on 4 Dec 2016, 08:42)

No success with inserting instead of appending. The client instance and the server instance configurations already use "dev tun0" and "dev tun1" respectively. Double NAT is not pleasant at all but it's all configured properly, I'm 100% sure that is not the problem.
I ruled out a firewall malfunction by
1) disabling the VPN client instance
2) using "route_noexec" for the VPN server instance, so this is the main routing table

default via 192.168.1.1 dev eth0  proto static 
10.255.0.2 dev tun1  proto kernel  scope link  src 10.255.0.1 #this gets added by the server instance even with route_noexec
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.2 
192.168.2.0/24 dev br-lan  proto kernel  scope link  src 192.168.2.1

3) creating this "no_vpn_provider" routing table

default via 192.168.1.1 dev eth0  proto static #copied by main table
10.255.0.0/24 via 10.255.0.2 dev tun1 #added for VPN server
10.255.0.2 dev tun1  proto kernel  scope link  src 10.255.0.1 #added for VPN server
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.2 #copied by main table
192.168.2.0/24 dev br-lan  proto kernel  scope link  src 192.168.2.1 #copied by main table

4) using ExaltedVanguard's iptables commands

iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW -i eth0 -j CONNMARK --set-mark 0x1
iptables -t mangle -A OUTPUT -m connmark --mark 0x1 -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW -i tun1 -j CONNMARK --set-mark 0x2
iptables -t mangle -A OUTPUT -m connmark --mark 0x2 -j CONNMARK --restore-mark

5) using these ip rules

0:     from all lookup 128 
1:     from all lookup local 
100:     from all fwmark 0x1 lookup no_vpn_provider
200:     from all fwmark 0x2 lookup no_vpn_provider
32766:     from all lookup main 
32767:     from all lookup default

Observations:
1) The connection to the server is established and I can ping hosts in my network
2) If I remove the fwmark rules I cannot ping hosts in my network, but I can still establish the connection!

Conclusions:
1) The firewall is doing its job, since the fwmark rules are being enforced
2) The fwmark rules are not responsible for the initial connection, the lookup to the main table is. And since the main table gets modified by the VPN client instance, the connection will fail when that instance is active.

This is killing me.

So, this works...

100:    from all to 91.252.0.0/16 lookup no_vpn_provider

...but this doesn't

iptables -t mangle -A OUTPUT -d 91.252.0.0/16 -j MARK --set-mark 0x1
100:    from all fwmark 0x1 lookup no_vpn_provider

where 91.252.0.0/16 is the public network of the client.
Shouldn't the two methods have exactly the same result? Is this a sign the firewall is not working properly?

Firewall seems to be working: if i do

iptables -t mangle -A PREROUTING -s 192.168.2.0/24 -j MARK --set-mark 0x1
ip rule add fwmark 0x1 table no_vpn_provider priority 100

packets from my LAN (192.168.2.0/24) don't get forwarded to the VPN provider. So it's just a question of finding the right iptables command. Ideas?

In the hope of finding a solution to this I've also posted my question here: https://superuser.com/questions/1152318 … ame-router
In case someone is reading this topic just now and wants to help, you might want to have a look there as well, as the question is perhaps stated more clearly and my attempts have been summarized

Solved.
No need for marking connections/packets, this simple ip rule is enough:

ip rule add from 192.168.1.2 table no_vpn_provider priority 100

The problem is that an OpenVPN server doesn't bind to any specific IP address by default, so the above rule won't work unless one adds this line to the server configuration file

local 192.168.1.2

The discussion might have continued from here.