Static route to VPN server

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello everyone,

I'm seeing a strange problem with static routing of one network to a local VPN server. Let's start with my setup:

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '10.0.99.1'

There's a VPN server running on a dedicated machine on the local network: 10.0.99.161.

VPN clients receive an IP from the range 172.22.16.0/24. The VPN server masquerades VPN traffic.

I also want LAN clients to be able to reach VPN clients, so I added a static route:

config route
        option target '172.22.16.0/24'
        option gateway '10.0.99.161'
        option interface 'lan'

I also added a firewall rule:

config rule
        option src 'lan'
        list dest_ip '172.22.16.0/24'
        option target 'ACCEPT'
        option dest '*'
        option name 'Allow-LAN-VPN-Forward'
        list proto 'all'

Nearly everything is working as expected. LAN clients can reach VPN clients and the other way around.

But there is one LAN client (10.0.99.142, a Unifi Access Point) which behaves strangely. It can ping VPN clients, but a HTTP GET request (curl) is not successful until I add the same static route as on the OpenWrt router (see above) to it.

The very same HTTP GET request works fine on all other LAN clients I tested. There is no special treatment of the problematic LAN client's IP anywhere.

I did tcpdumps on the problematic LAN client, the VPN server and the VPN client and the initial establishment of the TCP connection goes back and forth successfully:

10:40:04.554748 IP 10.0.99.142.41466 > 172.22.16.199.8080: Flags [S], seq 3647197637, win 29200, options [mss 1460,sackOK,TS val 172377 ecr 0,nop,wscale 5], length 0
10:40:04.666728 IP 172.22.16.199.8080 > 10.0.99.142.41466: Flags [S.], seq 2197915094, ack 3647197638, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
10:40:04.666933 IP 10.0.99.142.41466 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0

But when the actual GET request is sent by the LAN client, it does not reach the VPN server (10.0.99.161) and thus not the VPN client. Thus, the following line only appears on the LAN client's tcpdump:

10:40:05.679283 IP 10.0.99.142.41466 > 172.22.16.199.8080: Flags [P.], seq 1:75, ack 1, win 913, length 74: HTTP: GET / HTTP/1.1

Can anyone give me a clue as to what might be going wrong here? I'm glad to supply more details, if needed.

2

Do you have other UAP in your network?
If yes, do they behave the same or only this?
If no, have you overridden the "inform host" in Unifi-Controller?
image

4

Just a though so bear with me :slight_smile:

The AP sends to e.g. 172.22.16.1 (the VPN client) this will be send to the default gateway 10.0.99.1 but as your server NATs traffic it will return with the address of the server 10.0.99.161.

Not every firewall will allow this and this traffic can be classified as Invalid.

With the static route traffic will be send directly to the VPN server.

What might work in that case is to remove the NAT on the VPN server so traffic will keeps its original address.

5

I only have the one AP, at least at the moment. And note that this applies to a curl command, not the inform host - yet. Of course that's where I'm headed once the curl command works :smiley:

6

Thank you for this idea. I disabled masquerading on the VPN server, but it didn't change the behavior.

The packet with the timestamp 10:40:05.679283 (see above) is sent out from the LAN client to the VPN client's IP. But the packet never reaches the VPN server which I think is the main issue. Why does OpenWrt not route it like the two packets (10:40:04.554748 and 10:40:04.666933) before?

7

Better apply this first. The UAP is configured to look for the controller in the same subnet, so I wouldn't be surprised if the packet is blocked from that.

1 Like
8

Try capturing both request and response when running tcpdump on OpenWrt and VPN gateway.
Note that your setup likely involves ICMP redirects since the VPN gateway is not the default gateway, but on the same LAN as the clients.

1 Like
9

I did
set-inform http://172.22.16.199:8080/inform
on the AP, but curl still behaves the same way and info on the AP shows the inform host is "unreachable". Is that what you meant?

10

I did, but there is only the request on the LAN client. The tcpdump on the VPN gateway does not show an incoming packet.

Thank you for hinting towards ICMP redirects, as I didn't know about them. Is it possible that they create the problem, although ICMP pings work fine for an infinite amount of packets?

From what I read online, it's not unusual for things to break after a packet or two when ICMP redirects are involved. So this could be a good explanation for why it goes ok at first and then breaks after two / three packets.

1 Like
11

SInce the request from the AP is on the same network (which means the same zone) as the third-party gateway to the VPN that it is being routed to, you don't need a firewall forward and it could be breaking things. In the main router you only need the static route. The default lan intra-zone (lan-lan) forwarding should be ACCEPT.

Also option dest * does not work here-- when you do write a forward rule, the destination zone needs to be explicitly specified as the one particular zone where the destination host/network can be reached. The firewall does not figure that out by itself.

1 Like
12

Correct, provided this:

It should be harmless, but redundant.

This actually works, at least for traffic rules.

Assuming the default firewall configuration, I guess the forwarding should work even if ICMP redirects are disabled on the router or filtered by the client, but your testing results make me doubtful.

You mean the HTTP request is reaching the router, but not the VPN gateway?

13

You don't do anything on the access point.
You need to configure the inform host and if the DHCP server is not cooperating with Unifi controller, you should also add the IP of the controller in Option 43.

1 Like
14

Without that firewall rule, only the OpenWrt router itself can reach VPN clients, LAN clients can not.

This is my /etc/config/firewall configuration for the LAN zone:

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

I understand that with ACCEPT on the last line, I wouldn't need the firewall rule?

1 Like
15

Oh sorry, I misunderstood. The default inform host is "unifi", isn't it? That hostname resolves to the IP 172.22.16.199 so everything should be working fine from that point?

16

You should definitely change the LAN forward policy to ACCEPT.

17

I double-checked by installing tcpdump on the router: Correct, the last packet from my first post (10:40:05.679283) is reaching the router but not the VPN server.

I will now try to disable the firewall rule from my first post and instead set the default policy for forwarding to ACCEPT. Be right back with the results :wink:

1 Like
18

Disabling the firewall rule from my first posting and changing /etc/config/firewall to the following did not change the strange behavior of the AP. Other LAN clients can still reach VPN clients without issues.

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
19

Trying to give a more complete picture. Please correct me with more useful tcpdump calls, if needed.

I issued curl --local-port 4000 http://172.22.16.199:8080 on the problematic AP and tcpdumped on the AP itself, the OpenWrt router and the VPN server:

Access Point

U6-Lite-BZ.6.2.39# tcpdump -i br0 host 172.22.16.199 and port 4000
16:59:31.570219 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [S], seq 104323679, win 29200, options [mss 1460,sackOK,TS val 328800 ecr 0,nop,wscale 5], length 0
16:59:31.679918 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:31.680177 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0
16:59:31.680627 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.008162 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.338157 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.652263 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:32.652428 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0
16:59:32.668109 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.998161 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.328160 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.658154 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.988252 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:34.318173 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:34.652722 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:34.652887 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0
16:59:34.978183 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:36.298172 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:38.666464 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:38.666650 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0
16:59:38.948163 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:44.248199 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1:83, ack 1, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:46.670834 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:46.671029 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0
16:59:52.680947 IP 172.22.16.199.8080 > 10.0.99.142.4000: Flags [R], seq 1296771737, win 0, length 0

OpenWrt router

root@router:~# tcpdump -i br-lan host 10.0.99.142 and 172.22.16.199 and port 4000
16:59:31.573883 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [S], seq 104323679, win 29200, options [mss 1460,sackOK,TS val 328800 ecr 0,nop,wscale 5], length 0
16:59:31.574070 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [S], seq 104323679, win 29200, options [mss 1460,sackOK,TS val 328800 ecr 0,nop,wscale 5], length 0
16:59:31.683788 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0
16:59:31.684392 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.011793 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.341784 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:32.656036 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0
16:59:32.671741 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.001788 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.331786 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.661784 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:33.991886 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:34.321805 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:34.656496 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0
16:59:34.981816 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:36.301841 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:38.670259 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0
16:59:38.951801 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:44.251841 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [P.], seq 1627241545:1627241627, ack 27338385, win 913, length 82: HTTP: GET / HTTP/1.1
16:59:46.674635 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0

VPN server

root@vpn-server:~# tcpdump -i enp0s3 host 10.0.99.142 and 172.22.16.199 and port 4000
16:59:31.572533 IP 10.0.99.142.4000 > 172.22.16.199.http-alt: Flags [S], seq 104323679, win 29200, options [mss 1460,sackOK,TS val 328800 ecr 0,nop,wscale 5], length 0
16:59:31.680578 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:32.652923 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:34.653388 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:38.667113 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:46.671465 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [S.], seq 1296771736, ack 104323680, win 65535, options [mss 1385,nop,wscale 8,nop,nop,sackOK], length 0
16:59:52.681581 IP 172.22.16.199.http-alt > 10.0.99.142.4000: Flags [R], seq 1296771737, win 0, length 0
20

You have asymmetric routing.

uci set firewall.lan.masq_allow_invalid='1'
uci commit firewall
service firewall restart
1 Like
21

I think I'm on to something:

The following packet is dumped on the AP:

16:59:31.680177 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 1, win 913, length 0

On the router the relevant line seems to be:

16:59:31.683788 IP 10.0.99.142.4000 > 172.22.16.199.8080: Flags [.], ack 27338385, win 913, length 0

which is only dumped once instead of twice, which I would expect (incoming from the AP and outgoing to the VPN server). Also the ack 1 vs ack 27338385 seemed suspicious.

Why would the router drop it like that? I disabled "Drop invalid packets" on the OpenWrt firewall settings and guess what... It works.

Now I'm very interested to understand this better. I've enabled drop_invalid since I started using OpenWrt years ago, because it seemed logical and never had any unwanted effects.

Is the package leaving the AP invalid because it uses ack 1 and thus refers to a non-active connection / not previously existing seq? If so: Any guesses on why it does that?