Wireguard server with double nat and vpn client

Installing and Using OpenWrt Network and Wireless Configuration
1

Hey everyone! I am here with a problem which has been driving me nuts the past few weeks. My setup looks as following:

  • ISP-provided router which has direct internet access
  • Behind that router is my OpenWrt router
  • all my local devices connect to the OpenWrt router either via lan or wifi
  • it runs an OpenVPN client routing all my traffic though it (this has been working fine for years)
  • it also runs a web server serving a nextcloud instance which also works fine
  • now I want to add a wireguard server to connect to my network from outside

What already works:

  • connecting to the wireguard server when associated to the openwrt router, using its local ip as end point
    • handshake works for multiple clients and all traffic goes through the wireguard server
  • connecting to the wireguard server when associated to the ISP-provided internet facing router, using the lan ip provided by the isp router as end point
    • same as above, works perfectly
  • this tells me that wireguard is at least correctly set up

What is the problem:

  • connecting to the wireguard server from the internet does not work
  • I connect via a dyndns provider
    • this works fine for my web server
    • port forwards from the ISP-provided router to the OpenWrt router are set up and working
    • a scan via nmap shows that the ports for the webserver are open while the udp port for wireguard is open|filtered
  • using tcpdump on the wireguard port I see incoming traffic from the internet but for some reason my OpenWrt router does not answer when the connection comes from the internet

My conclusion so far is that something with my firewall settings is not correct. I've gone through all of it a dozen times but I cannot find the error. Here are my configs:

vpn-policy-routing is required as the default route would otherwise go via the vpc client. This works fine for my webserver. I've mostly followed this tutorial to set up the firewall and vpn policy rules: https://github.com/stangri/openwrt_packages/blob/master/vpn-policy-routing/files/README.md#local-wireguard-server--wireguard-client-scenario-1

A few notes about the setup:

  • at 192.168.1.177 is my webserver as well as a dns server (pihole)
  • the wireguard server runs on port 5892
  • 192.168.178.1 is the ip of the isp router
  • the openvpn client and the wireguard server have their own firewall zones (vpn_zone and wg_server)

Thank you everyone for any help!

2

You are using scenario 1 so option append_src_rules '! -d 192.168.200.0/24' is not needed in the policy config.

3

Alright, I have removed that setting. But the problem remains the same unfortunately. Wireguard clients can connect from the OpenWrt Lan and from ISP router Lan but not from the internet (again, I get incoming traffic on port 5892 according to tcpdump but my wireguard server does not answer...)

4

Remove masquerade from wg_server zone, it is not necessary.
Also post here the following:
iptables-save -c; ip -4 addr; ip -4 ru; ip -4 ro ls tab all

1 Like
5

I really appreciate you taking the time to help me, thank you! I've removed the masquerade from the wg_server zone; thanks for the hint.

Here is the output you asked for: https://pastebin.com/uRMt9GVS

6

I don't see anything wrong and I can also see the hits on the firewall that everything works as it should.

[298:86672] -A VPR_OUTPUT -p udp -m multiport --sports 5892 -m comment --comment wg-server -j MARK --set-xmark 0x10000/0xff0000
...
[4:704] -A INPUT -p udp -m udp --dport 5892 -m comment --comment "!fw3: Allow WG Incoming" -j ACCEPT

Can you verify that packets are returning from correct interface?

tcpdump -i eth0.2 -vn -w /tmp/wan.pcap udp port 5892 &
tcpdump -i tun0 -vn -w /tmp/vpn.pcap udp port 5892 &
[start some traffic]
kill $(pidof tcpdump)
tcpdump -r /tmp/wan.pcap
tcpdump -r /tmp/vpn.pcap

Post here the output of the last 2.

7

Since I don't have have a eth0.2 interface and was not sure which one you meant, I adapted the script from here: https://serverfault.com/a/907454 to this: https://pastebin.com/necjpxRT

This way it should listen for all interfaces on my device, only on udp port 5892 and print the corresponding interface. Here is the output when I try to connect from the internet: https://pastebin.com/JDDymYc6
Note that I have replaced my public internet ip with a few ***
Also note that 192.168.178.55 is the lan/wan ip which my openwrt gets from the isp router. Apparently the OpenWrt router just doesn't send an answer back.

Now if I connect my phone to the isp router wifi and set wireguard end point to 192.168.178.55 wireguard works fine: https://pastebin.com/YTqceHF5

Let me know whether this is the output you wanted.

8

It was a typo, I meant eth1.2 , but the script did the job as well anyway.

  1. Disable the vpn client and PBR. Is it working now that everything goes out of wan and not vpn?
  2. Try something else, define the ingress interface for WG as wan and not everything. It shouldn't matter, but I remember another thread that it was conflicting.
  3. Add wg0 interface in lan firewall zone and remove it from wg_server zone. It also shouldn't matter, but we need to rule out all possibilities.
9

I think we are getting close!

  1. Disabling vpn client and PBR: connecting from the internet now works fine!
  2. Not sure I understood this one correctly. You mean the firewall rule for allowing incoming wg traffic? If so I did that and set the source zone to wan: no change; still not working
  3. So I did that, restarted the network and it works. With VPN client and PBR enabled...

Which means something is not right with the wireguard zone. Since I would really like to work with separated zones, do you have any further ideas? Again, thank you for taking the time!

EDIT: I am sorry, point 3 is not working after all. After restarting the network the default route was going through wan and not the vpn client. After making sure the default route is set to the vpn client, it makes no difference whether wg0 interface is assigned to the lan zone or wg server zone: it does not work in any case. So the problem is rather related to PBR not working? Which would be strange since it works fine for my web server and I can also create working rules for specific domains...

10

Can you switch the policy in PBR for wireguard from UDP to TCP and try it. Only there, not in firewall.

11

I did that, no change :frowning:
Still incoming traffic on port 5892 but no answer...

12

Ok restore back the protocol to udp, change verbosity setting to 2 and post the following:
cat /etc/config/vpn-policy-routing , /etc/init.d/vpn-policy-routing support, as well as the output of /etc/init.d/vpn-policy-routing reload

@stangri is it a typo in the readme that although WG uses UDP, your example uses TCP?

13
14

Which line number in the readme are we talking about?

15

Line 485, you have proto tcp, but wireguard is udp.

16

Fantastic catch, thank you! Will be fixed in the next PR. Are you on github?

17

Only as a user, don't have any code.
Do you see anything weird here by any chance? I've been looking at it for a couple of days but all seems correct. And when PBR is disabled it works fine.

19

Is OpenVPN tunnel utilizing tcp ot udp? I'd recommend trying to switch it to TCP and see if it helps. I can't recall the reason exactly, but for the VPN server/client scenarios README recommends using different protocols for them, could it be the same here?

Do other policies (not OUTPUT chain) work?

20

I have changed the vpn client to tcp. Unfortunately it does not help with the problem.

Other policies work fine btw. I have several prerouting polices which work nicely

21

It is Wireguard which doesn't support tcp.