Very recent convert from pfSense here. I switched over to Lede because my VPN provider is so kind to offer wireguard as an alternative to OpenVPN and I wanted to make use of the superior performance. Among other things. Also learning is fun.
I humbly ask for forgiveness if this is posted in the wrong forum, or inappropriate in some other way.
Except for the difference that the Wireguard VPN was an OpenVPN vpn in my pfsense setup I had all this working as intended. I've tried to sketch out my intended setup here, apologies for the imperfections.
With the router itself not using the WG tunnel as it's default route, and using policies to route "Most clients" via the WG tunnel, both the OpenVPN server and "Service A" worked fine.
But then I noticed incoming traffic to the "Service B" port on the WAN interface. So the responses that was supposed to go via the WG tunnel must have exited via WAN and leaked my actual ip rather than translating it and sending it via the WG tunnel. I hope I'm making sense here.
At the moment I'm back to using the WG tunnel as the default route, which also means that my OpenVPN clients can't connect. The OpenVPN server sees the incoming connection on the WAN interface but the response goes out on the WG interface so no connection can be established. And "Service B" clients can't connect. Same story, incoming traffic via WAN, responses via WG.
If anyone could give any pointers on how to accomplish what I'm trying to do, it would be much appreciated. I suppose the solution lies in policy based routing somehow, but I can't quite wrap my head around it.
As my VPN provider allows multiple connections, a workaround could be to have "Service B" run it's own WG client to the VPN provider. But if at all possible I'd much rather have all of this stuff on the lede box.
If you have inbound WAN services, you must ensure that these devices use WAN for their outbound reply traffic, the easiest way to accomplish this is permanently configure the host to use WAN ONLY.
If you need to do something more advanced - like change the outbound routes on this host based on SRC IP or inbound traffic interface, feel free to reply and ask.
If you have inbound WAN services, you must ensure that these devices use WAN for their outbound reply traffic, the easiest way to accomplish this is permanently configure the host to use WAN ONLY.
The thread you're referring to is what lead me to the "openvpn-policy routing" thing. Which worked fine except for the "Service B" leaking IP issue. Which is a bit of a special case since it's forwarded through my VPN provider.
Service B is hosted on a dedicated IP on the LAN side. When I had it set up like this regular outgoing connections from that host was routed trough the VPN. But, from what I've been able to gather, traffic going out from this host on connections initiated from "Service B client" actually went out on the WAN interface. Which I suppose has something to do with it being forwarded from my VPN provider and then DNATed in my Lede box to the Service B host.
If you need to do something more advanced - like change the outbound routes on this host based on SRC IP or inbound traffic interface, feel free to reply and ask.
I suppose this is something along the lines of what I need. I.e. if the connection comes in on the WG interface, on the Service B port, I need the corresponding outgoing packets to leave via the WG interface as well. Which it would appear they didn't do. I verified this with tcpdump.
Maybe I missed something. All I did was add a port forward on the WG interface for the Service B port pointing to the Service B host on the LAN.
I can't get this to work properly. I've tried going back to using the WAN as default route for the Lede box itself, and trying to use routing policies. But for some reason I can't understand, this set up sometimes leaks connections. Monitoring the WAN interface with tcpdump, I can see that every now and then packages get routed directly out on that interface from my LAN.
When using WG as the default route the only traffic I see on WAN (with UDP and some other stuff filtered out) is the regular background noise, trying to connect to telnet, ssh, mssql, rdp and so on.
So it appears the only way to get reliable usage of the VPN is to have it as the default route. This doesn't make any sense to me.
Any pointers on how to get the forwarded services - "Service A" in the diagram and OpenVPN to work while having the default route still be the WG interface?
While Lede is interesting in many ways, my current thinking is to abandon this and roll my own Arch set up to try and do this. The issue above that random packets stray from my routing set up doesn't make any sense.
Maybe I'll revisit if i need to do something tricky with router/ap hardware, but for my current x86 setup I'll go for Arch.
Well that was pretty much hopeless as well. Policy base routing is really hard to get to work as I want. There seem so be a lot of gotchas and incompatibilities. Or I'm just too stupid to understand it.
What will happen now is that I'll go back to pfsense for my main router, have an additional WG-bridge black box (a vm) that will provide a fast internet connection for pfsense to route traffic to.
Pfsense will also be running an openvpn client to the same vpn provider as a fall back. WG is running on a VM after all and the host may need the occasional reboot or maintenence every now and then.
I believe I can get this working with pfsense way of handling multiple WAN connections.
Pfsense will also handle all incoming traffic splendidly without hours of hair pulling around policy based routing. And for the occasional exception which I may want to bypass VPNs entirely that's also easily managed in pfsene.
I gave it an honest shot but couldn't figure it out, so I'm reverting to a mature platform I'm comfortable with while still being able to get the performance out of wireguard.
Maybe I was asking to much from this project, but it's disappointing to not get more help from the community when intrepidly trying to push the evelope beyond the normal use case.
Not that I would personally have been much help, but I've noticed that people are way more inclined to help to people who post configs. Keep in mind that people who can help with pushing the envelope might not be checking the forum very often, especially during the holiday season. Also, besides the vpn-policy-routing package that you've linked there's just a policy-routing package and mwan3, either of which might have helped and/or worked better for you.
Finally, while it might be a very little consolation to you, I have both OVPN and WG clients running on my router and the local (in-LAN) server I want exposed and LEDE with vpn-policy-routing handles this just fine -- no need for occasional reboot or maintenance every now and then.
It's good that you got it working the way you needed it to work!
I'm sorry if I didn't post the right info. But I did draw a diagram of what I was trying to accomplish. That ought to be enough to at least start a discussion.
As I did kind of have it working with the policy routing, except that it leaked connections randomly, and I couldn't make sense of why, I gave up on it.
The plan worked out quite well. The isolated VM does the high performance connection to my VPN provider I so much wanted, I get ~110Mbps straight and ~90 via WG. If it fails pfSense fails over to it's own OpenVPN connection to the same provider, and recovers quite nicely when it's back. Incoming services work as they should and there's no unexpected traffic on the WAN. It's a bit of a contraption of an architecture but it works.
I'm sure I'll revisit LEDE again some day when/if I need to do something more straight forward. For more complex stuff I'm sticking to *BSD.
