Wireguard with 2 WANs - responds through wrong WAN

Hello,
I am trying to set up the Wireguard server on my OpenWrt router. When the client connects I get no RX traffic, so I checked the tcpdump and the traffic is trying to return via wrong WAN interface.

This is my setup:

config interface 'wan'
        option proto 'dhcp' // IP: 10.76.227.120
        option device 'eth0.2'

config interface 'wanp'
        option proto 'dhcp' // IP: 10.76.227.115
        option peerdns '0'
        option defaultroute '0'
        option device 'veth0'

The wan interface doesn't have public IP and is used for normal traffic from LAN (so that my normal traffic goes to the internet with non public shared IP address from the ISP). This interface is the default route.
The wanp interface has public IP which is routed to my router, all incoming traffic from the internet goes to this interface, either directly to the router or is port forwarded to lan.

My problem is that according to TCP dump I receive the Winguard traffic on the wanp interface but it is trying to respond on the wan interface:

01:05:56.812078 IP <client_ip>.17333 > 10.76.227.115.51820: UDP, length 148
01:05:56.812078 IP <client_ip>.17333 > 10.76.227.115.51820: UDP, length 148
01:05:56.815154 IP 10.76.227.120.51820 > <client_ip>.17333: UDP, length 92

How can I tell Wireguard to respond on the wanp interface (or use the one on which the original packet was received). I guess this is not specifically problem of Wireguard, more like IPtables or routing.

What I tried:
Using SNAT:

iptables -t nat -A POSTROUTING -p udp --sport 51820 -j SNAT --to-source 10.76.227.115

after this I don't see any response in the tcpdump and it still doesn't work.
Setting up different routing table, this also didn't work:

echo "1 wanp" >> /etc/iproute2/rt_tables
ip route add default via 10.76.227.65  dev veth0 src 10.76.227.115 table wanp
iptables -t mangle -A OUTPUT -p udp --sport 51820 -j MARK --set-mark 1
ip rule add from all fwmark 1 table wanp

For completeness this is my WireGuard config:


config interface 'wg0'
        option proto 'wireguard'
        option private_key '<private_key>'
        option listen_port '51820'
        list addresses '10.20.1.1/24'

config wireguard_wg0
        option description 'phone'
        option public_key '<public_key>'
        list allowed_ips '10.20.1.2/32'

config rule 'wg'
        option name 'Allow-WireGuard'
        option src 'wan'
        option proto 'udp'
        option target 'ACCEPT'
        option dest_port '51820'

Route list:

default via 10.76.227.65 dev eth0.2  src 10.76.227.120
10.10.1.0/24 dev br-lan scope link  src 10.10.1.1
10.10.2.0/24 dev br-lan_iot scope link  src 10.10.2.1
10.10.3.0/24 dev eth1.30 scope link  src 10.10.3.1
10.20.1.0/24 dev wg0 scope link  src 10.20.1.1
10.76.227.64/26 dev veth0 scope link  src 10.76.227.115
10.76.227.64/26 dev eth0.2 scope link  src 10.76.227.120

Is there any other way how to tell OpenWrt to respond on the same wan interface it received the traffic on?

Thank you

Possibly related: WAN ping echo-reply going over the wrong interface

One workaround that seemed to work for me when having a Wireguard Server with multiple WANs was setting a specific fwmark on the Wireguard server interface then adding this rule into the firewall

iptables -t mangle -I mwan3_hook 2 -p udp --dport 51820 -j MARK --set-mark 0x100

This then allowed connections to work. I experienced the same issue where replies to Wireguard traffic was being sent over the wrong interface. For context, this seems to allow the Wireguard server to work on the my primary WAN, it isn't necessarily load balanced doing this.

The linked reference about ping traffic generally is valid to a point, but this specifically is more to do with Wireguard itself. There is a lengthy GitHub issue around wireguard and mwan3 issues with multiple WANs when running a server @wackejohn has been testing patches to the Wireguard source code itself around the issue. It's a bit unclear if it's a OpenWrt/mwan3 issue, Wireguard issue or both.

@woodhead thank you for the response, however as @jamesmacwhite pointed out this is more to do with mwan3 which I am not using (since both WANs are connected to the same eth port and going from same ISP I don't need failover)

@jamesmacwhite thank you, I have read through the github issue and found a few mentions about the fwmark aswell. However I am not sure if this can be applied if I don't use the mwan3 aswell.
By setting the fwmark to the Wireguard interface, you mean the wg0 as per my config? But then I am not familiar with mwan3 so I don't know what mwan3_hook chain is, should I swap it with OUTPUT in my case? And isn't this a similar approch that I already tried using as I specified in my original post?

echo "1 wanp" >> /etc/iproute2/rt_tables
ip route add default via 10.76.227.65  dev veth0 src 10.76.227.115 table wanp
iptables -t mangle -A OUTPUT -p udp --sport 51820 -j MARK --set-mark 1
ip rule add from all fwmark 1 table wanp

Ah sorry, I didn't read the original post as carefully as I should. Yes the iptables rule would only work for mwan3, given it is targeting a specific mwan3 chain.

In theory the same concepts could be applied with iptables, although you do appear to be mimicking similar behaviour with fwmark in your example already.

I think ultimately the issue is with Wireguard itself. @wackejohn seems to have gone really deep into the Wireguard source code with patches and such, it sounds like there's a bit of a disagreement with the Wireguard developers on "fixing" the issue, but I must admit it is way over my head in terms of understanding the issues fully.

Normally with multiple WANs I'd suggest you use mwan3 but I understand these two connections have the same gateway, which probably isn't suitable for mwan3 in this case.

thanks for the clarification. I looked at some of the patches by @wackejohn but without first understanding how it all works it's way over my head too.
I think the problem for me is as someone mentioned in the github issue that it uses the default table and gets the route wit lowest metric, I think for this reason it doesn't work with my custom table. I will try few more things, otherwise I am afraid I will have to go back to OpenVPN for the time being or set the wanp as a default route and route all my lan traffic via custom table to wan iface.

Good luck, sorry I can't be more help. I think you are right with the default route issue. While again, acknowledging I'm not an expert on the Linux kernel and networking stack side of things, I do get the impression this is more on the Wireguard implementation side rather than OpenWrt itself.

You have helped a bunch, at least I know I am not crazy haha. Yes I think this is Wireguards problem, it should respond on the same interface the traffic was received on and not automatically send every response on the default route. We will have to wait if it gets fixed. As I feel a bit adventurous I think I will try to redo my routing table and set my wanp interface as default and put wan on a separate table, then mark all traffic coming from LAN and send it to this other table, this should fix it. But I have to wait till I get home because I was too fast with my fingers and typed ip del route default, instead of ip del route default table wanp so I killed my connection sighs

Does OpenWrt use the correct source address in the return traffic? If that's the case then I would try if defining network.wanp.ip4table=<custom table number> improves the situation. Since it will add an ip rule for traffic from the wanp address. But you also need to default route via wanp, otherwise it will be impossible to route.

No it doesn't have the correct source address as you can see in the tcpdump in the first post. I am connecting to server via wanp but the return traffic is going through wan with its IP address. Unfortunately setting a different route table won't work as well as I said earlier, Wireguard ignores this and uses the default route in default table. What I will do instead is use custom table for wan interface and set the default table to the wanp interface.

Ok, I managed to get it working. I will share the details in case somebody has similar scenario to mine:

  • 2x wan ifaces, not using mwan3
  • wanp iface - with Public IP, not used as default gw for internet access from lan, only to access services from internet
  • wan using ISPs shared IP, used for internet access
  • I also have 3 LAN interfaces which are VLANed, lan - PCs, phones and stuff, lan_iot - IoT devices with access to internet, lan_not - IoT devices without access to internet

Wireguard config:

config interface 'wg0'
        option proto 'wireguard'
        option private_key '<priv_key>'
        option listen_port '51820'
        list addresses '10.20.1.1/24'
  1. create a separate route table:
echo "1 wanshared" >> /etc/iproute2/rt_tables
  1. Config interfaces to use the wanshared table - this is used to create their specific routes using the ip4table option
    /etc/config/network:
config interface 'wan'
        option proto 'dhcp'
        option device 'eth0.2'
        option ip4table 'wanshared'

config interface 'wanp'
        option proto 'dhcp'
        option device 'veth0'

config interface 'lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.10.1.1'
        option ip6assign '64'
        option ip6hint '1'
        option device 'br-lan'
        option ip4table 'wanshared'

config interface 'lan_iot'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.10.2.1'
        option device 'br-lan_iot'
        option ip4table 'wanshared'

config interface 'lan_not'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.10.3.1'
        option device 'eth1.30'
        option ip4table 'wanshared'

This results in having two separate route tables:

default:

# ip route list
default via <isp_gw_ip> dev veth0  src <wanp_ip>
10.20.1.0/24 dev wg0 scope link  src 10.20.1.1
<isp_subnet>/26 dev veth0 scope link  src <wanp_ip>

wanshared:

# ip route list table wanshared
default via <isp_gw_ip> dev eth0.2  src <wan_ip>
10.10.1.0/24 dev br-lan scope link
10.10.2.0/24 dev br-lan_iot scope link
10.10.3.0/24 dev eth1.30 scope link
10.20.1.0/24 dev wg0 scope link
<isp_subnet>/26 dev eth0.2 scope link
  1. Mark traffic coming from lan, lan_iot, lan_not and set a ip rule to use the wanshared route table, I put this into the /etc/firewall.user script
iptables -t mangle -A PREROUTING -i br-lan -j MARK --set-mark 0xFF
iptables -t mangle -A PREROUTING -i br-lan_iot -j MARK --set-mark 0xFF
iptables -t mangle -A PREROUTING -i eth1.30 -j MARK --set-mark 0xFF
while ip rule delete fwmark 0xFF lookup wanshared  2>/dev/null; do true; done
ip rule add fwmark 0xFF lookup wanshared
  1. (optional) Set the route to the wg tunnel to the wanshared route table if you want to be able to access peers from your lan
ip route add 10.20.1.0/24 dev wg0 table wanshared
  1. I also needed to add the reload option to the /etc/config/firewall to the user script include so it is loaded after reboot and when all the interfaces are loaded
config include
        option path '/etc/firewall.user'
        option reload '1'

And that's it. Now I have a working tunnel, I can access all of my three lans from my peer (this can be further restricted by using zones and firewall) and I can use wg tunnel to access the internet through my router.
Thank you @jamesmacwhite and @woodhead for pointing me to the right direction.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.