Wireguard Server handshake not happening on specific muti-wan ISP

Hello Community,

I have a specific problem I am not sure how to solve. I have 2 ISP’s in multi-wan config for failover / load-balancing criteria. The 2 ISP’s are Airtel and ACT Broadband here in India. Airtel is using static wan interface and ACT Broadband is using pppoe wan interface. Airtel is called wan interface and ACT Broadband is called wanb interface.

My requirement is to have a wireguard server setup on OpenWrt with full tunnel for remote peers to connect to the internet through my tunnel. It should not be able to reach devices in other local networks on my router nor should it be able to reach my router page for administration. Additionally, I would prefer for the entire setup to dedicatedly use my ACT Broadband ISP for both handshakes and outgoing traffic from the peers. Incidentally, one of the peers I want working on this setup is using the same ISP on their side.

I followed this article to setup my wireguard server.

What works:

When I try to connect from the remote peer to my Airtel ISP Static Address endpoint, the handshake is successful and everything works. In the working configuration, I initially did not have tunlink set on the wireguard proto interface and no explicit PBR to route the wireguard network ‘192.168.30.0/26’ through wanb_only, but, in my efforts to get this to work, I made these changes and my remote peer is still able to successfully complete the handshake with these changes. It’s a mystery how this is still working.

What doesn’t work and I need it setup this way:

When I try to connect the same remote peer to my ACT Broadband Static Address endpoint, the handshake is not completed.

I figure this has something to do with my routing rules of some kind and I can’t figure for the life of me how to get this to work.

config interface 'wgvpn'
        option proto 'wireguard'
        option private_key 'private_key'
        option listen_port '51820'
        option tunlink 'wanb'
        list addresses '192.168.30.1/26'

config wireguard_wgvpn 'wgclient'
        option public_key 'public_key'
        option preshared_key 'psk'
        option persistent_keepalive '25'
        list allowed_ips '192.168.30.2/32'
        option route_allowed_ips '1'
root@openwrt:~# ip route
default via 122.166.252.1 dev wan proto static metric 1
default via 49.206.228.1 dev pppoe-wanb proto static metric 2
49.206.228.1 dev pppoe-wanb proto kernel scope link src A.C.T.IP
122.166.252.0/22 dev wan proto static scope link metric 1
192.168.1.0/24 dev br-lan.10 proto kernel scope link src 192.168.1.100
192.168.10.0/24 dev br-lan.20 proto kernel scope link src 192.168.10.1
192.168.20.0/24 dev br-lan.30 proto kernel scope link src 192.168.20.1
192.168.30.0/26 dev wgvpn proto kernel scope link src 192.168.30.1
192.168.30.2 dev wgvpn proto static scope link
root@openwrt:~#

root@openwrt:~# mwan3 status
Interface status:
 interface wan is online and tracking is active (online 00h:01m:20s, uptime 00h:04m:44s)
 interface wanb is online and tracking is active (online 00h:01m:21s, uptime 00h:04m:36s)
 interface openvpnclient is disconnecting and tracking is active (16)

Current ipv4 policies:
balanced:
 wanb (50%)
 wan (50%)
openvpn_only:
 openvpnclient (100%)
wan_only:
 wan (100%)
wan_wanb:
 wan (100%)
wanb_only:
 unreachable
wanb_wan:
 wan (100%)

Current ipv6 policies:
balanced:
 default
openvpn_only:
 blackhole
wan_only:
 unreachable
wan_wanb:
 unreachable
wanb_only:
 unreachable
wanb_wan:
 unreachable

Directly connected ipv4 networks:
192.168.30.63
192.168.20.1
192.168.10.0/24
192.168.1.100
192.168.30.1
192.168.10.255
192.168.30.0/26
127.0.0.0/8
A.C.T.IP
192.168.1.255
127.0.0.1
224.0.0.0/3
A.IR.TEL.IP
192.168.10.1
122.166.255.255
127.255.255.255
192.168.20.255
122.166.252.0/22
192.168.30.2
192.168.20.0/24
49.206.228.1
192.168.1.0/24

Directly connected ipv6 networks:
fe80::a67b:2cff:fef1:8181
fe80::/64
fe80::c472:1d52:ccb9:996c

Active ipv4 user rules:
    0     0 - wanb_only  0    --  *      *       192.168.30.0/26      0.0.0.0/0
    0     0 - openvpn_only  0    --  *      *       192.168.20.0/24      0.0.0.0/0
   86  4268 S https  6    --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 443
  101 58430 - balanced  0    --  *      *       0.0.0.0/0            0.0.0.0/0

Active ipv6 user rules:

root@openwrt:~# 

I am able to receive the packets on the wanb interface, but no responses go out.

root@openwrt:~# tcpdump -ni pppoe-wanb udp port 51820
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on pppoe-wanb, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
13:18:20.963358 IP 27.61.45.206.36984 > A.C.T.IP.51820: UDP, length 148
13:18:26.123808 IP 27.61.45.206.36984 > A.C.T.IP.51820: UDP, length 148
13:18:31.183308 IP 27.61.45.206.36984 > A.C.T.IP.51820: UDP, length 148
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@openwrt:~#

Handshakes are coming in on the wanb interface, but are being responded to on wan interface.

root@openwrt:~# tcpdump -ni wan udp port 51820
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:22:57.020524 IP A.IR.TEL.IP.51820 > 27.61.45.206.51248: UDP, length 148
13:22:57.103334 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 92
13:23:02.091355 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 92
13:23:07.209171 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 92
13:23:32.220434 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 148
13:23:37.340556 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 148
13:23:43.100525 IP A.IR.TEL.IP.51820 > 27.61.45.206.59742: UDP, length 148
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel
root@openwrt:~#

Take a look at this.

In short:

When receiving connection, the original wireguard will reset the incoming fwmark (but all the incoming connection info is in kernel sk buffer) then use the 0.0.0.0 as responding source ip, use incoming source ip as dest ip to serching the route from main routing table, this cause the wireguard always use the ip of lowest metric as respoding ip.

The easiest way would be to set a lower metric value for ACT Broadband in /etc/config/network. Otherwise, you need to use a custom IP rule/routing table.

AFAIK uci does not yet support ip rules that use sport as a selector.

So, tunlink specification didn’t work as expected, so I added fwmark ‘0x200’ which matches the ip rule output for my wanb interface, removed all PBR rules related to wireguard from mwan3 and everything works through wanb now.

config interface 'wgvpn'
        option proto 'wireguard'
        option private_key 'private_key'
        option listen_port '51820'
        option tunlink 'wanb'
        option fwmark '0x200'
        list addresses '192.168.30.1/26'

config wireguard_wgvpn 'wgclient'
        option public_key 'public_key'
        option preshared_key 'psk'
        option persistent_keepalive '25'
        list allowed_ips '192.168.30.2/32'
        option route_allowed_ips '1'

root@openwrt:~# ip rule
0:      from all lookup local
1001:   from all iif wan lookup 1
1002:   from all iif pppoe-wanb lookup 2
1003:   from all iif tun0 lookup 3
2001:   from all fwmark 0x100/0x3f00 lookup 1
2002:   from all fwmark 0x200/0x3f00 lookup 2
2003:   from all fwmark 0x300/0x3f00 lookup 3
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
3001:   from all fwmark 0x100/0x3f00 unreachable
3002:   from all fwmark 0x200/0x3f00 unreachable
3003:   from all fwmark 0x300/0x3f00 unreachable
32766:  from all lookup main
32767:  from all lookup default
root@openwrt:~#

Yes it does , I know because I made the PR's to add sport and dport a couple of months ago :slight_smile:

2 Likes