Wireguard site-to-site handshakes but don't pass traffic (opnsense + openwrt)

ReDaLeRt · September 18, 2021, 2:46pm

Hello.

I'm implementing a site-to-site wireguard configuration, between an opnsense site (A = 192.168.10.0/24) and another openwrt site (B = 192.168.0.0/24).

The configuration I used on openwrt:

WG_IF="WGPSA"
WG_PORT="63588"
WG_ADDR="10.0.0.2/24"


umask go=
wg genkey | tee wgserver.key | wg pubkey > wgserver.pub
wg genkey | tee wgclient.key | wg pubkey > wgclient.pub
 
WG_KEY="$(cat wgserver.key)"
WG_PUB="$(cat wgclient.pub)"


uci rename firewall.@zone[0]="lan"
uci rename firewall.@zone[1]="wan"
uci del_list firewall.lan.network="${WG_IF}"
uci add_list firewall.lan.network="${WG_IF}"
uci -q delete firewall.wg
uci set firewall.wg="rule"
uci set firewall.wg.name="Allow-WireGuard"
uci set firewall.wg.src="wan"
uci set firewall.wg.dest_port="${WG_PORT}"
uci set firewall.wg.proto="udp"
uci set firewall.wg.target="ACCEPT"
uci commit firewall
/etc/init.d/firewall restart


uci -q delete network.${WG_IF}
uci set network.${WG_IF}="interface"
uci set network.${WG_IF}.proto="wireguard"
uci set network.${WG_IF}.private_key="${WG_KEY}"
uci set network.${WG_IF}.listen_port="${WG_PORT}"
uci add_list network.${WG_IF}.addresses="${WG_ADDR}"


uci -q delete network.wgclient
uci set network.wgclient="wireguard_${WG_IF}"
uci set network.wgclient.public_key="${WG_PUB}"
uci add_list network.wgclient.allowed_ips="${WG_ADDR%.*}.1/32"
uci commit network
/etc/init.d/network restart


uci set network.wgserver.route_allowed_ips="1"
uci add_list network.wgserver.allowed_ips="192.168.10.1/24"
uci commit network
/etc/init.d/network restart

uci del_list firewall.wan.network="${WG_IF}"
uci add_list firewall.lan.network="${WG_IF}"
uci commit firewall
/etc/init.d/firewall restart

"A" Handshare with sucess with "B":

"B" Handshare with sucess with "A":
imagem

wg0 interface firewall rules:

Wireguard firewall rules:

I can ping from the opnsense (site "A") the host on the site "B". But I can't from hosts of "A" ping those on "B". Neither from the openwrt (site "B") or its host to "A".

I'm hoping someone can shed some light into this.

Thanks.

vgaetera · September 18, 2021, 3:09pm

You have added a peer named wgclient.

You need to configure the wgclient peer, not wgserver.

ReDaLeRt · September 18, 2021, 5:29pm

Still does the same behaviour:

Opnsense "A" pings hosts on "B"
Hosts on "A" don't ping Openwrt "B"
Hosts on "A" don't ping hosts on "B"
Openwrt "B" don't ping Opnsense "A"
Openwrt "B" don't ping Hosts on "A".

vgaetera · September 18, 2021, 5:36pm

Follow the instructions in the troubleshooting section of the wiki.
Post the collected diagnostics from both peers to pastebin.com.

ReDaLeRt · September 18, 2021, 5:53pm

For Openwrt: https://pastebin.com/eRmTS3zY
For Opnsense, the firewall and network rules are posted above.

ReDaLeRt · September 18, 2021, 6:10pm

I did previously:

uci set network.wgclient.route_allowed_ips="1"
uci add_list network.wgclient.allowed_ips="192.168.10.1/24"
uci commit network
/etc/init.d/network restart

For reference, I'm using OpenWrt SNAPSHOT r17528-aa344bcfa8 / LuCI Master git-21.226.86205-376af36, with kernel 5.10.64.

ReDaLeRt · September 18, 2021, 6:19pm

# uci show network.wgclient; wg show; ip route show
network.wgclient=wireguard_WGPSA
network.wgclient.description='WGPSA'
network.wgclient.route_allowed_ips='1'
network.wgclient.endpoint_host='REMOTE_WAN_SITE_A_PUBLIC_IP'
network.wgclient.endpoint_port='63588'
network.wgclient.persistent_keepalive='25'
network.wgclient.public_key='*****************************'
network.wgclient.allowed_ips='192.168.10.1/24' '192.168.11.1/24' '10.0.0.1/32' '192.168.10.1/24'
interface: WGPSA
  public key: ******************************
  private key: (hidden)
  listening port: 63588

peer: **********************************
  endpoint: REMOTE_WAN_SITE_A_PUBLIC_IP:63588
  allowed ips: 192.168.11.0/24, 10.0.0.1/32, 192.168.10.0/24
  transfer: 0 B received, 1.01 KiB sent
  persistent keepalive: every 25 seconds
default via 192.168.1.254 dev eth0.2  src 192.168.1.100
10.0.0.0/24 dev WGPSA scope link  src 10.0.0.2
10.0.0.1 dev WGPSA scope link
REMOTE_WAN_SITE_A_PUBLIC_IP via 192.168.1.254 dev eth0.2
192.168.0.0/24 dev br-lan scope link  src 192.168.0.1
192.168.1.0/24 dev eth0.2 scope link  src 192.168.1.100
192.168.20.0/24 via 192.168.0.12 dev br-lan

ReDaLeRt · September 18, 2021, 6:45pm

Did a total reconfiguration, from a previous "clean" backup, before begining with this wireguard thing:

WG_IF="WGPSA"
WG_PORT="63588"
WG_ADDR="10.0.0.2/24"


umask go=
wg genkey | tee wgserver.key | wg pubkey > wgserver.pub
wg genkey | tee wgclient.key | wg pubkey > wgclient.pub
 
WG_KEY="$(cat wgserver.key)"
WG_PUB="$(cat wgclient.pub)"


uci rename firewall.@zone[0]="lan"
uci rename firewall.@zone[1]="wan"
uci del_list firewall.lan.network="${WG_IF}"
uci add_list firewall.lan.network="${WG_IF}"
uci -q delete firewall.wg
uci set firewall.wg="rule"
uci set firewall.wg.name="Allow-WireGuard"
uci set firewall.wg.src="wan"
uci set firewall.wg.dest_port="${WG_PORT}"
uci set firewall.wg.proto="udp"
uci set firewall.wg.target="ACCEPT"
uci commit firewall
/etc/init.d/firewall restart


uci -q delete network.${WG_IF}
uci set network.${WG_IF}="interface"
uci set network.${WG_IF}.proto="wireguard"
uci set network.${WG_IF}.private_key="${WG_KEY}"
uci set network.${WG_IF}.listen_port="${WG_PORT}"
uci add_list network.${WG_IF}.addresses="${WG_ADDR}"


uci -q delete network.wgclient
uci set network.wgclient="wireguard_${WG_IF}"
uci set network.wgclient.public_key="${WG_PUB}"
uci add_list network.wgclient.allowed_ips="${WG_ADDR%.*}.1/32"
uci commit network
/etc/init.d/network restart


uci set network.wgclient.route_allowed_ips="1"
uci add_list network.wgclient.allowed_ips="192.168.10.1/24"
uci add_list network.wgclient.allowed_ips="192.168.11.0/24"
uci commit network
/etc/init.d/network restart

uci del_list firewall.wan.network="${WG_IF}"
uci add_list firewall.lan.network="${WG_IF}"
uci commit firewall
/etc/init.d/firewall restart

The issues remain.

vgaetera · September 18, 2021, 6:56pm

You are using the wrong prefix/suffix for the allowed IPs.
It must be 0/24, not 1/24, otherwise it results in the error:

Error: Invalid prefix for given prefix length.

So, it should be like this:

uci -q delete network.wgclient.allowed_ips
uci add_list network.wgclient.allowed_ips="10.0.0.1/32"
uci add_list network.wgclient.allowed_ips="192.168.10.0/24"
uci add_list network.wgclient.allowed_ips="192.168.11.0/24"
uci commit network
/etc/init.d/network restart

Or, you can combine those subnets into 192.168.10.0/23.

ReDaLeRt · September 18, 2021, 9:19pm

Wow! It works!

Many thanks for the quick replys and the tech support!

ReDaLeRt · September 19, 2021, 9:30am

One more thing:

I'm routing the IP range 213.13.24.0/24 through the tunnel, so that traffic starting from Openwer Site "B" could exit through Opnsense Site "A".

The traceroute from site "B" does not get to the destination:

tracert 213.13.24.12

Tracing route to 213.13.24.12 over a maximum of 30 hops

  1     1 ms    <1 ms    <1 ms  OpenWRT.lan [192.168.0.1]
  2    15 ms    16 ms    15 ms  10.0.0.1
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.

But from the Opnsense's Site "A" WG tunnel perspective, the traceroute completes perfectly:

traceroute -w 2 -n  -m '6' -s '10.0.0.1'   '213.13.24.12'
traceroute to 213.13.24.12 (213.13.24.12) from 10.0.0.1, 6 hops max, 40 byte packets
 1  * * ###########  6.484 ms
 2  213.13.24.12  1.793 ms
    213.13.24.13  1.593 ms
    213.13.24.12  1.277 ms

Am I missing something with the firewall rules?

vgaetera · September 19, 2021, 10:16pm

Make sure each destination IP/subnet routed via the tunnel is included in the allowed IPs.

ReDaLeRt · September 20, 2021, 7:08am

I'm sure it is on the Openwrt Site "B" ("origin"):

# ip route show table all
default via 192.168.1.254 dev eth0.2  src 192.168.1.100
10.0.0.0/24 dev WGPSA scope link  src 10.0.0.2
10.0.0.1 dev WGPSA scope link
******************* via 192.168.1.254 dev eth0.2
192.168.0.0/24 dev br-lan scope link  src 192.168.0.1
192.168.1.0/24 dev eth0.2 scope link  src 192.168.1.100
192.168.10.0/24 dev WGPSA scope link
192.168.11.0/24 dev WGPSA scope link
192.168.20.0/24 dev WGPSA scope link
213.13.24.0/24 dev WGPSA scope link
broadcast 10.0.0.0 dev WGPSA table local scope link  src 10.0.0.2
local 10.0.0.2 dev WGPSA table local scope host  src 10.0.0.2
broadcast 10.0.0.255 dev WGPSA table local scope link  src 10.0.0.2
broadcast 127.0.0.0 dev lo table local scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1
broadcast 192.168.0.0 dev br-lan table local scope link  src 192.168.0.1
local 192.168.0.1 dev br-lan table local scope host  src 192.168.0.1
broadcast 192.168.0.255 dev br-lan table local scope link  src 192.168.0.1
broadcast 192.168.1.0 dev eth0.2 table local scope link  src 192.168.1.100
local 192.168.1.100 dev eth0.2 table local scope host  src 192.168.1.100
broadcast 192.168.1.255 dev eth0.2 table local scope link  src 192.168.1.100
unreachable fd25:9b6f:f04b::/48 dev lo  metric 2147483647
fe80::/64 dev eth1  metric 256
fe80::/64 dev eth0  metric 256
fe80::/64 dev eth0.2  metric 256
fe80::/64 dev br-lan  metric 256
fe80::/64 dev wlan0  metric 256
fe80::/64 dev wlan1  metric 256
local ::1 dev lo table local  metric 0
anycast fe80:: dev eth1 table local  metric 0
anycast fe80:: dev eth0.2 table local  metric 0
anycast fe80:: dev eth0 table local  metric 0
anycast fe80:: dev br-lan table local  metric 0
anycast fe80:: dev wlan0 table local  metric 0
anycast fe80:: dev wlan1 table local  metric 0
local fe80::1691:82ff:fe7e:3254 dev eth0.2 table local  metric 0
local fe80::1691:82ff:fe7e:3254 dev br-lan table local  metric 0
local fe80::1691:82ff:fe83:c99 dev wlan0 table local  metric 0
local fe80::1691:82ff:fe83:c9a dev wlan1 table local  metric 0
local fe80::345c:4aff:fe1e:b84c dev eth1 table local  metric 0
local fe80::3c3e:bdff:fec1:cada dev eth0 table local  metric 0
multicast ff00::/8 dev eth1 table local  metric 256
multicast ff00::/8 dev eth0 table local  metric 256
multicast ff00::/8 dev eth0.2 table local  metric 256
multicast ff00::/8 dev br-lan table local  metric 256
multicast ff00::/8 dev wlan0 table local  metric 256
multicast ff00::/8 dev wlan1 table local  metric 256
multicast ff00::/8 dev WGPSA table local  metric 256

#  uci show network.wgclient.allowed_ips
network.wgclient.allowed_ips='10.0.0.1/32' '192.168.10.0/24' '192.168.11.0/24' '192.168.20.0/24' '213.13.24.0/24'

ReDaLeRt · September 20, 2021, 7:22am

The weird is the Openwrt itself can traceroute the destination correctly:

# traceroute 213.13.24.12
traceroute to 213.13.24.12 (213.13.24.12), 30 hops max, 38 byte packets
 1  10.0.0.1 (10.0.0.1)  13.198 ms  11.812 ms  12.551 ms
 2  *  *  *
 3  213.13.24.12 (213.13.24.12)  16.061 ms  213.13.24.13 (213.13.24.13)  14.694 ms  12.891 ms

But the its LAN hosts don't:

tracert 213.13.24.12
Tracing route to 213.13.24.12 over a maximum of 30 hops
  1     1 ms    <1 ms    <1 ms  OpenWRT.lan [192.168.0.1]
  2    16 ms    12 ms    13 ms  10.0.0.1
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.

ReDaLeRt · September 20, 2021, 12:08pm

# wg show; ip route get 213.13.24.12
interface: WGPSA
  public key: **********************************
  private key: (hidden)
  listening port: 63588

peer: ********************
  endpoint: ###############:63588
  allowed ips: 10.0.0.1/32, 192.168.10.0/24, 192.168.11.0/24, 213.13.24.0/24
  latest handshake: 33 seconds ago
  transfer: 465.50 KiB received, 5.20 MiB sent
  persistent keepalive: every 25 seconds
213.13.24.12 dev WGPSA src 10.0.0.2 uid 0
    cache

The route seems good, from the Openwrt Site "B" perspective. But its LAN hosts can't get to the 213.13.24.12.

ReDaLeRt · September 20, 2021, 12:55pm

From the Opnsense Site "A" site, viewing only ICMP packets (pings), arriving from the WG interface (wg0), from a Site "B" LAN host:

WG_S2S_P**
wg0	13:53:22.705259 IP 192.168.0.120 > 213.13.24.12: ICMP echo request, id 1, seq 18489, length 40
WG_S2S_P**
wg0	13:53:27.705480 IP 192.168.0.120 > 213.13.24.12: ICMP echo request, id 1, seq 18490, length 40
WG_S2S_P**
wg0	13:53:32.708285 IP 192.168.0.120 > 213.13.24.12: ICMP echo request, id 1, seq 18491, length 40
WG_S2S_P**
wg0	13:53:37.707324 IP 192.168.0.120 > 213.13.24.12: ICMP echo request, id 1, seq 18492, length 40

From that LAN Site "B" host:

Ping statistics for 213.13.24.12:
    Packets: Sent = 127, Received = 0, Lost = 127 (100% loss),
Control-C

vgaetera · September 20, 2021, 12:57pm

Try to ping from the LAN client and capture ICMP on the other side of the tunnel:

tcpdump -evni any icmp

ReDaLeRt · September 20, 2021, 1:07pm

Here it is: https://pastebin.com/bfdfBehK

ReDaLeRt · September 20, 2021, 1:19pm

I'm not sure about not being related to the openwrt. Because the openwrt can get to the 213.13.24.12 correctly. Only not its hosts:

This tcpdump was made while pinging 213.13.24.12 from the openwrt itself (CLI).

vgaetera · September 20, 2021, 1:30pm

Masqueraded packets are missing in the output.
You need to enable masquerading on OPNsense.
Or add a route to 192.168.0.0/24 on its gateway.