I've been working on a script to generate WireGuard configurations for a symmetric site-to-site VPN (https://openwrt.org/docs/guide-user/services/vpn/wireguard/site-to-site). While doing so, I tripped over an oddity while testing the resulting VPN.
I deleted the previous VPN and regenerated it, then while testing connectivity through the WireGuard VPN I found that the source IP address used when originating packets going out through the remote site through the new VPN tunnel is the WAN IP address of that system, rather than a LAN IP address. But, after a reboot with no configuration changes, the source IP address uses the LAN address.
How could I diagnose what's going on to see why the source IP is the WAN address?
The wireguard interface has no IP address assigned to it.
Example:
site A: LAN is 192.168.1.1/24, WAN (assigned by ISP) is 3.3.3.3/24
site B: LAN is 192.168.0.1/24, WAN (assigned by ISP) is 4.4.4.4/24
(see network configs below)
Site A's IPv4 network routes (netstat -n -r) (some addresses redacted to 4.4.4.x/3.3.3.x)
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 3.3.3.1 0.0.0.0 UG 0 0 0 eth0
4.4.4.4 3.3.3.1 255.255.255.255 UGH 0 0 0 eth0
3.3.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
172.29.34.0 0.0.0.0 255.255.255.0 U 0 0 0 br-iot
172.30.11.0 0.0.0.0 255.255.255.0 U 0 0 0 wg_vpn
172.30.11.2 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.3 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.4 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.5 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.6 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.7 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.8 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.9 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.30.11.10 0.0.0.0 255.255.255.255 UH 0 0 0 wg_vpn
172.31.66.0 0.0.0.0 255.255.255.0 U 0 0 0 br-guest
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 wg_s2s_b
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br-lan
192.168.6.0 0.0.0.0 255.255.255.0 U 0 0 0 br-local6
209.51.161.14 3.3.3.1 255.255.255.255 UGH 0 0 0 eth0
- wg_s2s_b is the WireGuard interface for the site-to-site config
- wg_vpn is a different WireGuard interface for road-warrior connections
- dhcpwan6 interface does not get an IP address from the ISP. (I've got it there so that if some day they do start supporting IPv6, I'll be ready to go.)
- 209.51.161.14 is a hurricane electric IPv6 tunnel server I'm using for wan6 (6in4)
When running a shell on this openwrt system, pings to 192.168.0.1 get no response. tcpdump on the tunnel shows:
root@LEDE:~# tcpdump -s0 -n -i wg_s2s_b &
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg_s2s_b, link-type RAW (Raw IP), capture size 262144 bytes
root@LEDE:~# ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
16:38:52.331049 IP 3.3.3.3 > 192.168.0.1: ICMP echo request, id 12031, seq 0, length 64
16:38:53.332238 IP 3.3.3.3 > 192.168.0.1: ICMP echo request, id 12031, seq 1, length 64
16:38:54.332820 IP 3.3.3.3 > 192.168.0.1: ICMP echo request, id 12031, seq 2, length 64
^C
--- 192.168.0.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Other systems on the 192.168.1.0/24 network can ping successfully through the tunnel:
16:49:54.862991 IP 192.168.1.81 > 192.168.0.1: ICMP echo request, id 44441, seq 0, length 64
16:49:54.917596 IP 192.168.0.1 > 192.168.1.81: ICMP echo reply, id 44441, seq 0, length 64
16:49:55.863688 IP 192.168.1.81 > 192.168.0.1: ICMP echo request, id 44441, seq 1, length 64
16:49:55.911608 IP 192.168.0.1 > 192.168.1.81: ICMP echo reply, id 44441, seq 1, length 64
16:49:56.864188 IP 192.168.1.81 > 192.168.0.1: ICMP echo request, id 44441, seq 2, length 64
16:49:56.913762 IP 192.168.0.1 > 192.168.1.81: ICMP echo reply, id 44441, seq 2, length 64
But if I reboot the openwrt system, then log in again and run ping locally, the system chooses 192.168.1.1 when tunneling to 192.168.0.0/24. The output of netstat -n -r after a reboot is identical.
Any ideas why the source IP address being chosen is the WAN address before reboot, but the LAN address after reboot?
/etc/config/network (redacted):
config interface 'loopback'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option device 'lo'
config globals 'globals'
option ula_prefix 'fdff:ffff:ffff::/48'
config interface 'lan'
option proto 'static'
option netmask '255.255.255.0'
option ipaddr '192.168.1.1'
option stp '1'
option device 'br-lan'
option delegate '0'
config interface 'wan'
option proto 'dhcp'
option device 'eth0'
option ipv6 '1'
config interface 'wan6'
option proto '6in4'
option peeraddr '209.51.161.14'
option ip6addr '2001:470:xxxx:xxx::2/64'
list ip6prefix '2001:470:xxxx::/48'
option tunnelid 'xxxxxx'
option username 'xxxx
option password 'xxxx'
option mtu '1480'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option description 'GUEST'
option vid '5'
option ports '1t 3t 6t'
config switch_vlan
option device 'switch0'
option vlan '2'
option vid '2'
option description 'WAN'
option ports '0 5'
config interface 'guest'
option proto 'static'
option netmask '255.255.255.0'
option ipaddr '172.31.66.1'
option device 'br-guest'
option delegate '0'
option ip6assign '64'
option ip6hint '2'
config interface 'iot'
option proto 'static'
option ipaddr '172.29.34.1'
option netmask '255.255.255.0'
option device 'br-iot'
option delegate '0'
config switch_vlan
option device 'switch0'
option vlan '3'
option vid '3'
option description 'IOT'
option ports '1t 3t 6t'
config switch_vlan
option device 'switch0'
option vlan '4'
option vid '4'
option description 'NEWLAN'
option ports '1t 2 3t 4 6t'
config device
list ports 'eth1.4'
option type 'bridge'
option name 'br-lan'
option macaddr 'xx:xx:xx:xx:xx:xx'
config device
option type 'bridge'
option name 'br-guest'
list ports 'eth1.5'
config device
list ports 'eth1.3'
option type 'bridge'
option name 'br-iot'
config switch_vlan
option device 'switch0'
option vlan '5'
option vid '1'
option description 'DONOTUSE'
config interface 'local6'
option device 'br-local6'
option proto 'static'
option ip6assign '64'
option ipaddr '192.168.6.1'
option netmask '255.255.255.0'
option ip6hint '1'
config device
option type 'bridge'
option name 'br-local6'
option bridge_empty '1'
list ports 'eth1.6'
config switch_vlan
option device 'switch0'
option vlan '6'
option vid '6'
option description 'IPV6'
option ports '1t 3t 6t'
config device
option type 'bridge'
option name 'br-only6'
option bridge_empty '1'
list ports 'eth1.7'
config switch_vlan
option device 'switch0'
option vlan '7'
option vid '7'
option description 'ONLY6'
option ports '1t 3t 6t'
config interface 'only6'
option proto 'static'
option device 'br-only6'
option ip6assign '64'
option ip6hint '6'
config interface 'wg_vpn'
option proto 'wireguard'
option private_key 'xxx'
option listen_port '51820'
list addresses '172.30.11.1/24'
list addresses '2001:470:xxxx:11::1/64'
option mtu '1420'
config wireguard_wg_vpn
option public_key 'xxxx'
option preshared_key 'xxxx'
option description 'xxxx'
list allowed_ips '172.30.11.2/32'
list allowed_ips '2001:470:xxxx:11::2/128'
option route_allowed_ips '1'
option persistent_keepalive '25'
< more wg_vpn road warriors elided >
config interface 'dhcpwan6'
option proto 'dhcpv6'
option reqaddress 'try'
option ip6assign '64'
option ip6hint 'e'
option device 'eth0'
option reqprefix '56'
config interface 'wg_s2s_b'
option proto 'wireguard'
option private_key 'xxxx'
option listen_port '51822'
config wireguard_wg_s2s_b 's2s_vpn_site_a'
option public_key 'xxxx'
option preshared_key 'xxxx'
option description 'Site A, xxxx.dynamic-dns.net'
list allowed_ips '192.168.0.0/24'
list allowed_ips 'fdee:eeee:eeee::/48'
option route_allowed_ips '1'
option persistent_keepalive '25'
option endpoint_host 'xxxx.dynamic-dns.net'
option endpoint_port '51822'
ubus call system board info:
{
"kernel": "5.4.154",
"hostname": "LEDE",
"system": "ARMv7 Processor rev 0 (v7l)",
"model": "Netgear Nighthawk X4S R7800",
"board_name": "netgear,r7800",
"release": {
"distribution": "OpenWrt",
"version": "21.02.1",
"revision": "r16325-88151b8303",
"target": "ipq806x/generic",
"description": "OpenWrt 21.02.1 r16325-88151b8303"
}
}