Port forwarding from WLAN across site-to-site VPN

brianm · April 10, 2023, 8:00pm

I have a site-to-site Wireguard VPN sort of working between a GL-X750V2 in an aircraft hangar / workshop and an Archer C7 at my home. I'm trying to fix a few issues.

Network diagram:

What works:

I can ping from myPC to home-router, hangar-router, and homeassistant
I can ping from home-router to hangar-router and homeassistant
HTTP connections from devices on the home network to homeassistant:8123 work fine
From homeassistant, I can ping hangar-router, home-router, myPC, and other devices on the home network
From myPC I can reach homeassistant via the port forward, myfakedomain.duckdns.org:8123

What doesn't work:

From hangar-router pings to anything on the home network fail (home-router, myPC, etc)
Using the dynamic DNS hostname and forwarded port from a device on the Internet fails

The goal is to be able to reach homeassistant from the Internet. I can't do dynamic DNS / port forwarding on hangar-router because it is a 4G connection with CG-NAT. The red line in the diagram illustrates what I want to happen, if possible.

Both routers are running OpenWrt 22.03.3.

home-router network config

root@router-home:~# uci show network
network.loopback=interface
network.loopback.device='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fdef:010e:48fc::/48'
network.@device[0]=device
network.@device[0].name='br-lan'
network.@device[0].type='bridge'
network.@device[0].ports='eth1.1'
network.lan=interface
network.lan.device='br-lan'
network.lan.proto='static'
network.lan.netmask='255.255.255.0'
network.lan.ip6assign='60'
network.lan.ipaddr='10.12.82.1'
network.wan=interface
network.wan.device='eth0.2'
network.wan.proto='dhcp'
network.wan6=interface
network.wan6.device='eth0.2'
network.wan6.proto='dhcpv6'
network.@switch[0]=switch
network.@switch[0].name='switch0'
network.@switch[0].reset='1'
network.@switch[0].enable_vlan='1'
network.@switch_vlan[0]=switch_vlan
network.@switch_vlan[0].device='switch0'
network.@switch_vlan[0].vlan='1'
network.@switch_vlan[0].ports='2 3 4 5 0t'
network.@switch_vlan[1]=switch_vlan
network.@switch_vlan[1].device='switch0'
network.@switch_vlan[1].vlan='2'
network.@switch_vlan[1].ports='1 6t'
network.vpn=interface
network.vpn.proto='wireguard'
network.vpn.listen_port='51820'
network.vpn.private_key='privateprivateprivate'
network.@device[1]=device
network.@device[1].name='vpn'
network.@wireguard_vpn[0]=wireguard_vpn
network.@wireguard_vpn[0].description='brian-pixel'
network.@wireguard_vpn[0].public_key='publicpublicpublic'
network.@wireguard_vpn[0].private_key='privateprivateprivate'
network.@wireguard_vpn[0].persistent_keepalive='25'
network.@wireguard_vpn[0].allowed_ips='10.12.84.10/32'
network.@wireguard_vpn[1]=wireguard_vpn
network.@wireguard_vpn[1].description='hangar'
network.@wireguard_vpn[1].public_key='publicpublicpublic='
network.@wireguard_vpn[1].private_key='privateprivateprivate'
network.@wireguard_vpn[1].persistent_keepalive='25'
network.@wireguard_vpn[1].route_allowed_ips='1'
network.@wireguard_vpn[1].allowed_ips='10.12.83.0/24'

home-router firewall config

root@router-home:~# uci show firewall
firewall.@defaults[0]=defaults
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='REJECT'
firewall.@defaults[0].synflood_protect='1'
firewall.@zone[0]=zone
firewall.@zone[0].name='wan'
firewall.@zone[0].input='REJECT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='REJECT'
firewall.@zone[0].masq='1'
firewall.@zone[0].mtu_fix='1'
firewall.@zone[0].network='wan' 'wan6'
firewall.@zone[1]=zone
firewall.@zone[1].name='lan'
firewall.@zone[1].input='ACCEPT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='ACCEPT'
firewall.@zone[1].network='lan' 'vpn'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='wan'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[2]=rule
firewall.@rule[2].name='Allow-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='ACCEPT'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-DHCPv6'
firewall.@rule[3].src='wan'
firewall.@rule[3].proto='udp'
firewall.@rule[3].dest_port='546'
firewall.@rule[3].family='ipv6'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-MLD'
firewall.@rule[4].src='wan'
firewall.@rule[4].proto='icmp'
firewall.@rule[4].src_ip='fe80::/10'
firewall.@rule[4].icmp_type='130/0' '131/0' '132/0' '143/0'
firewall.@rule[4].family='ipv6'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[5]=rule
firewall.@rule[5].name='Allow-ICMPv6-Input'
firewall.@rule[5].src='wan'
firewall.@rule[5].proto='icmp'
firewall.@rule[5].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type' 'router-solicitation' 'neighbour-solicitation' 'router-advertisement' 'neighbour-advertisement'
firewall.@rule[5].limit='1000/sec'
firewall.@rule[5].family='ipv6'
firewall.@rule[5].target='ACCEPT'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-ICMPv6-Forward'
firewall.@rule[6].src='wan'
firewall.@rule[6].dest='*'
firewall.@rule[6].proto='icmp'
firewall.@rule[6].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type'
firewall.@rule[6].limit='1000/sec'
firewall.@rule[6].family='ipv6'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-IPSec-ESP'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='lan'
firewall.@rule[7].proto='esp'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[8]=rule
firewall.@rule[8].name='Allow-ISAKMP'
firewall.@rule[8].src='wan'
firewall.@rule[8].dest='lan'
firewall.@rule[8].dest_port='500'
firewall.@rule[8].proto='udp'
firewall.@rule[8].target='ACCEPT'
firewall.@rule[9]=rule
firewall.@rule[9].name='wireguard'
firewall.@rule[9].proto='udp'
firewall.@rule[9].src='wan'
firewall.@rule[9].dest_port='51820'
firewall.@rule[9].target='ACCEPT'
firewall.@redirect[0]=redirect
firewall.@redirect[0].target='DNAT'
firewall.@redirect[0].name='homeassistant'
firewall.@redirect[0].src='wan'
firewall.@redirect[0].src_dport='8123'
firewall.@redirect[0].dest_port='8123'
firewall.@redirect[0].dest='lan'
firewall.@redirect[0].dest_ip='10.12.83.5'

hangar-router network config

root@router-hangar:~# uci show network
network.loopback=interface
network.loopback.device='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fd98:4268:ee13::/48'
network.@device[0]=device
network.@device[0].name='br-lan'
network.@device[0].type='bridge'
network.@device[0].ports='eth0'
network.lan=interface
network.lan.device='br-lan'
network.lan.proto='static'
network.lan.ip6assign='60'
network.lan.ipaddr='10.12.83.1/24'
network.wan=interface
network.wan.device='eth1'
network.wan.proto='dhcp'
network.wan.metric='1024'
network.wan6=interface
network.wan6.device='eth1'
network.wan6.proto='dhcpv6'
network.4g=interface
network.4g.proto='qmi'
network.4g.device='/dev/cdc-wdm0'
network.4g.apn='h2g2'
network.4g.auth='none'
network.4g.pdptype='ipv4'
network.wg0=interface
network.wg0.proto='wireguard'
network.wg0.private_key='privateprivateprivate'
network.@wireguard_wg0[0]=wireguard_wg0
network.@wireguard_wg0[0].description='vpnserver'
network.@wireguard_wg0[0].public_key='publicpublicpublic'
network.@wireguard_wg0[0].endpoint_host='myfakedomain.duckdns.org'
network.@wireguard_wg0[0].endpoint_port='51820'
network.@wireguard_wg0[0].persistent_keepalive='25'
network.@wireguard_wg0[0].route_allowed_ips='1'
network.@wireguard_wg0[0].allowed_ips='10.12.82.0/24'

hangar-router firewall config

root@router-hangar:~# uci show firewall
firewall.@defaults[0]=defaults
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='REJECT'
firewall.@defaults[0].synflood_protect='1'
firewall.@zone[0]=zone
firewall.@zone[0].name='lan'
firewall.@zone[0].input='ACCEPT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='ACCEPT'
firewall.@zone[0].network='lan' 'wg0'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].input='REJECT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='REJECT'
firewall.@zone[1].masq='1'
firewall.@zone[1].mtu_fix='1'
firewall.@zone[1].network='wan' 'wan6' '4g'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='wan'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[2]=rule
firewall.@rule[2].name='Allow-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='ACCEPT'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-DHCPv6'
firewall.@rule[3].src='wan'
firewall.@rule[3].proto='udp'
firewall.@rule[3].dest_port='546'
firewall.@rule[3].family='ipv6'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-MLD'
firewall.@rule[4].src='wan'
firewall.@rule[4].proto='icmp'
firewall.@rule[4].src_ip='fe80::/10'
firewall.@rule[4].icmp_type='130/0' '131/0' '132/0' '143/0'
firewall.@rule[4].family='ipv6'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[5]=rule
firewall.@rule[5].name='Allow-ICMPv6-Input'
firewall.@rule[5].src='wan'
firewall.@rule[5].proto='icmp'
firewall.@rule[5].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type' 'router-solicitation' 'neighbour-solicitation' 'router-advertisement' 'neighbour-advertisement'
firewall.@rule[5].limit='1000/sec'
firewall.@rule[5].family='ipv6'
firewall.@rule[5].target='ACCEPT'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-ICMPv6-Forward'
firewall.@rule[6].src='wan'
firewall.@rule[6].dest='*'
firewall.@rule[6].proto='icmp'
firewall.@rule[6].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type'
firewall.@rule[6].limit='1000/sec'
firewall.@rule[6].family='ipv6'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-IPSec-ESP'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='lan'
firewall.@rule[7].proto='esp'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[8]=rule
firewall.@rule[8].name='Allow-ISAKMP'
firewall.@rule[8].src='wan'
firewall.@rule[8].dest='lan'
firewall.@rule[8].dest_port='500'
firewall.@rule[8].proto='udp'
firewall.@rule[8].target='ACCEPT'

brianm · April 10, 2023, 8:07pm

I also tried tcpdump to capture any traffic.

A capture on home-router, making an HTTP request from my phone (disconnected from wifi) to myfakedomain.duckdns.org:8123:

root@router-home:~# tcpdump -i any port 8123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
20:02:05.988075 ethertype IPv4, IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514654392 ecr 0,nop,wscale 9], length 0
20:02:05.988075 IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514654392 ecr 0,nop,wscale 9], length 0
20:02:05.988245 IP my-phone-ip.3580 > homeassistant.lan.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514654392 ecr 0,nop,wscale 9], length 0
20:02:05.995124 ethertype IPv4, IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514654393 ecr 0,nop,wscale 9], length 0
20:02:05.995124 IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514654393 ecr 0,nop,wscale 9], length 0
20:02:05.995282 IP my-phone-ip.3582 > homeassistant.lan.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514654393 ecr 0,nop,wscale 9], length 0
20:02:06.118150 ethertype IPv4, IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514654528 ecr 0,nop,wscale 9], length 0
20:02:06.118150 IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514654528 ecr 0,nop,wscale 9], length 0
20:02:13.255164 ethertype IPv4, IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514661603 ecr 0,nop,wscale 9], length 0
20:02:13.255164 IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514661603 ecr 0,nop,wscale 9], length 0
20:02:13.255299 IP my-phone-ip.3582 > homeassistant.lan.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514661603 ecr 0,nop,wscale 9], length 0
20:02:13.255397 ethertype IPv4, IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514661604 ecr 0,nop,wscale 9], length 0
20:02:13.255397 IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514661604 ecr 0,nop,wscale 9], length 0
20:02:13.255488 IP my-phone-ip.3580 > homeassistant.lan.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514661604 ecr 0,nop,wscale 9], length 0
20:02:13.255516 ethertype IPv4, IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514661603 ecr 0,nop,wscale 9], length 0
20:02:13.255516 IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514661603 ecr 0,nop,wscale 9], length 0
20:02:21.404501 ethertype IPv4, IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
20:02:21.404501 IP my-phone-ip.3570 > wlan-public-address.comcast.net.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
20:02:21.404636 IP my-phone-ip.3570 > homeassistant.lan.8123: Flags [S], seq 1428396498, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
20:02:21.405479 ethertype IPv4, IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514669796 ecr 0,nop,wscale 9], length 0
20:02:21.405479 IP my-phone-ip.3582 > wlan-public-address.comcast.net.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514669796 ecr 0,nop,wscale 9], length 0
20:02:21.405610 IP my-phone-ip.3582 > homeassistant.lan.8123: Flags [S], seq 641812270, win 65535, options [mss 1318,sackOK,TS val 3514669796 ecr 0,nop,wscale 9], length 0
20:02:21.405678 ethertype IPv4, IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
20:02:21.405678 IP my-phone-ip.3580 > wlan-public-address.comcast.net.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
20:02:21.405757 IP my-phone-ip.3580 > homeassistant.lan.8123: Flags [S], seq 1736039414, win 65535, options [mss 1318,sackOK,TS val 3514669795 ecr 0,nop,wscale 9], length 0
^C
25 packets captured
45 packets received by filter
20 packets dropped by kernel

A capture made at the same time on hangar-router:

root@router-hangar:~# tcpdump -i any port 8123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes

^C
0 packets captured
2 packets received by filter
0 packets dropped by kernel

brianm · April 11, 2023, 12:01am

I messed around a bit further and got it more-or-less working. I had the vpn interface in the lan zone on home-router. I moved it to it's own zone, then configured the new vpn zone to forward to lan and wan, accept/accept/accept and masquerading enabled. I also updated the port forwarding to forward from wan port 8123 to the vpn zone, 10.12.83.5:8123. Now from the internet I can access the homeassistant machine on the hangar network via home network and vpn.

To access homeassistant from inside the network I added a DNS entry for the dynamic DNS hostname that points to the internal IP on both routers.

The only thing I can't do is ping into the 10.12.82.0/24 network from 10.12.83.1. Other machines in 10.12.83.0/24 can do it. But I can live with that for now.