Port forward/NATing with OpenVPN bridge tap0 used as WAN

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi,

The best internet connection I can have in my location (B) is based on 5G cell network. Unfortunately, the ISP does not provide a public IP.
So I want to use the public IP from another location (A) to reach my network & devices at location (B).

For this, I have configured an OpenWrt device at location B (Netgear EX7300v2) to run an OpenVPN client. It creates a bridge (TAP) with Location A's OpenVPN bridge server. Note that I have no control over the OpenVPN server configuration as it's part of Location A's ISP-provided router.

The WiFi radio on the OpenWrt device is disabled, so it's a 1-ethernet-port device connected directly to Location B's ISP 5G router via ethernet. It has 2 interfaces configured:

  • eth0: static IP 192.168.4.2 in the LAN firewall zone
  • tap0: static IP 192.168.1.100 in the WAN firewall zone

Here is a quick diagram:

ChaletMougly.drawio
ChaletMougly.drawio1986×1497 225 KB

From a connectivity standpoint, everything seems to work fine:

  • the OpenWrt device can be reached from devices in both Locations
  • the OpenWrt device can reach devices in both Locations
  • the OpenWrt device can reach the internet (google.com) using either interface
pings 
root@EX7300v2-OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.4.1     0.0.0.0         UG    10     0        0 eth0
default         192.168.1.254   0.0.0.0         UG    20     0        0 tap0
192.168.1.0     *               255.255.255.0   U     20     0        0 tap0
192.168.4.0     *               255.255.255.0   U     10     0        0 eth0
root@EX7300v2-OpenWrt:~#
root@EX7300v2-OpenWrt:~# ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1): 56 data bytes
64 bytes from 192.168.4.1: seq=0 ttl=64 time=0.804 ms
64 bytes from 192.168.4.1: seq=1 ttl=64 time=0.696 ms
^C
--- 192.168.4.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.696/0.750/0.804 ms
root@EX7300v2-OpenWrt:~# ping 192.168.1.254
PING 192.168.1.254 (192.168.1.254): 56 data bytes
64 bytes from 192.168.1.254: seq=0 ttl=64 time=68.947 ms
64 bytes from 192.168.1.254: seq=1 ttl=64 time=76.258 ms
^C
--- 192.168.1.254 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 68.947/72.602/76.258 ms
root@EX7300v2-OpenWrt:~# ping 192.168.1.40
PING 192.168.1.40 (192.168.1.40): 56 data bytes
64 bytes from 192.168.1.40: seq=0 ttl=64 time=81.982 ms
64 bytes from 192.168.1.40: seq=1 ttl=64 time=41.417 ms
^C
--- 192.168.1.40 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 41.417/61.699/81.982 ms
root@EX7300v2-OpenWrt:~# ping google.com
PING google.com (142.250.179.110): 56 data bytes
64 bytes from 142.250.179.110: seq=0 ttl=112 time=34.609 ms
64 bytes from 142.250.179.110: seq=1 ttl=112 time=31.884 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 31.884/33.246/34.609 ms
root@EX7300v2-OpenWrt:~# ping -I eth0 google.com
PING google.com (172.217.20.174): 56 data bytes
64 bytes from 172.217.20.174: seq=0 ttl=112 time=33.134 ms
64 bytes from 172.217.20.174: seq=1 ttl=112 time=32.614 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 32.614/32.874/33.134 ms
root@EX7300v2-OpenWrt:~# ping -I tap0 google.com
PING google.com (142.250.179.110): 56 data bytes
64 bytes from 142.250.179.110: seq=0 ttl=117 time=70.606 ms
64 bytes from 142.250.179.110: seq=1 ttl=117 time=77.091 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 70.606/73.848/77.091 ms

On location A's ISP router, port 48041 is opened and forwarded to 192.168.1.100.
On the OpenWrt device, the same port is opened and forwarded to 192.168.4.1 on port 80 (Location B's ISP router web interface).

Problem: connecting to http://IPofLocationA:48041 doesn't work (even when using an internet connection different from Location A or B).

From adding some logging in the firewall, I can see that the rule I put in is triggered.

root@EX7300v2-OpenWrt:~# logread -e "Gateway management"
Mon Jan 27 11:09:15 2025 kern.warn kernel: [29865.652792] Gateway management: IN=tap0 OUT= MAC=52:5d:68:29:63:c7:14:0c:76:b0:5c:bc:08:00 SRC=37.170.13.66 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x20 TTL=56 ID=50453 DF PROTO=TCP SPT=40457 DPT=48041 WINDOW=64240 RES=0x00 SYN URGP=0

But I am not sure how to troubleshoot further.
Am I missing a traffic rule or anything else?

Thanks for the help :slight_smile:

configs 
root@EX7300v2-OpenWrt:~# uci export network
package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd3c:6ced:effd::/48'

config interface 'lan1'
        option proto 'static'
        option device 'eth0'
        option ipaddr '192.168.4.2'
        option netmask '255.255.255.0'
        option gateway '192.168.4.1'
        option broadcast '192.168.2.255'
        option delegate '0'
        option ip6assign '60'
        list dns '192.168.4.1'
        option metric '10'

config device
        option name 'eth0'
        option ipv6 '1'

config interface 'bridgedVPN'
        option proto 'static'
        option device 'tap0'
        option ipaddr '192.168.1.100'
        option netmask '255.255.255.0'
        option gateway '192.168.1.254'
        list dns '192.168.1.254'
        option metric '20'
        option ip6assign '60'

config device
        option name 'tap0'
        option macaddr '52:5D:68:29:63:C7'
        option acceptlocal '0'

root@EX7300v2-OpenWrt:~# uci export firewall
package firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option log '1'
        list network 'lan1'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option mtu_fix '1'
        option log '1'
        list network 'bridgedVPN'
        option masq '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'Gateway management'
        option src 'wan'
        option src_dport '48041'
        option dest_ip '192.168.4.1'
        option dest_port '80'
        option log '1'
        list proto 'tcp'

root@EX7300v2-OpenWrt:~# cat /etc/openvpn/bridged.conf
client
remote <Location A public IP> <port>
proto udp
nobind

dev-type tap
dev tap0

log /var/log/bridged.log

pull

auth-user-pass <auth file>

fragment 1452
mssfix 1452

explicit-exit-notify 3

data-ciphers-fallback AES-256-CBC
remote-cert-tls server
verify-x509-name "C=FR, O=XXXXX SA, CN=XXXX OpenVPN server XXXX"
<ca>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</cert>
<extra-certs>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</extra-certs>
<key>
-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----
</key>
2

On router B you have to set a static route for return traffic of 192.168.1.0/24 via 192.168.4.2

3

Thanks for the quick suggestion, it helped but it still doesn't work all the way.

Adding the static route allowed me to reach Location B's ISP router web UI from a device on Location A's network at http://192.168.1.100:48041.

However, it still won't load when trying from a device connected outside of both networks (i.e. the WAN side of Location A).

4

Router B returns replies to requests originating from public IPs via its own default gateway, instead of sending them back to the OpenWrt device (and the tunnel accordingly).

You need to SNAT the requests to the OpenWrt lan IP.

uci add firewall nat
uci set firewall.@nat[-1].name='SNAT-to-RouterB'
uci set firewall.@nat[-1].src='lan'
uci set firewall.@nat[-1].target='SNAT'
uci set firewall.@nat[-1].dest_ip='192.168.4.1'
uci set firewall.@nat[-1].snat_ip='192.168.4.2'
uci set firewall.@nat[-1].proto='tcp'
uci set firewall.@nat[-1].dest_port='80'
uci commit firewall
/etc/init.d/firewall restart
1 Like
5

Thanks again for this new suggestion.
I just tried it and unfortunately, it doesn't make it work.
I do see the 2 log prints of the Port Forwarding & SNAT rules:

root@EX7300v2-OpenWrt:~# logread -e SRC=172.58.160.164
Tue Jan 28 14:57:47 2025 kern.warn kernel: [43576.335896] Gateway management: IN=tap0 OUT= MAC=52:5d:68:29:63:c7:14:0c:76:34:51:8d:08:00 SRC=172.58.160.164 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=46 ID=13028 DF PROTO=TCP SPT=18496 DPT=48041 WINDOW=65535 RES=0x00 SYN URGP=0
Tue Jan 28 14:57:47 2025 kern.warn kernel: [43576.357685] SNAT to Nokia 5G GW: IN=tap0 OUT=eth0 MAC=52:5d:68:29:63:c7:14:0c:76:34:51:8d:08:00 SRC=172.58.160.164 DST=192.168.4.1 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=13028 DF PROTO=TCP SPT=18496 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
root@EX7300v2-OpenWrt:~#
6

Please add option reflection '0' to this rule, restart the firewall service and test again.

If it still doesn't work, post the output of

for chain in dstnat_wan dstnat_lan srcnat_lan; do nft list chain inet fw4 $chain; done
7

Still no luck :frowning:

Here is the outpout you requested:

root@EX7300v2-OpenWrt:~# for chain in dstnat_wan dstnat_lan srcnat_lan; do nft list chain inet fw4 $chain; done
table inet fw4 {
        chain dstnat_wan {
                meta nfproto ipv4 tcp dport 48041 counter packets 3 bytes 180 log prefix "Gateway management: " dnat ip to 192.168.4.1:80 comment "!fw4: Gateway management"
        }
}
table inet fw4 {
        chain dstnat_lan {
        }
}
table inet fw4 {
        chain srcnat_lan {
                ip daddr 192.168.4.1 tcp dport 80 counter packets 3 bytes 180 log prefix "SNAT to Nokia 5G GW: " snat ip to 192.168.4.2 comment "!fw4: SNAT to Nokia 5G GW"
        }
}
8

Having Site B's VPN terminus being a lan appendage instead of the site's main router makes this a bit complicated. I would suggest having site B's DHCP advertise the OpenWrt router instead of the ISP router as the default route. This allows simple symmetric routing between the two LANs. Also install conditional routing into the OpenWrt router:

  • Anything to site A's LAN via tunnel (this is inherent since the tunnel interface already holds a site A LAN IP)
  • Anything from a site B server to the Internet via tunnel (so Internet clients tunneled from site A will be properly answered)
  • Other uses of the Internet from site B LAN via the ISP router
  • OpenVPN encrypted packets to site A via ISP router (could make all internal Internet from OpenWrt box B to ISP router.)
9

The rules look correct, but the counters don't show any hits, which is really strange.

For more in-depth debugging you should install tcpdump (hopefully you have enough free space).

tcpdump -nnvvi any port 48041 or port 80 -c 10

Make sure you are not connected to the router via http at the same time to avoid setting an additional filter.

Redact the public IPs.

10

Sorry I had reset the counters when I ran the command. I edited my reply with an updated output. They do get hit everytime I run a test.

11

Yeah, I know but I really don't have a capable OpenWrt router available for that.
The DHCP server for Location B is already running on the OpenWrt device and pushes route 192.168.1.0/24 via 192.168.4.2 (option121).

12

Then you should definitely run a tcpdump session to see what's going on.

1 Like
13 
root@EX7300v2-OpenWrt:~# tcpdump -nnvvi any port 48041 or port 80 -c 10
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:41:31.349175 tap0  In  IP (tos 0x0, ttl 46, id 49856, offset 0, flags [DF], proto TCP (6), length 60)
    172.58.160.164.32884 > 192.168.1.100.48041: Flags [S], cksum 0x39ad (correct), seq 1936756195, win 65535, options [mss 1324,sackOK,TS val 2200715087 ecr 0,nop,wscale 9], length 0
18:41:31.392781 eth0  Out IP (tos 0x0, ttl 45, id 49856, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.4.2.32884 > 192.168.4.1.80: Flags [S], cksum 0x7a9e (correct), seq 1936756195, win 65535, options [mss 1324,sackOK,TS val 2200715087 ecr 0,nop,wscale 9], length 0
18:41:31.394423 eth0  In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.4.1.80 > 192.168.4.2.32884: Flags [S.], cksum 0x073c (correct), seq 1271532907, ack 1936756196, win 65160, options [mss 1460,sackOK,TS val 1137366850 ecr 2200715087,nop,wscale 7], length 0
18:41:31.394519 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.100.48041 > 172.58.160.164.32884: Flags [S.], cksum 0xc64a (correct), seq 1271532907, ack 1936756196, win 65160, options [mss 1460,sackOK,TS val 1137366850 ecr 2200715087,nop,wscale 7], length 0
18:41:31.396125 tap0  In  IP (tos 0x0, ttl 46, id 54026, offset 0, flags [DF], proto TCP (6), length 60)
    172.58.160.164.55922 > 192.168.1.100.48041: Flags [S], cksum 0xe1e0 (correct), seq 2715376970, win 65535, options [mss 1324,sackOK,TS val 2200715085 ecr 0,nop,wscale 9], length 0
18:41:31.439741 eth0  Out IP (tos 0x0, ttl 45, id 54026, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.4.2.55922 > 192.168.4.1.80: Flags [S], cksum 0x22d2 (correct), seq 2715376970, win 65535, options [mss 1324,sackOK,TS val 2200715085 ecr 0,nop,wscale 9], length 0
18:41:31.441304 eth0  In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.4.1.80 > 192.168.4.2.55922: Flags [S.], cksum 0xea24 (correct), seq 3077334756, ack 2715376971, win 65160, options [mss 1460,sackOK,TS val 1137366897 ecr 2200715085,nop,wscale 7], length 0
18:41:31.441408 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.100.48041 > 172.58.160.164.55922: Flags [S.], cksum 0xa933 (correct), seq 3077334756, ack 2715376971, win 65160, options [mss 1460,sackOK,TS val 1137366897 ecr 2200715085,nop,wscale 7], length 0
18:41:31.473946 tap0  In  IP (tos 0x0, ttl 46, id 15551, offset 0, flags [DF], proto TCP (6), length 60)
    172.58.160.164.38575 > 192.168.1.100.48041: Flags [S], cksum 0xf0e0 (correct), seq 1862911075, win 65535, options [mss 1324,sackOK,TS val 2200715207 ecr 0,nop,wscale 9], length 0
18:41:31.517548 eth0  Out IP (tos 0x0, ttl 45, id 15551, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.4.2.38575 > 192.168.4.1.80: Flags [S], cksum 0x31d2 (correct), seq 1862911075, win 65535, options [mss 1324,sackOK,TS val 2200715207 ecr 0,nop,wscale 9], length 0
10 packets captured
14 packets received by filter
0 packets dropped by kernel
14

So far the dialogue looks by the book.
Then surprisingly a new connection from a different port is initiated.

The reason might be that the MSS value in the SYN and SYN/ACK packets differs (1324 vs. 1460), but I'm not sure.

I'll need some time to investigate the problem.

@trendy , if you've seen this before, please help.

15

A closer look with a clear head shows that the last packet leaves eth0 instead of tap0.

That's because the OpenWrt device uses 192.168.4.1 as a default gateway with a higher priority, so a kind of loop is created.

The easy option would be to remove the gateway option on lan1 and only set a static route for RouterA via RouterB (not applicable if RouterA's public IP is not static).

Otherwise you should try playing with fmark. Run these commands to see if it makes a difference:

nft add rule inet fw4 mangle_prerouting ip saddr 192.168.4.1 tcp sport 80 counter meta mark set 0x1
ip rule add fwmark 0x1 table 100 prio 1
ip route add default via 192.168.1.254 table 100
1 Like
16

Unfortunately, this Nokia 5G router is a piece of :face_with_symbols_over_mouth: and does not provide any static routing option in its interface.

17

Yeap that worked! Many thanks!

Would you mind telling me what these commands do?

18

The changes need to be made on OpenWrt and I will try to explain why.

For proper communication, the response must be returned from the point to which the request was sent, or in your case

Request:   Initiator(Internet)->RouterA->VPN->OpenWrt->RouterB
Response:  RouterB->OpenWrt->VPN->RouterA->Initiator

What was actually happening initially.

RouterB receives a request from a public IP. For the reply it looks at its routing table, the network is unknown, so it sends the response to its default gateway. Thus, the initiator receives the response directly from RouterB (from a different IP address) and the connection is dropped.

To avoid this, we created the SNAT rule. This way RouterB sees the request coming from OpenWrt (192.168.4.2), so the response is sent back there. Here comes the unexpected second problem. OpenWrt needs to return the response to a public IP, it looks at its routing table, the network is unknown, so it sends the packet to its default gateway, which is RouterB. Finally, after a bit of ping-pong, the connection is dropped again.

Set OpenWrt to use RouterA as the default gateway, and create a static route so that OpenWrt can establish the VPN tunnel via RouterB.

#/etc/config/network


config interface 'lan1'
        option proto 'static'
        option device 'eth0'
        option ipaddr '192.168.4.2'
        option netmask '255.255.255.0'
        list dns '192.168.4.1'
        
config interface 'bridgedVPN'
        option proto 'static'
        option device 'tap0'
        option ipaddr '192.168.1.100'
        option netmask '255.255.255.0'
        option gateway '192.168.1.254'
	    list dns '192.168.1.254'

config route
        option target 'A.B.C.D/32' # RouterA fixed public IP 
        option gateway '192.168.4.1'
        option interface 'lan1'

This more complicated option is based on the fact that RouterB replies to this type of requests only from source port 80.

  • The first command creates a firewall rule that marks traffic entering the router from source IP 192.168.4.1 and source port 80.
  • The second creates an ip rule that causes marked traffic to use custom routing table 100.
  • The third one creates a custom routing table 100, where RouterA is the default gateway.

This way, that specific marked traffic is sent back to RouterA and the initiator respectively.

These commands are not persistent and were for testing purposes only.
If you prefer to use this option, they should be translated to uci.

Keep in mind that exposing the device's web management interface to the Internet is a bad idea and using some random port does not bring any additional security.

1 Like
19

Again, many thanks for all of these details, very clear now.
I was using Router B's management interface as an example/test. I am not planning to keep exposing it going forward.

20

Maybe the problem can be traced back to the fact that the OpenVPN TAP interface is not doing SNAT if it should have done then all traffic from A originated from 192.168.1.254 and a simple static route on router B should do the trick?

But anyway glad it is working :slight_smile: