PPPoE+Wireguard+PBR Not possible forward port to lan client

Hello,
I tried to forward port to lan client through my VPS with wireguard server but it's impossible.
I did a test with wwan connected to my phone in hotspot and it work flawless.

schematic:
public address VPS 1.1.1.1:8081 ---> NAT to VPN WG 10.6.6.9 (ip assigned for openwrt) ---> assigned port 8081 with PBR to lan client ---> openwrt port forward.

Same configuration wwan it work, not with PPPoE wan.

Network config Openwrt router

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd37:98b4:9247::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wan'
	option proto 'pppoe'
	option username 'timadsl'
	option password 'timadsl'
	option device 'eth1.835'
	option ipv6 '0'
	option force_link '1'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option vid '1'
	option ports '0t 0t 1 2 3 4'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option vid '835'
	option ports '0t 0t 5t'

config interface 'wg0'
	option proto 'wireguard'
	list addresses '10.66.66.9/32'
	list addresses 'fd42:42:42::9/128'
	option force_link '1'
	option private_key 'xxxxxx'
	list dns '8.8.8.8'

config wireguard_wg0
	option endpoint_host '89.40.143.251'
	option description 'wg0'
	option persistent_keepalive '25'
	list allowed_ips '0.0.0.0/0'
	option public_key 'xxxxx'
	option preshared_key 'xxxxxxx'
	option endpoint_port '51568'
	option route_allowed_ips '0'

config interface 'wwan'
	option proto 'dhcp'
	option auto '0'

Firewall Openwrt


config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option family 'ipv4'
	list network 'wan'
	list network 'wwan'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'wg0'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	option forward 'REJECT'
	list network 'wg0'
	option input 'REJECT'

config forwarding
	option src 'lan'
	option dest 'wg0'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

config redirect
	option target 'DNAT'
	option family 'ipv4'
	list proto 'tcp'
	option src 'wg0'
	option src_dport '8081'
	option dest_ip '192.168.1.10'
	option dest_port '8081'
	option dest 'lan'

VPS iptables rule

-A PREROUTING -d vps-public-ip -i eth0 -p tcp -m tcp --dport 8081 -j DNAT --to-destination 10.66.66.9:8081

It looks like a routing issue and you haven't posted the pbr configuration.

For testing purposes, try SNAT-ing the requests to the VPS wireguard server IP address (assuming it's 10.66.66.1):

iptables -t nat -I POSTROUTING -p tcp -m tcp --dport 8081 -d 10.66.66.9/32 -j SNAT --to-source 10.66.66.1

with add postrouting rule work but is normal this configuration?

If use hotspot wwan work with only the prerouting rule.

VPS iptables

root@stevps:~# iptables -L -n -t nat --line-n
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DNAT       6    --  0.0.0.0/0            xxx.xxx.xxx.251        tcp dpt:8081 to:10.66.66.9:8081

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    SNAT       6    --  0.0.0.0/0            10.66.66.9           tcp dpt:8081 to:10.66.66.1
2    MASQUERADE  0    --  0.0.0.0/0            0.0.0.0/0
3    MASQUERADE  0    --  0.0.0.0/0            0.0.0.0/0
4    MASQUERADE  0    --  10.66.66.0/24        0.0.0.0/0

PBR openwrt

config pbr 'config'
	option verbosity '2'
	option strict_enforcement '1'
	option resolver_set 'none'
	option ipv6_enabled '0'
	list ignored_interface 'vpnserver'
	list ignored_interface 'wgserver'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '1'
	option webui_show_ignore_target '0'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	option enabled '1'

config policy
	option src_addr '192.168.1.10'
	option interface 'wg0'
	option proto 'tcp'
	option src_port '8081'

If it works using the wwan interface, then it should work the same way over the wan interface.

The problem would be difficult to debug without comparing the ip rules, routing tables and the pbr status in both cases. You may also need to use tcpdump.

This generally works, but better change the subnet mask something larger (like /24) to have a route to the hole wireguard network through the wg0 interface.

You have multiple masquerading rules on the VPS and it is not clear what they do, because you posted the configuration without using the -v option and the in/out interfaces are not listed.

Does port forwarding (when using wan) still work if you move the SNAT rule to the end of the chain?

Thanks for reply.
I took some time to find a solution but unfortunately I didn't find it.

I reinstalled the vps and wireguard.

iptables -L -v -n -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   23  1220 DNAT       6    --  eth0   *       0.0.0.0/0            000.000.000.000        tcp dpt:8081 to:192.168.110.2:8081

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  0    --  *      eth0    0.0.0.0/0            0.0.0.0/0
iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   29  1612 ACCEPT     0    --  wg0    *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Yes, wan port forwarding work with dnat e snat rule. only with both.

I found an explanation on the internet on the use of SNAT and NAT

The reason you need the SNAT is because otherwise your VPN client will send its return packets straight to the host which initiated the connection (z.z.z.z) via its default gateway (i.e. Verizon 3G), and not via the VPN. Thus the source IP address on the return packets will be your Verizon 3G address, and not x.x.x.x. This causes all sorts of problems, since z.z.z.z really initiated the connection to x.x.x.x.

In most port forwarding setups, the SNAT is not needed because the host performing the port forwarding is also the default gateway for the destination host (e.g. a home router).

Also note that if you want to forward port 6000 to a different port (say 7000), then the SNAT rule should match on 7000, not 6000

the strangeness is that it only occurs with the wan set to PPPoE

The problem is not that the return route is different the problem is that the router ( the connection tracking system) can be setup to classify this trafic as invalid and can block it.

Just as I explained

so is the provider the problem?

I do not thinks so it really looks like asymmetric routing see also this thread:

Start with posting the troubleshooting output of pbr.

I don't think pbr is the problem!
I set VPN with default gateway, I stopped and disabled the pbr but nothing changes.

With use rules DNAT and SNAT at the same time it's possible can it cause security issues ?

DNAT is almost always used in conjunction with SNAT. SNAT is a requirement for any LAN to share one IPv4 address from an ISP or VPS. DNAT establishes a known NAT relationship for one port in order to allow incoming connections to pass backward through SNAT. The replies from the internal server are then given SNAT to change their source IP back to the public one that the Internet user knows.

The fact that there is also a VPN tunnel involved is actually only a minor concern if you can set up symmetric routing between the sites. This means that the VPS has a static route to your home LAN 192.168.1.0/24 through the Wireguard tunnel, and also that 192.168.1.0/24 is an allowed_ip on the VPS end of Wireguard. Then at the home end, do not NAT from the LAN into the Wireguard tunnel. Packets will traverse the tunnel with their original 192.168.1.X source IP and then be NATd out to the Internet by the VPS.

Then when you configure the VPS to forward (DNAT) a port to for example 192.168.1.20, that IP is internally known to the VPS. The packet from the Internet will reach your LAN server via VPN automatically without further configuration of OpenWrt, and the reply packet will go into the tunnel and be de-DNATd and returned to the Internet in the VPS.

Configuration of a VPS not running OpenWrt to do this is outside the scope of this forum.

All of this can work without PBR. Once this is working you can add PBR to make other uses of the VPN tunnel conditional, for example having some home hosts go direct to the Internet via the local ISP instead of via the VPS. Run without PBR at first in order to get the basic connection going. That would mean installing a default route via Wireguard (route_allowed_ips on the 0.0.0.0/0 allowed_ips) at home, so all Internet use goes to VPS. In any case, traffic from the public port on the home server must be returned to the VPS so that Internet users of the service get the same source IP that they used as the initial destination.

There isn't a security implication to this other than that of course anyone on the Internet can now reach a port on your server at home, so it needs to be internally hardened against that. A good security practice is to sequester that server into a separate LAN which is firewalled off from reaching https / SSH of your router and VPS, anything on your main LAN (and in some cases, even from making unrelated outgoing connections to the Internet) so that if the server is hijacked it can't be used to do internal damage.

Thanks for detailed explanation! :+1: