Configuring pbr to selectively route traffic based on port

Hi all,

I've created this thread to have a dedicated place to discuss some trouble I've been having using the pbr package as well as be a dedicated resource for future users hoping to do what is described in the title. I'm sure the folks at Policy-Based-Routing (pbr) package discussion will appreciate me keeping this discussion and my questions elsewhere.

My goal is to route traffic coming in on a VPN interface to a specific LAN host based on port number. Similarly my goal is to route any egress traffic from said LAN host back through the VPN before it is transmitted to the destination address. It is my understanding this can be achieved using pbr but due to my limited understanding of firewall zones/iptables etc. I find myself needing a little bit of help.

Thus far, I have been able to use 3 simple iptables rules on my VPS (running the vpn server) to redirect incoming traffic on a specified port/proto to the vpn client:

iptables -t nat -A PREROUTING -d pubIP/32 -i eth0 -p tcp -m tcp --dport 34 -j DNAT --to-destination vpnIP:34
iptables -A FORWARD -p tcp -d vpnIP --dport 34 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -s vpnIP/24 ! -d vpnIP/24 -j SNAT --to-source pubIP

Now I'd like to do something similar in openwrt to route the traffic from the VPN to a specific LAN host but am running into some trouble in that I don't know which tables to assign a similar rule set to, and I'm unsure of how to use nf_tables/pbr package to do this. Any help is appreciated and I'm more than happy to provide cfg or screenshots if needed. Thanks

1 Like

For example,

Attempting to recreate the same rule in openwrt, to change the destination addr from the vpnIP to the expected LAN IP I see this error.

Also I don't have any familiarity with what is required in terms of firewall zones/forwarding etc so as of now I simply have this.

One option is to avoid the double DNAT. The VPS can redirect straight to the lan host. This means that the VPS must have a route for the 192.168.2.0/24
Second option is to use the luci to create the redirect, because it seems that you are using nftables.

Hey Trendy,

I understand why avoiding double NAT could be easier but I, unfortunately have some reasons as to why that won't work with my setup. Funnily enough my setup is pretty easy in terms of reserving specific ports and simply chaning the destination addr and source addr at a couple points in the routing process.

I'd really like to be able to replicate the example on openwrt's own wiki. I find it odd I can't create a rule where I change the destination addr in prerouting.

As for creating the redirect using luci I believe i have tried that to no avail. Here's a section of my /etc/config/firewall:


config forwarding
        option src 'lan'
        option dest 'vpn'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'test'
        list proto 'tcp'
        option src 'vpn'
        option src_dport '34'
        option dest_port '34'
        option reflection '0'
        option src_port '34'
        option dest_ip '192.168.2.218'

Also is there any way to dump the list of rules that are actively in effect from the CLI, I'd like to verify what this rule is and if it is actually being applied.

Remove this and it will work.

I see there is duplicate src_port option. I dont recall doing that but can remove it.

Will changing this file automatically regen the rules or do I need to issue some cmd to regenerate the rules?

It's not duplicate. One is src_port the other is src_ d port. You need the src_dport to match the destination port of the source packet. The src_port is rarely matched, at the source port is usually random.

I removed it then ran service firewall restart, and I still dont believe I am seeing any traffic on my LAN machine.

So is it working fine now?

Unfortunately that would be a no.

Still not seeing any traffic coming in on that LAN host.

Even using wireshark to sniff if any traffic coming into the machine matches the port specified and I'm seeing a whole lot of nothing.

I may do a reset on the router and just cfg the vpn and apply this rule to eliminate other variables.

Do I need to be concious of any firewall zone settings that would prevent this rule from working properly?

Can I monitor any incoming packet activity to the vpn interface on openwrt itself?

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export firewall; \
head -n -0 /etc/firewall.user; \
iptables-save -c; nft list ruleset
ubus call system board; \
> uci export network; uci export firewall; \
> head -n -0 /etc/firewall.user; \
> iptables-save -c; nft list ruleset
{
        "kernel": "5.10.146",
        "hostname": "OpenWrt",
        "system": "MediaTek MT7621 ver:1 eco:3",
        "model": "Netgear WAX202",
        "board_name": "netgear,wax202",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.2",
                "revision": "r19803-9a599fee93",
                "target": "ramips/mt7621",
                "description": "OpenWrt 22.03.2 r19803-9a599fee93"
        }
}
package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '1'
        option ula_prefix 'fdb7:42d9:7c46::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'

config device
        option name 'lan1'
        option macaddr '34:98:b5:12:d7:fc'

config device
        option name 'lan2'
        option macaddr '34:98:b5:12:d7:fc'

config device
        option name 'lan3'
        option macaddr '34:98:b5:12:d7:fc'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.2.1'

config device
        option name 'wan'
        option macaddr '34:98:b5:12:d7:fd'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config interface 'client3'
        option proto 'wireguard'
        option private_key 'kM5lAFYdKRY1NQA4aOsbsB+WN+1fMhjHv5bz1Bwb5UA='
        list addresses '10.7.0.2/24'
        option peerdns '0'
        list dns '94.140.14.14'
        list dns '94.140.15.15'

config wireguard_client3
        option description 'client3.conf'
        option public_key 'vNiu9aA/HzLBzrbyehgLOgHUiXUGNkkwxwSmIJN0V20='
        option preshared_key '4cIYLlWe1jKBn36EQ/SQGhUTvT3KoNohaT7UUngp8S4='
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '::/0'
        option persistent_keepalive '25'
        option endpoint_host 'publicVPNIP'
        option endpoint_port '51820'

package firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option mtu_fix '1'
        list network 'client3'

config forwarding
        option src 'vpn'
        option dest 'lan'

config forwarding
        option src 'lan'
        option dest 'vpn'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'test'
        list proto 'tcp'
        option src 'vpn'
        option src_dport '34'
        option dest_port '34'
        option reflection '0'
        option dest_ip '192.168.2.218'

config rule
        option name 'ftp'
        list proto 'tcp'
        option src 'vpn'
        option src_port '34'
        option dest 'lan'
        list dest_ip '192.168.2.218'
        option dest_port '34'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option enabled '0'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/pbr.firewall.include'

head: /etc/firewall.user: No such file or directory
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
table inet fw4 {
        chain input {
                type filter hook input priority filter; policy accept;
                iifname "lo" accept comment "!fw4: Accept traffic from loopback"
                ct state established,related accept comment "!fw4: Allow inbound established and related flows"
                tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
                iifname "br-lan" jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
                iifname "wan" jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"
                iifname "client3" jump input_vpn comment "!fw4: Handle vpn IPv4/IPv6 input traffic"
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
                iifname "br-lan" jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
                iifname "wan" jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
                iifname "client3" jump forward_vpn comment "!fw4: Handle vpn IPv4/IPv6 forward traffic"
                jump handle_reject
        }

        chain output {
                type filter hook output priority filter; policy accept;
                oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
                ct state established,related accept comment "!fw4: Allow outbound established and related flows"
                oifname "br-lan" jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
                oifname "wan" jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
                oifname "client3" jump output_vpn comment "!fw4: Handle vpn IPv4/IPv6 output traffic"
        }

        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
                iifname "br-lan" jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
                iifname "client3" jump helper_vpn comment "!fw4: Handle vpn IPv4/IPv6 helper assignment"
        }

        chain handle_reject {
                meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
                reject comment "!fw4: Reject any other traffic"
        }

        chain syn_flood {
                limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
                drop comment "!fw4: Drop excess packets"
        }

        chain input_lan {
                jump accept_from_lan
        }

        chain output_lan {
                jump accept_to_lan
        }

        chain forward_lan {
                jump accept_to_wan comment "!fw4: Accept lan to wan forwarding"
                jump accept_to_vpn comment "!fw4: Accept lan to vpn forwarding"
                jump accept_to_lan
        }

        chain helper_lan {
        }

        chain accept_from_lan {
                iifname "br-lan" counter packets 3667 bytes 304220 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
        }

        chain accept_to_lan {
                oifname "br-lan" counter packets 1069 bytes 79854 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
        }

        chain input_wan {
                meta nfproto ipv4 udp dport 68 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCP-Renew"
                icmp type echo-request counter packets 0 bytes 0 accept comment "!fw4: Allow-Ping"
                meta nfproto ipv4 meta l4proto igmp counter packets 0 bytes 0 accept comment "!fw4: Allow-IGMP"
                meta nfproto ipv6 udp dport 546 counter packets 0 bytes 0 accept comment "!fw4: Allow-DHCPv6"
                ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 0 bytes 0 accept comment "!fw4: Allow-MLD"
                icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply, nd-router-solicit, nd-router-advert } limit rate 1000/second counter packets 17 bytes 952 accept comment "!fw4: Allow-ICMPv6-Input"
                icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, nd-neighbor-solicit . no-route, nd-neighbor-advert . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Input"
                jump reject_from_wan
        }

        chain output_wan {
                jump accept_to_wan
        }

        chain forward_wan {
                icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
                icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
                meta l4proto esp counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-IPSec-ESP"
                udp dport 500 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-ISAKMP"
                jump reject_to_wan
        }

        chain accept_to_wan {
                oifname "wan" counter packets 5370 bytes 1262862 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
        }

        chain reject_from_wan {
                iifname "wan" counter packets 4478 bytes 299905 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
        }

        chain reject_to_wan {
                oifname "wan" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
        }

        chain input_vpn {
                ct status dnat accept comment "!fw4: Accept port redirections"
                jump accept_from_vpn
        }

        chain output_vpn {
                jump accept_to_vpn
        }

        chain forward_vpn {
                ip daddr 192.168.2.218 tcp sport 34 tcp dport 34 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: ftp"
                jump accept_to_lan comment "!fw4: Accept vpn to lan forwarding"
                ct status dnat accept comment "!fw4: Accept port forwards"
                jump accept_to_vpn
        }

        chain helper_vpn {
        }

        chain accept_from_vpn {
                iifname "client3" counter packets 2 bytes 80 accept comment "!fw4: accept vpn IPv4/IPv6 traffic"
        }

        chain accept_to_vpn {
                oifname "client3" counter packets 0 bytes 0 accept comment "!fw4: accept vpn IPv4/IPv6 traffic"
        }

        chain dstnat {
                type nat hook prerouting priority dstnat; policy accept;
                iifname "client3" jump dstnat_vpn comment "!fw4: Handle vpn IPv4/IPv6 dstnat traffic"
        }

        chain srcnat {
                type nat hook postrouting priority srcnat; policy accept;
                oifname "wan" jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
        }

        chain srcnat_wan {
                meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 wan traffic"
        }

        chain dstnat_vpn {
                meta nfproto ipv4 tcp dport 34 counter packets 5 bytes 300 dnat ip to 192.168.2.218:34 comment "!fw4: test"
                ip daddr 10.7.0.2 tcp dport 34 dnat ip to 192.168.2.218:34
        }

        chain raw_prerouting {
                type filter hook prerouting priority raw; policy accept;
        }

        chain raw_output {
                type filter hook output priority raw; policy accept;
        }

        chain mangle_prerouting {
                type filter hook prerouting priority mangle; policy accept;
                jump pbr_prerouting comment "Jump into pbr prerouting chain"
        }

        chain mangle_postrouting {
                type filter hook postrouting priority mangle; policy accept;
                jump pbr_postrouting comment "Jump into pbr postrouting chain"
        }

        chain mangle_input {
                type filter hook input priority mangle; policy accept;
                jump pbr_input comment "Jump into pbr input chain"
        }

        chain mangle_output {
                type route hook output priority mangle; policy accept;
                jump pbr_output comment "Jump into pbr output chain"
        }

        chain mangle_forward {
                type filter hook forward priority mangle; policy accept;
                iifname "wan" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
                oifname "wan" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
                iifname "client3" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone vpn IPv4/IPv6 ingress MTU fixing"
                oifname "client3" tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone vpn IPv4/IPv6 egress MTU fixing"
                jump pbr_forward comment "Jump into pbr forward chain"
        }

        chain pbr_forward {
        }

        chain pbr_input {
        }

        chain pbr_output {
        }

        chain pbr_prerouting {
        }

        chain pbr_postrouting {
        }
}

meta nfproto ipv4 tcp dport 34 counter packets 5 bytes 300 dnat ip to 192.168.2.218:34 comment "!fw4: test"
You have already some hits, so the packets are coming fine and are forwarded. Are you saying that you don't see the packets getting to the host 192.168.2.218?

I see that now,

I played around with some other cfg and I'm now seeing traffic on the LAN host. Great!

Now for the fun part, I'd like to force any egress traffic on LAN with a specific port through the VPN interface.

I understand DNAT cannot be used as it modifies the dest address of the packet which would remove the intended recipient. I was hoping to "forward?" it, sort of like forcing OpenWRT to believe the next hop is the VPN server so the packet is sent over the interface and the VPN can handle the final SNAT and sending it out over its WAN.

You need to do Policy Based Routing and you have 3 options:

  1. mwan3 package
  2. pbr package
  3. a set of rules/routes for each internet connection.
    Unfortunately the uci implementation of ip rules does not support ports, so you can either do it by matching something else, or by using the native ip rule add ... which supports ports, but it is not so much integrated to OpenWrt.

Not sure how familiar you are with the pbr package. But I'm struggling to understand which chain the rule should belong to and which fields should be populated with which address.

I'd assume I should choose the VPN interface as the interface on pbr? With local IP being the LAN IP and Destination IP being the VPN gateway?

Do I only populate the source port, thus filtering only by traffic originating from said IP:port?

Can I track what pbr rules have been created by checking some file, nft -a list chain inet fw4 pbr_x, or conntrack?

uci add pbr policy
uci set pbr.@policy[-1].enabled='1'
uci set pbr.@policy[-1].name='test'
uci set pbr.@policy[-1].src_addr='192.168.2.218'
uci set pbr.@policy[-1].src_port='34'
uci set pbr.@policy[-1].interface='vpn'
uci commit pbr
service pbr restart

Does this cover you? Rename the interface name, obviously.

1 Like

Whoa that worked...

I must say I'm a bit surprised I hit so many nuances. Perhaps I was a little too reliant on the luci-app or was too specific when setting the rules?

Regardless I'll definitely want to play around with this some more but it appears to be working, so for that I thank you.

I'm happy enough with pbr assuming for simple forwarding such as this doesn't result in high CPU usage.

I must admit I'm a little curious about the mwan approach if it doesn't require extensive explanation (just pointing me to a thread or man page would help). I also may be wanting to dive into that package soon as I'll have a fallback WAN connection in my new setup.

mwan3 and pbr are achieving the same goal, using the same means. Therefore I don't recommend to use them both. Pick the one that works best for you.

EDIT: Nevermind, windows firewall.