Policy-Based-Routing (pbr) package discussion

egc · February 1, 2025, 7:40am

I have made a new version for you to test (pbr-1.1.8-r6-egc-1.bash) if you like.
In this version, if you set the wg server on the ignored interfaces list on Advanced tab then PBR should do nothing, so make no escape rule for the WAN.
However I am not convinced this is the best solution.
From earlier versions people are used to place the vpn server (openvpn , wireguard) on the ignored interfaces list, for WireGuard this is no longer necessary as it is ignored already but with my hack the automatic routing out via the WAN which can be very useful is not happening if you place the wg server on the ignored list.

If @stangri still thinks this is a good idea then at least we have to change the accompanying text in Luci app PBR and in the read.me

terrytw · February 1, 2025, 9:21pm

Hi, I have some confusion regarding the DNS hijack. In the document it was stated that:

Please note that the use of DNS Policies will override local DNS Hijacking (if enabled) 
and also will prevent the domain-based policies from working for the local devices 
specified in the DNS Policy, as your dnsmasq will not be queried by those local 
devices anymore.

This is what I wanted (pbr DNS policy over local DNS hijack), but in reality the local DNS hijack will render DNS policies useless.

Can someone please enlighten me?

Thank you!

egc · February 2, 2025, 7:10am

These are my notes about DNS Policies : https://github.com/egc112/OpenWRT-egc-add-on/blob/main/stop-dns-leak/README.md#pbr-dns-policies

Take those for what they are worth

terrytw · February 2, 2025, 6:59pm

Thanks for the info!

It feels like @stangri should fix the document.

bd0426 · February 3, 2025, 11:47pm

Thank you @egc. Confirming your pbr-1.1.8-r6-egc-1.bash works as advertised (not that you didn't already know that).

egc · February 4, 2025, 6:34am

Thanks for testing, I will make a PR and will ask @stangri to take a look, as said not sure that this is the best solution but at least we know the problem.
To Be Continued

stangri · February 5, 2025, 7:25pm

@egc thanks for PRs with wg server on ignored interfaces! I may have to follow up with you exploring various use cases to make sure that's the desired behaviour in all scenarios.

For the DNS hijack, I feel that pbr dns policies taking over the DNS hijack (as described in the README) is a preferable scenario, so can you please confirm that just moving the nft file as per note 2 in https://github.com/egc112/OpenWRT-egc-add-on/blob/main/stop-dns-leak/README.md#pbr-dns-policies in the package is enough to achieve that, wherever the hijack is set thru procd firewall entries (like adblock-fast, https-dns-proxy) or manually thru firewall config?

I feel that your earlier PR and the moving the nft file from post-chain to pre-chain are significant enough fixes to warrant 1.1.10 release.

egc · February 6, 2025, 8:41am

Yes that is a sound plan, we are dealing with an edge case (user not wanted to routed out the WG listen port via the WAN port ) so no hurry in dealing with this.

But if you decide to go ahead then it is important that the help text is adapted (see my PR for luci-app-pbr) and maybe some explanation in the read.me is useful.

But I think the basics are covered with my PR added, this is my take on the wg_server situation:

A wg_servers is a wg interface with a listen_port which is not disabled.

By default there is no routing table / fwmark made for a wg_server.
To be able to use the wg_server for rotuing ( in case of a site-to-site setup), the wg_server can be added to the "Supported interfaces" so that a routing table/fwmark are made

By default the listen_port is routed out via the WAN.
New (my PR) if the wg_server is added to the "Ignored Interfaces" then this route out of the listen port via the WAN is not made

These mechanisms are independent of each other

I just checked again and my hack will move the DNS policy up so that it is hit before other DNS hijacking rules
Current:

        chain dstnat {
                type nat hook prerouting priority dstnat; policy accept;
                iifname "br-lan" jump dstnat_lan comment "!fw4: Handle lan IPv4/IPv6 dstnat traffic"
                iifname "wan" jump dstnat_wan comment "!fw4: Handle wan IPv4/IPv6 dstnat traffic"
                jump upnp_prerouting comment "Hook into miniupnpd prerouting chain"
                jump pbr_dstnat comment "Jump into pbr dstnat chain"   <<<<<<<<
        }

New:

        chain dstnat {
                type nat hook prerouting priority dstnat; policy accept;
                jump pbr_dstnat comment "Jump into pbr dstnat chain"    <<<<<<<<
                iifname "br-lan" jump dstnat_lan comment "!fw4: Handle lan IPv4/IPv6 dstnat traffic"
                iifname "wan" jump dstnat_wan comment "!fw4: Handle wan IPv4/IPv6 dstnat traffic"
                jump upnp_prerouting comment "Hook into miniupnpd prerouting chain"
        }

It is no guarantee, other packages can also use this to get on top so it also depends on when the packages are executed relative to each other but at least it gives a better chance

rawd · February 6, 2025, 8:20pm

Running 24.10.0 and I'm noticing that when I apply any policy changes, sqm does not restart as it used to with 23.05. Is this working as expected?

pbr is working fine for me. Thats the only difference I've noticed so far vs 23.05

yxtc934 · February 9, 2025, 9:51pm

It's even worse with 1.1.8-r6 as the (sleep 15; /etc/init.d/pbr restart) & won't clear it.
It just sits there, creating a bunch of instances:

until I kill them all, after what I can just start it normally and after that it just works like it should.

So, back to my original question, are there any logs to check what goes wrong at the startup?
I'm at 24.10 btw.

stangri could you please take a look?

fdauti · February 9, 2025, 10:32pm

maybe check the latest version 1.1.8-r10

yxtc934 · February 10, 2025, 6:16pm

ok, thanks, 1.1.8-r10 starts by itself (although a bit slowly)

By the way, why the repo
https://raw.githubusercontent.com/stangri/repo.openwrt.melmac.net/master src/gz stangri_repo https://repo.openwrt.melmac.net

wasn't showing the 1.1.8-r10 ? The opkg update was showing the 1.1.8-r6 being the last

egc · February 10, 2025, 6:42pm

It can take a couple of days before it trickles down, so check the coming days

Sam-arch99 · February 10, 2025, 8:20pm

The only way to get 1.1.8.r10 is to use stangris repo? I'm on 24.10.0 and r6 got installed with no updates offered via opkg/software in luci.

stas336 · February 11, 2025, 12:16am

Hi. Installed PBR 2 days ago and domain-based policy to redirect some traffic to vpn from wan aren't working at all for me. DNS has been flushed at 2 devices several times + reboot after that. Can someone help? Logs are attached below. Thanks in advance

System

root@AX3600:~# ubus call system board
{
	"kernel": "6.6.73",
	"hostname": "AX3600",
	"system": "ARMv8 Processor rev 4",
	"model": "Xiaomi AX3600",
	"board_name": "xiaomi,ax3600",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "24.10.0",
		"revision": "r28427-6df0e3d02a",
		"target": "qualcommax/ipq807x",
		"description": "OpenWrt 24.10.0 r28427-6df0e3d02a",
		"builddate": "1738624177"
	}
}

DHCP

root@AX3600:~# uci export dhcp

package dhcp

config dnsmasq

option domainneeded '1'

option boguspriv '1'

option filterwin2k '0'

option localise_queries '1'

option rebind_protection '1'

option rebind_localhost '1'

option local '/lan/'

option domain 'lan'

option expandhosts '1'

option nonegcache '0'

option cachesize '1000'

option authoritative '1'

option readethers '1'

option leasefile '/tmp/dhcp.leases'

option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'

option nonwildcard '1'

option localservice '1'

option ednspacket_max '1232'

option filter_aaaa '0'

option filter_a '0'

option confdir '/tmp/dnsmasq.d'

config dhcp 'lan'

option interface 'lan'

option start '100'

option limit '150'

option leasetime '12h'

option dhcpv4 'server'

option dhcpv6 'server'

option ra 'server'

option ra_slaac '1'

list ra_flags 'managed-config'

list ra_flags 'other-config'

config dhcp 'wan'

option interface 'wan'

option ignore '1'

config odhcpd 'odhcpd'

option maindhcp '0'

option leasefile '/tmp/hosts/odhcpd'

option leasetrigger '/usr/sbin/odhcpd-update'

option loglevel '4'

Firewall

root@AX3600:~# uci export firewall

package firewall

config defaults

option syn_flood '1'

option input 'REJECT'

option output 'ACCEPT'

option forward 'REJECT'

config zone

option name 'lan'

list network 'lan'

option input 'ACCEPT'

option output 'ACCEPT'

option forward 'ACCEPT'

config zone

option name 'wan'

option input 'REJECT'

option output 'ACCEPT'

option forward 'REJECT'

option masq '1'

option mtu_fix '1'

list network 'wan'

list network 'wan6'

list network 'vpn'

config forwarding

option src 'lan'

option dest 'wan'

config rule

option name 'Allow-DHCP-Renew'

option src 'wan'

option proto 'udp'

option dest_port '68'

option target 'ACCEPT'

option family 'ipv4'

config rule

option name 'Allow-Ping'

option src 'wan'

option proto 'icmp'

option icmp_type 'echo-request'

option family 'ipv4'

option target 'ACCEPT'

config rule

option name 'Allow-IGMP'

option src 'wan'

option proto 'igmp'

option family 'ipv4'

option target 'ACCEPT'

config rule

option name 'Allow-DHCPv6'

option src 'wan'

option proto 'udp'

option dest_port '546'

option family 'ipv6'

option target 'ACCEPT'

config rule

option name 'Allow-MLD'

option src 'wan'

option proto 'icmp'

option src_ip 'fe80::/10'

list icmp_type '130/0'

list icmp_type '131/0'

list icmp_type '132/0'

list icmp_type '143/0'

option family 'ipv6'

option target 'ACCEPT'

config rule

option name 'Allow-ICMPv6-Input'

option src 'wan'

option proto 'icmp'

list icmp_type 'echo-request'

list icmp_type 'echo-reply'

list icmp_type 'destination-unreachable'

list icmp_type 'packet-too-big'

list icmp_type 'time-exceeded'

list icmp_type 'bad-header'

list icmp_type 'unknown-header-type'

list icmp_type 'router-solicitation'

list icmp_type 'neighbour-solicitation'

list icmp_type 'router-advertisement'

list icmp_type 'neighbour-advertisement'

option limit '1000/sec'

option family 'ipv6'

option target 'ACCEPT'

config rule

option name 'Allow-ICMPv6-Forward'

option src 'wan'

option dest '*'

option proto 'icmp'

list icmp_type 'echo-request'

list icmp_type 'echo-reply'

list icmp_type 'destination-unreachable'

list icmp_type 'packet-too-big'

list icmp_type 'time-exceeded'

list icmp_type 'bad-header'

list icmp_type 'unknown-header-type'

option limit '1000/sec'

option family 'ipv6'

option target 'ACCEPT'

config rule

option name 'Allow-IPSec-ESP'

option src 'wan'

option dest 'lan'

option proto 'esp'

option target 'ACCEPT'

config rule

option name 'Allow-ISAKMP'

option src 'wan'

option dest 'lan'

option dest_port '500'

option proto 'udp'

option target 'ACCEPT'

config include 'qcanssecm'

option type 'script'

option path '/etc/firewall.d/qca-nss-ecm'

config include 'pbr'

option fw4_compatible '1'

option type 'script'

option path '/usr/share/pbr/firewall.include'

Network

root@AX3600:~# uci export network

package network

config interface 'loopback'

option device 'lo'

option proto 'static'

option ipaddr '127.0.0.1'

option netmask '255.0.0.0'

config globals 'globals'

option ula_prefix 'fd5d:fe1:f062::/48'

option packet_steering '0'

config device

option name 'br-lan'

option type 'bridge'

list ports 'lan1'

list ports 'lan2'

list ports 'lan3'

config interface 'lan'

option device 'br-lan'

option proto 'static'

option ipaddr '192.168.1.1'

option netmask '255.255.255.0'

option ip6assign '60'

config interface 'wan'

option device 'wan'

option proto 'dhcp'

option peerdns '0'

list dns '1.1.1.1'

list dns '8.8.8.8'

config interface 'wan6'

option device 'wan'

option proto 'dhcpv6'

config interface 'vpn'

option proto 'none'

option device 'tun0'

option auto '0'

option defaultroute '0'

PBR

root@AX3600:~# uci export pbr

package pbr

config pbr 'config'

option enabled '0'

option verbosity '2'

option strict_enforcement '1'

option resolver_set 'dnsmasq.nftset'

list resolver_instance '*'

option ipv6_enabled '0'

list ignored_interface 'vpnserver'

option boot_timeout '30'

option rule_create_option 'add'

option procd_reload_delay '1'

option webui_show_ignore_target '0'

option nft_rule_counter '1'

option nft_set_auto_merge '1'

option nft_set_counter '1'

option nft_set_flags_interval '1'

option nft_set_flags_timeout '0'

option nft_set_policy 'performance'

list webui_supported_protocol 'all'

list webui_supported_protocol 'tcp'

list webui_supported_protocol 'udp'

list webui_supported_protocol 'tcp udp'

list webui_supported_protocol 'icmp'

config include

option path '/usr/share/pbr/pbr.user.aws'

option enabled '0'

config include

option path '/usr/share/pbr/pbr.user.netflix'

option enabled '0'

config dns_policy

option name 'Redirect Local IP DNS'

option src_addr '192.168.1.5'

option dest_dns '1.1.1.1'

option enabled '0'

config policy

option name 'Ignore Local Requests'

option interface 'ignore'

option dest_addr '10.0.0.0/24 10.0.1.0/24 192.168.100.0/24 192.168.1.0/24'

option enabled '0'

config policy

option name 'Plex/Emby Local Server'

option interface 'wan'

option src_port '8096 8920 32400'

option enabled '0'

config policy

option name 'Plex/Emby Remote Servers'

option interface 'wan'

option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'

option enabled '0'

config policy

option name 'Test Route'

option src_addr '192.168.1.150 192.168.1.161’

option dest_addr 'iplocation.net 2ip.ru'

option interface 'vpn'

PBR Start

root@AX3600:~# /etc/init.d/pbr status

pbr - environment

pbr 1.1.8-r6 running on OpenWrt 24.10.0.

Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley

Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr chains - policies

chain pbr_forward { # handle 33

}

chain pbr_input { # handle 34

}

chain pbr_output { # handle 35

}

chain pbr_postrouting { # handle 37

}

chain pbr_prerouting { # handle 36

}

chain pbr_dstnat { # handle 32

}

pbr chains - marking

pbr nft sets

pbr tables & routing

root@AX3600:~# /etc/init.d/pbr reload

Using wan interface (on_start): wan

Found wan gateway (on_start): 89.70.126.1

Setting up routing for 'wan/89.70.126.1' [✓]

Setting up routing for 'vpn/tun0/192.168.241.250' [✓]

Routing 'Test Route' via vpn [✓]

Installing fw4 nft file [✓]

Restarting dnsmasq [✓]

pbr 1.1.8-r6 monitoring interfaces: wan vpn

pbr 1.1.8-r6 (fw4 nft file mode) started with gateways:

wan/89.70.126.1 [✓]

vpn/tun0/192.168.241.250

WARNING: Please set 'dhcp.lan.force=1' to speed up service start-up.

PBR Status

root@AX3600:~# /etc/init.d/pbr status

pbr - environment
pbr 1.1.8-r6 running on OpenWrt 24.10.0.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 counter mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add set inet fw4 pbr_vpn_4_dst_ip_cfg086ff5 { type ipv4_addr;  		 auto-merge; 		 counter; 		 flags interval; 		 	 policy performance; 		 		 comment "Test Route";}
add rule inet fw4 pbr_prerouting ip saddr { 192.168.1.150, 192.168.1.161 } ip daddr @pbr_vpn_4_dst_ip_cfg086ff5 counter goto pbr_mark_0x020000 comment "Test Route"

pbr chains - policies
	chain pbr_forward { # handle 33
	}
	chain pbr_input { # handle 34
	}
	chain pbr_output { # handle 35
	}
	chain pbr_postrouting { # handle 37
	}
	chain pbr_prerouting { # handle 36
		ip saddr { 192.168.1.150, 192.168.1.161 } ip daddr @pbr_vpn_4_dst_ip_cfg086ff5 counter packets 0 bytes 0 goto pbr_mark_0x020000 comment "Test Route" # handle 521
	}
	chain pbr_dstnat { # handle 32
	}

pbr chains - marking
	chain pbr_mark_0x010000 { # handle 513
		counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 514
		return # handle 515
	}
	chain pbr_mark_0x020000 { # handle 516
		counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 517
		return # handle 518
	}

pbr nft sets
	set pbr_vpn_4_dst_ip_cfg086ff5 { # handle 519
		type ipv4_addr
		flags interval
		counter
		auto-merge
		comment "Test Route"
	}

dnsmasq sets
nftset=/iplocation.net/4#inet#fw4#pbr_vpn_4_dst_ip_cfg086ff5 # Test Route
nftset=/2ip.ru/4#inet#fw4#pbr_vpn_4_dst_ip_cfg086ff5 # Test Route

pbr tables & routing
IPv4 table 256 pbr_wan route:
default via 89.70.126.1 dev wan 
IPv4 table 256 pbr_wan rule(s):
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan

IPv4 table 257 pbr_vpn route:
default via 192.168.241.250 dev tun0 
IPv4 table 257 pbr_vpn rule(s):
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_vpn

andreid1303 · February 11, 2025, 1:20am

I have switched to Openwrt x86 with ext4 image and pbr is having problem to start on boot. I suppose this is related to pbr nft rule files are somehow persistent.
When I disable pbr in config and reboot, the old rules are still loaded. pbr.lock and pbr.nft are in /var/run, pbr status is reporting as running but table inet fw4 is empty.
What is actually going on?

P.S. Well I found that just editing option enabled in /etc/config/pbr does not actually enables or disables service on boot. I have to run /etc/init.d/pbr enable/disble, stop/start manually after changing config. But still why previously running nft rules are persistent?

And /etc/init.d/pbr status is always running!

/etc/init.d/pbr stop

Command failed: Not found

/etc/init.d/pbr status

pbr - environment
pbr 1.1.8-r6 running on OpenWrt 24.10.0.

egc · February 11, 2025, 6:59am

On first glance everything looks OK.
See the focus points in this link if they apply to your situation.

PBR is in place but the nft set has to be filled by DNSMasq.

With nft list ruleset you can see if the set is filled.

egc · February 11, 2025, 7:29am

I think it works like this:
All changes regarding this 1.1.8 branch start at:

After internal testing a package is made which is then available in stangri's repo and which you can use to upgrade if you set stangri's repo as update source in your routers feeds, see the How to Use section in the PBR guide

Usually within days after commits are made the new build is available in stangri's repo for you to upgrade.
At this moment 1.1.8-r10 is available for you to test

After more testing and after feedback from users a build which is deemed good enough is added to the official OpenWRT master repo
See the Makefile for the version.
Master and 24.10 are on 1.1.8-r6.
23.05 is on 1.1.6-r22
Usually only Master is updated, sometimes a backport is done in case of serious bugs.

But as described above you can add stangr's repo in your feeds and update from his own repo.

odhiambo · February 12, 2025, 11:08am

Out of curiosity, when was the option procd_boot_delay removed from the config?
I updated to Version 1.1.8-r10 and diff-ing the bundled config against mine showed that the option wasn't in the bundled config.

TIA

egc · February 12, 2025, 11:15am

See my earlier post how things work: Policy-Based-Routing (pbr) package discussion - #1984 by egc

To make it easy for you:

If that is not easy enough:

But now you know where to look