Policy-Based-Routing (pbr) package discussion

O-S-T · 2024-11-04T13:43:53.373Z

If a set the DNS policy rule named "redirect local IP DNS" as in my example above.

The result is all the DNS requests are sent through the VPN tunnel to the VPN providers DNS server. I get no DNS leaks.

However if the aforementioned rule is enabled. Any DNS requests sent over the wan interface are not resolved. Despite having a public DNS server listed in the wan interface.

I hope that makes sense.

RSHARM · 2024-11-05T06:00:32.167Z

egc:

/usr/share/pbr/pbr.user.netflix

It is /usr/share/pbr/pbr.user.netflix with facebook ASN changed

Betonmischer · 2024-11-10T04:55:29.240Z

I've encountered a problem with PBR on the latest snapshot. When dnsmasq nft set is used domain name-based policies no longer work. Everything works as intended though when the resolver set option is switched back to none. I've flushed the DNS cache before every try and disabled the DoH proxy built into the web browser.

Any tips will be much appreciated. I've collected all the diagnostic info below.

{
	"kernel": "6.6.59",
	"hostname": "router",
	"system": "ARMv8 Processor rev 0",
	"model": "Bananapi BPI-R4",
	"board_name": "bananapi,bpi-r4",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r28034-ca53f2d430",
		"target": "mediatek/filogic",
		"description": "OpenWrt SNAPSHOT r28034-ca53f2d430",
		"builddate": "1731082951"
	}
}
package dhcp

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	option ednspacket_max '1232'
	option filter_aaaa '0'
	option filter_a '0'
	list server '/mask.icloud.com/'
	list server '/mask-h2.icloud.com/'
	list server '/use-application-dns.net/'
	list server '127.0.0.1#5053'
	list server '127.0.0.1#5054'
	option doh_backup_noresolv '-1'
	option noresolv '1'
	list doh_backup_server '/mask.icloud.com/'
	list doh_backup_server '/mask-h2.icloud.com/'
	list doh_backup_server '/use-application-dns.net/'
	list doh_backup_server '127.0.0.1#5053'
	list doh_backup_server '127.0.0.1#5054'
	list doh_server '127.0.0.1#5053'
	list doh_server '127.0.0.1#5054'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	option ra_slaac '1'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option force '1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'guest'
	option interface 'guest'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option force '1'

package firewall

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading '1'
	option flow_offloading_hw '1'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	list network 'tun0'

config zone
	option name 'guest'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'guest'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config forwarding
	option src 'guest'
	option dest 'wan'

config rule
	option name 'Allow-DNS-Guest'
	option src 'guest'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCP-Guest'
	list proto 'udp'
	option src 'guest'
	option dest_port '67'
	option target 'ACCEPT'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/firewall.include'

package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdca:8fab:878e::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'wan'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.2.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'eth2'
	option macaddr '46:9b:8a:36:18:fb'

config interface 'wan'
	option device 'eth2'
	option proto 'dhcp'

config interface 'wan6'
	option device 'eth2'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'

config interface 'guest'
	option proto 'static'
	option device 'phy1-ap1'
	option ipaddr '192.168.3.1'
	option netmask '255.255.255.0'

config interface 'tun0'
	option proto 'none'
	option device 'tun0'
	option defaultroute '0'

package pbr

config pbr 'config'
	option enabled '1'
	option verbosity '2'
	option strict_enforcement '1'
	option resolver_set 'dnsmasq.nftset'
	option ipv6_enabled '0'
	list ignored_interface 'vpnserver'
	list ignored_interface 'wgserver'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '10'
	option webui_show_ignore_target '0'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	option nft_rule_counter '0'
	option nft_set_auto_merge '1'
	option nft_set_counter '0'
	option nft_set_flags_interval '1'
	option nft_set_flags_timeout '0'
	option nft_set_policy 'performance'
	option procd_boot_delay '20'

config include
	option path '/usr/share/pbr/pbr.user.aws'
	option enabled '0'

config include
	option path '/usr/share/pbr/pbr.user.netflix'
	option enabled '0'

config policy
	option name 'facebook.com'
	option dest_addr 'facebook.com static.xx.fbcdn.net fbcdn.com meta.com'
	option interface 'tun0'

config policy
	option name 'instagram.com'
	option dest_addr 'instagram.com'
	option interface 'tun0'

config policy
	option interface 'tun0'
	option name 'twitter.com'
	option dest_addr 'twitter.com x.com twimg.com '

config policy
	option name 'digitalocean.com'
	option dest_addr 'digitalocean.com'
	option interface 'tun0'

config policy
	option name 'openvpn.net'
	option dest_addr 'openvpn.net'
	option interface 'tun0'

config policy
	option name 'chatgpt.com'
	option dest_addr 'chatgpt.com'
	option interface 'tun0'

config policy
	option name 'discord.com'
	option dest_addr 'dis.gd discord.co discord.com discord.design discord.dev discord.gg discord.gift discord.gifts discord.media discord.new discord.store discordactivities.com discordapp.com discordapp.net discordmerch.com discordpartygames.com discordsays.com discordstatus.com discordcdn.com '
	option interface 'tun0'

config policy
	option name 'protonmail.com'
	option dest_addr 'protonmail.com proton.me'
	option interface 'tun0'

config policy
	option name 'DUT1'
	option src_addr '192.168.1.217'
	option interface 'tun0'
	option enabled '0'

config policy
	option name 'DUT2'
	option src_addr '192.168.1.147'
	option interface 'tun0'
	option enabled '0'


pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000  mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000  mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return

pbr chains - policies
	chain pbr_forward { # handle 37
	}
	chain pbr_input { # handle 38
	}
	chain pbr_output { # handle 39
	}
	chain pbr_postrouting { # handle 41
	}
	chain pbr_prerouting { # handle 40
	}
	chain pbr_dstnat { # handle 36
	}

pbr chains - marking
	chain pbr_mark_0x010000 { # handle 577
		meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1074
		return # handle 1075
	}
	chain pbr_mark_0x020000 { # handle 580
		meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1076
		return # handle 1077
	}

pbr nft sets

IPv4 table 256 route: default via 192.168.1.1 dev eth2 
IPv4 table 256 rule(s):
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.8.0.5 dev tun0 
IPv4 table 257 rule(s):
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_tun0

pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000  mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000  mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add set inet fw4 pbr_tun0_4_dst_ip_cfg046ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "facebook.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg046ff5  goto pbr_mark_0x020000 comment "facebook.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg056ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "instagram.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg056ff5  goto pbr_mark_0x020000 comment "instagram.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg066ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "twitter.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg066ff5  goto pbr_mark_0x020000 comment "twitter.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg076ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "digitalocean.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg076ff5  goto pbr_mark_0x020000 comment "digitalocean.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg086ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "openvpn.net";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg086ff5  goto pbr_mark_0x020000 comment "openvpn.net"
add set inet fw4 pbr_tun0_4_dst_ip_cfg096ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "chatgpt.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg096ff5  goto pbr_mark_0x020000 comment "chatgpt.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg0a6ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "discord.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg0a6ff5  goto pbr_mark_0x020000 comment "discord.com"
add set inet fw4 pbr_tun0_4_dst_ip_cfg0b6ff5 { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "protonmail.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun0_4_dst_ip_cfg0b6ff5  goto pbr_mark_0x020000 comment "protonmail.com"

pbr chains - policies
	chain pbr_forward { # handle 37
	}
	chain pbr_input { # handle 38
	}
	chain pbr_output { # handle 39
	}
	chain pbr_postrouting { # handle 41
	}
	chain pbr_prerouting { # handle 40
		ip daddr @pbr_tun0_4_dst_ip_cfg046ff5 goto pbr_mark_0x020000 comment "facebook.com" # handle 1202
		ip daddr @pbr_tun0_4_dst_ip_cfg056ff5 goto pbr_mark_0x020000 comment "instagram.com" # handle 1204
		ip daddr @pbr_tun0_4_dst_ip_cfg066ff5 goto pbr_mark_0x020000 comment "twitter.com" # handle 1206
		ip daddr @pbr_tun0_4_dst_ip_cfg076ff5 goto pbr_mark_0x020000 comment "digitalocean.com" # handle 1208
		ip daddr @pbr_tun0_4_dst_ip_cfg086ff5 goto pbr_mark_0x020000 comment "openvpn.net" # handle 1210
		ip daddr @pbr_tun0_4_dst_ip_cfg096ff5 goto pbr_mark_0x020000 comment "chatgpt.com" # handle 1212
		ip daddr @pbr_tun0_4_dst_ip_cfg0a6ff5 goto pbr_mark_0x020000 comment "discord.com" # handle 1214
		ip daddr @pbr_tun0_4_dst_ip_cfg0b6ff5 goto pbr_mark_0x020000 comment "protonmail.com" # handle 1216
	}
	chain pbr_dstnat { # handle 36
	}

pbr chains - marking
	chain pbr_mark_0x010000 { # handle 1195
		meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1196
		return # handle 1197
	}
	chain pbr_mark_0x020000 { # handle 1198
		meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1199
		return # handle 1200
	}

pbr nft sets
	set pbr_tun0_4_dst_ip_cfg046ff5 { # handle 1201
		type ipv4_addr
		flags interval
		auto-merge
		comment "facebook.com"
	}
	set pbr_tun0_4_dst_ip_cfg056ff5 { # handle 1203
		type ipv4_addr
		flags interval
		auto-merge
		comment "instagram.com"
	}
	set pbr_tun0_4_dst_ip_cfg066ff5 { # handle 1205
		type ipv4_addr
		flags interval
		auto-merge
		comment "twitter.com"
	}
	set pbr_tun0_4_dst_ip_cfg076ff5 { # handle 1207
		type ipv4_addr
		flags interval
		auto-merge
		comment "digitalocean.com"
	}
	set pbr_tun0_4_dst_ip_cfg086ff5 { # handle 1209
		type ipv4_addr
		flags interval
		auto-merge
		comment "openvpn.net"
	}
	set pbr_tun0_4_dst_ip_cfg096ff5 { # handle 1211
		type ipv4_addr
		flags interval
		auto-merge
		comment "chatgpt.com"
	}
	set pbr_tun0_4_dst_ip_cfg0a6ff5 { # handle 1213
		type ipv4_addr
		flags interval
		auto-merge
		comment "discord.com"
	}
	set pbr_tun0_4_dst_ip_cfg0b6ff5 { # handle 1215
		type ipv4_addr
		flags interval
		auto-merge
		comment "protonmail.com"
	}

dnsmasq sets
nftset=/facebook.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg046ff5 # facebook.com
nftset=/static.xx.fbcdn.net/4#inet#fw4#pbr_tun0_4_dst_ip_cfg046ff5 # facebook.com
nftset=/fbcdn.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg046ff5 # facebook.com
nftset=/meta.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg046ff5 # facebook.com
nftset=/instagram.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg056ff5 # instagram.com
nftset=/twitter.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg066ff5 # twitter.com
nftset=/twimg.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg066ff5 # twitter.com
nftset=/digitalocean.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg076ff5 # digitalocean.com
nftset=/openvpn.net/4#inet#fw4#pbr_tun0_4_dst_ip_cfg086ff5 # openvpn.net
nftset=/chatgpt.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg096ff5 # chatgpt.com
nftset=/dis.gd/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.co/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.design/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.dev/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.gg/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.gift/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.gifts/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.media/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.new/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discord.store/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordactivities.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordapp.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordapp.net/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordmerch.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordpartygames.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordsays.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordstatus.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/discordcdn.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0a6ff5 # discord.com
nftset=/protonmail.com/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0b6ff5 # protonmail.com
nftset=/proton.me/4#inet#fw4#pbr_tun0_4_dst_ip_cfg0b6ff5 # protonmail.com

IPv4 table 256 route: default via 192.168.1.1 dev eth2 
IPv4 table 256 rule(s):
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.8.0.5 dev tun0 
IPv4 table 257 rule(s):
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_tun0

animusastralis · 2024-11-10T09:19:50.464Z

I've got an issue with PBR creating nft sets for single-letter domains. All domain policies work fine, yet policies for x.com or t.co don't work at all. It probably happens because no nft sets are being generated for these policies.

For troubleshooting purposes, I've limited it to only two domain-based policies: dw.com and x.com. The first one works ok - nft set is generated and later populated by dnsmasq-full. Yet no nft set to populate is generated for x.com (you can see it in the output below).

This bug seems to happen only in the case of single-letter domains. For example, if I combine x.com and twitter.com into a single policy, PBR will generate nft set, yet it will be populated only with IPs belonging to twitter.com.

Here is my debug info:

{
        "kernel": "6.6.54",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 0",
        "model": "FriendlyElec NanoPi R6S",
        "board_name": "friendlyarm,nanopi-r6s",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r0-8a5031c",
                "target": "rockchip/armv8",
                "description": "OpenWrt SNAPSHOT r0-8a5031c"
        }
}

package dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d'
        list server '/.themoviedb.org/9.9.9.9'
        list server '/.tmdb.org/9.9.9.9'
        list server '/tmdb-image-prod.b-cdn.net/9.9.9.9'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '24h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

package firewall

config defaults
        option flow_offloading '1'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone 'docker'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option name 'docker'
        list network 'docker'

config zone
        option name 'singbox'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        list network 'sb0'
        list network 'sb1'

config forwarding
        option src 'lan'
        option dest 'singbox'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/firewall.include'

package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'xxxx:xxxx:xxxx::/48'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'
        list ports 'eth2'
        option ipv6 '0'

config device
        option name 'eth0'
        option macaddr 'xx:xx:xx:xx:xx:xx'
        option ipv6 '0'

config device
        option name 'eth2'
        option macaddr 'xx:xx:xx:xx:xx:xx'
        option ipv6 '0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '10.10.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device
        option name 'eth1'
        option macaddr 'xx:xx:xx:xx:xx:xx'
        option ipv6 '0'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'
        option peerdns '0'
        list dns '1.1.1.1'
        list dns '8.8.8.8'
        list dns '9.9.9.9'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'

config interface 'docker'
        option device 'docker0'
        option proto 'none'
        option auto '0'

config device
        option type 'bridge'
        option name 'docker0'
        option ipv6 '0'

config interface 'sb0'
        option proto 'none'
        option device 'sb0'

config interface 'sb1'
        option proto 'none'
        option device 'sb1'

~# uci export pbr

package pbr

config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        list resolver_instance '*'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_boot_delay '0'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        option nft_rule_counter '0'
        option nft_set_auto_merge '1'
        option nft_set_counter '0'
        option nft_set_flags_interval '1'
        option nft_set_flags_timeout '0'
        option nft_set_policy 'performance'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config policy
        option name 'dw.com'
        option dest_addr 'dw.com'
        option interface 'sb0'

config policy
        option name 'x.com'
        option dest_addr 'x.com'
        option interface 'sb0'

~# /etc/init.d/pbr status

pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000  mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000  mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000  mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add set inet fw4 pbr_sb0_4_dst_ip_cfg586ff5 { type ipv4_addr;            auto-merge;                             flags interval;                  policy performance;                             comment "dw.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_sb0_4_dst_ip_cfg586ff5  goto pbr_mark_0x020000 comment "dw.com"

pbr chains - policies
        chain pbr_forward { # handle 45
        }
        chain pbr_input { # handle 46
        }
        chain pbr_output { # handle 47
        }
        chain pbr_postrouting { # handle 49
        }
        chain pbr_prerouting { # handle 48
                ip daddr @pbr_sb0_4_dst_ip_cfg586ff5 goto pbr_mark_0x020000 comment "dw.com" # handle 12599
        }
        chain pbr_dstnat { # handle 44
        }

pbr chains - marking
        chain pbr_mark_0x010000 { # handle 12589
                meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 12590
                return # handle 12591
        }
        chain pbr_mark_0x020000 { # handle 12592
                meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 12593
                return # handle 12594
        }
        chain pbr_mark_0x030000 { # handle 12595
                meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 12596
                return # handle 12597
        }

pbr nft sets
        set pbr_sb0_4_dst_ip_cfg586ff5 { # handle 12598
                type ipv4_addr
                flags interval
                auto-merge
                comment "dw.com"
        }

dnsmasq sets
nftset=/dw.com/4#inet#fw4#pbr_sb0_4_dst_ip_cfg586ff5 # dw.com

IPv4 table 256 route: default via x.x.x.x dev eth1
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 172.16.250.1 dev sb0
IPv4 table 257 rule(s):
29998:  from all fwmark 0x20000/0xff0000 lookup pbr_sb0
IPv4 table 258 route: default via 172.16.251.1 dev sb1
IPv4 table 258 rule(s):
29996:  from all fwmark 0x30000/0xff0000 lookup pbr_sb1

~# /etc/init.d/pbr reload

Loading environment (on_start) Using wan interface (on_start): wan
Found wan gateway (on_start): x.x.x.x
Setting up routing for 'wan/eth1/x.x.x.x' [✓]
Setting up routing for 'sb0/172.16.250.1' [✓]
Setting up routing for 'sb1/172.16.251.1' [✓]
Routing 'dw.com' via sb0 [✓]
Routing 'x.com' via sb0 [✓]
Installing fw4 nft file [✓]
pbr 1.1.7-47 monitoring interfaces: wan sb0 sb1
pbr 1.1.7-47 (fw4 nft file mode) started with gateways:
wan/eth1/x.x.x.x [✓]
sb0/172.16.250.1
sb1/172.16.251.1
WARNING: Please set 'dhcp.lan.force=1' to speed up service start-up.

~# /etc/init.d/pbr status

pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000  mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000  mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000  mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add set inet fw4 pbr_sb0_4_dst_ip_cfg586ff5 { type ipv4_addr;            auto-merge;                             flags interval;                  policy performance;                             comment "dw.com";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_sb0_4_dst_ip_cfg586ff5  goto pbr_mark_0x020000 comment "dw.com"

pbr chains - policies
        chain pbr_forward { # handle 45
        }
        chain pbr_input { # handle 46
        }
        chain pbr_output { # handle 47
        }
        chain pbr_postrouting { # handle 49
        }
        chain pbr_prerouting { # handle 48
        }
        chain pbr_dstnat { # handle 44
        }

pbr chains - marking

pbr nft sets

dnsmasq sets
nftset=/dw.com/4#inet#fw4#pbr_sb0_4_dst_ip_cfg586ff5 # dw.com

IPv4 table 256 route: default via x.x.x.x dev eth1
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 172.16.250.1 dev sb0
IPv4 table 257 rule(s):
29998:  from all fwmark 0x20000/0xff0000 lookup pbr_sb0
IPv4 table 258 route: default via 172.16.251.1 dev sb1
IPv4 table 258 rule(s):
29996:  from all fwmark 0x30000/0xff0000 lookup pbr_sb1

egc · 2024-11-10T09:30:20.707Z

This could be due to the fact you are using HTTPS-DNS proxy with Force router DNS enabled, this might bypass the DNS policy

egc · 2024-11-10T09:47:02.083Z

I can confirm your finding, I think the problem is in:

is_domain(){ echo "$1" | grep -qE '^([a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9]\.)*[a-zA-Z]{2,}$'; }

I do not have time today to make a PR but usually stangri also keeps an eye on this thread

egc · 2024-11-10T10:45:34.331Z

If you feel adventurous you can try my hacked pbr, if that works for you then we have some more proof that I am on the right track.

See: https://github.com/egc112/OpenWRT-egc-add-on/tree/main/pbr

odhiambo · 2024-11-10T11:53:30.805Z

Does this make sense?
Although I think it does (to me) because I have never understood why my luci-app-pbr was always having a version higher than pbr. Now they are the same!

root@DL-WRX36:~# opkg install pbr
Package pbr (1.1.7-r47) installed in root is up to date.
root@DL-WRX36:~# opkg install pbr luci-app-pbr
Package pbr (1.1.7-r47) installed in root is up to date.
Upgrading luci-app-pbr on root from 1.1.7-53 to 1.1.7-r47...
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/luci/luci-app-pbr_1.1.7-r47_all.ipk
Configuring luci-app-pbr.

egc · 2024-11-10T13:06:07.330Z

This has been discussed before, see

Policy-Based-Routing (pbr) package discussion Community Builds, Projects & Packages

Yes, any package release with the r prefix is considered to be higher than any package release without the r. The back story of this is that for the APK support the build system for snapshots now adds r to all package release versions (while the APK is not a default package manager even for snapshots and most likely won't even be an option for 24.xx), while the current release (23.05) build tools create packages without the r in the package release. You can try opkg install --force-downgrade /…

Betonmischer · 2024-11-10T13:55:02.610Z

You are spot-on! I'm testing a new router behind the NAT of the old one with the same settings, including DoH proxy. That's what was messing up the whole setup. I hope this will help someone avoid my mistake and save hours of troubleshooting.

Betonmischer · 2024-11-10T17:00:23.196Z

I was too quick to jump to conclusions. Now that I have my new router connected directly to the ISP in place of the old one, PBR policies fail to take effect every time I switch to dnsmasq nft set, regardless if DoH proxy is enabled or not (all traffic goes through the default WAN gateway as the result). The old router had the same PBR settings. Are there any Ideas what else I could check next?

stangri · 2024-11-10T23:41:07.324Z

Betonmischer:

"description": "OpenWrt SNAPSHOT r28034-ca53f2d430",

The snapshots recently switched the default dnsmasq instance confdir, I've created a PR to revert that, and until that PR is either merged or rejected I don't want to waste electrons developing a workaround for the new dir.

If you explicitly specify the confdir for the dnsmasq instance, the pbr should pick it up however.

stangri · 2024-11-10T23:42:00.159Z

animusastralis:

All domain policies work fine, yet policies for x.com or t.co don't work at all.

Replied to you on reddit as well, pbr 1.1.7-55 should have this addressed, hopefully with no ill-effects.

PS. Forgot to thank @dave14305 and @egc for pointing out an exact issue which was very evident with the information @animusastralis provided.

dave14305 · 2024-11-11T00:04:30.805Z

I think @egc beat me to it a few posts ago, but I saw the Reddit post before this thread.

Policy-Based-Routing (pbr) package discussion Community Builds, Projects & Packages

I can confirm your finding, I think the problem is in: is_domain(){ echo "$1" | grep -qE '^([a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9]\.)*[a-zA-Z]{2,}$'; } I do not have time today to make a PR but usually stangri also keeps an eye on this thread

Betonmischer · 2024-11-11T00:21:44.947Z

Thanks! Adding option confdir '/tmp/dnsmasq.d' to /etc/config/dhcp solved the issue.

animusastralis · 2024-11-11T15:20:02.321Z

Thanks to everyone for helping me! I have a newbie question though - how do I upgrade to 1.1.7-55 from the stock 1.1.7-47 version? I've added stangri_repo to custom feeds, but I'm not sure what to do next - opkg doesn't offer any package upgrades.

kuzevich1983 · 2024-11-11T20:24:33.796Z

Hi @stangri. I have installed latest version of luci-app-pbr-1.1.7.57 from your github repository and i have same issue with @animusastralis. If i set all domain policies to openconnect vpn all work fine, if i set 2ip.io or another domain, policies not work. What i need to create working config. I used latest snapshot on banana pi bpi-r4.

stangri · 2024-11-11T21:03:25.312Z

kuzevich1983:

i have same issue

same answer.

stangri · 2024-11-11T21:23:32.993Z

PSA: do not use pbr on snapshots for two reasons:

Borked dnsmasq instances default confdir, PR to fix this is: https://github.com/openwrt/openwrt/pull/16806
Switch to APK -- you wouldn't be able to upgrade to a newer version as I don't have an APK-compatible repo set up.

If you use the 24.10-snapshots (or rc images when they are available), at least you would be able to explicitly set the confdir path for dnsmasq instance(s) and force-downgrade to install a newer version from my repo.

kuzevich1983 · 2024-11-12T03:24:19.970Z

Good day @stangri. I added option confdir '/tmp/dnsmasq.d' to /etc/config/dhcp, but no changes on policies option if i set remote domains (2ip.io, dnsleaktest.com). DNS policies option work fine.
P.S I use snapshot version because my bpi-r4 not have a stable version of openwrt at this time.