Set procd_wan_interface option to your upstream interface.
That's what dnsmasq.nftset option is expected/supposed to do.
Remove the listen_port option from your client wg tunnel.
I have updated to the latest SNAPSHOT (r27283-85f0128851) and now i1.1.7-5 emits an error.
root@DL-WRX36:~# opkg install pbr luci-app-pbr
Upgrading pbr on root from 1.1.6-r20 to 1.1.7-5...
Downloading https://repo.openwrt.melmac.net/pbr_1.1.7-5_all.ipk
Stopping pbr service... OK
Removing rc.d symlink for pbr... OK
Removing obsolete file /usr/share/pbr/pbr.user.wg_server_and_client.
Upgrading luci-app-pbr on root from 1.1.6-20 to 1.1.7-5...
Downloading https://repo.openwrt.melmac.net/luci-app-pbr_1.1.7-5_all.ipk
Configuring pbr.
Installing rc.d symlink for pbr... OK
Using wan interface (on_start): wan
Found wan gateway (on_start): 192.168.1.1
Setting up routing for 'wan/192.168.1.1' [✓]
Setting up routing for 'wgc0/10.5.0.2' [✓]
Setting up routing for 'nordvpntun0/tun0/0.0.0.0' [✓]
Routing 'Via WAN' via wan [✓]
Routing 'Paypal' via wan [✓]
Routing 'chatgpt_via_wan' via wan [✓]
Routing 'stanbic_internet_banking' via wan [✓]
Routing 'aliexpress.com' via wan [✓]
Routing 'odoo.com' via wan [✓]
Routing 'Wyze' via wan [✓]
Routing 'VanessaFireTV' via wan [✓]
Routing 'Panafcon' via wan [✓]
Routing 'eCitizen' via wan [✓]
Routing 'Oraimo' via wan [✓]
Routing 'eBay' via wan [✓]
Running /usr/share/pbr/pbr.user.ke.lst [✓]
Installing fw4 nft file [✓]
pbr 1.1.7-5 monitoring interfaces: wan wgc0 nordvpntun0
pbr 1.1.7-5 (fw4 nft file mode) started with gateways:
wan/192.168.1.1
wgc0/10.5.0.2 [✓]
nordvpntun0/tun0/0.0.0.0
WARNING: Resolver set (dnsmasq.nftset) is not supported on this system.
WARNING: Resolver set (dnsmasq.nftset) is not supported on this system.
Configuring luci-app-pbr.
See the readme if this applies: https://docs.openwrt.melmac.net/pbr/#Requirements
1 Like
I removed the option resolver_set 'dnsmasq.nftset' from my config to silence the above warning, but not sure if this is the proper solution since the default pbr config still comes with that option.
I forgot to compile dnsmasq-full when building the firmware 
Mea culpa.
No problem but instead of just posting everything in this thread consider doing some research first e.g. consult the readme 
But of course users are encouraged to post possible bugs/problems so that this fantastic package can be even improved further 
I went through the README when I started initially so I thought I knew
I even have my procedure documented in a notepad. I follow this forum at least 3x weekly to keep up with the changes. However, today when updating (firmware-selector.openwrt.org) I forgot to edit dnsmasq to dnsmasq-full.
And then I started making useless noices here!
I am sorry for the noice.
I have updated my router running 23.05.4 from 1.1.7-4 to 1.1.7-5 and it tells me that my custom config file is incompatible. I was recently assisted by @egc to fix this file.
And this error did not show up with my router running SNAPHOT.
root@Belkin-RT3200:~# opkg install pbr luci-app-pbr
Upgrading pbr on root from 1.1.7-4 to 1.1.7-5...
Downloading https://repo.openwrt.melmac.net/pbr_1.1.7-5_all.ipk
Stopping pbr service... OK
Removing rc.d symlink for pbr... OK
Upgrading luci-app-pbr on root from 1.1.7-4 to 1.1.7-5...
Downloading https://repo.openwrt.melmac.net/luci-app-pbr_1.1.7-5_all.ipk
Configuring pbr.
Installing rc.d symlink for pbr... OK
Using wan interface (on_start): wan
Found wan gateway (on_start): 192.168.1.1
Setting up routing for 'wan/192.168.1.1' [✓]
Setting up routing for 'vpnclient0/ovpnc0/10.100.0.2' [✓]
Setting up routing for 'vpnclient1/ovpnc1/0.0.0.0' [✓]
Setting up routing for 'vpnclient2/ovpnc2/0.0.0.0' [✓]
Setting up routing for 'wgc0/10.5.0.2' [✓]
Routing 'Via WAN' via wan [✓]
Routing 'Paypal' via wan [✓]
Routing 'chatgpt_via_wan' via wan [✓]
Routing 'stanbic_internet_banking' via wan [✓]
Routing 'aliexpress.com' via wan [✓]
Routing 'odoo.com' via wan [✓]
Routing 'Wyze' via wan [✓]
Routing 'gmail' via wan [✓]
[✗]
Installing fw4 nft file [✓]
pbr 1.1.7-5 monitoring interfaces: wan vpnclient0 vpnclient1 vpnclient2 wgc0
Restarting dnsmasq [✓]
pbr 1.1.7-5 (fw4 nft file mode) started with gateways:
wan/192.168.1.1
vpnclient0/ovpnc0/10.100.0.2 [✓]
vpnclient1/ovpnc1/0.0.0.0
vpnclient2/ovpnc2/0.0.0.0
wgc0/10.5.0.2
ERROR: Incompatible custom user file detected '/usr/share/pbr/pbr.user.ke.lst'!
Configuring luci-app-pbr.
The file:
#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
TARGET_INTERFACE='wan'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_URL="http://www.ipdeny.com/ipblocks/data/countries/ke.zone"
TARGET_DL_FILE_4="/var/pbr_tmp_ke_ip_ranges"
_ret=0
if [ ! -s "$TARGET_DL_FILE_4" ]; then
uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null > "$TARGET_DL_FILE_4"
fi
if [ -s "$TARGET_DL_FILE_4" ]; then
params=
while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
[ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1
fi
return $_ret
1 Like
Oh, sorry, my bad, the logic of bad calls detection was inverted, thanks for testing so quickly, your file is fine, please update to 1.1.7-7.
1 Like
That fixed it! Thank you. Why did the issue not show up on SNAPSHOT?
Is normal the pbr config bellow do a dnsmasq proccess to run at 99% of cpu, when accessing the remote host?
# uci export pbr
config policy
option name 'twitter'
option dest_addr 'file:///data/tools/pbr/lists/twitter.txt'
option interface 'wg0'
option enabled '0'
# /data/tools/pbr/lists/twitter.txt
pi-0-5-0.twitter.com
api-24-0-0.twitter.com
api-26-0-0.twitter.com
api-37-0-0.twitter.com
api-41-0-0.twitter.com
api-stream.twitter.com
api.twitter.com
global.cftls.t.co
pbs.twimg.com
abs.twimg.com
probe.twitter.com
video.twimg.com
twimg.com
twitter.com
t.co
x.com
And If only twimg.com as dest, any *.twimg.com host will be pre-routed to the policy?
Definitely not, especially given that this policy is disabled.
@stangri
Is that normal that with the new version the 70-pbr hotplug disappear?
How it is reloading in case vpn tunnel start later than pbr?
my default gateway is tun0, should I set this interface as procd_wan_interface
Thanks
Edit...
Sorry I didn't see well the reload function before
Just upgraded my package to 1.1.7-7 (from 6-20), and when comparing pbr config files (as I knew there were changes) noticed that the value of nft_set_counter has changed from 1 to 0.
I checked the docs, which say " Use counter option when creating all nft sets" and point to nftables wiki for more info. The wiki just basically says that this enables a counter that counts when the elements have a hit/match.
Fair enough.
The question is: What value (or lack there-of) does setting the config value to 1 provide? If none, I assume 0 is now the default because there is no need for it, and there is a performance hit for enabling?
I also noticed dnsmasq going 100% cpu and crashing when I used a PBR nftset domain and had that domain in uptimekuma being resolved every 60 seconds.
Didn't really have time to troubleshoot it further and just added the IPs in manually and moved on.
Also, I don't know if it will help anyone but I just noticed this comment here: OpenWrt 23.05.4 - Service Release - #258 by brada4
Which made me wonder if I had software offload enabled. (I did)
I ran uci del firewall.@defaults[0].flow_offloading='1', uci commit and I'm monitoring to see if that helps with any of my PBR interface reload issues.
1 Like
sets are not reloaded, only offload states (conntrack -L -u offload) get destroyed in place of being downgraded back to conntrack states. Like NAT LAN or docker0. I have 3 thermometers using fixed source port and taking 1h to recover from frozen state.
Yes, the new versions add the firewall include file instead as part of the uci-defaults script, but you bring up a good point, the include file install should probably be part of the Makefile postinst and prerm scripts.
Yeah, it makes the output of status quite a bit wider without meaningful data unless debugging.
I don't think the performance penalty would be substantial, it's more for the benefit of compact/neater output of status.
PS. There's also nft_rule_counter.
Do either dnsmasq or pbr get restarted frequently? Like every 60 seconds?
dnsmasq was being restarted frequently because it was crashing
Don't really have the logs or the time to look into this fully right now, was thinking I should upgrade to the .4 release of openwrt first in case it's fixed in that, and was putting off troubleshooting until I've done that.
Domain based routing does not work properly after restarting pbr.
After restarting pbr, the domain's ip address is not added to "nft list sets". However, after restarting pbr and manually restarting my dnsmasq, the address is added.
Summary
root@OpenWrt-Jun:~# service pbr restart
Removing routing for 'wan/eth1.10/IPADDRESS' [✓]
Removing routing for 'wg0/10.99.99.2' [✓]
Removing routing for 'netflix/eth1.20/192.168.1.1' [✓]
Restarting dnsmasq [✓]
pbr 1.1.7-7 (fw4 nft file mode) stopped [✓]
Using wan interface (on_start): wan
Found wan gateway (on_start): IPADDRESS
Setting up routing for 'wan/eth1.10/IPADDRESS' [✓]
Setting up routing for 'wg0/10.99.99.2' [✓]
Setting up routing for 'netflix/eth1.20/192.168.1.1' [✓]
Routing 'netflix' via netflix [✓]
Installing fw4 nft file [✓]
pbr 1.1.7-7 monitoring interfaces: wan netflix wg0
pbr 1.1.7-7 (fw4 nft file mode) started with gateways:
wan/eth1.10/IPADDRESS [✓]
wg0/10.99.99.2
netflix/eth1.20/192.168.1.1
root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
set pbr_netflix_4_dst_ip_cfg046ff5 {
type ipv4_addr
flags interval
auto-merge
comment "netflix"
}
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}
root@OpenWrt-Jun:~# dig +noall +stats netflix.com
;; Query time: 20 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sun Sep 08 15:43:09 KST 2024
;; MSG SIZE rcvd: 88
root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
set pbr_netflix_4_dst_ip_cfg046ff5 {
type ipv4_addr
flags interval
auto-merge
comment "netflix"
}
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}
root@OpenWrt-Jun:~# service dnsmasq restart
root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
set pbr_netflix_4_dst_ip_cfg046ff5 {
type ipv4_addr
flags interval
auto-merge
comment "netflix"
}
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}
root@OpenWrt-Jun:~# dig +noall +stats netflix.com
;; Query time: 80 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sun Sep 08 15:43:23 KST 2024
;; MSG SIZE rcvd: 88
root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
set pbr_netflix_4_dst_ip_cfg046ff5 {
type ipv4_addr
flags interval
auto-merge
comment "netflix"
elements = { 44.234.232.238, 44.237.234.25,
44.242.60.85 }
}
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}
Right now I'm using stop and start, not restart.
Greetings.
I am using a custom file with the following content:
#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
TARGET_INTERFACE='tun'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_URL="https://antifilter.download/list/allyouneed.lst"
TARGET_DL_FILE_4="/var/pbr_tmp_rkn_ip_ranges"
_ret=0
if [ ! -s "$TARGET_DL_FILE_4" ]; then
uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null > "$TARGET_DL_FILE_4"
fi
if [ -s "$TARGET_DL_FILE_4" ]; then
params=
while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
[ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1
fi
return $_ret
When I reboot pbr I get the following message:
Running /usr/share/pbr/pbr.user.rkn /etc/rc.common: /usr/share/pbr/pbr.user.rkn: line 742: grep: Argument list too long
[✓]
I understand that this is due to the fact that TARGET_DL_FILE_4="/var/pbr_tmp_rkn_ip_ranges" contains a large number of records? Should I worry that not all subnets will be included in the nft lists?