Policy-Based-Routing (pbr) package discussion

odhiambo · 2024-09-03T16:09:21.436Z

That fixed it! Thank you. Why did the issue not show up on SNAPSHOT?

powtrix · 2024-09-04T02:35:34.509Z

Is normal the pbr config bellow do a dnsmasq proccess to run at 99% of cpu, when accessing the remote host?

# uci export pbr
config policy
	option name 'twitter'
	option dest_addr 'file:///data/tools/pbr/lists/twitter.txt'
	option interface 'wg0'
	option enabled '0'

# /data/tools/pbr/lists/twitter.txt 
pi-0-5-0.twitter.com
api-24-0-0.twitter.com
api-26-0-0.twitter.com
api-37-0-0.twitter.com
api-41-0-0.twitter.com
api-stream.twitter.com
api.twitter.com
global.cftls.t.co
pbs.twimg.com
abs.twimg.com
probe.twitter.com
video.twimg.com
twimg.com
twitter.com
t.co
x.com

And If only twimg.com as dest, any *.twimg.com host will be pre-routed to the policy?

stangri · 2024-09-04T20:26:27.604Z

powtrix:

Is normal the pbr config bellow do a dnsmasq proccess to run at 99% of cpu, when accessing the remote host?

Definitely not, especially given that this policy is disabled.

pesa1234 · 2024-09-06T07:35:26.927Z

@stangri
Is that normal that with the new version the 70-pbr hotplug disappear?
How it is reloading in case vpn tunnel start later than pbr?

my default gateway is tun0, should I set this interface as procd_wan_interface

Thanks

Edit...

Sorry I didn't see well the reload function before

Edrikk · 2024-09-07T01:37:16.570Z

Just upgraded my package to 1.1.7-7 (from 6-20), and when comparing pbr config files (as I knew there were changes) noticed that the value of nft_set_counter has changed from 1 to 0.

I checked the docs, which say " Use counter option when creating all nft sets" and point to nftables wiki for more info. The wiki just basically says that this enables a counter that counts when the elements have a hit/match.

Fair enough.

The question is: What value (or lack there-of) does setting the config value to 1 provide? If none, I assume 0 is now the default because there is no need for it, and there is a performance hit for enabling?

NoahO · 2024-09-07T05:40:45.078Z

I also noticed dnsmasq going 100% cpu and crashing when I used a PBR nftset domain and had that domain in uptimekuma being resolved every 60 seconds.

Didn't really have time to troubleshoot it further and just added the IPs in manually and moved on.

Also, I don't know if it will help anyone but I just noticed this comment here: OpenWrt 23.05.4 - Service Release - #258 by brada4

Which made me wonder if I had software offload enabled. (I did)

I ran uci del firewall.@defaults[0].flow_offloading='1', uci commit and I'm monitoring to see if that helps with any of my PBR interface reload issues.

brada4 · 2024-09-07T05:45:48.960Z

sets are not reloaded, only offload states (conntrack -L -u offload) get destroyed in place of being downgraded back to conntrack states. Like NAT LAN or docker0. I have 3 thermometers using fixed source port and taking 1h to recover from frozen state.

stangri · 2024-09-07T20:13:50.518Z

pesa1234:

Is that normal that with the new version the 70-pbr hotplug disappear?

Yes, the new versions add the firewall include file instead as part of the uci-defaults script, but you bring up a good point, the include file install should probably be part of the Makefile postinst and prerm scripts.

Edrikk:

Just upgraded my package to 1.1.7-7 (from 6-20), and when comparing pbr config files (as I knew there were changes) noticed that the value of nft_set_counter has changed from 1 to 0.

Yeah, it makes the output of status quite a bit wider without meaningful data unless debugging.

Edrikk:

The question is: What value (or lack there-of) does setting the config value to 1 provide? If none, I assume 0 is now the default because there is no need for it, and there is a performance hit for enabling?

I don't think the performance penalty would be substantial, it's more for the benefit of compact/neater output of status.

PS. There's also nft_rule_counter.

NoahO:

I also noticed dnsmasq going 100% cpu and crashing when I used a PBR nftset domain and had that domain in uptimekuma being resolved every 60 seconds.

Do either dnsmasq or pbr get restarted frequently? Like every 60 seconds?

NoahO · 2024-09-08T00:36:48.810Z

dnsmasq was being restarted frequently because it was crashing

Don't really have the logs or the time to look into this fully right now, was thinking I should upgrade to the .4 release of openwrt first in case it's fixed in that, and was putting off troubleshooting until I've done that.

odkrys · 2024-09-08T06:59:29.918Z

Domain based routing does not work properly after restarting pbr.

After restarting pbr, the domain's ip address is not added to "nft list sets". However, after restarting pbr and manually restarting my dnsmasq, the address is added.

Summary

root@OpenWrt-Jun:~# service pbr restart
Removing routing for 'wan/eth1.10/IPADDRESS' [✓]
Removing routing for 'wg0/10.99.99.2' [✓]
Removing routing for 'netflix/eth1.20/192.168.1.1' [✓]
Restarting dnsmasq [✓]
pbr 1.1.7-7 (fw4 nft file mode) stopped [✓]
Using wan interface (on_start): wan
Found wan gateway (on_start): IPADDRESS
Setting up routing for 'wan/eth1.10/IPADDRESS' [✓]
Setting up routing for 'wg0/10.99.99.2' [✓]
Setting up routing for 'netflix/eth1.20/192.168.1.1' [✓]
Routing 'netflix' via netflix [✓]
Installing fw4 nft file [✓]
pbr 1.1.7-7 monitoring interfaces: wan netflix wg0
pbr 1.1.7-7 (fw4 nft file mode) started with gateways:
wan/eth1.10/IPADDRESS [✓]
wg0/10.99.99.2
netflix/eth1.20/192.168.1.1

root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
        set pbr_netflix_4_dst_ip_cfg046ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "netflix"
        }
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}

root@OpenWrt-Jun:~# dig +noall +stats netflix.com
;; Query time: 20 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sun Sep 08 15:43:09 KST 2024
;; MSG SIZE  rcvd: 88

root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
        set pbr_netflix_4_dst_ip_cfg046ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "netflix"
        }
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}

root@OpenWrt-Jun:~# service dnsmasq restart

root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
        set pbr_netflix_4_dst_ip_cfg046ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "netflix"
        }
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}

root@OpenWrt-Jun:~# dig +noall +stats netflix.com
;; Query time: 80 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sun Sep 08 15:43:23 KST 2024
;; MSG SIZE  rcvd: 88

root@OpenWrt-Jun:~# nft list sets
table inet fw4 {
        set pbr_netflix_4_dst_ip_cfg046ff5 {
                type ipv4_addr
                flags interval
                auto-merge
                comment "netflix"
                elements = { 44.234.232.238, 44.237.234.25,
                             44.242.60.85 }
        }
}
table ip mangle {
}
table ip nat {
}
table ip filter {
}

Right now I'm using stop and start, not restart.

kot-nikot · 2024-09-08T08:57:25.255Z

Greetings.
I am using a custom file with the following content:

#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh

TARGET_INTERFACE='tun'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_URL="https://antifilter.download/list/allyouneed.lst"
TARGET_DL_FILE_4="/var/pbr_tmp_rkn_ip_ranges"
_ret=0

if [ ! -s "$TARGET_DL_FILE_4" ]; then
 uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null > "$TARGET_DL_FILE_4"
fi


if [ -s "$TARGET_DL_FILE_4" ]; then
 params=
 while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
 [ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1
fi

return $_ret

When I reboot pbr I get the following message:

Running /usr/share/pbr/pbr.user.rkn /etc/rc.common: /usr/share/pbr/pbr.user.rkn: line 742: grep: Argument list too long
[✓]

I understand that this is due to the fact that TARGET_DL_FILE_4="/var/pbr_tmp_rkn_ip_ranges" contains a large number of records? Should I worry that not all subnets will be included in the nft lists?

egc · 2024-09-08T14:13:09.474Z

Can you try with changing the following config parameters:

	option procd_reload_delay '10'
	option procd_boot_delay '30'

That seems to work for me

egc · 2024-09-08T14:28:44.297Z

@stangri the handling of the wgserver is nice but there are setups where you want a WG server with a listen port (e.g. site-to site setup) to be able to use as an interface you want to route to.
But as it is now it is not possible to deal with these situations because it is hard coded.
Consider making the whole code optional with a config switch or if the interface is added as "supported interface" make it possible to actually use as supported interface, I am using that now on my test setup, do not exit the interface code for WG interfaces with a listen port if they are added as "supported interface":

--- pbr-1.1.7-7-egc-1.bash	2024-09-08 12:34:40.096781205 +0200
+++ pbr-1.1.7-7-egc-2.bash	2024-09-08 13:58:54.097756200 +0200
@@ -1859,7 +1859,9 @@
 				fi
 			;;
 		esac
-		return 0
+		if ! str_contains_word "$supported_interface" "$iface"; then
+			return 0
+		fi
 	fi
 
 	is_supported_interface "$iface" || return 0

stangri · 2024-09-08T16:43:08.190Z

odkrys:

Domain based routing does not work properly after restarting pbr.

How do you test?

odkrys:

After restarting pbr, the domain's ip address is not added to "nft list sets".

Domain needs to be resolved to be added to the nft set. Dnsmasq also periodically (when idle) resolves domains from the config and adds them to the nft set, but it does it immediately on resolution request.

kot-nikot:

I am using a custom file with the following content:

I'll test locally and get back to you.

egc:

the handling of the wgserver is nice but there are setups where you want a WG server with a listen port (e.g. site-to site setup) to be able to use as an interface you want to route to.

Thank you for bringing this to my attention, I'll implement your solution.

odkrys · 2024-09-09T03:16:55.283Z

egc:

That seems to work for me

It doesn't work for me.

stangri:

How do you test?

Restart in gui. Clear cache using "ipconfig /flushdns" command on windows laptop and find dns using "nslookup netflix.com".

stangri:

it does it immediately on resolution request.

But it didn't work for me. The ip list was empty.

I modified pbr's init.d file slightly to force dnsmasq to restart on restart. It works now.

line 2166

start_service() {

#! nft_file 'exists' && resolver 'compare_hash' && resolver 'restart'
to
resolver 'compare_hash' && resolver 'restart'

stangri · 2024-09-09T20:31:27.021Z

kot-nikot:

I am using a custom file with the following content:

Fixed in 1.1.7-9.

egc:

the handling of the wgserver is nice but there are setups where you want a WG server with a listen port (e.g. site-to site setup) to be able to use as an interface you want to route to.

Fixed as per your solution in 1.1.7-9.

odkrys:

After restarting pbr, the domain's ip address is not added to "nft list sets".

Thanks again for bringing this up, I've reworked how/when dnsmasq is restarted by pbr in 1.1.7-9, however, if fw4 is restarted (not reloaded), without restarting dnsmasq, it would also fail to add resolved IPs to nft sets. Since because of the fw4 nft file support, the pbr is no longer reloaded/restarted on fw4 reload/restart, I need to think on how to best implement dnsmasq restart on fw4 restart.

powtrix · 2024-09-11T00:43:42.673Z

The luci-app-pbr-1.1.7-9 shows iptables as dep

10000495011080×2340 276 KB

10000495021080×2340 298 KB

stangri · 2024-09-11T20:42:00.779Z

powtrix:

The luci-app-pbr-1.1.7-9 shows iptables as dep

Can't reproduce:

edge620 in /tmp # opkg depends luci-app-pbr
luci-app-pbr depends on:
	libc
	luci-base
	jsonfilter
	pbr

mvi · 2024-09-12T13:42:54.223Z

I have custom script for pbr v1.1.1 where the main command to load the list is like this "nft -f file_with_rules"
On the new version of the pbr (1.1.6) this command does not work.
should i rewrite the command as in the example file pbr.user.aws - "nft add element ..." and why?

oliverwalkerjp · 2024-09-13T03:37:59.944Z

I have been using pbr for a while now to route to my vm's with vpns. suddenly stop working.

when check the status service firewall status here is the result

active with no instances

when i restart server firewall restart here is the result

ubus rule (ubus:https-dns-proxy[instance1] rule 1) option 'src' specifies invalid value 'lan'
ubus rule (ubus:https-dns-proxy[instance1] rule 1) skipped due to invalid options
ubus redirect (ubus:https-dns-proxy[instance1] redirect 0) option 'src' specifies invalid value 'lan'
ubus redirect (ubus:https-dns-proxy[instance1] redirect 0) skipped due to invalid options
Section @redirect[0] (XYZ.lan) is disabled, ignoring section
Section @redirect[1] (vid100) external address range cannot be determined, disabling reflection
Section @redirect[1] (vid100) external address range cannot be determined, disabling reflection
Section @redirect[2] (vid200) external address range cannot be determined, disabling reflection
Section @redirect[2] (vid200) external address range cannot be determined, disabling reflection
Section @redirect[3] (vid300) external address range cannot be determined, disabling reflection
Section @redirect[3] (vid300) external address range cannot be determined, disabling reflection
Section @redirect[4] (vid400) external address range cannot be determined, disabling reflection
Section @redirect[4] (vid400) external address range cannot be determined, disabling reflection
Section @redirect[5] (vid500) external address range cannot be determined, disabling reflection
Section @redirect[5] (vid500) external address range cannot be determined, disabling reflection
Section @redirect[6] (vid110) is disabled, ignoring section
Section @redirect[7] (vid210) is disabled, ignoring section
Section @redirect[8] (vid310) is disabled, ignoring section
Section openclash specifies unreachable path '/var/etc/openclash.include', ignoring section
Automatically including '/usr/share/nftables.d/ruleset-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/table-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/dstnat_lan/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_forward/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_input/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_output/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_postrouting/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_prerouting/30-pbr.nft'

Please, What is stopping the firewalls from responding correctly

What is the possible solutions to this.