PBR in 24.10 creates set for custom file after its execution

In the documentation said that nft sets for all network interfaces will be created: https://docs.openwrt.melmac.net/pbr/1.1.8/#ProcessingCustomUserFilesnftmode
I use pbr_myinterface_4_dst_ip_user in my custom script file but there's a problem that this set is not created. PBR creates it AFTER script is being executed.
I tried to to create my own set and forward it to pbr_mark_0x020000 chain manually, but it doesn't exist too.
So after restarting PBR I have this record (I cut most):

        chain pbr_prerouting {
                ip daddr @pbr_myinterface_4_dst_ip_user goto pbr_mark_0x020000
        }
        chain pbr_mark_0x020000 {
                meta mark set meta mark & 0xff02ffff | 0x00020000
                return
        }

But while script execution nothing of this created yet and it fails.
I tried to add sleep 5s to the script but there's no delay, it's ignored somehow.
What else I can do to delay script executed by PBR or maybe there's a way to fix PBR?

This happens in Owrt 24.10 release. In 23.05 it works fine.
23.05 (official build for ax6s): pbr 1.1.6-20 (nft mode), dnsmasq 2.90-2
24.10 (custom with small patches for ax3000t): pbr 1.1.8-r10 (fw4 nft file mode), dnsmasq-full 2.90-r4

P.S. And also pbr after restart (in luci) says "Incompatible custom user file detected" but I cannot find any logs. The file itself can be executed without errors in ssh terminal.

P.P.S. nft list ruleset returns nothing.

Tagging @egc

I ran some tests and looked at the code and that actually is true at least on my 1.1.8-r10

There are two user scripts available and both run.

All chains are made if you enable a userscript.

Just try with the Netflix script when you enable it all user chains are made:

        chain pbr_prerouting {
                ip daddr @pbr_wan_4_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip6 daddr @pbr_wan_6_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip saddr @pbr_wan_4_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip6 saddr @pbr_wan_6_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ether saddr @pbr_wan_4_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ether saddr @pbr_wan_6_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip daddr @pbr_mullvad_se_4_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip6 daddr @pbr_mullvad_se_6_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip saddr @pbr_mullvad_se_4_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip6 saddr @pbr_mullvad_se_6_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ether saddr @pbr_mullvad_se_4_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ether saddr @pbr_mullvad_se_6_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x020000
        }

you might need to do a:

service pbr stop && sleep 10 && service pbr start

and wait about 10 seconds before the chains appear but the netflix set is filled:

table inet fw4 {
        set pbr_wan_4_dst_ip_user {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment ""
                elements = { 23.246.0.0/18 counter packets 0 bytes 0, 37.77.184.0/21 counter packets 0 bytes 0,
                             45.57.0.0/17 counter packets 0 bytes 0, 108.175.32.0/20 counter packets 0 bytes 0,
                             185.2.220.0/22 counter packets 0 bytes 0, 185.9.188.0/22 counter packets 0 bytes 0,
                             192.173.64.0/18 counter packets 0 bytes 0, 198.38.96.0/19 counter packets 0 bytes 0,
                             198.45.48.0/20 counter packets 0 bytes 0, 207.45.72.0/22 counter packets 0 bytes 0,
                             208.75.76.0/22 counter packets 0 bytes 0 }
        }

Have you tested whether the rules work,
for example that they pass through the WAN interface?

I do not have netflix but in the script I added the IP address of ipleak.net and I tested that was routed via the wan.
So it does look like this is working

1 Like

Thank you for helping. But I guess I will not use this not handy way with console :).
I'm going to use UI and some workaround - nft table. I've tried and it seem it works at first glance. Pbr user script creates ntf file with required rules with low priority (99). There's a problem that unchecking the check box in pbr ui doesn't remove nft elements. I made it so the nft file is a link to a file in /tmp so it will be removed at least after reboot. Not perfect but easier to use.

P.S. Don't you know is there a way that when I uncheck pbr user file then pbr could run this or another script with parameter and I will know about this? Or maybe pbr can run some script before applying everything? It will help me to delete nft files and they will not be used if not checked.

I just tried with Netflix script and can see that all chains are created.
But routing via wan doesn't work. (Have stop/started after loading script)
Tested with ipleak.net (95.85.16.212) as you did.
Can you share your setup so I can compare with mine?

I don't know what is wrong with pbr or something else, but it behaviors very strange.
In these examples I have tun2vless interface. change it to your if you will try.
First, I wrote this script and it works just fine:

pbr.user.1
#!/bin/sh

# forum.openwrt.org
# 139.59.210.197

_ret=1

nft "add element inet fw4 pbr_tun2vless_4_dst_ip_user { 139.59.210.197 }" && _ret=0 || _ret=1

return $_ret

Then I unchecked pbr.user.1, checked pbr.youtube.tun2vless.my and restarted pbr (I use luci).

pbr.youtube.tun2vless.my
#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
TARGET_SET='pbr_tun2vless_4_dst_ip_user'
TARGET_TABLE='inet fw4'
TARGET_DL_FILE="/etc/pbr.as.youtube"
TARGET_NFT_FILE="/etc/pbr.list.youtube.nft"
[ -z "$nft" ] && nft="$(command -v nft)"
_ret=1

# took from https://host.io/youtube.com and https://host.io/google.com
whois -h whois.radb.net -- '-i origin AS15169' > "$TARGET_DL_FILE"

printf "add element %s %s { " "$TARGET_TABLE" "$TARGET_SET" > "$TARGET_NFT_FILE"
awk '/^route:\s*([^0-9a-f]+)/{printf $2 ", "}' "$TARGET_DL_FILE" >> "$TARGET_NFT_FILE"
printf " } " >> "$TARGET_NFT_FILE"
echo "" >> "$TARGET_NFT_FILE"
"$nft" -f "$TARGET_NFT_FILE"

_ret=0

return $_ret

Note that this script works with pbr in 23.05 and works in 24.10 if run from sh separately.

If I check this script in pbr and restart (in luci) then "Service Errors" section appears with "Incompatible custom user file detected '/usr/share/pbr/pbr.youtube.tun2vless.my'!" text.

I found the line which causes the problem, it's "$nft" -f "$TARGET_NFT_FILE".
But if I change "$nft" to /usr/sbin/nft (this is the value of nft variable, I checked) then no error after pbr restart. But anyway rules dont' apply.

It's very strange because

  1. nft has to work with both "$nft" and /usr/sbin/nft (and it works in 23.05 and command line in 24.10)
  2. pbr.youtube.tun2vless.my adds rules if I run it from command line
  3. if I check pbr.user.1 and pbr.youtube.tun2vless.my (in luci, in this order) then 139.59.210.197 is added, but youtube IPs are not (and there's no error if I use /usr/sbin/nft).

I have a fairly standard test setup.
Mullvad client with default route via Mullvad

This is the script I test with, nothing else is enabled for PBR.
I added ipchicken.com to the script for testing.

#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
# Credits to https://forum.openwrt.org/u/dscpl for api.hackertarget.com code.
# Credits to https://github.com/kkeker and https://github.com/tophirsch for api.bgpview.io code.

TARGET_INTERFACE='wan'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_ASN='2906'
TARGET_DL_FILE_4="/var/pbr_tmp_AS${TARGET_ASN}.ipv4"
# Uncomment the following line if you enabled ipv6 for pbr and want IPv6 entries added to the IPv6 set
# TARGET_DL_FILE_6="/var/pbr_tmp_AS${TARGET_ASN}.ipv6"
DB_SOURCE='ipinfo.io'
#DB_SOURCE='api.hackertarget.com'
#DB_SOURCE='api.bgpview.io'
REGEX_IPV4='[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\/[0-9]\{1,\}'
REGEX_IPV6='.*::.*'
_ret=0

if [ ! -s "$TARGET_DL_FILE_4" ]; then
	if [ "$DB_SOURCE" = "ipinfo.io" ]; then
		TARGET_URL="https://ipinfo.io/AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed -n "s|\(.*\)/AS${TARGET_ASN}/\($REGEX_IPV4\)\"|\2|p" > "$TARGET_DL_FILE_4"
	fi
	if [ "$DB_SOURCE" = "api.hackertarget.com" ]; then
		TARGET_URL="https://api.hackertarget.com/aslookup/?q=AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed '1d' > "$TARGET_DL_FILE_4"
	fi
	if [ "$DB_SOURCE" = "api.bgpview.io" ]; then
		TARGET_URL="https://api.bgpview.io/asn/${TARGET_ASN}/prefixes"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | jsonfilter -e '@.data.ipv4_prefixes[*].prefix' > "$TARGET_DL_FILE_4"
	fi
fi

if [ -s "$TARGET_DL_FILE_4" ]; then
	params=
	while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
	[ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1

	#egc test rule for ipleak.net 95.85.16.212 and whatismyip.com: 172.66.40.87, 172.66.43.169 , ipchicken: 104.26.7.112, 104.26.6.112, 172.67.68.101
	# NOTE these are subjetc to change so alwasy do an ndslookup to verify currenlty added ipchicken.com
	nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { 104.26.7.112, 104.26.6.112, 172.67.68.101 }"

fi

if [ -n "$TARGET_DL_FILE_6" ] && [ ! -s "$TARGET_DL_FILE_6" ]; then
	if [ "$DB_SOURCE" = "ipinfo.io" ]; then
		TARGET_URL="https://ipinfo.io/AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed -n "s|\(.*\)/AS${TARGET_ASN}/\($REGEX_IPV6\)\"|\2|p" > "$TARGET_DL_FILE_6"
	fi
fi
if [ -s "$TARGET_DL_FILE_6" ]; then
	params=
	while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_6"
	[ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_6 { $params }" || _ret=1
fi

return $_ret

nftl list ruleset shows all chains are made and targeted and when surfing to ipchicken.com from a LAN client (this is prerouting so only works for a LAN client) I see my WAN ip address.

        chain pbr_prerouting {
                ip daddr @pbr_wan_4_dst_ip_user counter packets 33 bytes 5127 goto pbr_mark_0x010000
                ip6 daddr @pbr_wan_6_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip saddr @pbr_wan_4_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip6 saddr @pbr_wan_6_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ether saddr @pbr_wan_4_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ether saddr @pbr_wan_6_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x010000
                ip daddr @pbr_mullvad_se_4_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip6 daddr @pbr_mullvad_se_6_dst_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip saddr @pbr_mullvad_se_4_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ip6 saddr @pbr_mullvad_se_6_src_ip_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ether saddr @pbr_mullvad_se_4_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x020000
                ether saddr @pbr_mullvad_se_6_src_mac_user counter packets 0 bytes 0 goto pbr_mark_0x020000
        }

You can see the counter for ipchicken.com 04.26.6.112 counter packets 18 bytes 4540 being hit

        set pbr_wan_4_dst_ip_user {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment ""
                elements = { 23.246.0.0/18 counter packets 0 bytes 0, 37.77.184.0/21 counter packets 0 bytes 0,
                             45.57.0.0/17 counter packets 0 bytes 0, 104.26.6.112 counter packets 18 bytes 4540,
                             104.26.7.112 counter packets 0 bytes 0, 108.175.32.0/20 counter packets 0 bytes 0,
                             172.67.68.101 counter packets 0 bytes 0, 185.2.220.0/22 counter packets 0 bytes 0,
                             185.9.188.0/22 counter packets 0 bytes 0, 192.173.64.0/18 counter packets 0 bytes 0,
                             198.38.96.0/19 counter packets 0 bytes 0, 198.45.48.0/20 counter packets 0 bytes 0,
                             207.45.72.0/22 counter packets 0 bytes 0, 208.75.76.0/22 counter packets 0 bytes 0 }

when I have more time I will take a look into your script

I took a very quick look and did not test anything but as you want to run an nft file did you follow the rules:
https://wiki.nftables.org/wiki-nftables/index.php/Scripting

e.g. start with:

#!/usr/sbin/nft -f

I've tried, adding !/usr/sbin/nft -f doesn't help. But now I have another error for some reason.

Failed to install fw4 nft file '/var/run/pbr.nft'!
Errors encountered, please check the [README]

Note that it says error in /var/run/pbr.nft although I have /etc/pbr.list.youtube.nft file and directly call nft to apply it. But it appears in pbr's nft. This is it:

/var/run/pbr.nft
#!/usr/sbin/nft -f

add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000  mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000  mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add set inet fw4 pbr_wan_4_dst_ip_user { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_wan_4_dst_ip_user  goto pbr_mark_0x010000
add set inet fw4 pbr_wan_4_src_ip_user { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "";}
add rule inet fw4 pbr_prerouting ip saddr @pbr_wan_4_src_ip_user  goto pbr_mark_0x010000
add set inet fw4 pbr_wan_4_src_mac_user { type ether_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment ""; }
add rule inet fw4 pbr_prerouting ether saddr @pbr_wan_4_src_mac_user  goto pbr_mark_0x010000
add set inet fw4 pbr_tun2vless_4_dst_ip_user { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun2vless_4_dst_ip_user  goto pbr_mark_0x020000
add set inet fw4 pbr_tun2vless_4_src_ip_user { type ipv4_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment "";}
add rule inet fw4 pbr_prerouting ip saddr @pbr_tun2vless_4_src_ip_user  goto pbr_mark_0x020000
add set inet fw4 pbr_tun2vless_4_src_mac_user { type ether_addr;  		 auto-merge; 		 		 flags interval; 		 		 policy performance; 		 		 comment ""; }
add rule inet fw4 pbr_prerouting ether saddr @pbr_tun2vless_4_src_mac_user  goto pbr_mark_0x020000
-f /etc/pbr.list.youtube.nft

It already contents #!/usr/sbin/nft -f so I removed it. This is my current script I'm testing and which gives the error above:

/etc/pbr.youtube.tun2vless.my
#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
TARGET_SET='pbr_tun2vless_4_dst_ip_user'
TARGET_TABLE='inet fw4'
TARGET_DL_FILE="/etc/pbr.as.youtube"
TARGET_NFT_FILE="/etc/pbr.list.youtube.nft"
[ -z "$nft" ] && nft="$(command -v nft)"
_ret=1

# took from https://host.io/youtube.com and https://host.io/google.com
whois -h whois.radb.net -- '-i origin AS15169' > "$TARGET_DL_FILE"

> "$TARGET_NFT_FILE"
echo "" >> "$TARGET_NFT_FILE"
#echo "#!/usr/sbin/nft -f" >> "$TARGET_NFT_FILE"
#echo "" >> "$TARGET_NFT_FILE"
printf "add element %s %s { " "$TARGET_TABLE" "$TARGET_SET" >> "$TARGET_NFT_FILE"
awk '/^route:\s*([^0-9a-f]+)/{printf $2 ", "}' "$TARGET_DL_FILE" >> "$TARGET_NFT_FILE"
echo " } " >> "$TARGET_NFT_FILE"
nft -f "$TARGET_NFT_FILE"

_ret=0

return $_ret

It seems I understood how it works.
New PBR somehow replaces nft with it's own and when I call nft then pbr takes other part of the command and appends it to its own /var/run/pbr.nft nft file.
And if I use "$nft" -f text in custom user script it looks for exactly this substring (even if the line commented) and if find then show error Incompatible custom user file detected.
And if I use /usr/sbin/nft then pbr... I don't know what he does, maybe fw is not started yet, or it will clear everything later and doesn't show errors.

Anyway it seems this is the working solution:

/etc/pbr.youtube.tun2vless.my
#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
TARGET_SET='pbr_tun2vless_4_dst_ip_user'
TARGET_TABLE='inet fw4'
TARGET_DL_FILE="/etc/pbr.as.youtube"
TARGET_NFT_FILE="/etc/pbr.list.youtube.nft"
[ -z "$nft" ] && nft="$(command -v nft)"
_ret=1

# took from https://host.io/youtube.com and https://host.io/google.com
whois -h whois.radb.net -- '-i origin AS15169' > "$TARGET_DL_FILE"

> "$TARGET_NFT_FILE"
echo "" >> "$TARGET_NFT_FILE"
#echo "#!/usr/sbin/nft -f" >> "$TARGET_NFT_FILE"
#echo "" >> "$TARGET_NFT_FILE"
printf "add element %s %s { " "$TARGET_TABLE" "$TARGET_SET" >> "$TARGET_NFT_FILE"
awk '/^route:\s*([^0-9a-f]+)/{printf $2 ", "}' "$TARGET_DL_FILE" >> "$TARGET_NFT_FILE"
echo " } " >> "$TARGET_NFT_FILE"
nft "include \"$TARGET_NFT_FILE\""
#/usr/sbin/nft -f "$TARGET_NFT_FILE"
#"$nft" "include \"$TARGET_NFT_FILE\""

_ret=0

return $_ret

And this how /var/run/pbr.nft looks with it (it has include at the end):

/var/run/pbr.nft

#!/usr/sbin/nft -f

add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add set inet fw4 pbr_wan_4_dst_ip_user { type ipv4_addr; auto-merge; flags interval; policy performance; comment "";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_wan_4_dst_ip_user goto pbr_mark_0x010000
add set inet fw4 pbr_wan_4_src_ip_user { type ipv4_addr; auto-merge; flags interval; policy performance; comment "";}
add rule inet fw4 pbr_prerouting ip saddr @pbr_wan_4_src_ip_user goto pbr_mark_0x010000
add set inet fw4 pbr_wan_4_src_mac_user { type ether_addr; auto-merge; flags interval; policy performance; comment ""; }
add rule inet fw4 pbr_prerouting ether saddr @pbr_wan_4_src_mac_user goto pbr_mark_0x010000
add set inet fw4 pbr_tun2vless_4_dst_ip_user { type ipv4_addr; auto-merge; flags interval; policy performance; comment "";}
add rule inet fw4 pbr_prerouting ip daddr @pbr_tun2vless_4_dst_ip_user goto pbr_mark_0x020000
add set inet fw4 pbr_tun2vless_4_src_ip_user { type ipv4_addr; auto-merge; flags interval; policy performance; comment "";}
add rule inet fw4 pbr_prerouting ip saddr @pbr_tun2vless_4_src_ip_user goto pbr_mark_0x020000
add set inet fw4 pbr_tun2vless_4_src_mac_user { type ether_addr; auto-merge; flags interval; policy performance; comment ""; }
add rule inet fw4 pbr_prerouting ether saddr @pbr_tun2vless_4_src_mac_user goto pbr_mark_0x020000
include "/etc/pbr.list.youtube.nft"

My mistake was I didn't notice that in /usr/share/pbr/pbr.user.aws example now it's nft instead of "$nft" that was in OpenWrt 23.05.

1 Like

I took your script as is and ran it.
All chains are created.
Tested with ipchicken.com but it didn't show wan ip number but my vpn ip number.

Every time I restart pbr, (Version 1.1.8-r10) I get this in log.

Tue Feb 18 20:18:40 2025 daemon.err dnsmasq[1]: nftset inet fw4 pbr_wan_4_dst_ip_cfg096ff5 Error: No such file or directory
Tue Feb 18 20:18:40 2025 daemon.err dnsmasq[1]: nftset inet fw4 pbr_wan_4_dst_ip_cfg096ff5 Error: No such file or directory
Tue Feb 18 20:18:40 2025 daemon.err dnsmasq[1]: nftset inet fw4 pbr_wan_6_dst_ip_cfg096ff5 Error: No such file or directory
Tue Feb 18 20:18:40 2025 daemon.err dnsmasq[1]: nftset inet fw4 pbr_wan_6_dst_ip_cfg096ff5 Error: No such file or directory

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.