Policy-Based-Routing (pbr) package discussion

This is the version which might work:

#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh
# Credits to https://forum.openwrt.org/u/dscpl for api.hackertarget.com code.
# Credits to https://github.com/kkeker and https://github.com/tophirsch for api.bgpview.io code.

TARGET_INTERFACE='wan'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_ASN='2906'
TARGET_DL_FILE_4="/var/pbr_tmp_AS${TARGET_ASN}.ipv4"
# Uncomment the following line if you enabled ipv6 for pbr and want IPv6 entries added to the IPv6 set
# TARGET_DL_FILE_6="/var/pbr_tmp_AS${TARGET_ASN}.ipv6"
DB_SOURCE='ipinfo.io'
#DB_SOURCE='api.hackertarget.com'
#DB_SOURCE='api.bgpview.io'
REGEX_IPV4='[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\/[0-9]\{1,\}'
REGEX_IPV6='.*::.*'
_ret=0

if [ ! -s "$TARGET_DL_FILE_4" ]; then
	if [ "$DB_SOURCE" = "ipinfo.io" ]; then
		TARGET_URL="https://ipinfo.io/AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed -n "s|\(.*\)/AS${TARGET_ASN}/\($REGEX_IPV4\)\"|\2|p" > "$TARGET_DL_FILE_4"
	fi
	if [ "$DB_SOURCE" = "api.hackertarget.com" ]; then
		TARGET_URL="https://api.hackertarget.com/aslookup/?q=AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed '1d' > "$TARGET_DL_FILE_4"
	fi
	if [ "$DB_SOURCE" = "api.bgpview.io" ]; then
		TARGET_URL="https://api.bgpview.io/asn/${TARGET_ASN}/prefixes"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | jsonfilter -e '@.data.ipv4_prefixes[*].prefix' > "$TARGET_DL_FILE_4"
	fi
fi

if [ -s "$TARGET_DL_FILE_4" ]; then
	params=
	while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
	nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1
fi

if [ -n "$TARGET_DL_FILE_6" ] && [ ! -s "$TARGET_DL_FILE_6" ]; then
	if [ "$DB_SOURCE" = "ipinfo.io" ]; then
		TARGET_URL="https://ipinfo.io/AS${TARGET_ASN}"
		uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null | sed -n "s|\(.*\)/AS${TARGET_ASN}/\($REGEX_IPV6\)\"|\2|p" > "$TARGET_DL_FILE_6"
	fi
fi
if [ -s "$TARGET_DL_FILE_6" ]; then
	params=
	while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_6"
	nft "add element $TARGET_TABLE $TARGET_NFTSET_6 { $params }" || _ret=1
fi

return $_ret

@stangri both pbr.user.netflix and pbr.user.aws need to change:
$p to $params in all lines with

nft "add element $TARGET_TABLE $TARGET_NFTSET_X { $p }" || _ret=1

1 Like

Even thought I have the watchdog running which keeps the wg tunnel up and running, I find that PBR some times stops working unless I restart PBR from the GUI. Is there a way to automatically restart PBR every night to ensure it's working correctly?

I'm running 1.1.4-16.

Edit: Answering my own question.. I have set up a cron job in "scheduled services" to start the pbr service at 2am every night, lets see how I get on with that. :slight_smile:

image

Thank you, this is a brilliant idea, I'll try to experiment with this!

The reason I'm not using built in functions is that if the VPN tunnel is set to be default gateway, the built in functions will return that tunnel interface/gateway as WAN.

@odhiambo @egc thank you for the report and the fix accordingly, the pbr 1.1.6-17 has the user files updated with @egc fix and they should be working properly now.

The speed difference is due to two things:

  1. The previous user files added one address at a time to the nft set, which was a time-consuming operation, this new version adds all the addresses from the downloaded files in one command, which is much faster
  2. This new version uses the nft_file mode by default, so instead of executing actual nft commands when policies, dns policies and user files are processed, one file with the nft commands is created which is then added to the OpenWrt firewall.
2 Likes

I believe it's safer to use /etc/init.d/pbr restart instead of invoking service call in the cron.

I'd love to address the underlaying issue tho, so if you could elaborate on "stops working" a bit, maybe I can suggest additional diagnosing steps.

Yes Im all for trying to resolve the issue rather then having a workaround.

The setup:
PBR is configured to send all traffic from 2 given subnets (192.168.2.0/24 and 192.168.3.0/24 through to the wg tunnel.. (the source of these 2 subnets are 2 WAPs (running DD-WRT) with separate VAPs where all traffic is routed to the main router (Openwrt SNAPSHOT (r26915-7a96d36188)).

image

Wireguard tunnel permanently set up against the endpoint, and last checked this was working about week or so ago (not used very often, but I am happy to monitor more frequently going forward if it helps.

I tried connecting today and upon checking my IP I got my WAN IP rather than the WG IP, so no traffic appeared to be routed through through wg0 by PBR.

I restarted PBR through the GUI and it started working again (got correct internet IP).

I am happy to remove the cron job and monitor going forward - what I would need from you is what to look out for, or provide, in the event it happens again.. I will then leave it as it is until you get a chance to review my findings and make suggestions as to what to try.

Can you try upgrading to the latest 1.1.6 version from https://dev.melmac.net/repo/ ?

It should be more resistant to external firewall changes which is what I think may be happening.

If you could update to the latest 1.1.6 and in this incorrect flow of packets capture the full output of service pbr status, that would be helpful.

1 Like

That didnt go to plan. :slight_smile:

Whats the best way to go back to 1.1.4-16? 1.1.6 will have to wait until Im ready to upgrade openwrt it seems.


root@RPi5:~# opkg install http://dev.melmac.net/repo/pbr_1.1.6-17_all.ipk
Downloading http://dev.melmac.net/repo/pbr_1.1.6-17_all.ipk
Upgrading pbr on root from 1.1.4-r16 to 1.1.6-17...
Stopping pbr service... OK
Removing rc.d symlink for pbr... OK
Configuring pbr.
//usr/lib/opkg/info/pbr.postinst: /etc/uci-defaults/91-pbr-nft: line 17: check_agh_ipset: not found
//usr/lib/opkg/info/pbr.postinst: /etc/uci-defaults/91-pbr-nft: line 20: check_dnsmasq_ipset: not found
Setting resolver_set to 'none'... [✓]
Installing rc.d symlink for pbr... OK
Using wan interface (on_start): wan
Found wan gateway (on_start): 84.65.64.1
Setting up routing for 'wan/pppoe-wan/84.11.14.1' [✓]
Setting up routing for 'wg0/10.8.0.5' [✓]
Routing 'Route R7800 through Wireguard' via wg0 [✓]
Routing 'Route AC68U through Wireguard' via wg0 [✓]
pbr 1.1.6-17 monitoring interfaces: wan wg0
pbr 1.1.6-17 (nft mode) started with gateways:
wan/pppoe-wan/84.11.61.1 [✓]
wg0/10.8.0.5
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-reject
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-reject6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-log
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-log6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-crypto-crc32c
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nft-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nft-nat
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-ipt
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-ipt-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-ipt-ipopt
 * resolve_conffiles: Existing conffile /etc/config/pbr is different from the conffile in the new package. The new conffile will be placed at /etc/config/pbr-opkg.

Most of the times that is no problem

I also updated and have the same log and it works none the less but YMMV.

I do copy my own settings from /etc/config/pbr to /etc/config/pbr-opkg`.
Rename pbr -> pbr.old
Rename pbr-opkg -> pbr and then enable pbr again and check the settings

You should also update luci-pbr

1 Like

Im back up and running, sort of.. PBR routes traffic according to rules, but GUI is reporting it as not running:

image

service pbr status and openwrt log also indicates it is running..

root@RPi5:~# service pbr status

pbr - environment
pbr 1.1.6-17 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-nftset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile

pbr chains - policies
        chain pbr_forward { # handle 40
        }
        chain pbr_input { # handle 41
        }
        chain pbr_output { # handle 42
        }
        chain pbr_postrouting { # handle 44
        }
        chain pbr_prerouting { # handle 43
                ip saddr 192.168.2.0/24 goto pbr_mark_0x020000 comment "Route R7800 through Wireguard" # handle 583
                ip saddr 192.168.3.0/24 goto pbr_mark_0x020000 comment "Route AC68U through Wireguard" # handle 584
        }
        chain pbr_dstnat_lan { # handle 39
        }

pbr chains - marking
        chain pbr_mark_0x010000 { # handle 577
                meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 578
                return # handle 579
        }
        chain pbr_mark_0x020000 { # handle 580
                meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 581
                return # handle 582
        }

pbr nft sets

IPv4 table 256 route: default via 84.XX.XX.1 dev pppoe-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.8.0.5 dev wg0
IPv4 table 257 rule(s):
29998:  from all fwmark 0x20000/0xff0000 lookup pbr_wg0

Yes that is a known problem just refresh your screen e.g. ctrl+F5

No amount of ctrl+F5 or incognito seems to show it as running.

Thanks, this is fixed in 1.1.6-18.

For the WebUI to function, you need to update the luci-app-pbr as well (like @egc already suggested).

1 Like

In my TP-Link Archer C7v5 running 23.05.4, and pbr-1.1.6-18, I am having a problem with a custom file that @stangri graciously did for me. The error:

root@Archer-C7-v5-EU:~/opkg# /etc/init.d/pbr reload
Using wan interface (on_start): wan
Found wan gateway (on_start): 192.168.1.1
Setting up routing for 'wan/eth0.2/192.168.1.1' [✓]
Setting up routing for 'nordvpntun0/tun0/10.100.0.2' [✓]
Routing 'amzn' via wan [✓]
Running /usr/share/pbr/pbr.user.ke.lst /var/pbr_tmp_ke_ip_ranges.nft:1:46-48: Error: syntax error, unexpected add
add element inet fw4 pbr_wan_4_dst_ip_user { add element }  { 41.57.96.0/20, 41.72.160.0/19, 41.75.144.0/20, 41.76.168.0/21, 41.76.184.0/21, 41.78.24.0/22, 41.79.8.0/22, 41.79.168.0/22, 41.79.228.0/22, 41.79.252.0/22, 41.80.0.0/15, 41.89.0.0/16, 41.90.0.0/16, 41.138.240.0/20, 41.139.128.0/17, 41.191.192.0/21, 41.203.208.0/20, 41.204.160.0/19, 41.206.32.0/19, 41.207.64.0/19, 41.207.96.0/19, 41.209.0.0/18, 41.212.0.0/17, 41.215.0.0/17, 41.215.128.0/20, 41.217.220.0/22, 41.220.112.0/20, 41.222.8.0/21, 41.222.160.0/21, 41.223.56.0/22, 41.223.148.0/22, 41.242.0.0/22, 41.242.4.0/22, 62.8.64.0/19, 62.12.112.0/21, 62.24.96.0/19, 77.220.0.0/19, 80.72.96.0/20, 80.88.4.0/23, 87.255.96.0/19, 102.0.0.0/13, 102.22.108.0/22, 102.22.208.0/21, 102.23.136.0/22, 102.67.152.0/22, 102.68.20.0/24, 102.68.21.0/24, 102.68.76.0/22, 102.68.141.0/24, 102.68.142.0/24, 102.69.224.0/21, 102.69.232.0/22, 102.69.239.0/24, 102.130.102.0/24, 102.134.129.0/24, 102.135.168.0/21, 102.140.192.0/18, 102.164.52.0/22, 102.164.56.0/22, 102.164.60.0/
                                             ^^^
[✗]
pbr 1.1.6-18 monitoring interfaces: wan nordvpntun0
pbr 1.1.6-18 (nft mode) started with gateways:
wan/eth0.2/192.168.1.1
nordvpntun0/tun0/10.100.0.2 [✓]
ERROR: Error running custom user file '/usr/share/pbr/pbr.user.ke.lst'!
root@Archer-C7-v5-EU:~/opkg#

And the custom file:

#!/bin/sh

TARGET_SET='pbr_wan_4_dst_ip_user'
TARGET_IPSET='pbr_wan_4_dst_net_user'
TARGET_TABLE='inet fw4'
TARGET_URL="http://www.ipdeny.com/ipblocks/data/countries/ke.zone"
TARGET_DL_FILE="/var/pbr_tmp_ke_ip_ranges"
TARGET_NFT_FILE="/var/pbr_tmp_ke_ip_ranges.nft"
[ -z "$nft" ] && nft="$(command -v nft)"
_ret=1

if [ ! -s "$TARGET_DL_FILE" ]; then
        uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null > "$TARGET_DL_FILE"
fi

if [ -s "$TARGET_DL_FILE" ]; then
        if ipset -q list "$TARGET_IPSET" >/dev/null 2>&1; then
                if awk -v ipset="$TARGET_IPSET" '{print "add " ipset " " $1}' "$TARGET_DL_FILE" | ipset restore -!; then
                        _ret=0
                fi
        elif [ -n "$nft" ] && [ -x "$nft" ] && "$nft" list set "$TARGET_TABLE" "$TARGET_SET" >/dev/null 2>&1; then
                printf "add element %s %s { " "$TARGET_TABLE" "$TARGET_SET" } > "$TARGET_NFT_FILE"
                awk '{printf $1 ", "}' "$TARGET_DL_FILE" >> "$TARGET_NFT_FILE"
                printf " } " >> "$TARGET_NFT_FILE"
                if "$nft" -f "$TARGET_NFT_FILE"; then
                        rm -f "$TARGET_NFT_FILE"
                        _ret=0
                fi
        fi
fi

return $_ret

I am clueless on what I need to change to fix this.

Confirmed, the 2 errors went when upgrade to -18.

GUI upgrade also now reports "running" :slight_smile:

Are these anything to worry about:

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-reject
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-ipt
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-log
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-ipt-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-reject6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nf-log6
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-crypto-crc32c
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-nft-core
 * pkg_hash_check_unresolved: cannot find dependency kernel (= 6.6.43~f102ae4e623e0312df17ffa74182ca02-r1) for kmod-ipt-ipopt

Totally noob question. I am on 23.05.4 OpenWrt and opkg update lists only the 1.1.1-7 of pbr. Is this the current stable one or something fishy is happening?

Try this:

#!/bin/sh
# This file is heavily based on code from https://github.com/Xentrk/netflix-vpn-bypass/blob/master/IPSET_Netflix.sh

TARGET_INTERFACE='wan'
TARGET_NFTSET_4="pbr_${TARGET_INTERFACE}_4_dst_ip_user"
TARGET_NFTSET_6="pbr_${TARGET_INTERFACE}_6_dst_ip_user"
TARGET_TABLE='inet fw4'
TARGET_URL="http://www.ipdeny.com/ipblocks/data/countries/ke.zone"
TARGET_DL_FILE_4="/var/pbr_tmp_ke_ip_ranges"
_ret=0

if [ ! -s "$TARGET_DL_FILE_4" ]; then
	uclient-fetch --no-check-certificate -qO- "$TARGET_URL" 2>/dev/null > "$TARGET_DL_FILE_4"
fi

if [ -s "$TARGET_DL_FILE_4" ]; then
	params=
	while read -r p; do params="${params:+$params, }${p}"; done < "$TARGET_DL_FILE_4"
	[ -n "$params" ] && nft "add element $TARGET_TABLE $TARGET_NFTSET_4 { $params }" || _ret=1
fi

return $_ret
1 Like

I'm not up-to-date with kernel/kmod upgrades with current snapshots, looks like opkg tried to update some kernel mods as they are dependencies for the nft version of the package, but failed. As long as older version of those kmods (compatible with your currently installed kernel) are installed, I believe you should be OK and you should ignore those warnings from opkg.

For stability/compatibility reasons, I have not updated the pbr in the 23.05 packages for a while. There were multiple updates in snapshots packages, but with a lot of help with @egc and volunteers willing to test the current bleeding edge version: 1.1.6-xx, I've been making good progress on it and I'll probably push this new version to snapshot packages within weeks.

If there are news about 24.xx forking in the near future, I will not backport new version to 23.05. If there's no 24.xx fork happening in August, I might backport the new version to 23.05.

TLDR: if you want the most up-to-date version, try 1.1.6 from https://dev.melmac.net/repo/ (make sure to upgrade both pbr and luci-app-pbr packages).

2 Likes