Policy-Based-Routing (pbr) package discussion

Trying today I get this

Executing package manager
apk add pbr
(1/1) Installing pbr (1.1.7-r59)
pbr-1.1.7-r59: installing file to etc/config/pbr.apk-new
Executing pbr-1.1.7-r59.post-install
e[0;31mERRORe[0m: Failed to install fw4 nft file '/var/run/pbr.nft'!
Installing rc.d symlink for pbr... OK
OK: 69 MiB in 387 packages
Errors
uci: Invalid argument

image

image
That came after flashing new main snapshot -> reboot -> install pbr.
Restarting pbr from Luci got it working.

1 Like

Thanks for posting, as far as I understand, apk is still going thru some teething problems. If there are any issues with apk packages/my apk repo say a few weeks from now, please let me know, otherwise it's hard to know what are the pbr issues and what are the apk issues. :wink:

4 Likes

I have just moved my router to the latest snapshot and automated the installation of pbr and luci-app-pbr according to your documentation. Thank you very much for making life easier. I will shout out in case there is any issues with apk.

2 Likes

@ stangri Maybe the option procd_boot_delay should be activated by default with a timer. Because otherwise it can happen that the pbr does not work properly although the rules are supposedly loaded correctly. For example, I had the problem that the Postfix relay did not work properly and all sent larger mails expired with a data timeout when sending to the relay server. After I restarted pbr or set the timeout it worked.

I don't collect telemetry, so it's impossible for me to know what percentage of users needs to change/set that option, however with most of the world moving towards faster connections which are up faster, I don't see the need for the default setting of that option to change.

1 Like

Well, it's hard to say, many users don't even know that they need it in the end. It's not that Pbr doesn't work at all, it's just incomplete. I myself searched like a fool until I figured it out because the mail dispatch sometimes worked and sometimes not. Maybe you should at least introduce that pbr is only loaded 10 seconds after the vpn is up, that would probably be enough. The problem seems to be triggered because other services are not fully loaded.

There's no way I could make default settings of pbr work for everybody in all the cases if users are unwilling to go over README which is linked from the package info and WebUI.

1 Like

You can restart the PBR.

Well, that's clear, but it must be possible to delay the start of pbr a little when booting up to aviod this problem. At least make the boot start timeout adjustable from Luci and write a note about the problem in the readme.

@xiaohei I did that in the beginning for testing but the boot timeout is better because it does not interfere with other services like banip and ad blocker that download data at startup.

Hello!

Somehow pbr stopped working for me and I am getting a lot of daemon.err dnsmasq[1]: nftset inet fw4 pbr_xray_4_dst_ip_cfg096ff5 Error: No such file or directory errors. Nothing have changed in my pbr configuration, I've just restarted it.

I am using 1.1.6-22. What might be an issue?

Here is my configuration:

config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        list resolver_instance '*'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_boot_delay '0'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        option nft_rule_counter '0'
        option nft_set_auto_merge '1'
        option nft_set_counter '0'
        option nft_set_flags_interval '1'
        option nft_set_flags_timeout '0'
        option nft_set_policy 'performance'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        list supported_interface 'xray'

Something with your xray interface is it up and running?
Did you change anything or upgrade anything?

Im trying to set up pbr on a brand new router, but when I test a simple rule to send a given IP (192.168.0.5, using 'test' rule in PBR) over a Wireguard tunnel nothing works on that machine.

ubus call system board

{
"kernel": "6.6.61",
"hostname": "MT6000",
"system": "ARMv8 Processor rev 4",
"model": "GL.iNet GL-MT6000",
"board_name": "glinet,gl-mt6000",
"rootfs_type": "squashfs",
"release": {
"distribution": "OpenWrt",
"version": "SNAPSHOT",
"revision": "r28320-e01af5e8d6",
"target": "mediatek/filogic",
"description": "OpenWrt SNAPSHOT r28320-e01af5e8d6",
"builddate": "1732212090"
}
}

uci export dhcp

package dhcp

config dnsmasq
option domainneeded '1'
option boguspriv '1'
option filterwin2k '0'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option nonegcache '0'
option cachesize '1000'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option nonwildcard '1'
option localservice '1'
option ednspacket_max '1232'
option filter_aaaa '0'
option filter_a '0'

config dhcp 'lan'
option interface 'lan'
option start '100'
option limit '150'
option leasetime '12h'
option dhcpv4 'server'
option dhcpv6 'server'
option ra 'server'
list ra_flags 'managed-config'
list ra_flags 'other-config'

config dhcp 'wan'
option interface 'wan'
option ignore '1'

config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'

uci export firewall

package firewall

config defaults
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'

config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
list network 'lan'

config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wan'

config forwarding
option src 'lan'
option dest 'wan'

config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'

config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'

config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'

config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'

config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'

config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'

config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'

config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'

config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'

config include 'pbr'
option fw4_compatible '1'
option type 'script'
option path '/usr/share/pbr/firewall.include'

config zone
option name 'wg_taa'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
list network 'wg_taa'

config forwarding
option src 'wg_taa'
option dest 'wan'

config forwarding
option src 'lan'
option dest 'wg_taa'

uci export network

package network

config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd3e:252a:bdf0::/48'

config device
option name 'br-lan'
option type 'bridge'
list ports 'lan1'
list ports 'lan2'
list ports 'lan3'
list ports 'lan4'
list ports 'lan5'

config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '192.168.0.1'
option netmask '255.255.255.0'
option ip6assign '60'

config interface 'wan'
option device 'eth1.911'
option proto 'pppoe'
option username 'XXXXXXXXXXXXXX'
option password 'YYYYYYYYYYYYYY'
option ipv6 'auto'
option sourcefilter '0'
option delegate '0'

config interface 'wg_taa'
option proto 'wireguard'
option private_key 'XXXXXXXXXXXXXXXXXXXXXXXXXXX'
list addresses '10.8.0.3/24'
list dns '1.1.1.1'
option route_allowed_ips '0'

config wireguard_wg_taa
option description 'YYYYY'
option public_key 'XXXXXXXXXXXXXX'
option preshared_key 'XXXXXXXXXXXXXXXXXXXXX'
list allowed_ips '0.0.0.0/0'
list allowed_ips '::/0'
option endpoint_host 'xxxxxxxx.org'
option endpoint_port '51820'

uci export pbr

package pbr

config pbr 'config'
option enabled '1'
option verbosity '2'
option strict_enforcement '1'
option resolver_set 'dnsmasq.nftset'
list resolver_instance '*'
option ipv6_enabled '0'
list ignored_interface 'vpnserver'
option boot_timeout '30'
option rule_create_option 'add'
option procd_boot_delay '0'
option procd_reload_delay '1'
option webui_show_ignore_target '0'
option nft_rule_counter '0'
option nft_set_auto_merge '1'
option nft_set_counter '0'
option nft_set_flags_interval '1'
option nft_set_flags_timeout '0'
option nft_set_policy 'performance'
list webui_supported_protocol 'all'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'

config include
option path '/usr/share/pbr/pbr.user.aws'
option enabled '0'

config include
option path '/usr/share/pbr/pbr.user.netflix'
option enabled '0'

config dns_policy
option name 'Redirect Local IP DNS'
option src_addr '192.168.1.5'
option dest_dns '1.1.1.1'
option enabled '0'

config policy
option name 'Ignore Local Requests'
option interface 'ignore'
option dest_addr '10.0.0.0/24 10.0.1.0/24 192.168.100.0/24 192.168.1.0/24'
option enabled '0'

config policy
option name 'Plex/Emby Local Server'
option interface 'wan'
option src_port '8096 8920 32400'
option enabled '0'

config policy
option name 'Plex/Emby Remote Servers'
option interface 'wan'
option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
option enabled '0'

config policy
option name 'test'
option src_addr '192.168.0.5'
option interface 'wg_taa'
option enabled '0'

/etc/init.d/pbr status

pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return

pbr chains - policies
chain pbr_forward { # handle 37
}
chain pbr_input { # handle 38
}
chain pbr_output { # handle 39
}
chain pbr_postrouting { # handle 41
}
chain pbr_prerouting { # handle 40
}
chain pbr_dstnat { # handle 36
}

pbr chains - marking
chain pbr_mark_0x010000 { # handle 1193
meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1194
return # handle 1195
}
chain pbr_mark_0x020000 { # handle 1196
meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1197
return # handle 1198
}

pbr nft sets

IPv4 table 256 route: default via 84.65.64.1 dev pppoe-wan
IPv4 table 256 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.8.0.3 dev wg_taa
IPv4 table 257 rule(s):
29998: from all fwmark 0x20000/0xff0000 lookup pbr_wg_taa

/etc/init.d/pbr reload

Loading environment (on_start) Using wan interface (on_start): wan
Found wan gateway (on_start): 84.11.11.1
Setting up routing for 'wan/pppoe-wan/84.11.11.1' [✓]
Setting up routing for 'wg_taa/10.8.0.3' [✓]
Installing fw4 nft file [✓]
pbr 1.1.7-47 monitoring interfaces: wan wg_taa
pbr 1.1.7-47 (fw4 nft file mode) started with gateways:
wan/pppoe-wan/84.65.64.1 [✓]
wg_taa/10.8.0.3
WARNING: Please set 'dhcp.lan.force=1' to speed up service start-up.

/etc/init.d/pbr status

pbr - environment
pbr 1.1.7-47 running on OpenWrt SNAPSHOT.

Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile

pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return

pbr chains - policies
chain pbr_forward { # handle 37
}
chain pbr_input { # handle 38
}
chain pbr_output { # handle 39
}
chain pbr_postrouting { # handle 41
}
chain pbr_prerouting { # handle 40
}
chain pbr_dstnat { # handle 36
}

pbr chains - marking
chain pbr_mark_0x010000 { # handle 1278
meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1279
return # handle 1280
}
chain pbr_mark_0x020000 { # handle 1281
meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1282
return # handle 1283
}

pbr nft sets

IPv4 table 256 route: default via 84.65.64.1 dev pppoe-wan
IPv4 table 256 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.8.0.3 dev wg_taa
IPv4 table 257 rule(s):
29998: from all fwmark 0x20000/0xff0000 lookup pbr_wg_taa

This is a very simple scenario, so im sure its something silly I have/havent done..

I've had it set up for a similar scenario before and it worked fine then.

Any advice as to what is going wrong?

All your policies, including the one you're testing, are disabled, pbr works as expected.

README -> Getting Help.

@stangri : Sorry, I should have mentioned.. I am using that machine (192.168.0.5), so I had to disable the rule in order for me post on the forum. :slight_smile:

The Wireguard tunnel is handshaking OK, and just to prove the wireguard interface works I have ran this command (I believe it is the correct one to route the ping through the wg_taa tunnel):

root@MT6000:/etc/config# ping -I wg_taa 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=114 time=96.660 ms
64 bytes from 8.8.8.8: seq=1 ttl=114 time=54.539 ms

You have to enable masquerading on the wg_taa zone

1 Like

You were spot on, thank you very much. :slight_smile:

Yeah, you are right, restarting xray helped. Thank you!

Though didn't upgrade anything or changed anything about xray config.

is there any way to have pbr restart itself if it crashes? (or maybe a script that does a pbr restart if it detects that pbr has crashed?)

It is better to solve the problem, what exactly is the problem?