Policy-Based-Routing (pbr) package discussion

Does the following produce any output:

ipv6_leases_to_nftset() { [ -s '/tmp/hosts/odhcpd' ] || return 1; grep "$1" '/tmp/hosts/odhcpd' | awk '{print $1}' | sed -n 'H;${x;s/\n/,/g;s/^,//;p;};d' | tr '\n' ' '; }
ipv6_leases_to_nftset "C2VQF93F24"

pbr switch to nft...
Running (version: 0.9.9-19 using nft) with dnsmasq 2.87 and new kernel 5.15 on my wrt1900acs is working great! THANKS

If I use Imagebuilder I get this message...

Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency ipset for pbr-iptables

but it is not a problem

Don't include pbr-iptables, include pbr in your image.

only in imagebuilder...
I include only pbr and luci-app-pbr... but I get that message...

Weird, do you include them in this specific order: pbr luci-app-pbr? I'll monitor messages from IB next time I'm building an image and try to figure out what's what.

Hi, yes

make image PROFILE=linksys_wrt1900acs PACKAGES="pbr luci-app-pbr luci-app-sqm kmod-sched openssl-util iperf3 libopenssl-devcrypto e2fsprogs f2fs-tools kmod-fs-f2fs kmod-rtl8192cu kmod-nls-base kmod-usb2 kmod-usb3 kmod-usb-core kmod-usb-ohci kmod-usb-uhci wireless-regdb mwlwifi-firmware-88w8864 wpad-basic curl luci luci-app-firewall luci-mod-admin-full luci-theme-bootstrap luci-app-advanced-reboot lm-sensors kmod-usb-serial-ch341 openvpn-openssl luci-app-openvpn luci-app-adblock tcpdump-mini luci-app-vnstat2 luci-app-ksmbd ksmbd-avahi-service wsdd2 kmod-usb-storage kmod-usb-storage-uas usbutils kmod-fs-exfat kmod-fs-ext4 kmod-fs-ntfs kmod-fs-vfat ntfs-3g blkid dnsmasq-full -dnsmasq -wpad-basic-wolfssl -ppp -ppp-mod-pppoe -ip6tables -odhcp6c -kmod-ipv6 -kmod-ip6tables -odhcpd-ipv6only -odhcpd" FILES=files/

Thanks for your prompt replies, I'll take a look.

1 Like

It does...

# ipv6_leases_to_nftset "C2VQF93F24"
2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,

...which is puzzling as even a freshly created rule just after running the above, gives this on 0.9.9-19

	set pbr_van_6_src_ip_cfg036ff5 {
		type ipv6_addr
		flags interval
		auto-merge
		comment "C2VQF93F24"
	}

Does nft complain about the extra comma maybe?

nft add element inet fw4 pbr_van_6_src_ip_cfg036ff5 "{ 2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81, }"

I wondered that, but sadly not.

Instead we get the desired outcome!

	set pbr_van_6_src_ip_cfg036ff5 {
		type ipv6_addr
		flags interval
		auto-merge
		comment "C2VQF93F24"
		elements = { 2001:8b0:dc1b:1da1::81,
			     fdc9:a26:3e41::81 }
	}

Can you replace the init script with one from this gist: https://gist.githubusercontent.com/stangri/8afddf154a806465f2f9f3febbb5554a/raw/60b89821b490d0c5692c83d018dace3585561ad2/pbr and let me know what the output was during restart/reload?

I'm sure I'm missing something simple, it's one of those things...

Thanks yet again - I feel like I should be doing more to help!

Looking at the output from the new script could it be the hash character at the end?

# /etc/init.d/pbr reload
Activating traffic killswitch [✓]
Setting up routing for 'wan/pppoe-wan/81.187.81.187/2001:8b0:1111:1111:0:ffff:51bb:a8f9/128
fe80::a8f2:c1a6:bad2:949d' [✓]
Setting up routing for 'van/eth1/86.16.56.1/::/0' RTNETLINK answers: File exists
[✓]
Routing 'Consoles' via van [✓]
Routing 'C2VQF93F24' via van target: src, param: C2VQF93F24, param6: 2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,# 
param: C2VQF93F24, param6: 2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,# 
ipv4_error: 0, ipv6_error: 1
[✓]
Deactivating traffic killswitch [✓]
pbr dev-test monitoring interfaces: wan 
pbr dev-test (nft) started with gateways:
wan/pppoe-wan/81.187.81.187/2001:8b0:1111:1111:0:ffff:51bb:a8f9/128
fe80::a8f2:c1a6:bad2:949d [✓]
van/eth1/86.16.56.1/::/0

If I try the command from before with the trailing comma and hash I do get an error thus:

# nft add element inet fw4 pbr_van_6_src_ip_cfg036ff5 "{ 2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,#}"
Error: syntax error, unexpected end of file, expecting comma or '}'
add element inet fw4 pbr_van_6_src_ip_cfg036ff5 { 2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,#}
^

And in fact, thinking about it, this is my fault. Apologies. When you asked me to run the command earlier, the exact output was as shown below, but I scanned the line by eye and assumed that the first hash in line 2 was just some rogue formatting because the prompt was on the same row (as there was no newline). Looking closer, actually the first hash was part of the string output by the command. I'm sorry about that.

root@A_A_x86:/etc/init.d# ipv6_leases_to_nftset "C2VQF93F24"
2001:8b0:dc1b:1da1::81,fdc9:a26:3e41::81,# root@A_A_x86:/etc/init.d# 

Can you post/PM me the whole file please?

Changed position, still same messages...

So amusingly I've switched my configuration to MAC addresses as the device identifier, and discovered that it doesn't like lowercase hexadecimal.

Specifically, C8:89:F3:DB:0B:E8 works but c8:89:f3:db:0b:e8 gives service error as below.

All configs as before apart from this rule change.

Policy insertion failed for both IPv4 and IPv6!
nft 'add rule inet fw4 pbr_prerouting ip saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000 comment "C2VQF93F24"'
nft 'add rule inet fw4 pbr_prerouting ip6 saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000

It's not a problem once I figured it out, but if you wanted to force uppercase, which seems to be what nft is expecting, that would be neat I guess.

Thanks for testing so many things and the reports! This issue should have been addressed in pbr 0.9.9-21.

1 Like

good evening stangri what are the chains for?

I installed my vpn for the console and it changed the localization but it doesn't find any games,

I have tried for the moment prerouting and output thank you

Happy to help.

On 0.9.9-21 I'm afraid I still get case-related errors. Full set of config files below again just in case something else is awry, but I think it's the same issue? I tried deleting and re-adding the rule in case you were uppercasing them on entry to the config file.

config policy
	option name 'C2VQF93F24'
	option interface 'van'
	option src_addr 'c8:89:f3:db:0b:e8'

leads to

ERROR: Policy insertion failed for both IPv4 and IPv6!
ERROR: nft 'add rule inet fw4 pbr_prerouting ip saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000 comment "C2VQF93F24"'
ERROR: nft 'add rule inet fw4 pbr_prerouting ip6 saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000 comment "C2VQF93F24"'
Full set of config files
/etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '1'
	option ednspacket_max '1232'
	option cachesize '10000'
	list notinterface 'pppoe-wan'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	option ra_slaac '1'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option leasetime '2h'
	option ra_default '1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

/etc/config/firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	list network 'vodem'
	list network 'van'
	list network 'modem'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option masq6 '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config redirect 'dns_int'
	option name 'Intercept-DNS'
	option src 'lan'
	option src_dport '53'
	option proto 'tcp udp'
	option target 'DNAT'
	option family 'all'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdc9:0a26:3e41::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	option ports 'eth2'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wan'
	option device 'eth0'
	option proto 'pppoe'
	option username ''
	option password ''
	option pppd_options 'debug'
	option ipv6 '1'
	option metric '10'
	option peerdns '0'
	list dns '45.90.28.123'
	list dns '45.90.30.123'

config interface 'wan6'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	list dns '2a07:a8c0::31:527d'
	list dns '2a07:a8c1::31:527d'
	option device 'pppoe-wan'

config interface 'van'
	option device 'eth1'
	option proto 'dhcp'
	option ipv6 '0'
	option peerdns '0'
	list dns '45.90.28.181'
	list dns '45.90.30.181'
	option metric '20'

config interface 'modem'
	option device 'eth0'
	option proto 'static'
	option ipaddr '192.168.2.2'
	option netmask '255.255.255.0'

config interface 'vodem'
	option proto 'static'
	option ipaddr '192.168.100.20'
	option netmask '255.255.255.0'
	option device 'eth1'

/etc/config/pbr

config pbr 'config'
	option verbosity '2'
	option resolver_set 'none'
	list ignored_interface 'vpnserver'
	list ignored_interface 'wgserver'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '1'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	option strict_enforcement '1'
	option ipv6_enabled '1'
	option enabled '1'

config policy
	option name 'Consoles'
	option interface 'van'
	option src_addr '00:E4:21:9E:FC:01 0C:DD:24:79:7F:F2 4C:3B:DF:8E:CD:91 4C:3B:DF:8E:CD:93 98:5F:D3:F6:49:57 BC:83:85:82:3E:6D CC:6B:1E:A8:1B:7B'

config policy
	option name 'C2VQF93F24'
	option interface 'van'
	option src_addr 'c8:89:f3:db:0b:e8'

config policy
	option name 'mobuntu'
	option interface 'van'
	option src_addr '00:CE:39:D0:BE:1C'

/etc/init.d/pbr status

============================================================
pbr - environment
pbr 0.9.9-21 running on OpenWrt 22.03.2. WAN (IPv4): van/eth1/86.16.56.1. WAN (IPv6): wan6/pppoe-wan/2001:8b0:1111:1111:0:ffff:51bb:a8f9.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
	chain pbr_forward {
	}
	chain pbr_input {
	}
	chain pbr_output {
	}
	chain pbr_prerouting {
		ether saddr @pbr_van_4_src_mac_cfg026ff5 goto pbr_mark_0x020000 comment "Consoles"
		ether saddr @pbr_van_6_src_mac_cfg026ff5 goto pbr_mark_0x020000 comment "Consoles"
		ether saddr @pbr_van_4_src_mac_cfg046ff5 goto pbr_mark_0x020000 comment "mobuntu"
		ether saddr @pbr_van_6_src_mac_cfg046ff5 goto pbr_mark_0x020000 comment "mobuntu"
	}
	chain pbr_postrouting {
	}
============================================================
pbr chains - marking
	chain pbr_mark_0x010000 {
		counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
		return
	}
	chain pbr_mark_0x020000 {
		counter packets 112 bytes 14268 meta mark set meta mark & 0xff02ffff | 0x00020000
		return
	}
============================================================
pbr nft sets
	set pbr_van_4_src_mac_cfg026ff5 {
		type ether_addr
		flags interval
		auto-merge
		comment "Consoles"
		elements = { 00:e4:21:9e:fc:01,
			     0c:dd:24:79:7f:f2,
			     4c:3b:df:8e:cd:91,
			     4c:3b:df:8e:cd:93,
			     98:5f:d3:f6:49:57,
			     bc:83:85:82:3e:6d,
			     cc:6b:1e:a8:1b:7b }
	}
	set pbr_van_6_src_mac_cfg026ff5 {
		type ether_addr
		flags interval
		auto-merge
		comment "Consoles"
		elements = { 00:e4:21:9e:fc:01,
			     0c:dd:24:79:7f:f2,
			     4c:3b:df:8e:cd:91,
			     4c:3b:df:8e:cd:93,
			     98:5f:d3:f6:49:57,
			     bc:83:85:82:3e:6d,
			     cc:6b:1e:a8:1b:7b }
	}
	set pbr_van_4_src_ip_cfg036ff5 {
		type ipv4_addr
		flags interval
		auto-merge
		comment "C2VQF93F24"
	}
	set pbr_van_6_src_ip_cfg036ff5 {
		type ipv6_addr
		flags interval
		auto-merge
		comment "C2VQF93F24"
	}
	set pbr_van_4_src_mac_cfg046ff5 {
		type ether_addr
		flags interval
		auto-merge
		comment "mobuntu"
		elements = { 00:ce:39:d0:be:1c }
	}
	set pbr_van_6_src_mac_cfg046ff5 {
		type ether_addr
		flags interval
		auto-merge
		comment "mobuntu"
		elements = { 00:ce:39:d0:be:1c }
	}
============================================================
IPv4 table 256 route: default via 81.187.81.187 dev pppoe-wan 
default via 86.16.56.1 dev eth1 proto static src 86.16.59.159 metric 20 
IPv4 table 256 rule:  30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 86.16.56.1 dev eth1 
default via 86.16.56.1 dev eth1 proto static src 86.16.59.159 metric 20 
IPv4 table 257 rule:  29999:	from all fwmark 0x20000/0xff0000 lookup pbr_van


/etc/init.d/pbr reload with verbosity setting set to 2

Activating traffic killswitch [✓]
Setting up routing for 'wan/pppoe-wan/81.187.81.187/2001:8b0:1111:1111:0:ffff:51bb:a8f9/128
fe80::4131:a0cb:523:a5a1' [✓]
Setting up routing for 'van/eth1/86.16.56.1/::/0' RTNETLINK answers: File exists
[✓]
Routing 'Consoles' via van [✓]
Routing 'C2VQF93F24' via van [✗]
Routing 'mobuntu' via van [✓]
Deactivating traffic killswitch [✓]
pbr 0.9.9-21 monitoring interfaces: wan 
pbr 0.9.9-21 (nft) started with gateways:
wan/pppoe-wan/81.187.81.187/2001:8b0:1111:1111:0:ffff:51bb:a8f9/128
fe80::4131:a0cb:523:a5a1 [✓]
van/eth1/86.16.56.1/::/0
ERROR: Policy insertion failed for both IPv4 and IPv6!
ERROR: nft 'add rule inet fw4 pbr_prerouting ip saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000 comment "C2VQF93F24"'
ERROR: nft 'add rule inet fw4 pbr_prerouting ip6 saddr c8:89:f3:db:0b:e8 goto pbr_mark_0x020000 comment "C2VQF93F24"'

Tiny UI bug

If I disable the service in System > Startup I see this


But then if I navigate to Services > Policy Routing I see an inconsistent state of the buttons - I think Enable should be highlighted at this point?

The reverse is also true. Starting with service being enabled, if I push the button in the service pane to disable the service I see this

But in System > Startup it still shows as Enabled.

1 Like

To allow to add policies to specific nft/iptables chains.

Turns out it's not nft (which I should have been clued in when mac address was being added to the saddr parameter), it was that pbr only allowed upper case letters in the MAC address. I've adjusted how pbr identifies MAC addresses and it should work now (0.9.9-22).

No, these pages control different things (even tho the buttons are similar). The startup page controls wherever the service is started on the router startup (boot). The button at the Policy Routing page controls wherever the service is enabled/disabled generally speaking.

2 Likes