VPN Policy-Based Routing + Web UI -- Discussion

Hi,
what is the exact procedure to update?

If you are having issues with the nft version make sure you update to the latest as stangri has renamed it

pbr 0.9.8-4
luci-app-pbr 0.9.8-4

Did some testing with the latest version.
The service doesn't seem to start properly (in the ui at least).

Hitting start doesn't change this. Rebooting and trying again does not solve this.
The policy I added to test did not work.

root@OpenWrt:/etc/config# fw4 reload
Section @include[0] is not marked as compatible with fw4, ignoring section
Section @include[0] requires 'option fw4_compatible 1' to be considered compatib                                              le
Automatically including '/usr/share/nftables.d/table-post/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/table-post/30-pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/dstnat/20-miniupnpd.nf                                              t'
Automatically including '/usr/share/nftables.d/chain-post/forward/20-miniupnpd.n                                              ft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_forward/30-pbr.                                              nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_input/30-pbr.nf                                              t'
Automatically including '/usr/share/nftables.d/chain-post/mangle_output/30-pbr.n                                              ft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_postrouting/30-                                              pbr.nft'
Automatically including '/usr/share/nftables.d/chain-post/mangle_prerouting/30-p                                              br.nft'
Automatically including '/usr/share/nftables.d/chain-post/srcnat/20-miniupnpd.nf                                              t'
Activating Traffic Killswitch [✓]
Deactivating Traffic Killswitch [✓]
pbr 0.9.8-4 (nft) started with gateways:
wan/eth0.100/136.143.112.1 [✓]
nordvpntun/tun0/10.7.0.2
wg0/10.5.0.2
root@OpenWrt:/etc/config#

I am unsure what the command fw4 reload should give.

The errors that I saw previously during changing/adding policies have gone.

root@OpenWrt:~#  opkg list-installed | grep nft
kmod-nft-core - 5.10.138-1
kmod-nft-fib - 5.10.138-1
kmod-nft-nat - 5.10.138-1
kmod-nft-offload - 5.10.138-1
libnftnl11 - 1.2.1-1
miniupnpd-nftables - 2022-08-31-68c8ec50-1
nftables-json - 1.0.2-2.1

pbr service seems to be running but the luci app says stopped.

pbr - environment
pbr 0.9.8-4 running on OpenWrt 22.03-SNAPSHOT. WAN (IPv4): wan/pppoe-wan/10.zz.zz.zz

Tue Sep 20 08:55:14 2022 user.notice pbr: service (nft) started with gateways: wan/pppoe-wan/10.zz.zz.zz [✓] tun0_Nordvpn/tun0/10.zz.zz.zz tun1_NordVPN/tun1/10.zz.zz.zz tun2_NordVPN/tun2/10.zz.zz.zz tun3_NordVPN/tun3/10.zz.zz.zz

### Service Status [pbr 0.9.8-4]
Service Status
Stopped

0.9.8-4

Dnsmasq goes in crash loop

Doesn't matter.. I come back to vpnbypass and 21.02.3

Not need to go back at 21.02. if you want to set specific upstream to single interface or ip address you can use netifd VPN pbr on 22.03, this is not well documentated but it work really good.

1 Like

Sorry my mistake... I forgot to install ipset with imagebuilder. sorry!

Thanks for the link. Was able to get it working by creating a route table and use IP rules to select the route table for specific hosts. Works great except for newly created route table which doesn't survive reboot.

The contents of /etc/iproute2/rt_tables resets to defaults after reboot. Therefore, none of the IP rules get added. Adding this file to list of files to backup kind of solves the problem.

Tried adding a line in rc.local to insert a line to ip route tables file, but it seems to be running the command after openwrt sets up ip rules, so it doesn't work.

Not sure if there is a nicer way to solve this problem

Yesterday with 0.9.7-11 works, today with 0.9.8-4 nothing
After I enable these rules

config policy
	option interface 'wan'
	option name 'WebByPass_3'
	option dest_addr 'playstation.com google.com google.it youtube.com googlevideo.com googleapis.com youtu.be youtube-nocookie.com youtubeeducation.com ggpht.com ytimg.com teamviewer.com misurainternet.it qnclouds.com quickconnect.cn quickconnect.to synology.com synology.cn synology.me spotify.com akamaized.net pscdn.co scdn.co edgesuite.net'

dnsmasq crash
it crash also with pbr or pbr-iptables...

Changes for 0.9.8-6:

  • changed package names to pbr and pbr-iptables
  • updated the README
  • updated WebUI to reflect the principal package running status
  • cleaned up some extra calls to flush/delete sets/rules in a preparation for atomic nft

If you have any previous version of pbr installed, uninstall it fully (it's save to safe config) then install either of those new versions.

Rest of config? Output of reload/status?

dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '1'
	option ednspacket_max '1232'
	option confdir '/tmp/dnsmasq.d'
	list server '/amazonaws.com/llnwi.net/aiv-cdn.net/footprint.net/aiv-delivery.net/amazonvideo.com/amazon.com/cloudfront.net/media-amazon.com/primevideo.com/a2z.com/amazon.fr/amazon.it/paypal.com/1.1.1.1'
	list server '/netflix.com/nflxext.com/nflximg.com/nflximg.net/nflxso.net/nflxvideo.net/fast.com/aws.dev/amazonalexa.com/1.1.1.1'
	list server '/timvision.it/cubovision.it/ticdn.it/tim.it/telecomitalia.it/1.1.1.1'
	list server '/browserleaks.com/browserleaks.org/dslreports.com/1.1.1.1'
	list server '/esp8266.com/needrom.com/microsoft.com/1.1.1.1'
	list server '/santanderconsumer.it/ausl.re.it/ausl.mo.it/inps.it/lepida.it/cupweb.it/fascicolo-sanitario.it/poste.it/gottardospa.it/1.1.1.1'
	list server '/sky.it/gazzetta.it/rcsobjects.it/repubblica.it/lefrecce.it/trenitalia.com/1.1.1.1'
	list server '/zalando.it/ebay.it/fondoest.it/unieuro.it/playstation.com/1.1.1.1'
	list server '/google.com/google.it/youtube.com/googlevideo.com/googleapis.com/youtu.be/youtube-nocookie.com/youtubeeducation.com/ggpht.com/ytimg.com/1.1.1.1'
	list server '/teamviewer.com/misurainternet.it/1.1.1.1'
	list server '/qnclouds.com/giustizia.it/agenziaentrate.gov.it/gov.it/1.1.1.1'
	list server '/quickconnect.cn/quickconnect.to/synology.com/synology.cn/synology.me/1.1.1.1'
	list server '/spotify.com/akamaized.net/pscdn.co/scdn.co/edgesuite.net/demdex.net/edgekey.net/fastly.net/spotifycdn.com/spotifycdn.net/spotilocal.com/1.1.1.1'
	list server '/akamaihd.net/rai.it/raiplay.it/raiplayradio.it/akamaiedge.net/accedo.tv/mediaset.net/theplatform.eu/mediaset.it/gigya.com/1.1.1.1'
	list interface 'br-lan'
	option cachesize '1000'

config dhcp 'lan'
	option interface 'lan'
	option start '105'
	option limit '145'
	option leasetime '12h'
	option dhcpv4 'server'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

firewall


config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wwan'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'vpn'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'VPN_TUN'
	option input 'REJECT'
	option masq '1'

config forwarding
	option src 'lan'
	option dest 'vpn'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'


network



config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd0xxxxx218e::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.1xx.1'
	list dns '1.1.1.1'
	list dns '8.8.8.8'
	list dns '8.8.4.4'

config device
	option name 'wan'
	option macaddr '26:f5:xxxxxxx'
	option ipv6 '0'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'

config interface 'VPN_TUN'
	option proto 'none'
	option device 'tun0'

config interface 'wwan'
	option proto 'dhcp'

pbr


config pbr 'config'
	option verbosity '2'
	option strict_enforcement '1'
	option ipv6_enabled '0'
	list ignored_interface 'vpnserver'
	list ignored_interface 'wgserver'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '1'
	option webui_enable_column '1'
	option webui_protocol_column '1'
	option webui_chain_column '1'
	option webui_show_ignore_target '1'
	option webui_sorting '1'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	list webui_supported_protocol 'all'
	option resolver_set 'dnsmasq.ipset'
	option enabled '1'

config include
	option path '/usr/share/pbr/pbr.user.aws'
	option enabled '0'

config include
	option path '/usr/share/pbr/pbr.user.netflix'
	option enabled '0'

config policy
	option name 'Plex/Emby Local Server'
	option interface 'wan'
	option src_port '8096 8920 32400'
	option enabled '0'

config policy
	option name 'Plex/Emby Remote Servers'
	option interface 'wan'
	option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
	option enabled '0'

config policy
	option name 'WireGuard Server'
	option interface 'wan'
	option src_port '51820'
	option chain 'OUTPUT'
	option proto 'udp'
	option enabled '0'

config policy
	option interface 'wan'
	option name 'WebByPass_1'
	option dest_addr 'amazonaws.com llnwi.net aiv-cdn.net footprint.net aiv-delivery.net amazonvideo.com amazon.com cloudfront.net media-amazon.com primevideo.com a2z.com amazon.fr amazon.it aws.dev amazonalexa.com'

config policy
	option interface 'wan'
	option name 'WebByPass_2'
	option dest_addr 'browserleaks.com browserleaks.org dslreports.com esp8266.com needrom.com microsoft.com santanderconsumer.it ausl.re.it ausl.mo.it inps.it lepida.it cupweb.it fascicolo-sanitario.it poste.it gottardospa.it sky.it gazzetta.it rcsobjects.it repubblica.it lefrecce.it trenitalia.com zalando.it ebay.it fondoest.it unieuro.it'

config policy
	option interface 'wan'
	option name 'WebByPass_3'
	option dest_addr 'playstation.com google.com google.it youtube.com googlevideo.com googleapis.com youtu.be youtube-nocookie.com youtubeeducation.com ggpht.com ytimg.com teamviewer.com misurainternet.it qnclouds.com quickconnect.cn quickconnect.to synology.com synology.cn synology.me spotify.com akamaized.net pscdn.co scdn.co edgesuite.net'

config policy
	option interface 'wan'
	option name 'WebByPass_4'
	option dest_addr 'demdex.net edgekey.net fastly.net spotifycdn.com spotifycdn.net spotilocal.com akamaihd.net rai.it raiplay.it raiplayradio.it akamaiedge.net accedo.tv mediaset.net theplatform.eu mediaset.it gigya.com'

config policy
	option interface 'wan'
	option name 'WebByPass_5'
	option dest_addr 'paypal.com netflix.com nflxext.com nflximg.com nflximg.net nflxso.net nflxvideo.net fast.com timvision.it cubovision.it ticdn.it tim.it telecomitalia.it'

config policy
	option interface 'wan'
	option name 'WebByPass_6'
	option dest_addr 'giustizia.it agenziaentrate.gov.it gov.it'

config policy
	option interface 'wan'
	option name 'WorkPC'
	option src_addr '192.168.181.31'

config policy
	option interface 'wan'
	option name 'Playstation'
	option src_addr '192.168.181.20'
/etc/init.d/pbr status
pbr 0.9.8-4 running on OpenWrt 22.03.0. WAN (IPv4): wan/wan/192.168.178.1.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         10.x.x.1        128.0.0.0       UG    0      0        0 tun0
default         192.168.1xx.1   0.0.0.0         UG    0      0        0 wan

IPv4 Table 201: default via 192.168.1xx.1 dev wan 
192.168.1xx.0/24 dev br-lan proto kernel scope link src 192.168.1xx.1 
IPv4 Table 201 Rules:
30000:	from all fwmark 0x10000/0xff0000 lookup wan

IPv4 Table 202: unreachable default 
192.168.1xx.0/24 dev br-lan proto kernel scope link src 192.168.1xx.1 
IPv4 Table 202 Rules:
29999:	from all fwmark 0x20000/0xff0000 lookup wwan
============================================================
Mangle IP Table: PREROUTING
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_PREROUTING
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg076ff5 dst -m comment --comment WebByPass_1 -c 45 6512 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg086ff5 dst -m comment --comment WebByPass_2 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg096ff5 dst -m comment --comment WebByPass_3 -c 222 107248 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0a6ff5 dst -m comment --comment WebByPass_4 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0b6ff5 dst -m comment --comment WebByPass_5 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0c6ff5 dst -m comment --comment WebByPass_6 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -s 192.168.1xx.31/32 -m comment --comment WorkPC -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -s 192.168.1xx.20/32 -m comment --comment Playstation -c 84 22775 -g PBR_MARK_0x010000
============================================================
Mangle IP Table MARK Chain: PBR_MARK_0x010000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK_0x010000
-A PBR_MARK_0x010000 -c 2893 690389 -j MARK --set-xmark 0x10000/0xff0000
-A PBR_MARK_0x010000 -c 2893 690389 -j RETURN
============================================================
Mangle IP Table MARK Chain: PBR_MARK_0x020000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK_0x020000
-A PBR_MARK_0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A PBR_MARK_0x020000 -c 0 0 -j RETURN
============================================================
Current ipsets
create pbr_wan_4_dst_ip_cfg076ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
add pbr_wan_4_dst_ip_cfg076ff5 3.67.1.169
add pbr_wan_4_dst_ip_cfg076ff5 18.194.0.165
add pbr_wan_4_dst_ip_cfg076ff5 18.195.246.68
add pbr_wan_4_dst_ip_cfg076ff5 18.196.126.21
add pbr_wan_4_dst_ip_cfg076ff5 3.65.16.46
add pbr_wan_4_dst_ip_cfg076ff5 99.86.253.63
add pbr_wan_4_dst_ip_cfg076ff5 35.157.138.105
add pbr_wan_4_dst_ip_cfg076ff5 18.197.237.101
add pbr_wan_4_dst_ip_cfg076ff5 52.22.220.116
add pbr_wan_4_dst_ip_cfg076ff5 18.193.233.126
add pbr_wan_4_dst_ip_cfg076ff5 35.157.230.137
add pbr_wan_4_dst_ip_cfg076ff5 3.124.69.229
add pbr_wan_4_dst_ip_cfg076ff5 3.67.114.41
add pbr_wan_4_dst_ip_cfg076ff5 3.122.138.18
add pbr_wan_4_dst_ip_cfg076ff5 18.204.41.26
add pbr_wan_4_dst_ip_cfg076ff5 3.123.69.239
add pbr_wan_4_dst_ip_cfg076ff5 18.193.160.115
add pbr_wan_4_dst_ip_cfg076ff5 3.125.151.91
add pbr_wan_4_dst_ip_cfg076ff5 54.236.110.134
add pbr_wan_4_dst_ip_cfg076ff5 18.195.31.25
add pbr_wan_4_dst_ip_cfg076ff5 3.73.157.134
add pbr_wan_4_dst_ip_cfg076ff5 3.72.144.95
add pbr_wan_4_dst_ip_cfg076ff5 18.196.44.57
add pbr_wan_4_dst_ip_cfg076ff5 3.120.255.147
add pbr_wan_4_dst_ip_cfg076ff5 18.157.245.241
add pbr_wan_4_dst_ip_cfg076ff5 3.123.213.13
add pbr_wan_4_dst_ip_cfg076ff5 18.185.81.70
add pbr_wan_4_dst_ip_cfg076ff5 52.58.139.24
create pbr_wan_4_dst_ip_cfg086ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg096ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
add pbr_wan_4_dst_ip_cfg096ff5 216.58.212.234
add pbr_wan_4_dst_ip_cfg096ff5 159.122.182.201
add pbr_wan_4_dst_ip_cfg096ff5 188.172.233.166
add pbr_wan_4_dst_ip_cfg096ff5 188.172.246.179
add pbr_wan_4_dst_ip_cfg096ff5 37.252.231.133
add pbr_wan_4_dst_ip_cfg096ff5 37.252.232.100
add pbr_wan_4_dst_ip_cfg096ff5 142.250.178.10
add pbr_wan_4_dst_ip_cfg096ff5 213.227.168.151
add pbr_wan_4_dst_ip_cfg096ff5 188.172.198.141
add pbr_wan_4_dst_ip_cfg096ff5 213.227.168.135
add pbr_wan_4_dst_ip_cfg096ff5 158.176.86.4
add pbr_wan_4_dst_ip_cfg096ff5 217.146.4.136
add pbr_wan_4_dst_ip_cfg096ff5 37.252.247.101
add pbr_wan_4_dst_ip_cfg096ff5 213.227.185.134
add pbr_wan_4_dst_ip_cfg096ff5 213.227.162.108
add pbr_wan_4_dst_ip_cfg096ff5 217.146.4.134
add pbr_wan_4_dst_ip_cfg096ff5 213.227.186.142
add pbr_wan_4_dst_ip_cfg096ff5 178.255.154.136
add pbr_wan_4_dst_ip_cfg096ff5 188.172.246.168
add pbr_wan_4_dst_ip_cfg096ff5 172.217.16.234
add pbr_wan_4_dst_ip_cfg096ff5 188.172.233.170
add pbr_wan_4_dst_ip_cfg096ff5 213.227.168.142
add pbr_wan_4_dst_ip_cfg096ff5 217.146.14.132
create pbr_wan_4_dst_ip_cfg0a6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg0b6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg0c6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_src_ip_cfg0d6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_src_ip_cfg0e6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
============================================================
DNSMASQ sets
ipset=/amazonaws.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonaws.com
ipset=/llnwi.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: llnwi.net
ipset=/aiv-cdn.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aiv-cdn.net
ipset=/footprint.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: footprint.net
ipset=/aiv-delivery.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aiv-delivery.net
ipset=/amazonvideo.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonvideo.com
ipset=/amazon.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.com
ipset=/cloudfront.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: cloudfront.net
ipset=/media-amazon.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: media-amazon.com
ipset=/primevideo.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: primevideo.com
ipset=/a2z.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: a2z.com
ipset=/amazon.fr/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.fr
ipset=/amazon.it/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.it
ipset=/aws.dev/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aws.dev
ipset=/amazonalexa.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonalexa.com
ipset=/browserleaks.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: browserleaks.com
ipset=/browserleaks.org/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: browserleaks.org
ipset=/dslreports.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: dslreports.com
ipset=/esp8266.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: esp8266.com
ipset=/needrom.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: needrom.com
ipset=/microsoft.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: microsoft.com
ipset=/santanderconsumer.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: santanderconsumer.it
ipset=/ausl.re.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ausl.re.it
ipset=/ausl.mo.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ausl.mo.it
ipset=/inps.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: inps.it
ipset=/lepida.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: lepida.it
ipset=/cupweb.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: cupweb.it
ipset=/fascicolo-sanitario.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: fascicolo-sanitario.it
ipset=/poste.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: poste.it
ipset=/gottardospa.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: gottardospa.it
ipset=/sky.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: sky.it
ipset=/gazzetta.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: gazzetta.it
ipset=/rcsobjects.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: rcsobjects.it
ipset=/repubblica.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: repubblica.it
ipset=/lefrecce.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: lefrecce.it
ipset=/trenitalia.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: trenitalia.com
ipset=/zalando.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: zalando.it
ipset=/ebay.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ebay.it
ipset=/fondoest.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: fondoest.it
ipset=/unieuro.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: unieuro.it
ipset=/playstation.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: playstation.com
ipset=/google.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: google.com
ipset=/google.it/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: google.it
ipset=/youtube.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtube.com
ipset=/googlevideo.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: googlevideo.com
ipset=/googleapis.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: googleapis.com
ipset=/youtu.be/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtu.be
ipset=/youtube-nocookie.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtube-nocookie.com
ipset=/youtubeeducation.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtubeeducation.com
ipset=/ggpht.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: ggpht.com
ipset=/ytimg.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: ytimg.com
ipset=/teamviewer.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: teamviewer.com
ipset=/misurainternet.it/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: misurainternet.it
ipset=/qnclouds.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: qnclouds.com
ipset=/quickconnect.cn/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: quickconnect.cn
ipset=/quickconnect.to/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: quickconnect.to
ipset=/synology.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.com
ipset=/synology.cn/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.cn
ipset=/synology.me/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.me
ipset=/spotify.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: spotify.com
ipset=/akamaized.net/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: akamaized.net
ipset=/pscdn.co/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: pscdn.co
ipset=/scdn.co/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: scdn.co
ipset=/edgesuite.net/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: edgesuite.net
ipset=/demdex.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: demdex.net
ipset=/edgekey.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: edgekey.net
ipset=/fastly.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: fastly.net
ipset=/spotifycdn.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotifycdn.com
ipset=/spotifycdn.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotifycdn.net
ipset=/spotilocal.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotilocal.com
ipset=/akamaihd.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: akamaihd.net
ipset=/rai.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: rai.it
ipset=/raiplay.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: raiplay.it
ipset=/raiplayradio.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: raiplayradio.it
ipset=/akamaiedge.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: akamaiedge.net
ipset=/accedo.tv/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: accedo.tv
ipset=/mediaset.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: mediaset.net
ipset=/theplatform.eu/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: theplatform.eu
ipset=/mediaset.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: mediaset.it
ipset=/gigya.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: gigya.com
ipset=/paypal.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: paypal.com
ipset=/netflix.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: netflix.com
ipset=/nflxext.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxext.com
ipset=/nflximg.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflximg.com
ipset=/nflximg.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflximg.net
ipset=/nflxso.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxso.net
ipset=/nflxvideo.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxvideo.net
ipset=/fast.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: fast.com
ipset=/timvision.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: timvision.it
ipset=/cubovision.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: cubovision.it
ipset=/ticdn.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: ticdn.it
ipset=/tim.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: tim.it
ipset=/telecomitalia.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: telecomitalia.it
ipset=/giustizia.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: giustizia.it
ipset=/agenziaentrate.gov.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: agenziaentrate.gov.it
ipset=/gov.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: gov.it
============================================================
Your support details have been logged to '/var/pbr-support'. [✓]

pbr reload

/etc/init.d/pbr reload
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Activating Traffic Killswitch [✓]
Routing 'WebByPass_1' via wan [✓]
Routing 'WebByPass_2' via wan [✓]
Routing 'WebByPass_3' via wan [✓]
Routing 'WebByPass_4' via wan [✓]
Routing 'WebByPass_5' via wan [✓]
Routing 'WebByPass_6' via wan [✓]
Routing 'WorkPC' via wan [✓]
Routing 'Playstation' via wan [✓]
Deactivating Traffic Killswitch [✓]
pbr 0.9.8-4 (iptables) started with gateways:
wan/192.168.1xx.1
VPN_TUN/tun0/10.x.x.2 [✓]
wwan/0.0.0.0

In this boot seems that is working..., no dnsmasq just crash...

daemon.info procd: Instance dnsmasq::cfg01411c s in a crash loop 6 crashes, 4 seconds since last crash

The dnsmasq ipset file seems to be correct, I don't know why dnsmasq is crashing. I didn't see the output for service pbr status, does the installed dnsmasq support ipset?

opkg list-installed | grep dns
dnsmasq-full - 2.86-14
root@WRT1900ACS:~# service pbr status
pbr 0.9.8-4 running on OpenWrt 22.03.0. WAN (IPv4): wan/wan/192.168.178.1.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         10.8.3.1        128.0.0.0       UG    0      0        0 tun0
default         192.168.178.1   0.0.0.0         UG    0      0        0 wan

IPv4 Table 201: default via 192.168.178.1 dev wan 
192.168.181.0/24 dev br-lan proto kernel scope link src 192.168.181.1 
IPv4 Table 201 Rules:
30000:	from all fwmark 0x10000/0xff0000 lookup wan

IPv4 Table 202: default via 10.8.3.2 dev tun0 
192.168.181.0/24 dev br-lan proto kernel scope link src 192.168.181.1 
IPv4 Table 202 Rules:
29999:	from all fwmark 0x20000/0xff0000 lookup wwan
============================================================
Mangle IP Table: PREROUTING
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_PREROUTING
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg076ff5 dst -m comment --comment WebByPass_1 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg086ff5 dst -m comment --comment WebByPass_2 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg096ff5 dst -m comment --comment WebByPass_3 -c 19 2273 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0a6ff5 dst -m comment --comment WebByPass_4 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0b6ff5 dst -m comment --comment WebByPass_5 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -m set --match-set pbr_wan_4_dst_ip_cfg0c6ff5 dst -m comment --comment WebByPass_6 -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -s 192.168.181.31/32 -m comment --comment WorkPC -c 0 0 -g PBR_MARK_0x010000
-A PBR_PREROUTING -s 192.168.181.20/32 -m comment --comment Playstation -c 0 0 -g PBR_MARK_0x010000
============================================================
Mangle IP Table MARK Chain: PBR_MARK_0x010000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK_0x010000
-A PBR_MARK_0x010000 -c 295 84638 -j MARK --set-xmark 0x10000/0xff0000
-A PBR_MARK_0x010000 -c 295 84638 -j RETURN
============================================================
Mangle IP Table MARK Chain: PBR_MARK_0x020000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK_0x020000
-A PBR_MARK_0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A PBR_MARK_0x020000 -c 0 0 -j RETURN
============================================================
Current ipsets
create pbr_wan_4_dst_ip_cfg076ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg086ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg096ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
add pbr_wan_4_dst_ip_cfg096ff5 37.252.247.104
add pbr_wan_4_dst_ip_cfg096ff5 217.146.8.68
add pbr_wan_4_dst_ip_cfg096ff5 165.22.130.180
add pbr_wan_4_dst_ip_cfg096ff5 178.255.155.173
add pbr_wan_4_dst_ip_cfg096ff5 142.250.178.4
add pbr_wan_4_dst_ip_cfg096ff5 157.230.131.25
add pbr_wan_4_dst_ip_cfg096ff5 134.209.50.218
add pbr_wan_4_dst_ip_cfg096ff5 134.209.62.234
add pbr_wan_4_dst_ip_cfg096ff5 66.102.1.188
add pbr_wan_4_dst_ip_cfg096ff5 188.172.192.109
add pbr_wan_4_dst_ip_cfg096ff5 134.209.61.200
add pbr_wan_4_dst_ip_cfg096ff5 158.176.86.3
add pbr_wan_4_dst_ip_cfg096ff5 165.22.130.105
add pbr_wan_4_dst_ip_cfg096ff5 165.22.140.152
add pbr_wan_4_dst_ip_cfg096ff5 142.93.23.129
create pbr_wan_4_dst_ip_cfg0a6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg0b6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_dst_ip_cfg0c6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_src_ip_cfg0d6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
create pbr_wan_4_src_ip_cfg0e6ff5 hash:ip family inet hashsize 1024 maxelem 65536 comment
============================================================
DNSMASQ sets
ipset=/amazonaws.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonaws.com
ipset=/llnwi.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: llnwi.net
ipset=/aiv-cdn.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aiv-cdn.net
ipset=/footprint.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: footprint.net
ipset=/aiv-delivery.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aiv-delivery.net
ipset=/amazonvideo.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonvideo.com
ipset=/amazon.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.com
ipset=/cloudfront.net/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: cloudfront.net
ipset=/media-amazon.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: media-amazon.com
ipset=/primevideo.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: primevideo.com
ipset=/a2z.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: a2z.com
ipset=/amazon.fr/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.fr
ipset=/amazon.it/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazon.it
ipset=/aws.dev/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: aws.dev
ipset=/amazonalexa.com/pbr_wan_4_dst_ip_cfg076ff5 # WebByPass_1: amazonalexa.com
ipset=/browserleaks.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: browserleaks.com
ipset=/browserleaks.org/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: browserleaks.org
ipset=/dslreports.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: dslreports.com
ipset=/esp8266.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: esp8266.com
ipset=/needrom.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: needrom.com
ipset=/microsoft.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: microsoft.com
ipset=/santanderconsumer.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: santanderconsumer.it
ipset=/ausl.re.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ausl.re.it
ipset=/ausl.mo.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ausl.mo.it
ipset=/inps.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: inps.it
ipset=/lepida.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: lepida.it
ipset=/cupweb.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: cupweb.it
ipset=/fascicolo-sanitario.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: fascicolo-sanitario.it
ipset=/poste.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: poste.it
ipset=/gottardospa.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: gottardospa.it
ipset=/sky.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: sky.it
ipset=/gazzetta.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: gazzetta.it
ipset=/rcsobjects.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: rcsobjects.it
ipset=/repubblica.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: repubblica.it
ipset=/lefrecce.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: lefrecce.it
ipset=/trenitalia.com/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: trenitalia.com
ipset=/zalando.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: zalando.it
ipset=/ebay.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: ebay.it
ipset=/fondoest.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: fondoest.it
ipset=/unieuro.it/pbr_wan_4_dst_ip_cfg086ff5 # WebByPass_2: unieuro.it
ipset=/playstation.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: playstation.com
ipset=/google.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: google.com
ipset=/google.it/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: google.it
ipset=/youtube.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtube.com
ipset=/googlevideo.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: googlevideo.com
ipset=/googleapis.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: googleapis.com
ipset=/youtu.be/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtu.be
ipset=/youtube-nocookie.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtube-nocookie.com
ipset=/youtubeeducation.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: youtubeeducation.com
ipset=/ggpht.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: ggpht.com
ipset=/ytimg.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: ytimg.com
ipset=/teamviewer.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: teamviewer.com
ipset=/misurainternet.it/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: misurainternet.it
ipset=/qnclouds.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: qnclouds.com
ipset=/quickconnect.cn/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: quickconnect.cn
ipset=/quickconnect.to/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: quickconnect.to
ipset=/synology.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.com
ipset=/synology.cn/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.cn
ipset=/synology.me/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: synology.me
ipset=/spotify.com/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: spotify.com
ipset=/akamaized.net/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: akamaized.net
ipset=/pscdn.co/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: pscdn.co
ipset=/scdn.co/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: scdn.co
ipset=/edgesuite.net/pbr_wan_4_dst_ip_cfg096ff5 # WebByPass_3: edgesuite.net
ipset=/demdex.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: demdex.net
ipset=/edgekey.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: edgekey.net
ipset=/fastly.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: fastly.net
ipset=/spotifycdn.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotifycdn.com
ipset=/spotifycdn.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotifycdn.net
ipset=/spotilocal.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: spotilocal.com
ipset=/akamaihd.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: akamaihd.net
ipset=/rai.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: rai.it
ipset=/raiplay.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: raiplay.it
ipset=/raiplayradio.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: raiplayradio.it
ipset=/akamaiedge.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: akamaiedge.net
ipset=/accedo.tv/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: accedo.tv
ipset=/mediaset.net/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: mediaset.net
ipset=/theplatform.eu/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: theplatform.eu
ipset=/mediaset.it/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: mediaset.it
ipset=/gigya.com/pbr_wan_4_dst_ip_cfg0a6ff5 # WebByPass_4: gigya.com
ipset=/paypal.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: paypal.com
ipset=/netflix.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: netflix.com
ipset=/nflxext.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxext.com
ipset=/nflximg.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflximg.com
ipset=/nflximg.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflximg.net
ipset=/nflxso.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxso.net
ipset=/nflxvideo.net/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: nflxvideo.net
ipset=/fast.com/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: fast.com
ipset=/timvision.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: timvision.it
ipset=/cubovision.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: cubovision.it
ipset=/ticdn.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: ticdn.it
ipset=/tim.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: tim.it
ipset=/telecomitalia.it/pbr_wan_4_dst_ip_cfg0b6ff5 # WebByPass_5: telecomitalia.it
ipset=/giustizia.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: giustizia.it
ipset=/agenziaentrate.gov.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: agenziaentrate.gov.it
ipset=/gov.it/pbr_wan_4_dst_ip_cfg0c6ff5 # WebByPass_6: gov.it
============================================================
Your support details have been logged to '/var/pbr-support'. [✓]
root@WRT1900ACS:~# logread -f | grep crash
Wed Sep 21 02:59:48 2022 daemon.info procd: Instance dnsmasq::cfg01411c s in a crash loop 6 crashes, 9 seconds since last crash


That's all crash until I disable the domain based policies

[pbr 0.9.8-6]

Lots of improvements. Using nft.
It does not survive a reboot. I have to manually restart the service to get it working again after rebooting.

Just did another reboot to reproduce. Now pbr doesn't get working at all.

root@OpenWrt:~# service pbr status
============================================================
pbr - environment
pbr 0.9.8-6 running on OpenWrt 22.03.0. WAN (IPv4): wan/eth0.100/136.143.112.1.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6                                                                                      no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-de                                                                                     tect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward {
        }
        chain pbr_input {
        }
        chain pbr_output {
        }
        chain pbr_prerouting {
                ether saddr d0:57:7b:9b:fc:ad goto pbr_mark_0x010000 comment "St                                                                                     okbookG2"
                ether saddr 54:ef:44:c8:f8:ce goto pbr_mark_0x030000 comment "wi                                                                                     ttecamera"
                ether saddr 54:ef:44:c8:fa:cc goto pbr_mark_0x030000 comment "ge                                                                                     lecamera"
                ether saddr b4:9c:df:98:ef:c0 goto pbr_mark_0x010000 comment "st                                                                                     okpad"
                ether saddr d0:50:99:1a:d2:e1 goto pbr_mark_0x020000 comment "ko                                                                                     dibuntu"
        }
        chain pbr_postrouting {
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
        }
        chain pbr_mark_0x020000 {
        }
        chain pbr_mark_0x030000 {
        }
============================================================
pbr nft sets
        set pbr_wan_4_src_mac_cfg016ff5 {
                type ether_addr
                flags interval
                comment "StokbookG2: D0:57:7B:9B:FC:AD"
        }
        set pbr_wg0_4_src_mac_cfg026ff5 {
                type ether_addr
                flags interval
                comment "wittecamera: 54:EF:44:C8:F8:CE"
        }
        set pbr_wg0_4_src_mac_cfg036ff5 {
                type ether_addr
                flags interval
                comment "gelecamera: 54:EF:44:C8:FA:CC"
        }
        set pbr_wan_4_src_mac_cfg046ff5 {
                type ether_addr
                flags interval
                comment "stokpad: B4:9C:DF:98:EF:C0"
        }
        set pbr_nordvpntun_4_src_mac_cfg056ff5 {
                type ether_addr
                flags interval
                comment "kodibuntu: D0:50:99:1A:D2:E1"
        }
root@OpenWrt:~#

Is there anything in the log from pbr after a reboot? What does logread -e pbr produce immediately after the reboot?

I don't have an idea why. pbr makes a clean dnsmsaq file, maybe it's the conflict between all/some server and ipset entries with the same domain name? Have you tried commenting out all the list server options to see if dnsmasq will start with the ipset file from pbr?

1 Like

root@OpenWrt:~# logread -e pbr
Wed Sep 21 09:08:37 2022 user.notice pbr: Reloading pbr due to includes of firewall
Wed Sep 21 09:08:37 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:38 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:39 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:41 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:42 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:43 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:44 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:45 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:46 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:47 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:48 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:49 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:51 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:52 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:53 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:54 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:55 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:56 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:57 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:58 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:08:59 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:01 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:02 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:03 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:04 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:05 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:06 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:07 2022 daemon.notice procd: /etc/rc.d/S19firewall: Include '/usr/share/pbr/pbr.firewall.include' failed with exit code -9
Wed Sep 21 09:09:07 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:10 2022 user.notice pbr: service waiting for wan gateway...
Wed Sep 21 09:09:11 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:09:25 2022 user.notice pbr: Setting up routing for 'wan/eth0.100/136.143.112.1' [✓]
Wed Sep 21 09:09:26 2022 user.notice pbr: Setting up routing for 'wg0/0.0.0.0' [✓]
Wed Sep 21 09:09:26 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✗]
Wed Sep 21 09:09:26 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:09:26 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:09:26 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:09:26 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✗]
Wed Sep 21 09:09:26 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:09:27 2022 user.notice pbr: service monitoring interfaces: wan wg0
Wed Sep 21 09:09:27 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] wg0/0.0.0.0
Wed Sep 21 09:09:27 2022 user.notice pbr: ERROR: Policy 'StokbookG2' has an unknown interface: 'nordvpntun' ERROR: Policy 'kodibuntu' has an unknown interface: 'nordvpntun'
Wed Sep 21 09:09:27 2022 user.notice pbr: Reloading pbr due to includes of firewall
Wed Sep 21 09:09:27 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:09:28 2022 user.notice pbr: Setting up routing for 'wan/eth0.100/136.143.112.1' [✓]
Wed Sep 21 09:09:28 2022 user.notice pbr: Setting up routing for 'nordvpntun/tun0/10.7.1.2' [✓]
Wed Sep 21 09:09:28 2022 user.notice pbr: Setting up routing for 'wg0/10.5.0.2' [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✓]
Wed Sep 21 09:09:29 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:09:30 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] nordvpntun/tun0/10.7.1.2 wg0/10.5.0.2
Wed Sep 21 09:09:30 2022 user.notice pbr: service monitoring interfaces: wan nordvpntun wg0
Wed Sep 21 09:09:30 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:09:31 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:10:16 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] nordvpntun/tun0/10.7.1.2 wg0/10.5.0.2
Wed Sep 21 09:10:18 2022 user.notice pbr: Reloading pbr due to includes of firewall
Wed Sep 21 09:10:18 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:10:19 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✓]
Wed Sep 21 09:10:19 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:10:19 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:10:20 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:10:20 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✓]
Wed Sep 21 09:10:20 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:10:20 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] nordvpntun/tun0/10.7.1.2 wg0/10.5.0.2
Wed Sep 21 09:10:21 2022 user.notice pbr: Reloading pbr due to includes of firewall
Wed Sep 21 09:10:21 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:10:22 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✓]
Wed Sep 21 09:10:22 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:10:22 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:10:23 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:10:23 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✓]
Wed Sep 21 09:10:23 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:10:23 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] nordvpntun/tun0/10.7.1.2 wg0/10.5.0.2
Wed Sep 21 09:10:24 2022 user.notice pbr: Reloading pbr due to includes of firewall
Wed Sep 21 09:10:24 2022 user.notice pbr: Activating Traffic Killswitch [✓]
Wed Sep 21 09:10:25 2022 user.notice pbr: Routing 'StokbookG2' via nordvpntun [✓]
Wed Sep 21 09:10:25 2022 user.notice pbr: Routing 'wittecamera' via wg0 [✓]
Wed Sep 21 09:10:26 2022 user.notice pbr: Routing 'gelecamera' via wg0 [✓]
Wed Sep 21 09:10:26 2022 user.notice pbr: Routing 'stokpad' via wan [✓]
Wed Sep 21 09:10:26 2022 user.notice pbr: Routing 'kodibuntu' via nordvpntun [✓]
Wed Sep 21 09:10:26 2022 user.notice pbr: Deactivating Traffic Killswitch [✓]
Wed Sep 21 09:10:30 2022 user.notice pbr: service (nft) started with gateways: wan/eth0.100/136.143.112.1 [✓] nordvpntun/tun0/10.7.1.2 wg0/10.5.0.2
root@OpenWrt:~#

Net effect is that StokbookG2 is not routed over vpn.

can you run service pbr status in this condition? I'd like to see the nordvpn routing table as reported by pbr.

PS. Is strict mode enabled?

root@OpenWrt:~# service pbr status
============================================================
pbr - environment
pbr 0.9.8-6 running on OpenWrt 22.03.0. WAN (IPv4): wan/eth0.100/136.143.112.1.
============================================================
Dnsmasq version 2.86  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward {
        }
        chain pbr_input {
        }
        chain pbr_output {
        }
        chain pbr_prerouting {
                ether saddr d0:57:7b:9b:fc:ad goto pbr_mark_0x020000 comment "StokbookG2"
                ether saddr 54:ef:44:c8:f8:ce goto pbr_mark_0x030000 comment "wittecamera"
                ether saddr 54:ef:44:c8:fa:cc goto pbr_mark_0x030000 comment "gelecamera"
                ether saddr b4:9c:df:98:ef:c0 goto pbr_mark_0x010000 comment "stokpad"
                ether saddr d0:50:99:1a:d2:e1 goto pbr_mark_0x020000 comment "kodibuntu"
        }
        chain pbr_postrouting {
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 {
        }
        chain pbr_mark_0x020000 {
        }
        chain pbr_mark_0x030000 {
        }
============================================================
pbr nft sets
        set pbr_nordvpntun_4_src_mac_cfg016ff5 {
                type ether_addr
                flags interval
                comment "StokbookG2: D0:57:7B:9B:FC:AD"
        }
        set pbr_wg0_4_src_mac_cfg026ff5 {
                type ether_addr
                flags interval
                comment "wittecamera: 54:EF:44:C8:F8:CE"
        }
        set pbr_wg0_4_src_mac_cfg036ff5 {
                type ether_addr
                flags interval
                comment "gelecamera: 54:EF:44:C8:FA:CC"
        }
        set pbr_wan_4_src_mac_cfg046ff5 {
                type ether_addr
                flags interval
                comment "stokpad: B4:9C:DF:98:EF:C0"
        }
        set pbr_nordvpntun_4_src_mac_cfg056ff5 {
                type ether_addr
                flags interval
                comment "kodibuntu: D0:50:99:1A:D2:E1"
        }
root@OpenWrt:~#