Policy-Based-Routing (pbr) package discussion

First of all I want to say, great work on this package! I'm new to OpenWRT since I've always used pfSense/OPNsense as router, but needed a low power solution with decent performance. OpenWRT / Linux seem to perform much better on low end old hardware than FreeBSD. So I decided to switch to OpenWRT a few weeks ago and so far pretty happy with it.

Now I wanted to replicate the routing policies I had on OPNsense for wan / wireguard vpn, which this package seems to be perfect for. Unfortunately I can't really get it to work reliably, pbr is reloading every few minutes with the following log file:
user.notice pbr: Reloading pbr due to includes of firewall
This causes the connection getting interrupted for a few sec, resulting in major packet loss. Is there a way to debug what triggers this? I already figured out that putting 'exit 0' on top of /usr/share/pbr/pbr.firewall.include will stop this reloading behavior, but then the nft rules are then no longer being added. I'm using dnsmasq-full 2.88 with dnsmasq nft set, but had the same behavior with dnsmasq ipset on dnsmasq-full 2.86. I thought that upgrading to the latest greatest might solve my issue, but unfortunately it makes no difference.

You can disable secure_reload to prevent traffic killswitch while pbr reloads.

No need to debug, the cause is right there in the log message -- pbr is being reloaded due to the firewall restart/reload.

You do need to figure out why the firewall is getting restarted/reloaded every few minutes, this is not normal.

Dear community, thank you for your development efforts and support.
I am in the situation when I do route all the LAN traffic to WireGuard VPN and on other hand need to be able to access LAN network from outside of the world. Incoming connections works when I disable VPN client interface on the router. I have read the side note for PBR,

If your router is set to use default routing via VPN tunnel and the WAN-targeting policies do not work, you need to stop your VPN tunnel first and ensure that you still have internet connection. If your router is set up to use the default routing via VPN tunnel and when you stop the VPN tunnel you have no internet connection, this package can’t help you. You first need to make sure that you do have internet connection when the VPN tunnel is stopped.

but maybe there might be some technical workaround for overcoming this ? Like separating to secondary LAN interface, with default routing to WAN and allow incoming connection to router ?

You help with examples would help a lot since my technical skill set is still on beginner level. Thank you.

root@Router:~# cat /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fde3:d3f0:9ebc::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'
	option peerdns '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'

config interface 'guest'
	option proto 'static'
	option ipaddr '192.168.10.1'
	option netmask '255.255.255.0'

config interface 'WG0'
	option proto 'wireguard'
	option peerdns '0'
	option mtu '1280'
	option private_key ''
	list addresses '172.16.0.2/32'
	list addresses 'fd01:5ca1:ab1e:849e:158f:6b7d:d692:3965/128'
	list dns '1.1.1.1'
	list dns '1.0.0.1'
	list dns '2606:4700:4700::1111'
	list dns '2606:4700:4700::1'

config wireguard_WG0
	option route_allowed_ips '1'
	option endpoint_port '2408'
	list allowed_ips '0.0.0.0/0'
	list allowed_ips '::/0'
	option description 'cloudflare_warp'
	option endpoint_host 'engage.cloudflareclient.com'
	option public_key ''
	option persistent_keepalive '25'

config interface 'wg1'
	option proto 'wireguard'
	option private_key ''
	option listen_port '51821'
	list addresses '192.168.9.1/24'
	list addresses 'fdf1:7610:d152:3a9c::1/64'

config wireguard_wg1 'wgclient'
	option preshared_key ''
	option description 'home'
	option public_key ''
	option private_key ''
	list allowed_ips '192.168.9.0/24'
	list allowed_ips 'fdf1:7610:d152:3a9c::1/64'
	list allowed_ips '192.168.9.11/32'
root@Router:~# cat /etc/config/firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option synflood_protect '1'
	option forward 'REJECT'
	option flow_offloading '1'
	option flow_offloading_hw '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'
	list network 'wg1'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'guest'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'REJECT'
	list network 'guest'

config rule
	option name 'Guest DNS'
	option src 'guest'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option name 'Guest DHCP'
	list proto 'udp'
	option src 'guest'
	option dest_port '67-68'
	option target 'ACCEPT'

config zone
	option name 'wgzone'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	list network 'WG0'
	option forward 'REJECT'
	option input 'REJECT'

config zone
	option name 'wan'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	option forward 'REJECT'
	option input 'REJECT'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'WOL'
	option src_dport '9'
	option dest_ip '192.168.1.233'
	option dest_port '9'
	option src 'lan'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'WOL2'
	option src 'wan'
	option src_dport '9'
	option dest_ip '192.168.1.233'
	option dest_port '9'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name '8080'
	list proto 'tcp'
	option src 'wgzone'
	option src_dport '8080'
	option dest_ip '192.168.1.108'
	option dest_port '8080'
	option enabled '0'

config forwarding
	option src 'lan'
	option dest 'wgzone'

config forwarding
	option src 'guest'
	option dest 'wgzone'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'http'
	list proto 'tcp'
	option src 'wan'
	option src_dport '80'
	option enabled '0'

config rule
	list proto 'tcp'
	option src 'wan'
	option src_port '80'
	option dest_port '80'
	option target 'ACCEPT'
	option name 'Allow-HTTP'
	option enabled '0'

config rule 'wg'
	option name 'Allow-WireGuard'
	option src 'wan'
	option dest_port '51821'
	option proto 'udp'
	option target 'ACCEPT'

Thanks! Unfortunately only this line appears in the log, nothing else. Is there a way to make the firewall more verbose so I can see the reason of it constantly reloading? Otherwise I'll just backup current config and start with a fresh installation.

No need to cross-post, like instructed by @psherman and README in another thread, if your default routing is via a VPN tunnel, you need to run an OpenVPN server in TCP mode to be able to access your LAN from outside of home network.

1 Like

Hello there.
I am trying to make use of two ISP service subscription, from eth0 and eth1.
I want to route traffic to different ISP based on LAN IP address.
ISP-A is IPv4 only, on "wan" and "wan6".
ISP-B is IPv4 and IPv6, on "HKT4" and "HKT6".
I have got "ERROR: Unknown fw_mark" error and no IPv6 route was created.

Thanks.

/etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option localservice '1'
	option ednspacket_max '1232'
	option rebind_protection '0'
	option noresolv '1'
	list server '127.0.0.1'
	list server '::ffff:7f00:1'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option ra_management '1'
	option ndp 'relay'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'other-config'
	list ra_flags 'managed-config'
	option ra_default '1'
	list dns 'dd16:8314:83b0:0000:0000:0000:0000:0001'

config dhcp 'HKT'
	option interface 'HKT'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'
	option ra_default '1'
	list dns 'dd16:8314:83b0:0002:0000:0000:0000:0001'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'
	list ra_flags 'none'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'guest'
	option interface 'guest'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'
	option ra_default '1'
	list dns 'dd16:8314:83b0:0001:0000:0000:0000:0001'
	list ra_flags 'other-config'
	list ra_flags 'managed-config'

config host
	option mac '00:E0:4C:02:27:36'
	option name 'telephone-booth'
	option dns '1'
	option ip '192.168.50.2'

config host
	option mac 'FC:34:97:C5:BE:57'
	option name 'the-core'
	option dns '1'
	option ip '192.168.50.3'

config host
	option mac '44:E4:EE:63:39:9E'
	option name 'LSPX-S2'
	option dns '1'
	option ip '192.168.50.4'

config host
	option mac '00:B0:2C:00:15:CB'
	option name 'Karaoke'
	option dns '1'
	option ip '192.168.50.5'

config host
	option mac '48:B0:2D:2D:D3:C3'
	option name 'Nvidia-Shield'
	option dns '1'
	option ip '192.168.50.6'

config host
	option mac 'EC:0D:E4:84:6D:3C'
	option name 'FireTV'
	option dns '1'
	option ip '192.168.50.7'

config host
	option mac '4C:53:FD:37:D0:39'
	option name 'FireHD-10'
	option dns '1'
	option ip '192.168.50.8'

config host
	option mac 'D8:BE:65:E2:7D:0B'
	option name 'FireHD-8'
	option dns '1'
	option ip '192.168.50.9'

config host
	option mac 'a8:a1:59:a6:f3:d1'
	option ip '192.168.50.10'
	option name 'DeskMini'

config host
	option mac '94:83:C4:0E:59:D2'
	option name 'GL-AR150'
	option dns '1'
	option ip '192.168.50.50'

config host
	option mac 'EC:0B:AE:23:DF:57'
	option name 'BroadLink'
	option dns '1'
	option ip '192.168.50.51'

config host
	option mac 'A0:43:B0:72:9E:1C'
	option name 'BroadLink'
	option dns '1'
	option ip '192.168.50.52'

config host
	option mac '44:5D:5E:02:79:84'
	option name 'SOnOff-SwitchBotHub'
	option dns '1'
	option ip '192.168.50.53'

config host
	option mac '94:B9:7E:DA:95:90'
	option name 'ESP32-SwitchBotHub'
	option dns '1'
	option ip '192.168.50.54'

config host
	option mac '38:56:10:C9:98:6B'
	option ip '192.168.50.55'
	option name 'Sesame'

config host
	option mac '7C:DD:E9:02:49:33'
	option name 'ATOM-3603'
	option dns '1'
	option ip '192.168.50.60'

config host
	option mac '7C:DD:E9:03:0B:74'
	option name 'ATOM-FRONT'
	option dns '1'
	option ip '192.168.50.63'

config host
	option mac '7C:DD:E9:03:0B:88'
	option name 'ATOM-REAR'
	option dns '1'
	option ip '192.168.50.64'

config host
	option mac '7C:DD:E9:01:B0:A8'
	option name 'ATOM-DESK'
	option dns '1'
	option ip '192.168.50.65'

config host
	option mac '08:E9:F6:92:90:58'
	option ip '192.168.50.81'
	option name 'Atmoph'
	option dns '1'

config host
	option mac '08:E9:F6:92:7B:96'
	option name 'Atmoph'
	option dns '1'
	option ip '192.168.50.82'

config host
	option mac '08:E9:F6:92:0B:0C'
	option name 'Atmoph'
	option dns '1'
	option ip '192.168.50.83'

config host
	option mac '38:1A:52:2E:17:8D'
	option dns '1'
	option ip '192.168.50.11'
	option name 'EPSON-XP4101'

config host
	option mac 'EC:71:DB:37:E1:42'
	option name 'Reolink-Entrance'
	option dns '1'
	option ip '192.168.50.61'

config host
	option mac '7C:DD:E9:03:04:61'
	option name 'ATOM-ENTRANCE'
	option dns '1'
	option ip '192.168.50.62'

config host
	option mac '94:83:C4:24:6A:28'
	option ip '192.168.50.100'
	option name 'GL-MT3000'

config dhcp 'HKT4'
	option ignore '1'
	option interface 'HKT4'
	list ra_flags 'none'

/etc/config/firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option output 'ACCEPT'
	option forward 'REJECT'
	option mtu_fix '1'
	option input 'DROP'
	option masq '1'
	option masq6 '1'
	list network 'wwan'
	list network 'wan'
	list network 'wan6'
	list network 'HKT4'
	list network 'HKT6'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled 'false'

config include
	option path '/etc/firewall.user'

config include 'nat6'
	option path '/etc/firewall.nat6'
	option reload '1'

config rule 'process_mark'
	option name 'process_mark'
	option dest '*'
	option proto 'all'
	option extra '-m owner --gid-owner 65533'
	option target 'MARK'
	option set_xmark '0x80000/0x80000'

config include 'gls2s'
	option type 'script'
	option path '/var/etc/gls2s.include'
	option reload '1'

config include 'glblock'
	option type 'script'
	option path '/usr/bin/gl_block.sh'
	option reload '1'

config zone
	option name 'guest'
	option forward 'REJECT'
	option output 'ACCEPT'
	option input 'REJECT'
	list network 'guest'

config forwarding
	option src 'guest'
	option dest 'wan'

config rule
	option name 'Allow-DHCP'
	option src 'guest'
	option target 'ACCEPT'
	option proto 'udp'
	option dest_port '67-68'

config rule
	option name 'Allow-DNS'
	option src 'guest'
	option target 'ACCEPT'
	option proto 'tcp udp'
	option dest_port '53'

config include 'vpn_server_policy'
	option type 'script'
	option path '/etc/firewall.vpn_server_policy.sh'
	option reload '1'
	option enabled '1'

config rule 'glipv6_guest_dhcp'
	option name 'Allow-DHCP-IPV6'
	option src 'guest'
	option target 'ACCEPT'
	option proto 'udp'
	option dest_port '546:547'
	option family 'ipv6'

config rule 'glipv6_guest_icmp'
	option name 'Allow-ICMP-IPV6'
	option src 'guest'
	option target 'ACCEPT'
	option proto 'icmp'
	option dest_port '58'
	option family 'ipv6'

config rule 'sambasharewan'
	option src 'wan'
	option dest_port '137 138 139 445'
	option dest_proto 'tcpudp'
	option target 'DROP'

config rule 'sambasharelan'
	option src 'lan'
	option dest_port '137 138 139 445'
	option dest_proto 'tcpudp'
	option target 'ACCEPT'

config rule 'glnas_ser'
	option src 'wan'
	option dest_port '6000-6002'
	option dest_proto 'tcp'
	option target 'DROP'

config rule 'webdav_wan'
	option src 'wan'
	option dest_port '6008'
	option dest_proto 'tcp'
	option target 'DROP'

config rule 'wgserver_allow'
	option name 'wgserver_allow'
	option target 'ACCEPT'
	option src 'wan'
	option proto 'udp tcp'
	option family 'ipv4'
	option dest_port '51820'
	option enabled '1'

config zone 'wgserver'
	option name 'wgserver'
	option output 'ACCEPT'
	option mtu_fix '1'
	option input 'DROP'
	option masq '1'
	option masq6 '1'
	option enabled '1'
	option forward 'REJECT'
	list network 'wgserver'

config forwarding 'wgserver2wan'
	option src 'wgserver'
	option dest 'wan'
	option enabled '1'

config forwarding 'lan2wgserver'
	option src 'lan'
	option dest 'wgserver'
	option enabled '1'

config redirect 'adguard_home'
	option src 'lan'
	option src_dport '53'
	option dest 'lan'
	option dest_port '3053'
	option name 'AdGuard'
	option proto 'tcpudp'

config forwarding
	option dest 'wan'
	option src 'lan'

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'dd16:8314:83b0::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	option multicast_to_unicast '0'
	list ports 'eth2'
	list ports 'eth3'
	list ports 'eth4'

config device
	option name 'eth1'
	option macaddr '94:83:c4:18:9d:18'

config device
	option name 'eth2'
	option macaddr '94:83:c4:18:9d:18'

config device
	option name 'eth3'
	option macaddr '94:83:c4:18:9d:18'

config device
	option name 'eth4'
	option macaddr '94:83:c4:18:9d:18'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option isolate '0'
	option ipaddr '192.168.50.1'
	option ip6assign '64'
	option ip6hint '0000'
	option ip6ifaceid '::1'
	option ip6class 'local'

config device
	option name 'eth0'
	option macaddr '94:83:c4:18:9d:17'

config interface 'wan'
	option device 'eth0'
	option proto 'dhcp'
	option ipv6 '1'
	list dns '1.1.1.1'
	list dns '1.0.0.1'
	option peerdns '0'

config interface 'wan6'
	option proto 'dhcpv6'
	option disabled '0'
	option device '@wan'
	option reqprefix 'auto'
	option reqaddress 'try'
	option peerdns '0'

config switch
	option name 'switch0'
	option reset '0'
	option enable_vlan '0'

config interface 'tethering6'
	option proto 'dhcpv6'
	option disabled '0'
	option device '@tethering'

config interface 'wwan6'
	option proto 'dhcpv6'
	option disabled '0'
	option device '@wwan'

config interface 'guest'
	option force_link '1'
	option type 'bridge'
	option proto 'static'
	option ipaddr '192.168.9.1'
	option netmask '255.255.255.0'
	option multicast_querier '1'
	option igmp_snooping '0'
	option isolate '0'
	option bridge_empty '1'
	option ip6prefix 'dd16:8314:83b0::/48'
	option ip6assign '64'
	option ip6hint '0001'
	option ip6ifaceid '::1'
	option ip6class 'guest'

config interface 'wwan'
	option proto 'dhcp'

config rule 'policy_bypass_vpn'
	option mark '0x60000/0x60000'
	option lookup '53'
	option priority '53'

config rule 'policy_via_vpn'
	option mark '0x80000/0x80000'
	option lookup '52'
	option priority '52'

config rule 'policy_dns'
	option mark '0x100000/0x100000'
	option lookup '51'
	option priority '51'

config interface 'wgserver'
	option proto 'wgserver'
	option config 'main_server'
	option disabled '0'

config interface 'HKT4'
	option proto 'dhcp'
	list dns '1.1.1.1'
	list dns '1.0.0.1'
	option peerdns '0'
	option device 'eth1'

config interface 'HKT6'
	option device '@HKT4'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'

/etc/config/vpn-policy-routing

config vpn-policy-routing 'config'
	option verbosity '2'
	option strict_enforcement '1'
	option src_ipset '0'
	option dest_ipset '0'
	option resolver_ipset 'dnsmasq.ipset'
	list ignored_interface 'vpnserver wgserver'
	option boot_timeout '30'
	option iptables_rule_option 'append'
	option procd_reload_delay '1'
	option webui_enable_column '0'
	option webui_protocol_column '0'
	option webui_chain_column '0'
	option webui_show_ignore_target '0'
	option webui_sorting '1'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	list webui_supported_protocol 'all'
	option ipv6_enabled '1'
	list supported_interface 'HKT4'
	list supported_interface 'HKT6'
	option enabled '1'

config include
	option path '/etc/vpn-policy-routing.netflix.user'
	option enabled '0'

config include
	option path '/etc/vpn-policy-routing.aws.user'
	option enabled '0'

config policy
	option name 'telephone-booth'
	option src_addr '192.168.50.2'
	option interface 'HKT4'

config policy
	option interface 'HKT6'
	option name 'telephone-booth'
	option src_addr '::ffff:c0a8:3202'

config policy
	option name 'the-core'
	option src_addr '192.168.50.3'
	option interface 'HKT4'

config policy
	option name 'the-core'
	option interface 'HKT6'
	option src_addr '::ffff:c0a8:3203'

config policy
	option interface 'wan'
	option name 'deskmini'
	option src_addr '192.168.50.10'

/etc/init.d/vpn-policy-routing support

vpn-policy-routing 0.3.4-8 running on OpenWrt 21.02-SNAPSHOT.
============================================================
Dnsmasq version 2.85  Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default         42-3-156-252.pt 0.0.0.0         UG    0      0        0 eth1

IPv4 Table 201: default via 223.17.77.1 dev eth0 
10.0.0.0/24 dev wgserver proto kernel scope link src 10.0.0.1 
192.168.50.0/24 dev br-lan proto kernel scope link src 192.168.50.1 
IPv4 Table 201 Rules:
50:	from all fwmark 0x10000/0xff0000 lookup wan

IPv4 Table 202: unreachable default 
10.0.0.0/24 dev wgserver proto kernel scope link src 10.0.0.1 
192.168.50.0/24 dev br-lan proto kernel scope link src 192.168.50.1 
IPv4 Table 202 Rules:
49:	from all fwmark 0x20000/0xff0000 lookup wwan

IPv4 Table 203: default via 42.3.156.252 dev eth1 
10.0.0.0/24 dev wgserver proto kernel scope link src 10.0.0.1 
192.168.50.0/24 dev br-lan proto kernel scope link src 192.168.50.1 
IPv4 Table 203 Rules:
48:	from all fwmark 0x30000/0xff0000 lookup HKT4

IPv4 Table 204: 
IPv4 Table 204 Rules:

IPv4 Table 205: 
IPv4 Table 205 Rules:

IPv4 Table 206: 
IPv4 Table 206 Rules:

IPv4 Table 207: 
IPv4 Table 207 Rules:

IPv4 Table 208: 
IPv4 Table 208 Rules:

IPv4 Table 209: 
IPv4 Table 209 Rules:
IPv6 Table 201: default from 2404:c800:913b:1c9a::/64 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 201: default from 2404:c805:3b4f:7900::/56 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 201: 2404:c800:913b:1c9a::/64 dev eth1 proto static metric 256 pref medium
IPv6 Table 201: fe80::/64 dev eth1 proto kernel metric 256 pref medium
IPv6 Table 202: default from 2404:c800:913b:1c9a::/64 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 202: default from 2404:c805:3b4f:7900::/56 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 202: 2404:c800:913b:1c9a::/64 dev eth1 proto static metric 256 pref medium
IPv6 Table 202: fe80::/64 dev eth1 proto kernel metric 256 pref medium
IPv6 Table 203: default from 2404:c800:913b:1c9a::/64 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 203: default from 2404:c805:3b4f:7900::/56 via fe80::ff:fe02:202 dev eth1 proto static metric 512 pref medium
IPv6 Table 203: 2404:c800:913b:1c9a::/64 dev eth1 proto static metric 256 pref medium
IPv6 Table 203: fe80::/64 dev eth1 proto kernel metric 256 pref medium
============================================================
Mangle IP Table: PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -m set --match-set HKT4 dst -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -m set --match-set wwan dst -c 0 0 -g VPR_MARK0x020000
-A VPR_PREROUTING -m set --match-set wan dst -c 0 0 -g VPR_MARK0x010000
-A VPR_PREROUTING -s 192.168.50.2/32 -m comment --comment telephone-booth -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -s 192.168.50.3/32 -m comment --comment the-core -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -s 192.168.50.10/32 -m comment --comment deskmini -c 4455 515987 -g VPR_MARK0x010000
============================================================
Mangle IPv6 Table: PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -m set --match-set HKT46 dst -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -m set --match-set wwan6 dst -c 0 0 -g VPR_MARK0x020000
-A VPR_PREROUTING -m set --match-set wan6 dst -c 0 0 -g VPR_MARK0x010000
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x010000
-N VPR_MARK0x010000
-A VPR_MARK0x010000 -c 4463 516403 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_MARK0x010000 -c 4463 516403 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x020000
-N VPR_MARK0x020000
-A VPR_MARK0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_MARK0x020000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x030000
-N VPR_MARK0x030000
-A VPR_MARK0x030000 -c 0 0 -j MARK --set-xmark 0x30000/0xff0000
-A VPR_MARK0x030000 -c 0 0 -j RETURN
============================================================
NAT IP Table: PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -p udp -m udp --dport 443 -m set --match-set tor dst -m comment --comment TorHTTPS-UDP -c 0 0 -j REDIRECT --to-ports 9040
-A VPR_PREROUTING -p tcp -m tcp --dport 443 -m set --match-set tor dst -m comment --comment TorHTTPS-TCP -c 0 0 -j REDIRECT --to-ports 9040
-A VPR_PREROUTING -p udp -m udp --dport 80 -m set --match-set tor dst -m comment --comment TorHTTP-UDP -c 0 0 -j REDIRECT --to-ports 9040
-A VPR_PREROUTING -p tcp -m tcp --dport 80 -m set --match-set tor dst -m comment --comment TorHTTP-TCP -c 0 0 -j REDIRECT --to-ports 9040
-A VPR_PREROUTING -p udp -m udp --dport 53 -m set --match-set tor dst -m comment --comment TorDNS-UDP -c 0 0 -j REDIRECT --to-ports 9053
============================================================
NAT IPv6 Table: PREROUTING
-N VPR_PREROUTING
============================================================
Current ipsets
create GL_MAC_BLOCK hash:mac hashsize 1024 maxelem 65536
create via_vpn_domain hash:net family inet hashsize 1024 maxelem 65536
create bypass_vpn_domain hash:net family inet hashsize 1024 maxelem 65536
create via_vpn_mac hash:mac hashsize 1024 maxelem 65536
create bypass_vpn_mac hash:mac hashsize 1024 maxelem 65536
create mwan3_connected_v4 hash:net family inet hashsize 1024 maxelem 65536
add mwan3_connected_v4 10.0.0.0/24
add mwan3_connected_v4 192.168.50.0/24
add mwan3_connected_v4 223.17.77.0/24
add mwan3_connected_v4 224.0.0.0/3
add mwan3_connected_v4 127.0.0.0/8
add mwan3_connected_v4 42.3.156.0/24
add mwan3_connected_v4 192.168.9.0/24
create mwan3_connected_v6 hash:net family inet6 hashsize 1024 maxelem 65536
add mwan3_connected_v6 dd16:8314:83b0:0:e64:35bf:5e78:11b2
add mwan3_connected_v6 dd16:8314:83b0:0:5cde:d802:313e:98da
add mwan3_connected_v6 dd16:8314:83b0:0:b150:bb9a:c6d5:c94a
add mwan3_connected_v6 dd16:8314:83b0:0:f8c2:7be0:ed04:3f2b
add mwan3_connected_v6 dd16:8314:83b0:0:a48f:8798:17a:c58c
add mwan3_connected_v6 dd16:8314:83b0:0:1945:f369:32ed:619d
add mwan3_connected_v6 2404:c800:913b:1c9a::/64
add mwan3_connected_v6 fe80::/64
add mwan3_connected_v6 dd16:8314:83b0:0:3584:1849:12ed:3f2a
add mwan3_connected_v6 dd16:8314:83b0:1::/64
add mwan3_connected_v6 dd16:8314:83b0::e04
add mwan3_connected_v6 dd16:8314:83b0::/64
add mwan3_connected_v6 dd16:8314:83b0:0:1c8e:fff:fe7b:11b1
add mwan3_connected_v6 dd16:8314:83b0:0:99a8:8e0:b5dc:cdd
add mwan3_connected_v6 fd00:db8:0:abc::/64
create mwan3_source_v6 hash:net family inet6 hashsize 1024 maxelem 65536
add mwan3_source_v6 2404:c800:913b:1c9a:9683:c4ff:fe18:9d18
add mwan3_source_v6 dd16:8314:83b0::1
add mwan3_source_v6 fd00:db8:0:abc::1
add mwan3_source_v6 dd16:8314:83b0:1::1
create mwan3_dynamic_v4 hash:net family inet hashsize 1024 maxelem 65536
create mwan3_dynamic_v6 hash:net family inet6 hashsize 1024 maxelem 65536
create mwan3_custom_v4 hash:net family inet hashsize 1024 maxelem 65536
create mwan3_custom_v6 hash:net family inet6 hashsize 1024 maxelem 65536
create wan6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create wwan6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create wwan hash:net family inet hashsize 1024 maxelem 65536 comment
create HKT46 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create HKT4 hash:net family inet hashsize 1024 maxelem 65536 comment
create tor6 hash:net family inet6 hashsize 1024 maxelem 65536 comment
create tor hash:net family inet hashsize 1024 maxelem 65536 comment
create mwan3_connected list:set size 8
add mwan3_connected mwan3_connected_v4
add mwan3_connected mwan3_connected_v6
add mwan3_connected mwan3_dynamic_v4
add mwan3_connected mwan3_dynamic_v6
add mwan3_connected mwan3_custom_v4
add mwan3_connected mwan3_custom_v6
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [βœ“]

/etc/init.d/vpn-policy-routing reload

Creating table 'wan/eth0/223.17.77.1/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64' [βœ“]
Creating table 'wwan//0.0.0.0/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64' [βœ“]
Creating table 'HKT4/eth1/42.3.156.252/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64' [βœ“]
WARNING: Variable 'tor' does not exist or is not an array/object
WARNING: Variable 'instances' does not exist or is not an array/object
WARNING: Variable 'instance1' does not exist or is not an array/object
Creating TOR redirects [βœ“]
Routing 'telephone-booth' via HKT4 [βœ“]
Routing 'telephone-booth' via HKT6 [βœ—]
Routing 'the-core' via HKT4 [βœ“]
Routing 'the-core' via HKT6 [βœ—]
Routing 'deskmini' via wan [βœ“]
vpn-policy-routing 0.3.4-8 monitoring interfaces: wan wwan HKT4 HKT6 [βœ“]
vpn-policy-routing 0.3.4-8 started with gateways:
wan/eth0/223.17.77.1/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64
wwan//0.0.0.0/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64
HKT4/eth1/42.3.156.252/2404:c800:913b:1c9a:8403:720a:b149:ef6a/64
2404:c800:913b:1c9a:9683:c4ff:fe18:9d18/64
fe80::9683:c4ff:fe18:9d18/64 [βœ“]
tor/53->9053/80,443->9040
ERROR: Unknown fw_mark for HKT6

ERROR: Unknown fw_mark for HKT6

Sorry, that's too old for both package and the OS.

If you want to switch to a current version of pbr and something more modern and reproducible than 21.02-SNAPSHOT, please post the configs when you are on such a system/version and I'll see what can be done.

1 Like

Hi, I'm newbie here. I have IPs list and all traffic to that IPs must go through WireGuard connection.
Can't do that even with two provided example user files.

Read the very first post of this thread.

Installed on another system and put back my packages one by one. This way I figured out snmpd is the culprit, now disabled I have no longer pbr reloaded messages every few minutes. It seems unfortunate to disable snmpd, because I use it quite extensively. Now I need to figure out how to get this working without interfering with the firewall, but that's out of this thread's scope :slight_smile:

Edit: I have installed mini_snmpd instead of snmpd and so far it doesn't seem to interfere with the firewall!

1 Like

Trying to get pbr working using wan and 3 VPNs (1 Wireguard (tun_w) and 2 OpenVpn (tun_o1, tun_o2)).
I'm on OpenWrt 22.03.3 x86/64, tried both with and without the dnsmasq-full's nft sets.
Used to work fine on previous OpenWrt versions using the vpn-policy-routing.
No matter what I try, I'm only able to switch between wan and the tun_w. The strange thing is that occassionaly assigning the iterface to one of the tun_o works for some reason, but then it goes down by itself.

Screenshot

All in all, it goes as follows:

pbr setting:  effective:
tun_o1        tun_w
tun_o2        tun_w (sometimes tun_o1 (not a typo))
tun_w         tun_w
br-wan        br-wan

I've also opened a ticket at GitHub with more config details: https://github.com/stangri/repo.openwrt.melmac.net/issues/19

Not sure what is possibly wrong with my config and why it's not working.

Already replied in the issue you've also created: https://github.com/stangri/repo.openwrt.melmac.net/issues/19

Sorry, missed that Getting help part in the readme.
One question though, how do I set the verbosity to 2?

Nevermind, I thought that it's about some global OpenWrt verbosity setting. Posted the configs and logs at GitHub.

quick question is there a way to have a domain list that get routed via VPN?

i dont want to add via UI 40 domains

Footnote #5

sorry huh, how is that relevant to my question

adguardhome.ipset, dnsmasq.ipset or dnsmasq.nftset

i still dont follow you

@stangri
my vpn always gives me a moderate nat is it possible to make a hybrid vpn like dumOS by adding port numbers? thanks

like on the picture

config pbr 'config'
  list ignored_interface 'vpnserver'
  ...

config policy
  option name 'OpenVPN Server'
  option interface 'wan'
  option proto 'tcp'
  option src_port '1194'
  option chain 'output'

or rules ?

config rule
  option name 'Allow-OpenVPN-Inbound'
  option target 'ACCEPT'
  option src '*'
  option proto 'tcp'
  option dest_port '1194'