Problem with PBR getting to work with IPv6

Quokka · July 8, 2024, 12:57pm

@stangri: understanding that you are the author of the pbr package, I’d like to consult you (and others knowledgeable on the subject ) on the following.

Being still a noob when it comes to OpenWrt, I recently flashed a x86/64 device with OpenWrt. I succeeded in having my system not only function as a router and wireless access point, but I also configured OpenVPN to the point where I was able to make IPv4 connections with servers of my VPN provider. Thanks to the generous help of several contributors to this forum it is now also possible to establish VPN connections over IPv6. When connected to a specific VPN server and checking my connection at the website of my VPN provider, I see something like this:

This weekend I took the next step by starting to experiment with pbr, my guiding being your excellent tutorial Welcome to docs.openwrt.melmac.net!. What I would like to achieve is that, whereas most of the devices in my network access the internet through the default gateway (being the closest VPN server or the VPN server with the fastest connection), some of them should be subjected to a policy that dictates them to connect to a specific VPN server of my choice. I have gotten as far as to make this work for IPv4, but with respect to IPv6 I am running into a curious anomaly I am not able to solve myself. To explain my current situation, the following.

I configured a number of OpenVPN instances of which for experiment's sake only two are active: a VPN server in Amsterdam and one in Stockholm:

Given this, the VPN server in Amsterdam (vpnclient0) acts as the main gateway:

I now created a policy which tells my MacBook Pro to connect to the internet through the VPN server in Stockholm (vpnclient9):

When I subsequently tested my connections with the IP test of my VPN provider, I got the following astonishing result:

When it comes to IPv4, my MacBook is indeed connected to the VPN server in Stockholm (vpnclient9) as my policy dictates. With respect to IPv6 though, the policy is completely negated, since I am still connected to the default gateway, being the VPN server in Amsterdam (vpnclient0). The latter of course should also be the VPN server in Stockholm.

I do know problems with pbr in relation to IPv6 have been reported by some others as well (https://github.com/openwrt/packages/issues/20352), and I have tried to find a solution by following the advice given at https://gist.github.com/aliicex/3bd8413029b1f728c1f00bc1ac0e98b4. This, however, only led to my internet connection completely being disrupted, so I quickly returned to the status quo ante.

Anyway, whether my current issue is due to a configuration error on my part, or there is something amiss with package pbr in relation to IPv6, is a question I am not able to answer (although the former option is the most likely ). Any help you may be able to offer to reach a solution will be much appreciated, since being able to reliably configure policies is important to what I would like to achieve with my OpenWrt router. To enable you to analyse my issue, I attach the following output. Many thanks in advance for your insights!

ubus call system board:

Summary

root@OpenWrt:~# ubus call system board
{
	"kernel": "5.15.150",
	"hostname": "OpenWrt",
	"system": "Intel(R) Celeron(R) J6412 @ 2.00GHz",
	"model": "Protectli VP2420",
	"board_name": "protectli-vp2420",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.3",
		"revision": "r23809-234f1a2efa",
		"target": "x86/64",
		"description": "OpenWrt 23.05.3 r23809-234f1a2efa"
	}
}
root@OpenWrt:~#

uci export dhcp:

Summary

root@OpenWrt:~# uci export dhcp
package dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.vpn'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '2'
	option limit '150'
	option leasetime '2m'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option preferred_lifetime '12h'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'
	option preferred_lifetime '12h'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'vpnclient0'
	option interface 'vpnclient0'
	option ignore '1'

config dhcp 'vpnclient1'
	option interface 'vpnclient1'
	option ignore '1'

config dhcp 'vpnclient2'
	option interface 'vpnclient2'
	option ignore '1'

config dhcp 'vpnclient3'
	option interface 'vpnclient3'
	option ignore '1'

config dhcp 'vpnclient4'
	option interface 'vpnclient4'
	option ignore '1'

config dhcp 'vpnclient5'
	option interface 'vpnclient5'
	option ignore '1'

config dhcp 'vpnclient6'
	option interface 'vpnclient6'
	option ignore '1'

config dhcp 'vpnclient7'
	option interface 'vpnclient7'
	option ignore '1'

config dhcp 'vpnclient8'
	option interface 'vpnclient8'
	option ignore '1'

config dhcp 'vpnclient9'
	option interface 'vpnclient9'
	option ignore '1'

config host
	option name 'Mac-Studio-M1-Max'
	option ip '1x.xxx.x.8’
	list mac '9C:xx:xx:xx:xx:7B'
	option leasetime 'infinite'

config host
	option name 'MacBook-Pro-M3'
	option ip '1x.xxx.x.x7'
	option leasetime 'infinite'
	list mac '10:xx:xx:xx:xx:9C'
	option duid '0001xxxxxxxxxxxxxxxxxxxxxx9c'

root@OpenWrt:~#

uci export firewall:

Summary

root@OpenWrt:~# uci export firewall
package firewall

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	option masq6 '1'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'vpn_fw'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option masq6 '1'
	option mtu_fix '1'
	list network 'vpnclient0'
	list network 'vpnclient1'
	list network 'vpnclient2'
	list network 'vpnclient3'
	list network 'vpnclient4'
	list network 'vpnclient5'
	list network 'vpnclient6'
	list network 'vpnclient7'
	list network 'vpnclient8'
	list network 'vpnclient9'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/pbr.firewall.include'

config forwarding
	option src 'lan'
	option dest 'vpn_fw'

root@OpenWrt:~#

uci export network:

Summary

root@OpenWrt:~# uci export network
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ip6assign '60'
	option broadcast '1x.xxx.x.255'
	list ipaddr '1x.xxx.x.1/24'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	list dns '193.110.81.9'
	list dns '185.253.5.9'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	list dns '2a0f:fc80::9'
	list dns '2a0f:fc81::9'

config interface 'vpnclient0'
	option proto 'none'
	option device 'vpn_amsterdam'

config interface 'vpnclient1'
	option proto 'none'
	option device 'vpn_berlin'

config interface 'vpnclient2'
	option proto 'none'
	option device 'vpn_copenhagen'

config interface 'vpnclient3'
	option proto 'none'
	option device 'vpn_frankfurt'

config interface 'vpnclient4'
	option proto 'none'
	option device 'vpn_hamburg'

config interface 'vpnclient5'
	option proto 'none'
	option device 'vpn_malmoe'

config interface 'vpnclient6'
	option proto 'none'
	option device 'vpn_oslo'

config interface 'vpnclient7'
	option proto 'none'
	option device 'vpn_paris'

config interface 'vpnclient8'
	option proto 'none'
	option device 'vpn_rotterdam'

config interface 'vpnclient9'
	option proto 'none'
	option device 'vpn_stockholm'

root@OpenWrt:~#

uci export pbr:

Summary

root@OpenWrt:~# uci export pbr
package per

config pbr 'config'
	option enabled '1'
	option verbosity '2'
	option strict_enforcement '1'
	option resolver_set 'dnsmasq.nftset'
	option ipv6_enabled '1'
	list ignored_interface 'vpnserver'
	list ignored_interface 'wgserver'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '1'
	option webui_show_ignore_target '0'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	list supported_interface 'vpnclient0 vpnclient1 vpnclient2 vpnclient3 vpnclient4 vpnclient5 vpnclient6 vpnclient7 vpnclient8 vpnclient9'
	list supported_interface 'wan'

config include
	option path '/usr/share/pbr/pbr.user.aws'
	option enabled '0'

config include
	option path '/usr/share/pbr/pbr.user.netflix'
	option enabled '0'

config policy
	option name 'Apple Mac'
	option src_addr '1x.xxx.x.x7'
	option interface 'vpnclient9'

root@OpenWrt:~#

/etc/init.d/pbr status [1]:

Summary

root@OpenWrt:~# /etc/init.d/pbr status
============================================================
pbr - environment
pbr 1.1.1-7 running on OpenWrt 23.05.3. WAN (IPv4): wan/eth1/192.168.188.1. WAN (IPv6): wan6/eth1/2a02:a464:c5d0:1:6662:66ff:fe22:d598.
============================================================
Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
	chain pbr_forward { # handle 36
	}
	chain pbr_input { # handle 37
	}
	chain pbr_output { # handle 38
	}
	chain pbr_prerouting { # handle 39
		ip saddr @pbr_vpnclient9_4_src_ip_cfg046ff5 goto pbr_mark_0x0b0000 comment "Apple Mac" # handle 1008
		ip6 saddr @pbr_vpnclient9_6_src_ip_cfg046ff5 goto pbr_mark_0x0b0000 comment "Apple Mac" # handle 1009
	}
	chain pbr_postrouting { # handle 40
	}
============================================================
pbr chains - marking
	chain pbr_mark_0x010000 { # handle 973
		counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 974
		return # handle 975
	}
	chain pbr_mark_0x020000 { # handle 976
		counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 977
		return # handle 978
	}
	chain pbr_mark_0x030000 { # handle 979
		counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 980
		return # handle 981
	}
	chain pbr_mark_0x040000 { # handle 982
		counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000 # handle 983
		return # handle 984
	}
	chain pbr_mark_0x050000 { # handle 985
		counter packets 0 bytes 0 meta mark set meta mark & 0xff05ffff | 0x00050000 # handle 986
		return # handle 987
	}
	chain pbr_mark_0x060000 { # handle 988
		counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000 # handle 989
		return # handle 990
	}
	chain pbr_mark_0x070000 { # handle 991
		counter packets 0 bytes 0 meta mark set meta mark & 0xff07ffff | 0x00070000 # handle 992
		return # handle 993
	}
	chain pbr_mark_0x080000 { # handle 994
		counter packets 0 bytes 0 meta mark set meta mark & 0xff08ffff | 0x00080000 # handle 995
		return # handle 996
	}
	chain pbr_mark_0x090000 { # handle 997
		counter packets 0 bytes 0 meta mark set meta mark & 0xff09ffff | 0x00090000 # handle 998
		return # handle 999
	}
	chain pbr_mark_0x0a0000 { # handle 1000
		counter packets 0 bytes 0 meta mark set meta mark & 0xff0affff | 0x000a0000 # handle 1001
		return # handle 1002
	}
	chain pbr_mark_0x0b0000 { # handle 1003
		counter packets 8877 bytes 1246844 meta mark set meta mark & 0xff0bffff | 0x000b0000 # handle 1004
		return # handle 1005
	}
============================================================
pbr nft sets
	set pbr_vpnclient9_4_src_ip_cfg046ff5 { # handle 1006
		type ipv4_addr
		flags interval
		counter
		auto-merge
		comment "Apple Mac"
		elements = { 10.210.0.57 counter packets 8879 bytes 1246948 }
	}
	set pbr_vpnclient9_6_src_ip_cfg046ff5 { # handle 1007
		type ipv6_addr
		flags interval
		counter
		auto-merge
		comment "Apple Mac"
	}
============================================================
IPv4 table 256 route: default via 192.168.188.1 dev eth1 
IPv4 table 256 rule(s):
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv6 table 256 route: default from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
default from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
unreachable default dev lo metric 1024 pref medium
IPv6 table 256 rule(s):
default from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
default from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0::/48 from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0::/48 from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0:1::/64 dev eth1 proto static metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo metric 1024 pref medium
IPv4 table 257 route: default via 10.0.48.249 dev vpn_amsterdam 
IPv4 table 257 rule(s):
30001:	from all fwmark 0x20000/0xff0000 lookup pbr_vpnclient0
IPv6 table 257 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 257 rule(s):
::/3 dev vpn_amsterdam metric 1024 pref medium
2000::/4 dev vpn_amsterdam metric 1024 pref medium
3000::/4 dev vpn_amsterdam metric 1024 pref medium
2000::/3 dev vpn_amsterdam metric 1024 pref medium
fdbf:1d37:bbe0:0:3:8::/112 dev vpn_amsterdam proto kernel metric 256 pref medium
fc00::/7 dev vpn_amsterdam metric 1024 pref medium
fe80::/64 dev vpn_amsterdam proto kernel metric 256 pref medium
unreachable default dev lo metric 1024 pref medium
IPv4 table 258 route: unreachable default 
IPv4 table 258 rule(s):
30002:	from all fwmark 0x30000/0xff0000 lookup pbr_vpnclient1
IPv6 table 258 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 258 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 259 route: unreachable default 
IPv4 table 259 rule(s):
30003:	from all fwmark 0x40000/0xff0000 lookup pbr_vpnclient2
IPv6 table 259 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 259 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 260 route: unreachable default 
IPv4 table 260 rule(s):
30004:	from all fwmark 0x50000/0xff0000 lookup pbr_vpnclient3
IPv6 table 260 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 260 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 261 route: unreachable default 
IPv4 table 261 rule(s):
30005:	from all fwmark 0x60000/0xff0000 lookup pbr_vpnclient4
IPv6 table 261 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 261 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 262 route: unreachable default 
IPv4 table 262 rule(s):
30006:	from all fwmark 0x70000/0xff0000 lookup pbr_vpnclient5
IPv6 table 262 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 262 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 263 route: unreachable default 
IPv4 table 263 rule(s):
30007:	from all fwmark 0x80000/0xff0000 lookup pbr_vpnclient6
IPv6 table 263 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 263 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 264 route: unreachable default 
IPv4 table 264 rule(s):
30008:	from all fwmark 0x90000/0xff0000 lookup pbr_vpnclient7
IPv6 table 264 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 264 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 265 route: unreachable default 
IPv4 table 265 rule(s):
30009:	from all fwmark 0xa0000/0xff0000 lookup pbr_vpnclient8
IPv6 table 265 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 265 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 266 route: default via 10.4.19.16 dev vpn_stockholm 
IPv4 table 266 rule(s):
30010:	from all fwmark 0xb0000/0xff0000 lookup pbr_vpnclient9
IPv6 table 266 route: 
IPv6 table 266 rule(s):
::/3 dev vpn_stockholm metric 1024 pref medium
2000::/4 dev vpn_stockholm metric 1024 pref medium
3000::/4 dev vpn_stockholm metric 1024 pref medium
2000::/3 dev vpn_stockholm metric 1024 pref medium
fdbf:1d37:bbe0:0:65:11::/112 dev vpn_stockholm proto kernel metric 256 pref medium
fc00::/7 dev vpn_stockholm metric 1024 pref medium
fe80::/64 dev vpn_stockholm proto kernel metric 256 pref medium
root@OpenWrt:~#

/etc/init.d/pbr reload:

Summary

root@OpenWrt:~# /etc/init.d/pbr reload
Activating traffic killswitch [✓]
Setting up routing for 'wan/eth1/192.168.188.1/2a02:a464:c5d0:1:6662:66ff:fe22:d598/128' [✓]
Setting up routing for 'vpnclient0/vpn_amsterdam/10.0.48.249/fdbf:1d37:bbe0:0:3:8:0:f9/112' [✓]
Setting up routing for 'vpnclient1/vpn_berlin/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient2/vpn_copenhagen/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient3/vpn_frankfurt/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient4/vpn_hamburg/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient5/vpn_malmoe/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient6/vpn_oslo/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient7/vpn_paris/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient8/vpn_rotterdam/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient9/vpn_stockholm/10.4.19.16/fdbf:1d37:bbe0:0:65:11:0:10/112' [✓]
Routing 'Apple Mac' via vpnclient9 [✓]
Deactivating traffic killswitch [✓]
pbr 1.1.1-7 monitoring interfaces: wan vpnclient0 vpnclient1 vpnclient2 vpnclient3 vpnclient4 vpnclient5 vpnclient6 vpnclient7 vpnclient8 vpnclient9 
pbr 1.1.1-7 (nft) started with gateways:
wan/eth1/192.168.188.1/2a02:a464:c5d0:1:6662:66ff:fe22:d598/128
vpnclient0/vpn_amsterdam/10.0.48.249/fdbf:1d37:bbe0:0:3:8:0:f9/112 [✓]
vpnclient1/vpn_berlin/0.0.0.0/::/0
vpnclient2/vpn_copenhagen/0.0.0.0/::/0
vpnclient3/vpn_frankfurt/0.0.0.0/::/0
vpnclient4/vpn_hamburg/0.0.0.0/::/0
vpnclient5/vpn_malmoe/0.0.0.0/::/0
vpnclient6/vpn_oslo/0.0.0.0/::/0
vpnclient7/vpn_paris/0.0.0.0/::/0
vpnclient8/vpn_rotterdam/0.0.0.0/::/0
vpnclient9/vpn_stockholm/10.4.19.16/fdbf:1d37:bbe0:0:65:11:0:10/112
root@OpenWrt:~#

/etc/init.d/pbr status [2]:

Summary

root@OpenWrt:~# /etc/init.d/pbr status
============================================================
pbr - environment
pbr 1.1.1-7 running on OpenWrt 23.05.3. WAN (IPv4): wan/eth1/192.168.188.1. WAN (IPv6): wan6/eth1/2a02:a464:c5d0:1:6662:66ff:fe22:d598.
============================================================
Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
	chain pbr_forward { # handle 36
	}
	chain pbr_input { # handle 37
	}
	chain pbr_output { # handle 38
	}
	chain pbr_prerouting { # handle 39
		ip saddr @pbr_vpnclient9_4_src_ip_cfg046ff5 goto pbr_mark_0x0b0000 comment "Apple Mac" # handle 1086
		ip6 saddr @pbr_vpnclient9_6_src_ip_cfg046ff5 goto pbr_mark_0x0b0000 comment "Apple Mac" # handle 1087
	}
	chain pbr_postrouting { # handle 40
	}
============================================================
pbr chains - marking
	chain pbr_mark_0x010000 { # handle 1051
		counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1052
		return # handle 1053
	}
	chain pbr_mark_0x020000 { # handle 1054
		counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1055
		return # handle 1056
	}
	chain pbr_mark_0x030000 { # handle 1057
		counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 1058
		return # handle 1059
	}
	chain pbr_mark_0x040000 { # handle 1060
		counter packets 0 bytes 0 meta mark set meta mark & 0xff04ffff | 0x00040000 # handle 1061
		return # handle 1062
	}
	chain pbr_mark_0x050000 { # handle 1063
		counter packets 0 bytes 0 meta mark set meta mark & 0xff05ffff | 0x00050000 # handle 1064
		return # handle 1065
	}
	chain pbr_mark_0x060000 { # handle 1066
		counter packets 0 bytes 0 meta mark set meta mark & 0xff06ffff | 0x00060000 # handle 1067
		return # handle 1068
	}
	chain pbr_mark_0x070000 { # handle 1069
		counter packets 0 bytes 0 meta mark set meta mark & 0xff07ffff | 0x00070000 # handle 1070
		return # handle 1071
	}
	chain pbr_mark_0x080000 { # handle 1072
		counter packets 0 bytes 0 meta mark set meta mark & 0xff08ffff | 0x00080000 # handle 1073
		return # handle 1074
	}
	chain pbr_mark_0x090000 { # handle 1075
		counter packets 0 bytes 0 meta mark set meta mark & 0xff09ffff | 0x00090000 # handle 1076
		return # handle 1077
	}
	chain pbr_mark_0x0a0000 { # handle 1078
		counter packets 0 bytes 0 meta mark set meta mark & 0xff0affff | 0x000a0000 # handle 1079
		return # handle 1080
	}
	chain pbr_mark_0x0b0000 { # handle 1081
		counter packets 45 bytes 3170 meta mark set meta mark & 0xff0bffff | 0x000b0000 # handle 1082
		return # handle 1083
	}
============================================================
pbr nft sets
	set pbr_vpnclient9_4_src_ip_cfg046ff5 { # handle 1084
		type ipv4_addr
		flags interval
		counter
		auto-merge
		comment "Apple Mac"
		elements = { 10.210.0.57 counter packets 46 bytes 3222 }
	}
	set pbr_vpnclient9_6_src_ip_cfg046ff5 { # handle 1085
		type ipv6_addr
		flags interval
		counter
		auto-merge
		comment "Apple Mac"
	}
============================================================
IPv4 table 256 route: default via 192.168.188.1 dev eth1 
IPv4 table 256 rule(s):
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv6 table 256 route: default from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
default from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
unreachable default dev lo metric 1024 pref medium
IPv6 table 256 rule(s):
default from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
default from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0::/48 from 2a02:a464:c5d0:1:6662:66ff:fe22:d598 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0::/48 from 2a02:a464:c5d0:fc::/62 via fe80::2e91:abff:fe45:f3b5 dev eth1 proto static metric 384 pref medium
2a02:a464:c5d0:1::/64 dev eth1 proto static metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
unreachable default dev lo metric 1024 pref medium
IPv4 table 257 route: default via 10.0.48.249 dev vpn_amsterdam 
IPv4 table 257 rule(s):
30001:	from all fwmark 0x20000/0xff0000 lookup pbr_vpnclient0
IPv6 table 257 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 257 rule(s):
::/3 dev vpn_amsterdam metric 1024 pref medium
2000::/4 dev vpn_amsterdam metric 1024 pref medium
3000::/4 dev vpn_amsterdam metric 1024 pref medium
2000::/3 dev vpn_amsterdam metric 1024 pref medium
fdbf:1d37:bbe0:0:3:8::/112 dev vpn_amsterdam proto kernel metric 256 pref medium
fc00::/7 dev vpn_amsterdam metric 1024 pref medium
fe80::/64 dev vpn_amsterdam proto kernel metric 256 pref medium
unreachable default dev lo metric 1024 pref medium
IPv4 table 258 route: unreachable default 
IPv4 table 258 rule(s):
30002:	from all fwmark 0x30000/0xff0000 lookup pbr_vpnclient1
IPv6 table 258 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 258 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 259 route: unreachable default 
IPv4 table 259 rule(s):
30003:	from all fwmark 0x40000/0xff0000 lookup pbr_vpnclient2
IPv6 table 259 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 259 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 260 route: unreachable default 
IPv4 table 260 rule(s):
30004:	from all fwmark 0x50000/0xff0000 lookup pbr_vpnclient3
IPv6 table 260 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 260 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 261 route: unreachable default 
IPv4 table 261 rule(s):
30005:	from all fwmark 0x60000/0xff0000 lookup pbr_vpnclient4
IPv6 table 261 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 261 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 262 route: unreachable default 
IPv4 table 262 rule(s):
30006:	from all fwmark 0x70000/0xff0000 lookup pbr_vpnclient5
IPv6 table 262 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 262 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 263 route: unreachable default 
IPv4 table 263 rule(s):
30007:	from all fwmark 0x80000/0xff0000 lookup pbr_vpnclient6
IPv6 table 263 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 263 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 264 route: unreachable default 
IPv4 table 264 rule(s):
30008:	from all fwmark 0x90000/0xff0000 lookup pbr_vpnclient7
IPv6 table 264 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 264 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 265 route: unreachable default 
IPv4 table 265 rule(s):
30009:	from all fwmark 0xa0000/0xff0000 lookup pbr_vpnclient8
IPv6 table 265 route: unreachable default dev lo metric 1024 pref medium
IPv6 table 265 rule(s):
unreachable default dev lo metric 1024 pref medium
IPv4 table 266 route: default via 10.4.19.16 dev vpn_stockholm 
IPv4 table 266 rule(s):
30010:	from all fwmark 0xb0000/0xff0000 lookup pbr_vpnclient9
IPv6 table 266 route: 
IPv6 table 266 rule(s):
::/3 dev vpn_stockholm metric 1024 pref medium
2000::/4 dev vpn_stockholm metric 1024 pref medium
3000::/4 dev vpn_stockholm metric 1024 pref medium
2000::/3 dev vpn_stockholm metric 1024 pref medium
fdbf:1d37:bbe0:0:65:11::/112 dev vpn_stockholm proto kernel metric 256 pref medium
fc00::/7 dev vpn_stockholm metric 1024 pref medium
fe80::/64 dev vpn_stockholm proto kernel metric 256 pref medium
root@OpenWrt:~#

egc · July 8, 2024, 1:14pm

There is no element (IPv6 address of Mac) in this rule.

Not sure if it works but use the MAC address instead of IP address in your rule.
Make sure the your Mac book does not use private (=changing) MAC addresses

Quokka · July 8, 2024, 3:01pm

This is amazing, it works! I previously tried adding both the IPv4 and the IPv6 address of my MacBook to said policy, but this didn't make a difference. Replacing the IPv4 address with the MAC address of my MacBook does the trick: despite the VPN server in Amsterdam being the default gateway, traffic from my laptop is now sent through the VPN server in Stockholm, both over IPv4 and IPv6, as intended! @egc once again my sincere thanks for your help, this is absolutely appreciated!

When checking my connection at https://ipv6-test.com, there's one thing that stands out. This is that DNS6 + IP4 and DNS6 + IP6 do not seem to work:

When connecting to a VPN server directly, meaning without PBR, I most often see everything is 'in the green', with a score of 19 out of 20 (although sometimes even then one or two of the subtests fail). Whether this observation is meaningful to any extent, and if it can be solved I don't know, but at least I am a step further again!

Quokka · July 8, 2024, 4:31pm

My last issue appears to be solved as well:

It almost seems that, once established, VPN connections need to 'settle down' for a while for the DNS tests at https://ipv6-test.com to succesfully run. They're now all green, and I hence consider the problem this topic is about to be successfully solved. Kudos to @egc

egc · July 8, 2024, 4:31pm

Check with ipleak.net, you could have a DNS leak.

With the newest PBR package you can specify DNS servers per client to use but if that is not present use option 6 to specify a DNS server for you client. this will follow the same routing as the client.
See: https://github.com/egc112/OpenWRT-egc-add-on/tree/main/stop-dns-leak#policy-based-routing

Quokka · July 8, 2024, 6:51pm

@egc: my last post obviously crossed yours, otherwise I wouldn't have added 'Solved' to the subject line of the current topic.

Anyway, I checked with ipleak.net, and this is what I see:

The DNS server certainly doesn't belong to my ISP, and from this I conclude that I do not suffer from a DNS leak. However, I also do not immediately recognise this DNS server as one I have set for the interfaces wan and wan6 in /etc/config/network:

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	list dns '193.110.81.9'
	list dns '185.253.5.9'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	list dns '2a0f:fc80::9'
	list dns '2a0f:fc81::9'

With other words, I have no idea how I end up with this German DNS server at 185.14.45.185. To force my router to use the DNS server of my (current) preference, I tried to follow your advise as outlined under 'option 6' in your tutorial at the link you provided. I added the following to /etc/config/dhcp (without changing anything in /etc/config/network!):

config tag 'tag1'
    list dhcp_option '193.110.81.98,185.253.5.9'

config host
	option name 'Mac-Studio-M1-Max'
	option ip '1x.xxx.x.8'
	list mac '9x:xx:xx:xx:xx:xB'
	option leasetime 'infinite'
	list tag 'tag1'

config host
	option name 'MacBook-Pro-M3'
	option ip '1x.xxx.x.x7'
	option leasetime 'infinite'
	list mac '1x:xx:xx:xx:xx:xC'
	option duid '000xxxxxxxxxxxxxxxxxxxxxxxxc'
	list tag 'tag1'
root@OpenWrt:~#

This wasn't very succesful though, since I now lost access to internet completely, reason why I returned to the status quo ante. I'd be much obliged to hear what I did wrong here. Another question is whether it is possible to assign DNS servers at the VPN interface level in /etc/config/network (vpnclient0, vpnclient1, vpnclient2 etc.), if that makes any sense. TIA!

cat /etc/config/network:

Summary

root@OpenWrt:/etc/config# cat network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ip6assign '60'
	option broadcast '1x.xxx.x.255'
	list ipaddr '1x.xxx.x.x/24'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	list dns '193.110.81.9'
	list dns '185.253.5.9'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	list dns '2a0f:fc80::9'
	list dns '2a0f:fc81::9'

config interface 'vpnclient0'
	option proto 'none'
	option device 'vpn_amsterdam'

config interface 'vpnclient1'
	option proto 'none'
	option device 'vpn_berlin'

config interface 'vpnclient2'
	option proto 'none'
	option device 'vpn_copenhagen'

config interface 'vpnclient3'
	option proto 'none'
	option device 'vpn_frankfurt'

config interface 'vpnclient4'
	option proto 'none'
	option device 'vpn_hamburg'

config interface 'vpnclient5'
	option proto 'none'
	option device 'vpn_malmoe'

config interface 'vpnclient6'
	option proto 'none'
	option device 'vpn_oslo'

config interface 'vpnclient7'
	option proto 'none'
	option device 'vpn_paris'

config interface 'vpnclient8'
	option proto 'none'
	option device 'vpn_rotterdam'

config interface 'vpnclient9'
	option proto 'none'
	option device 'vpn_stockholm'
root@OpenWrt:/etc/config#

egc · July 8, 2024, 7:08pm

OK first what has gone wrong

It is called option 6 so you have to begin with a 6 e.g.
list dhcp_option '6,193.110.81.98,185.253.5.9'

Furthermore are you sure that 193.110.81.98 is a valid DNS server?

Yes you can do that just like any other interface, but it does not help. All DNS servers from all interfaces are thrown into one bucket and DNSMasq draws one of those DNS servers out of the bucket.
See: https://github.com/egc112/OpenWRT-egc-add-on/tree/main/stop-dns-leak

Anyway if you are happy , I am happy and this is the end

Quokka · July 9, 2024, 12:58pm

@egc: you have very sharp eyes; 193.110.81.98 is indeed not a valid DNS server. I made a typo here which I quickly corrected - my thanks for pointing out!

A little further down the road, I have chosen to assign option 6 to interface br-lan under the general DHCP options:

root@OpenWrt:~# cat /etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.vpn'
	option localservice '1'
	option ednspacket_max '1232'
	list dhcp_option 'br-lan,6,8.8.8.8,8.8.4.4'

My reasoning here is that br-lan covers all traffic from devices within my lan (wired, wireless and VPN), and that hence all DNS queries from within my network will pass through the VPN to one of the DNS servers shown above. Can you confirm this assumption is correct? If not, I'd be very glad to hear where I am taking a wrong turn,

Two more questions:

what about IPv6 DNS servers? Would the following option setting make any sense, or is it pointless to alter the setting like this:

list dhcp_option 'br-lan,6,8.8.8.8,8.8.4.4,2001:4860:4860::8888,2001:4860:4860::8844'

what about the DNS settings for wan and wan6 in /etc/config/network:

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	list dns '8.8.8.8'
	list dns '8.8.4.4'

config interface 'wan6'
	option device 'eth1'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option peerdns '0'
	list dns '2001:4860:4860::8888'
	list dns '2001:4860:4860::8844'

Can these be removed now I have set my preferences regarding DNS servers for bride-device br-lan in etc/config/dhcp, or do these settings for wan and wan6 still serve a purpose and should they therefore be left alone? My sincere thanks in advance for any advice you may be able to give me. I am definitely getting closer, but this would not have been possible without the help I have been given at this forum!

egc · July 9, 2024, 1:08pm

Setting option 6 on an interface is fine.

Just keep the DNS servers on the interfaces, they can/will be used by the router itself and for clients not using option 6.

Yes you can try this, but an IPv4 DNS server also will return the IPv6 address so necessary it is not as all clients have IPv4 access.
In case you have IPv6 only clients then you should specify an IPv6 DNS server

Quokka · July 10, 2024, 1:45pm

Thank you very much for your confirmation. Not that it would make any sense as you pointed out, but for curiosity's sake I tried out the dhcp_option already mentioned above:

list dhcp_option 'br-lan,6,8.8.8.8,8.8.4.4,2001:4860:4860::8888,2001:4860:4860::8844'

It appears that while the router starts up and a wireless connection is established, there is no access to the internet anymore. I therefore quickly removed the IPv6 DNS server addresses again, and now everything works like a charm. Topic solved, much obliged for your help!

system · July 20, 2024, 1:46pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.