Looked at the previous message and realized that I'm not the only one making large lists of domains in the dest_addr option.
@stangri What do you think about implementing a list of addresses via a list? So that it looks a little more readable than a long text line.
config rule
..
list dest_addr 'disneyplus.com'
list dest_addr 'bamgrid.com'
list dest_addr 'bam.nr-data.net'
.. etc
I'll need to look into why it's happening, which version of VPR is that?
It may (or may not) present a problem when mapping the list to the view in the WebUI. I'll look into it when the WebUI app is converted to javascript.
i'm using a 19.07 fork (turrisOS 5.3.5) with VPR 0.2.1-13 here is is my full config file
root@turris:~# cat /etc/config/vpn-policy-routing
config vpn-policy-routing 'config'
option strict_enforcement '1'
option resolver_ipset 'dnsmasq.ipset'
option ipv6_enabled '0'
list ignored_interface 'vpnserver wgserver'
option boot_timeout '30'
option iptables_rule_option 'append'
option procd_reload_delay '1'
option webui_enable_column '0'
option webui_protocol_column '0'
option webui_chain_column '0'
option webui_show_ignore_target '0'
option webui_sorting '1'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list webui_supported_protocol 'all'
list supported_interface 'tun0'
option verbosity '1'
option enabled '1'
option dest_ipset '0'
option src_ipset '0'
config include
option path '/etc/vpn-policy-routing.netflix.user'
option enabled '0'
config include
option path '/etc/vpn-policy-routing.aws.user'
option enabled '0'
config policy
option name 'vpn-disney'
option src_addr '192.168.1.9'
option dest_addr 'disneyplus.com edge.bamgrid.com bam.nr-data.net cdn.registerdisney.go.com cws.conviva.com d9.flashtalking.com disney-portal.my.onetrust.com disneyplus.bn5x.net js-agent.newrelic.com disney-plus.net dssott.com adobedtm.com'
option interface 'tun0'
dnsmasq-full is installed
root@turris:~# opkg list-installed | grep dnsm
dnsmasq-full - 2.80-16.3
If you use the source address in the policy, the dnsmasq is not being used. There's clearly a bug in this older version of VPR that it doesn't fully resolve the domain names and they end up in the iptables rules (expected behaviour is that the domain names are resolved and their IP addresses are used in iptables).
Given the status of 19.07, I'm reluctant to spend time to fix the bug in an old version. Maybe some of the other contributors to the vpn-policy-routing package can do a diff between the old version and the new version and send a PR for this bug.
Here's what you can try:
If you can remove the source address from that policy, the domains should end up in the dnsmasq ipset (if that option is selected) and everything should work.
If you can't remove the source address from that policy, you can try upgrading to a later version of VPR from my repository. I've had to downgrade a version for 19.07 because there were reports from some users on 19.07 that later versions of VPR didn't work for them, but it might work for you.
DO NOT replace vpn-policy-routing with pbr on your system tho, I can't recall exactly what, but there are certainly things in pbr which would only work on 21.02 and later.
thanks I've removed the source ip address and got no errors but the service is not working as intended (via vpn) - confirmed via traceroute (should I do other tests? )
root@turris:~# /etc/init.d/vpn-policy-routing support
vpn-policy-routing 0.2.1-13 running on TurrisOS 5.3.5.
============================================================
Dnsmasq version 2.80 Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth nettlehash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
default 10.0.0.1 0.0.0.0 UG 0 0 0 pppoe-wan
IPv4 Table 201: default via 10.0.0.1 dev pppoe-wan
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
172.17.17.0/24 via 172.17.17.2 dev tun1
172.17.17.2 dev tun1 proto kernel scope link src 172.17.17.1
IPv4 Table 201 Rules:
32761: from all fwmark 0x10000/0xff0000 lookup wan
IPv4 Table 202: default via 10.0.10.1 dev wg0
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
172.17.17.0/24 via 172.17.17.2 dev tun1
172.17.17.2 dev tun1 proto kernel scope link src 172.17.17.1
IPv4 Table 202 Rules:
32760: from all fwmark 0x20000/0xff0000 lookup wg0
IPv4 Table 203: default via 172.31.31.1 dev tun_turris
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
172.17.17.0/24 via 172.17.17.2 dev tun1
172.17.17.2 dev tun1 proto kernel scope link src 172.17.17.1
IPv4 Table 203 Rules:
32759: from all fwmark 0x30000/0xff0000 lookup vpn_turris
IPv4 Table 204: default via 10.28.0.12 dev tun0
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
172.17.17.0/24 via 172.17.17.2 dev tun1
172.17.17.2 dev tun1 proto kernel scope link src 172.17.17.1
IPv4 Table 204 Rules:
32758: from all fwmark 0x40000/0xff0000 lookup tun0
============================================================
Mangle IP Table: PREROUTING
-N VPR_PREROUTING
-A VPR_PREROUTING -m set --match-set tun0_mac src -c 0 0 -g VPR_MARK0x040000
-A VPR_PREROUTING -m set --match-set tun0_ip src -c 0 0 -g VPR_MARK0x040000
-A VPR_PREROUTING -m set --match-set tun0 dst -c 0 0 -g VPR_MARK0x040000
-A VPR_PREROUTING -m set --match-set vpn_turris_mac src -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -m set --match-set vpn_turris_ip src -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -m set --match-set vpn_turris dst -c 0 0 -g VPR_MARK0x030000
-A VPR_PREROUTING -m set --match-set wg0_mac src -c 0 0 -g VPR_MARK0x020000
-A VPR_PREROUTING -m set --match-set wg0_ip src -c 0 0 -g VPR_MARK0x020000
-A VPR_PREROUTING -m set --match-set wg0 dst -c 0 0 -g VPR_MARK0x020000
-A VPR_PREROUTING -m set --match-set wan_mac src -c 0 0 -g VPR_MARK0x010000
-A VPR_PREROUTING -m set --match-set wan_ip src -c 0 0 -g VPR_MARK0x010000
-A VPR_PREROUTING -m set --match-set wan dst -c 0 0 -g VPR_MARK0x010000
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x010000
-N VPR_MARK0x010000
-A VPR_MARK0x010000 -c 0 0 -j MARK --set-xmark 0x10000/0xff0000
-A VPR_MARK0x010000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x020000
-N VPR_MARK0x020000
-A VPR_MARK0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A VPR_MARK0x020000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x030000
-N VPR_MARK0x030000
-A VPR_MARK0x030000 -c 0 0 -j MARK --set-xmark 0x30000/0xff0000
-A VPR_MARK0x030000 -c 0 0 -j RETURN
============================================================
Mangle IP Table MARK Chain: VPR_MARK0x040000
-N VPR_MARK0x040000
-A VPR_MARK0x040000 -c 0 0 -j MARK --set-xmark 0x40000/0xff0000
-A VPR_MARK0x040000 -c 0 0 -j RETURN
============================================================
Current ipsets
create wan hash:net family inet hashsize 1024 maxelem 65536 comment
create wg0 hash:net family inet hashsize 1024 maxelem 65536 comment
create vpn_turris hash:net family inet hashsize 1024 maxelem 65536 comment
create tun0 hash:net family inet hashsize 1024 maxelem 65536 comment
create wan_ip hash:net family inet hashsize 1024 maxelem 65536 comment
create wan_mac hash:mac hashsize 1024 maxelem 65536 comment
create wg0_ip hash:net family inet hashsize 1024 maxelem 65536 comment
create wg0_mac hash:mac hashsize 1024 maxelem 65536 comment
create vpn_turris_ip hash:net family inet hashsize 1024 maxelem 65536 comment
create vpn_turris_mac hash:mac hashsize 1024 maxelem 65536 comment
create tun0_ip hash:net family inet hashsize 1024 maxelem 65536 comment
create tun0_mac hash:mac hashsize 1024 maxelem 65536 comment
============================================================
DNSMASQ ipsets
ipset=/disneyplus.com/tun0 # vpn-disney
ipset=/edge.bamgrid.com/tun0 # vpn-disney
ipset=/bam.nr-data.net/tun0 # vpn-disney
ipset=/cdn.registerdisney.go.com/tun0 # vpn-disney
ipset=/cws.conviva.com/tun0 # vpn-disney
ipset=/d9.flashtalking.com/tun0 # vpn-disney
ipset=/disney-portal.my.onetrust.com/tun0 # vpn-disney
ipset=/disneyplus.bn5x.net/tun0 # vpn-disney
ipset=/js-agent.newrelic.com/tun0 # vpn-disney
ipset=/disney-plus.net/tun0 # vpn-disney
ipset=/dssott.com/tun0 # vpn-disney
ipset=/adobedtm.com/tun0 # vpn-disney
============================================================
Your support details have been logged to '/var/vpn-policy-routing-support'. [✓]
LE - I've tried vpn-policy-routing_0.3.4-8_all.ipk with the same results
Using the following build with fw4/nft reverted to fw3/ipt:
OK, I just tried installing pbr after using the transition command to convert /etc/config/vpn-policy-routing to /etc/config/pbr:
opkg install vpn-policy-routing luci-app-vpn-policy-routing:
opkg install pbr luci-app-pbr
Installing pbr-ipt (0.9.4-10) to root...
Downloading https://repo.openwrt.melmac.net/pbr-ipt_0.9.4-10_all.ipk
Installing luci-app-pbr (0.9.4-10) to root...
Downloading https://repo.openwrt.melmac.net/luci-app-pbr_0.9.4-10_all.ipk
Configuring pbr-ipt.
Installing rc.d symlink for pbr... OK
uci: Parse error
uci: Parse error
uci: Parse error (invalid command) at line 6, byte 0
Processing Interfaces ✓✓
Processing Policies ✓✓✓✗
pbr 0.9.4-10 monitoring interfaces: wan wan6 wireguard
pbr 0.9.4-10 started with gateways:
wan/eth0/redacted/2603:9000:redacted/128
fe80::c453:a4ff:fea0:f935/64
wireguard/redacted/fc00:bbbb:redacted/128 [✓]
ERROR: Unknown fw_mark for wan6
Configuring luci-app-pbr.
uci: Parse error
uci: Parse error
uci: Parse error (invalid command) at line 6, byte 0
Collected errors:
* resolve_conffiles: Existing conffile /etc/config/pbr is different from the conffile in the new package. The new conffile will be placed at /etc/config/pbr-opkg.
This breaks LuCI so I reverted back to vpn-policy-routing for the moment. I'm wondering if the "ERROR: Unknown fw_mark for wan6" is causing my IPv6 routing issues with vpn-policy-routing. Regardless it looks like there are other issues abound as well.
Edit: the unknown fw_mark was from using an IPv6 CIDR as src_addr: 2603:9000::0/32
How are we supposed to enter a local device's IPv6 address so it can be routed via wan6?
I'd guess the config wasn't converted correctly. Can you run:
for i in /etc/config/*; do if ! uci show ${i##*/} > /dev/null 2>&1; then echo -e -n "$i: "; uci show ${i##*/} > /dev/null; fi; done;
and also paste the /etc/config/pbr ?
I see what the issue is, I store backup network configs using the nomenclature /etc/config/network.vpnprovider.location and it looks like pbr is reading them all in and not just /etc/config/network. I moved them to a sub directory and the two uci parse errors disappeared.
luci-app-pbr install still errors out, looks like it is in /etc/config/sysupgrade:
opkg install luci-app-pbr --force-reinstall:
Removing package luci-app-pbr from root...
Installing luci-app-pbr (0.9.4-10) to root...
Downloading https://repo.openwrt.melmac.net/luci-app-pbr_0.9.4-10_all.ipk
Configuring luci-app-pbr.
uci: Parse error (invalid command) at line 6, byte 0
for i in /etc/config/*; do if ! uci show ${i##*/} > /dev/null 2>&1; then echo -e -n "$i: "; uci show ${i##*/} > /dev/null; fi; done;:
/etc/config/bak: uci: Entry not found
/etc/config/sysupgrade: uci: Parse error (invalid command) at line 6, byte 0
/etc/config/wireless.bk: uci: Entry not found
cat /etc/config/sysupgrade:
## This file contains files and directories that should
## be preserved during an upgrade.
# /etc/example.conf
# /etc/openvpn/
/etc/wireguard/
Unfortunately luci app is still failing to load.
In browser:
/usr/lib/lua/luci/dispatcher.lua:781: bad argument #1 to 'pairs' (table expected, got nil)
stack traceback:
[C]: in function 'pairs'
/usr/lib/lua/luci/dispatcher.lua:781: in function 'resolve_firstchild'
/usr/lib/lua/luci/dispatcher.lua:861: in function 'resolve_page'
/usr/lib/lua/luci/dispatcher.lua:885: in function 'dispatch'
/usr/lib/lua/luci/dispatcher.lua:479: in function </usr/lib/lua/luci/dispatcher.lua:478>
Edit: I have done a force reinstall and now I have a working PBR LuCI instance. 
Now I can bug you with questions that matter.
OK, here goes. Wireguard is the primary route. I've got some simple IPv4 rules set up that work great for directing traffic on my lan vlan subnet to wan (bypassing wireguard) but I can't get IPv6 to work.
uci export network:
package network
config globals 'globals'
option ula_prefix 'fdc2:9aea:13b1::/48'
config interface 'loopback'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option device 'lo'
config interface 'lan'
option proto 'static'
option netmask '255.255.255.0'
option ipaddr '192.168.1.1'
option ip6assign '64'
option ip6ifaceid '::1'
option ip6hint '1'
option device 'br-lan'
config interface 'wan'
option proto 'dhcp'
option device 'eth0'
config interface 'wan6'
option proto 'dhcpv6'
option reqaddress 'try'
option reqprefix '56'
option device 'eth0'
option ip6table '2'
config interface 'wireguard'
option proto 'wireguard'
option private_key 'redacted'
list addresses 'redacted/32'
list addresses 'fc00:bbbb:redacted/128'
config wireguard_wireguard
option persistent_keepalive '25'
option public_key 'redacted'
option endpoint_host 'redacted'
option description 'redacted'
list allowed_ips '0.0.0.0/0'
list allowed_ips '::0/0'
option endpoint_port '51820'
option route_allowed_ips '1'
config interface 'vpn'
option proto 'static'
option ipaddr '192.168.2.1'
option netmask '255.255.255.0'
list dns 'redacted'
option device 'br-vpn'
option ip6ifaceid '::1'
option ip6assign '64'
option ip6hint '2'
list ip6class 'local'
config interface 'dmz'
option proto 'static'
option ipaddr '192.168.3.1'
option netmask '255.255.255.0'
option ip6assign '64'
option ip6ifaceid '::1'
option ip6hint '3'
option device 'br-dmz'
config interface 'iot'
option proto 'static'
option ipaddr '192.168.4.1'
option ip6assign '64'
option ip6hint '4'
option ip6ifaceid '::1'
option netmask '255.255.255.0'
option device 'br-iot'
config device
option name 'br-lan'
option type 'bridge'
option stp '1'
list ports 'eth1'
config device
option name 'br-vpn'
option type 'bridge'
list ports 'eth1.2'
config device
option name 'br-dmz'
option type 'bridge'
list ports 'eth1.3'
config device
option name 'br-iot'
option type 'bridge'
list ports 'eth1.4'
uci export firewall:
package firewall
config defaults
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
list network 'dmz'
list network 'lan'
config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wan'
list network 'wan6'
config zone
option name 'iot'
option output 'ACCEPT'
list network 'iot'
option forward 'REJECT'
option input 'ACCEPT'
config zone
option name 'vpn'
option output 'ACCEPT'
list network 'vpn'
option forward 'REJECT'
option input 'ACCEPT'
config zone
option name 'wireguard'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wireguard'
config forwarding
option src 'lan'
option dest 'wan'
config forwarding
option src 'lan'
option dest 'vpn'
config forwarding
option src 'lan'
option dest 'iot'
config forwarding
option src 'vpn'
option dest 'lan'
config forwarding
option src 'vpn'
option dest 'wireguard'
config forwarding
option src 'vpn'
option dest 'iot'
config forwarding
option src 'iot'
option dest 'wireguard'
config forwarding
option src 'iot'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option src_ip 'fc00::/6'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
option enabled ''\''0'\'''
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config include
option path '/etc/firewall.user'
config include 'miniupnpd'
option type 'script'
option path '/usr/share/miniupnpd/firewall.include'
option family 'any'
option reload '1'
config include 'pbr'
option type 'script'
option path '/usr/share/pbr/pbr.firewall.include'
option family 'any'
option reload '1'
/etc/init.d/pbr status -d:
pbr 0.9.4-10 running on OpenWrt SNAPSHOT.
============================================================
Dnsmasq version 2.86 Copyright (c) 2000-2021 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
Routes/IP Rules
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default * 0.0.0.0 U 0 0 0 wireguard
redacted * 255.255.192.0 U 0 0 0 eth0
redacted 035-143-192-001 255.255.255.255 UGH 0 0 0 eth0
192.168.1.0 * 255.255.255.0 U 0 0 0 br-lan
192.168.2.0 * 255.255.255.0 U 0 0 0 br-vpn
192.168.3.0 * 255.255.255.0 U 0 0 0 br-dmz
192.168.4.0 * 255.255.255.0 U 0 0 0 br-iot
0: from all lookup local
29999: from all fwmark 0x20000/0xff0000 lookup wireguard
30000: from all fwmark 0x10000/0xff0000 lookup wan
32766: from all lookup main
32767: from all lookup default
IPv4 Table 201: default via redacted dev eth0
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev br-vpn proto kernel scope link src 192.168.2.1
192.168.3.0/24 dev br-dmz proto kernel scope link src 192.168.3.1
192.168.4.0/24 dev br-iot proto kernel scope link src 192.168.4.1
IPv4 Table 201 Rules:
30000: from all fwmark 0x10000/0xff0000 lookup wan
IPv4 Table 202: default via 10.65.5.239 dev wireguard
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev br-vpn proto kernel scope link src 192.168.2.1
192.168.3.0/24 dev br-dmz proto kernel scope link src 192.168.3.1
192.168.4.0/24 dev br-iot proto kernel scope link src 192.168.4.1
IPv4 Table 202 Rules:
29999: from all fwmark 0x20000/0xff0000 lookup wireguard
Error: ipv6: FIB table does not exist.
Dump terminated
IPv6 Table 202: default dev wireguard proto static metric 1024 pref medium
============================================================
Mangle IP Table
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
TTL all -- anywhere base-address.mcast.net/4 TTL increment by 1
TTL all -- anywhere base-address.mcast.net/4 TTL increment by 1
PBR_PREROUTING all -- anywhere anywhere mark match 0x0/0xff0000
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain PBR_MARK0x010000 (5 references)
target prot opt source destination
MARK all -- anywhere anywhere MARK xset 0x10000/0xff0000
RETURN all -- anywhere anywhere
Chain PBR_MARK0x020000 (0 references)
target prot opt source destination
MARK all -- anywhere anywhere MARK xset 0x20000/0xff0000
RETURN all -- anywhere anywhere
Chain PBR_PREROUTING (1 references)
target prot opt source destination
PBR_MARK0x010000 all -- 192.168.1.0/24 anywhere [goto] /* dmz4 */
============================================================
Mangle IPv6 Table
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PBR_PREROUTING all anywhere anywhere mark match 0x0/0xff0000
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
TCPMSS tcp anywhere anywhere tcp flags:SYN,RST/SYN /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
TCPMSS tcp anywhere anywhere tcp flags:SYN,RST/SYN /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
TCPMSS tcp anywhere anywhere tcp flags:SYN,RST/SYN /* !fw3: Zone wireguard MTU fixing */ TCPMSS clamp to PMTU
TCPMSS tcp anywhere anywhere tcp flags:SYN,RST/SYN /* !fw3: Zone wireguard MTU fixing */ TCPMSS clamp to PMTU
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain PBR_MARK0x010000 (0 references)
target prot opt source destination
MARK all anywhere anywhere MARK xset 0x10000/0xff0000
RETURN all anywhere anywhere
Chain PBR_MARK0x020000 (0 references)
target prot opt source destination
MARK all anywhere anywhere MARK xset 0x20000/0xff0000
RETURN all anywhere anywhere
Chain PBR_PREROUTING (1 references)
target prot opt source destination
============================================================
Mangle IP Table MARK Chain: PBR_MARK0x010000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK0x010000
-A PBR_MARK0x010000 -c 77 4587 -j MARK --set-xmark 0x10000/0xff0000
-A PBR_MARK0x010000 -c 77 4587 -j RETURN
============================================================
Mangle IP Table MARK Chain: PBR_MARK0x020000
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-N PBR_MARK0x020000
-A PBR_MARK0x020000 -c 0 0 -j MARK --set-xmark 0x20000/0xff0000
-A PBR_MARK0x020000 -c 0 0 -j RETURN
============================================================
NAT IP Table
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
============================================================
NAT IPv6 Table
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
============================================================
Current ipsets
============================================================
Your support details have been logged to '/var/pbr-support'. [✓]
OK, first question. I am a little confused about directing IPv6 using PBR. At some point in the past I think I had this working correctly w/ VPR but it seems to have stopped working on my new device (that I need to use a snapshot on). I'm also unsure if IPv6 is even supported in this manner any longer and whether NAT6 is required or forbidden (I have completely removed NAT6 for the time being and am just using IPv4 on the vpn vlan, although I would like to eventually enable it for my vpn vlan in the future so I can NAT a single /128). The documentation is a little contradictory (clearly IPv6 is "supported" as it's an option although the README suggests setting up a route6 instead--which I can do am not sure if strictly required nor how to perform this per subnet).
The naive me figured I could just enter either the ULA/GUA IPv6 subnet in CIDR notation and route to wan6 interface but PBR gives me a Unknown fw_mark for wan6 error. I've also tried using a single devices /128 and that didn't seem to work either. So I assume that it is not possible to use PBR in this manner to route IPv6 traffic. Am I forced to set wan as default route and then route ipv4 to wireguard? What if I eventually want to route IPv6 through wireguard with or without NAT6?
I'm also confused about the use of the @device notation. Could I, for instance, just use my @br-lan device to direct all traffic from the lan network to wan/wan6 instead of the associated subnet (192.168.2.0/24). Would this also handle IPv6 (it doesn't seem to make a difference in my tests)? That would allow me to change IP subnets without having to fix in PBR. I've tried this but still no luck.
One common theme is that I don't think clients aren't receiving an IPv6 gateway address. I'm not sure if this is PBR's fault or how this works when making wireguard the default route (with 0.0.0.0/0 and ::0/0).
Thank you so much for this tool and any insight you can provide!
How can I setup multiple OpenVPN client connections and assign/force each VPN to one device on LAN? So that the local device don't use the normal WAN anymore?
This is what I want to achieve:
Each LAN device should have it's own VPN assigned.
If a device is not connected to a VPN it should not have an internet connection.
Upon setting up my new router I now cannot find the option append_local_rules.
I think I need this to run both wireguard server and client on my router: option append_local_rules '! -d 192.168.200.0/24'... at least it is in my old routers config.
Can anyone tell me how to set that in the latest vpn-policy-routing version?
I think you might want append_src_rules. I see no reference to append_local_rules in the documentation.
I noticed an update to ipset in master that allows sets to be used with nft.
Not sure if this helps towards supporting fw4
It took me a long time to realize this has been discontinued and PBR eg ( https://docs.openwrt.melmac.net/pbr/ ) is the way forward... It would be great if this page linked to PBR https://openwrt.org/docs/guide-user/network/routing/pbr and https://docs.openwrt.melmac.net/vpn-policy-routing/ had a notice that it's been discontinued...
Also, is there a different thread for support of PBR that could be linked to in this thread?
EDIT: PS: Thanks for all the hard work 
PBR is working well for me, and seems to have solved an issue I had with it not being applied at startup.
Good point. However VPR hasn't been actually discontinued, it's still available in 19.07, 21.02 and master branches, while pbr may be too much of a development version for now. I certainly feel that I need to update the top post and include information about pbr, but it's too soon to write VPR off completely.
I wonder whether pbr has some conflicts with AdGuardHome. When I turn on AdGuardHome, pbr stopped to work. Is there a way to block ads while using pbr at the same time? It is the same story with VPR.
Either pbr or vpn-policy-routing work fine together with simple-adblock and/or https-dns-proxy.
@stangri , Hey!
is there any news on adapting vpr or pbr for fw4? Updated my belkin RT3200 device to latest snapshot and broke package work. Reverting to the snapshot to the fw4 branch does not work, because breaks the installation of other packages that require the latest kernel. Of course, I did not make a backup of the working snapshot with the current packages 
Perhaps before the advent of a working vpr or pbr for fw4 is it possible to manually configure traffic routing? I would be grateful for any advice.
I still have a base build for RT3200 (UBI flavor) right before the switch to FW4 if you wanted it, it's from Jan 13th?
I will be grateful for the file
while in parallel I study nftables regarding manual routing. It seems to me that @stangri should have the first test instances soon.