Removing the number from listen_port helped. But the main question remains why the behavior differed between SNAPHOT and 23.05.4 which had identical configs.
And despite luci-app-pbr not showing the default gateway, by default gateway was still the wgc0 interface.
1 Like
Will explain it tomorrow
1 Like
I think that may be true for an ISP with DHCP where the route pushed doesn't include the modem, but mine is PPPoE tunnel, so without the interface there's no IP address for eth0, just a pppoe tunnel that uses the raw interface.
PPPoe(Wan) --> Eth0 --> Bridge mode --> PPPoe concentrator (ISP side of the fiber) --> Internet gateway
Eth0/3bbadmin(10.33.3.20) --> Modem(10.33.3.1)
Since it has the local interface a route does get defined for 10.33.3.*/3bbadmin but there's no default route on that interface, just an automatic interface route. Default route remains with pppoe, and 3bbadmin interface is not tied to WAN, so I'm not sure why PBR sees that interface as WAN instead of 3bbadmin.
I doubt the configs are identical, this seems like a setup error.
The WireGuard interfaces cannot have the same listen port.
I suspect the listen port of the both interfaces was the same.
PBR does do some things with WG interfaces and listen port, if it detects a listen port it will assume this is a WG server and will route the listen port via the WAN to make it possible to run a WG server and Client at the same time (necessary if the WG Client uses default routing).
A WG client does not need a fixed listen port, it will choose a random port which is free which prevents mistakes you seem to have made.
1 Like
Some progress but still not working with 2 subnets with their own VPN. Having 2 instances of openvpn defined (with 1 or both enabled) gives the following message on the PBR service luci page:
Service Status
Version 1.1.6-20 - Stopped.
Service Errors
Unknown error!
Failed to set up 'wan/phy1-sta0/192.168.137.1'!
Unknown error!
Failed to set up 'vpntun0/tun0/10.100.0.2'!
Unknown error!
Failed to set up 'vpntun1/tun1/10.100.0.2'!
Failed to set up any gateway!
There's nothing obvious reported on the system log.
Deleting one of the openvpn instances allows the other to work fine. Are there any potential clashes between the openvpn configurations I need to be aware of? The service gateway is the same for both. Is that a potential issue?
Having tunnels with the same IP address does not work.
So I need to change the gateway address from 10.100.0.2 for one of them? How can this be done as there's no mention of the gateway address in the openvpn config?
That is something you have to ask your VPN provider they handout the addresses.
Usually for every config they hand out has a different address.
(Theoretically it is possible that your VPN provider hands out a static address)
The 2 openvpn configs I am trying specify different IP addresses. Is there any other potential for a clash as it seems like when both configs are loaded by PBR, it throws a wobbly.
1.1.7 had additional checks for wireguard servers in the config in order to implement @egc's excellent code for running wg server and client at the same time within the init script itself, without the need for the external custom user file. Hence 1.1.7 may have behaved differently on the same config vs 1.1.6 (which is included in OpenWrt 23.05 packages now).
Since the device name is the same for both interfaces, it seems that pbr just grabs the gateway from whichever one was up first. There were some minor tweaks in the gateway detection/insertion recently, I'll have to have another look at some point this/next week. Or, if you have time, check last dozen or so commits into the 1.1.6 branch of https://github.com/stangri/pbr/tree/1.1.6
So I just reset everything from the UPS and discovered another problem:
When the ISP modem takes a while to initialize, it goes longer than the procd_boot_delay..
I could set this to a higher value but it begs the question,
What happens when the power fails and I'm away? The modem/openwrt might get power hours before the ISP gives a link, so then when PPPoE comes back the VPN will start at the same time and PBR will initialize before the VPN is finished and I'm back to the same race condition..
Is there some delay I can set for after WAN is up? boot delay seems to apply from boot regardless of WAN status.
There are triggers for pbr to be reloaded when WAN and tunnels are up. But you should experiment how it works in your case of slow ISP model boot up.
right I understand it will trigger after PPPoE comes back up,
But the boot timeout will have already expired by then so it won't wait for my VPN to be up before proceeding, it will trigger immediately when PPPoE comes back up.
I'm talking about interface monitoring here "service waiting for wan gateway..." not procd firewall reloads.
Maybe the procd will trigger again for my VPN after it comes up, but by then PBR has already included the wrong gateway IP for my VPN because it started while the tunnel was half-initialized.
EDIT: In reply to egc about same tunnel address
I think this is a problem that a lot of people will be vulnerable to but not very likely to trigger.
Most people using different country servers will use the same provider
Most providers will keep the configs on the servers the same because otherwise they'd need a stateful database of which tun IP range was assigned to which vpn server. And they might even have multiple identical VPN country endpoints for load balancing with DNS.
.. But, most providers will allocate IP based on the order you connected to that server. So you'd have to be "caller number 1" on both servers for example to cause a conflict. Which is unlikely to happen unless the servers are very lightly loaded.
I don't know enough about the NFT rules to know if you could bind the gateway to the interface rather than the IP and if that would be possible to solve it?
There is another solution: Configure one tunnel to use the tunX interface and the other on Wireguard - if your VPN provider supports it. Then route the subnets as desired.
That is correct so a duplicate address is very unlikely but not impossible.
It is not about NFT but about routing.
These VPN's are usually point to point links and can do without an address, I actually am testing something in that direction.
If you end up in the same situation I can send you something to try.
1 Like
Thanks - I'm getting closer to a solution using 2 VPN's on separate subnets but getting this:
Service Status
Version 1.1.6-20 - Stopped.
Service Errors
Unknown error!
Failed to set up 'wan/phy1-sta0/192.168.137.1'!
Unknown error!
Failed to set up 'vpntun0/tun0/10.100.0.2'!
Unknown error!
Failed to set up 'vpntun1/tun1/10.100.0.2'!
Failed to set up any gateway!
Removing the 2nd openvpn config yields a working VPN on one subnet
Service Status
Version 1.1.6-20 - Running (fw4 nft file mode).
Service Gateways
wan/phy1-sta0/192.168.137.1 ✓
vpntun0/tun0/10.100.0.2
'vpntun1/tun1/0.0.0.0
Is the error because 10.100.0.2 is assigned to both tunnels?
Thanks. I have a suspicion that it's because 10.100.0.2 is being assigned to both tunnels but don't have enough knowledge to know if this is so and, if it is, how to change it.
The wireguard solution sounds a way forward but I had hoped to increase the number of subnets once I got two working
Since updating to PBR v. 1.1.6-20 I no longer seem to be able to establish a IPv6 connection with any of the VPN servers I have access to. Without a PBR policy in action, I am getting dual IPv4/IPv6 connectivity with the VPN server that acts as default gateway just fine. However, as soon as I enable my PBR policy (which is meant to steer traffic from my laptop to a specific VPN server, other than the default gateway), it’s a IPv4 connection only - restarting PBR or my router or laptop doesn’t help. With the previous version of PBR (v. 1.1.1-7) things worked without issue, getting dual IPv4/IPv6 connectivity to my VPN server of choice as dictated by my PBR policy.
I am interested to hear how the problem described above can be solved. To facilitate looking into it, I’ll provide some details about my setup below. Should any additional information be helpful, please do let me know. TIA!
ubus call system board
root@OpenWrt:~# ubus call system board
{
"kernel": "5.15.162",
"hostname": "OpenWrt",
"system": "Intel(R) Celeron(R) J6412 @ 2.00GHz",
"model": "Protectli VP2420",
"board_name": "protectli-vp2420",
"rootfs_type": "ext4",
"release": {
"distribution": "OpenWrt",
"version": "23.05.4",
"revision": "r24012-d8dd03c46f",
"target": "x86/64",
"description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
}
}
root@OpenWrt:~#
uci export pbr
root@OpenWrt:~# uci export pbr
package pbr
config pbr 'config'
option enabled '1'
option verbosity '2'
option strict_enforcement '1'
option resolver_set 'dnsmasq.nftset'
option ipv6_enabled '1'
list ignored_interface 'vpnserver'
list ignored_interface 'wgserver'
option boot_timeout '30'
option rule_create_option 'insert'
option procd_reload_delay '1'
option webui_show_ignore_target '0'
list webui_supported_protocol 'all'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list supported_interface 'vpnclient0 vpnclient1 vpnclient2 vpnclient3 vpnclient4 vpnclient5 vpnclient6 vpnclient7 vpnclient8 vpnclient9'
config policy
option name 'MacBook Pro M3'
option src_addr ‘Redacted'
option interface 'vpnclient6'
root@OpenWrt:~#
etc/init.d/pbr status 1
root@OpenWrt:~# /etc/init.d/pbr status
pbr - environment
pbr 1.1.6-20 running on OpenWrt 23.05.4.
Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 mark set mark and 0xff00ffff xor 0x040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 mark set mark and 0xff00ffff xor 0x050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 mark set mark and 0xff00ffff xor 0x060000
add rule inet fw4 pbr_mark_0x060000 return
add chain inet fw4 pbr_mark_0x070000
add rule inet fw4 pbr_mark_0x070000 mark set mark and 0xff00ffff xor 0x070000
add rule inet fw4 pbr_mark_0x070000 return
add chain inet fw4 pbr_mark_0x080000
add rule inet fw4 pbr_mark_0x080000 mark set mark and 0xff00ffff xor 0x080000
add rule inet fw4 pbr_mark_0x080000 return
add chain inet fw4 pbr_mark_0x090000
add rule inet fw4 pbr_mark_0x090000 mark set mark and 0xff00ffff xor 0x090000
add rule inet fw4 pbr_mark_0x090000 return
add chain inet fw4 pbr_mark_0x0a0000
add rule inet fw4 pbr_mark_0x0a0000 mark set mark and 0xff00ffff xor 0x0a0000
add rule inet fw4 pbr_mark_0x0a0000 return
add chain inet fw4 pbr_mark_0x0b0000
add rule inet fw4 pbr_mark_0x0b0000 mark set mark and 0xff00ffff xor 0x0b0000
add rule inet fw4 pbr_mark_0x0b0000 return
add rule inet fw4 pbr_prerouting ether saddr { 10:9F:41:D6:AE:9C } goto pbr_mark_0x080000 comment "MacBook Pro M3"
pbr chains - policies
chain pbr_forward { # handle 37
}
chain pbr_input { # handle 38
}
chain pbr_output { # handle 39
}
chain pbr_postrouting { # handle 41
}
chain pbr_prerouting { # handle 40
ether saddr <Redacted> goto pbr_mark_0x080000 comment "MacBook Pro M3" # handle 2006
}
chain pbr_dstnat_lan { # handle 36
}
pbr chains - marking
chain pbr_mark_0x010000 { # handle 1631
meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 1984
return # handle 1985
}
chain pbr_mark_0x020000 { # handle 1634
meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 1986
return # handle 1987
}
chain pbr_mark_0x030000 { # handle 1637
meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 1988
return # handle 1989
}
chain pbr_mark_0x040000 { # handle 1640
meta mark set meta mark & 0xff04ffff | 0x00040000 # handle 1990
return # handle 1991
}
chain pbr_mark_0x050000 { # handle 1643
meta mark set meta mark & 0xff05ffff | 0x00050000 # handle 1992
return # handle 1993
}
chain pbr_mark_0x060000 { # handle 1646
meta mark set meta mark & 0xff06ffff | 0x00060000 # handle 1994
return # handle 1995
}
chain pbr_mark_0x070000 { # handle 1649
meta mark set meta mark & 0xff07ffff | 0x00070000 # handle 1996
return # handle 1997
}
chain pbr_mark_0x080000 { # handle 1652
meta mark set meta mark & 0xff08ffff | 0x00080000 # handle 1998
return # handle 1999
}
chain pbr_mark_0x090000 { # handle 1655
meta mark set meta mark & 0xff09ffff | 0x00090000 # handle 2000
return # handle 2001
}
chain pbr_mark_0x0a0000 { # handle 1658
meta mark set meta mark & 0xff0affff | 0x000a0000 # handle 2002
return # handle 2003
}
chain pbr_mark_0x0b0000 { # handle 1661
meta mark set meta mark & 0xff0bffff | 0x000b0000 # handle 2004
return # handle 2005
}
pbr nft sets
IPv4 table 256 route: default via 192.168.188.1 dev eth1
IPv4 table 256 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.0.65.20 dev vpn_amsterdam
IPv4 table 257 rule(s):
29998: from all fwmark 0x20000/0xff0000 lookup pbr_vpnclient0
IPv4 table 258 route: default via 10.4.165.22 dev vpn_berlin
IPv4 table 258 rule(s):
29996: from all fwmark 0x30000/0xff0000 lookup pbr_vpnclient1
IPv4 table 259 route: default via 10.4.3.18 dev vpn_copenhagen
IPv4 table 259 rule(s):
29994: from all fwmark 0x40000/0xff0000 lookup pbr_vpnclient2
IPv4 table 260 route: unreachable default
IPv4 table 260 rule(s):
29992: from all fwmark 0x50000/0xff0000 lookup pbr_vpnclient3
IPv4 table 261 route: default via 10.4.181.242 dev vpn_hamburg
IPv4 table 261 rule(s):
29990: from all fwmark 0x60000/0xff0000 lookup pbr_vpnclient4
IPv4 table 262 route: unreachable default
IPv4 table 262 rule(s):
29988: from all fwmark 0x70000/0xff0000 lookup pbr_vpnclient5
IPv4 table 263 route: default via 10.1.118.241 dev vpn_oslo
IPv4 table 263 rule(s):
29986: from all fwmark 0x80000/0xff0000 lookup pbr_vpnclient6
IPv4 table 264 route: unreachable default
IPv4 table 264 rule(s):
29984: from all fwmark 0x90000/0xff0000 lookup pbr_vpnclient7
IPv4 table 265 route: default via 10.1.199.16 dev vpn_rotterdam
IPv4 table 265 rule(s):
29982: from all fwmark 0xa0000/0xff0000 lookup pbr_vpnclient8
IPv4 table 266 route: unreachable default
IPv4 table 266 rule(s):
29980: from all fwmark 0xb0000/0xff0000 lookup pbr_vpnclient9
root@OpenWrt:~#
etc/init.d/pbr reload
root@OpenWrt:~# /etc/init.d/pbr reload
Using wan interface (on_start): wan
Found wan gateway (on_start): 192.168.188.1
Using wan6 interface (on_start): wan6
Found wan6 gateway (on_start): fe80::2e91:abff:fe45:f3b5
Setting up routing for 'wan/eth1/192.168.188.1/2a02:a464:c5d0:1:6662:66ff:fe22:d598/128' [✓]
Setting up routing for 'vpnclient0/vpn_amsterdam/10.0.65.20/fdbf:1d37:bbe0:0:4:9:0:14/112' [✓]
Setting up routing for 'vpnclient1/vpn_berlin/10.4.165.22/fdbf:1d37:bbe0:0:74:5:0:16/112' [✓]
Setting up routing for 'vpnclient2/vpn_copenhagen/10.4.3.18/fdbf:1d37:bbe0:0:64:11:0:12/112' [✓]
Setting up routing for 'vpnclient3/vpn_frankfurt/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient4/vpn_hamburg/10.4.181.242/fdbf:1d37:bbe0:0:75:5:0:f2/112' [✓]
Setting up routing for 'vpnclient5/vpn_malmoe/0.0.0.0/::/0' [✓]
Setting up routing for 'vpnclient6/vpn_oslo/10.1.118.241/fdbf:1d37:bbe0:0:23:6:0:f1/112' [✓]
Setting up routing for 'vpnclient7/vpn_paris/10.1.130.17/fdbf:1d37:bbe0:0:24:10:0:11/112' [✓]
Setting up routing for 'vpnclient8/vpn_rotterdam/10.1.199.16/fdbf:1d37:bbe0:0:28:7:0:10/112' [✓]
Setting up routing for 'vpnclient9/vpn_stockholm/0.0.0.0/::/0' [✓]
Routing 'MacBook Pro M3' via vpnclient6 [✓]
Installing fw4 nft file [✓]
pbr 1.1.6-20 monitoring interfaces: wan vpnclient0 vpnclient1 vpnclient2 vpnclient3 vpnclient4 vpnclient5 vpnclient6 vpnclient7 vpnclient8 vpnclient9
pbr 1.1.6-20 (fw4 nft file mode) started with gateways:
wan/eth1/192.168.188.1/2a02:a464:c5d0:1:6662:66ff:fe22:d598/128
vpnclient0/vpn_amsterdam/10.0.65.20/fdbf:1d37:bbe0:0:4:9:0:14/112 [✓]
vpnclient1/vpn_berlin/10.4.165.22/fdbf:1d37:bbe0:0:74:5:0:16/112
vpnclient2/vpn_copenhagen/10.4.3.18/fdbf:1d37:bbe0:0:64:11:0:12/112
vpnclient3/vpn_frankfurt/0.0.0.0/::/0
vpnclient4/vpn_hamburg/10.4.181.242/fdbf:1d37:bbe0:0:75:5:0:f2/112
vpnclient5/vpn_malmoe/0.0.0.0/::/0
vpnclient6/vpn_oslo/10.1.118.241/fdbf:1d37:bbe0:0:23:6:0:f1/112
vpnclient7/vpn_paris/10.1.130.17/fdbf:1d37:bbe0:0:24:10:0:11/112
vpnclient8/vpn_rotterdam/10.1.199.16/fdbf:1d37:bbe0:0:28:7:0:10/112
vpnclient9/vpn_stockholm/0.0.0.0/::/0
root@OpenWrt:~#
etc/init.d/pbr status 2
root@OpenWrt:~# /etc/init.d/pbr status
pbr - environment
pbr 1.1.6-20 running on OpenWrt 23.05.4.
Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
add chain inet fw4 pbr_mark_0x020000
add rule inet fw4 pbr_mark_0x020000 mark set mark and 0xff00ffff xor 0x020000
add rule inet fw4 pbr_mark_0x020000 return
add chain inet fw4 pbr_mark_0x030000
add rule inet fw4 pbr_mark_0x030000 mark set mark and 0xff00ffff xor 0x030000
add rule inet fw4 pbr_mark_0x030000 return
add chain inet fw4 pbr_mark_0x040000
add rule inet fw4 pbr_mark_0x040000 mark set mark and 0xff00ffff xor 0x040000
add rule inet fw4 pbr_mark_0x040000 return
add chain inet fw4 pbr_mark_0x050000
add rule inet fw4 pbr_mark_0x050000 mark set mark and 0xff00ffff xor 0x050000
add rule inet fw4 pbr_mark_0x050000 return
add chain inet fw4 pbr_mark_0x060000
add rule inet fw4 pbr_mark_0x060000 mark set mark and 0xff00ffff xor 0x060000
add rule inet fw4 pbr_mark_0x060000 return
add chain inet fw4 pbr_mark_0x070000
add rule inet fw4 pbr_mark_0x070000 mark set mark and 0xff00ffff xor 0x070000
add rule inet fw4 pbr_mark_0x070000 return
add chain inet fw4 pbr_mark_0x080000
add rule inet fw4 pbr_mark_0x080000 mark set mark and 0xff00ffff xor 0x080000
add rule inet fw4 pbr_mark_0x080000 return
add chain inet fw4 pbr_mark_0x090000
add rule inet fw4 pbr_mark_0x090000 mark set mark and 0xff00ffff xor 0x090000
add rule inet fw4 pbr_mark_0x090000 return
add chain inet fw4 pbr_mark_0x0a0000
add rule inet fw4 pbr_mark_0x0a0000 mark set mark and 0xff00ffff xor 0x0a0000
add rule inet fw4 pbr_mark_0x0a0000 return
add chain inet fw4 pbr_mark_0x0b0000
add rule inet fw4 pbr_mark_0x0b0000 mark set mark and 0xff00ffff xor 0x0b0000
add rule inet fw4 pbr_mark_0x0b0000 return
add rule inet fw4 pbr_prerouting ether saddr { 10:9F:41:D6:AE:9C } goto pbr_mark_0x080000 comment "MacBook Pro M3"
pbr chains - policies
chain pbr_forward { # handle 37
}
chain pbr_input { # handle 38
}
chain pbr_output { # handle 39
}
chain pbr_postrouting { # handle 41
}
chain pbr_prerouting { # handle 40
ether saddr <Redacted> goto pbr_mark_0x080000 comment "MacBook Pro M3" # handle 2324
}
chain pbr_dstnat_lan { # handle 36
}
pbr chains - marking
chain pbr_mark_0x010000 { # handle 2291
meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 2292
return # handle 2293
}
chain pbr_mark_0x020000 { # handle 2294
meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 2295
return # handle 2296
}
chain pbr_mark_0x030000 { # handle 2297
meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 2298
return # handle 2299
}
chain pbr_mark_0x040000 { # handle 2300
meta mark set meta mark & 0xff04ffff | 0x00040000 # handle 2301
return # handle 2302
}
chain pbr_mark_0x050000 { # handle 2303
meta mark set meta mark & 0xff05ffff | 0x00050000 # handle 2304
return # handle 2305
}
chain pbr_mark_0x060000 { # handle 2306
meta mark set meta mark & 0xff06ffff | 0x00060000 # handle 2307
return # handle 2308
}
chain pbr_mark_0x070000 { # handle 2309
meta mark set meta mark & 0xff07ffff | 0x00070000 # handle 2310
return # handle 2311
}
chain pbr_mark_0x080000 { # handle 2312
meta mark set meta mark & 0xff08ffff | 0x00080000 # handle 2313
return # handle 2314
}
chain pbr_mark_0x090000 { # handle 2315
meta mark set meta mark & 0xff09ffff | 0x00090000 # handle 2316
return # handle 2317
}
chain pbr_mark_0x0a0000 { # handle 2318
meta mark set meta mark & 0xff0affff | 0x000a0000 # handle 2319
return # handle 2320
}
chain pbr_mark_0x0b0000 { # handle 2321
meta mark set meta mark & 0xff0bffff | 0x000b0000 # handle 2322
return # handle 2323
}
pbr nft sets
IPv4 table 256 route: default via 192.168.188.1 dev eth1
IPv4 table 256 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.0.65.20 dev vpn_amsterdam
IPv4 table 257 rule(s):
29998: from all fwmark 0x20000/0xff0000 lookup pbr_vpnclient0
IPv4 table 258 route: default via 10.4.165.22 dev vpn_berlin
IPv4 table 258 rule(s):
29996: from all fwmark 0x30000/0xff0000 lookup pbr_vpnclient1
IPv4 table 259 route: default via 10.4.3.18 dev vpn_copenhagen
IPv4 table 259 rule(s):
29994: from all fwmark 0x40000/0xff0000 lookup pbr_vpnclient2
IPv4 table 260 route: unreachable default
IPv4 table 260 rule(s):
29992: from all fwmark 0x50000/0xff0000 lookup pbr_vpnclient3
IPv4 table 261 route: default via 10.4.181.242 dev vpn_hamburg
IPv4 table 261 rule(s):
29990: from all fwmark 0x60000/0xff0000 lookup pbr_vpnclient4
IPv4 table 262 route: unreachable default
IPv4 table 262 rule(s):
29988: from all fwmark 0x70000/0xff0000 lookup pbr_vpnclient5
IPv4 table 263 route: default via 10.1.118.241 dev vpn_oslo
IPv4 table 263 rule(s):
29986: from all fwmark 0x80000/0xff0000 lookup pbr_vpnclient6
IPv4 table 264 route: default via 10.1.130.17 dev vpn_paris
IPv4 table 264 rule(s):
29984: from all fwmark 0x90000/0xff0000 lookup pbr_vpnclient7
IPv4 table 265 route: default via 10.1.199.16 dev vpn_rotterdam
IPv4 table 265 rule(s):
29982: from all fwmark 0xa0000/0xff0000 lookup pbr_vpnclient8
IPv4 table 266 route: unreachable default
IPv4 table 266 rule(s):
29980: from all fwmark 0xb0000/0xff0000 lookup pbr_vpnclient9
root@OpenWrt:~#