Hi, here:
Thanks a lot for checking
I have adblock
I think that a startup delay is necessary in some case
Probably
The problem looks indeed to be a race condition at boot up.
The easy way for now is probably to add to /etc/rc.local (the startup):
sleep 60 && service pbr restart
In my case the problem seems to be solved, can you or @stangri please check this:
Just the rework on triggers solve the issue in my case. Thanks guys
Thanks @pesa1234 we are still studying about the root cause.
Unfortunately your solution did not work for me but the solution might indeed be in the region you are looking as the problem seems very early in the boot process.
I am pursuing a different angle but also not sure if that will work.
I have uploaded my test version: pbr-1.1.8-r28-egc-5.bash, see:https://github.com/egc112/OpenWRT-egc-add-on/tree/main/pbr
But if someone decides to test it make sure you make a backup
To be continued
1 Like
Hi, with 1.1.8-r29 seems ok on my side...
Thanks for testing, unfortunately it did not for me 
If you have the time can you test my version:
It is easy to switch back
Hi, I've tested your version, the problem is still present on my side sorry.
I confirm that r29 is ok!
1 Like
Hello,
I'm having an issue with routing a specific subnet through my WireGuard VPN tunnel. My host is an OpenWrt VM in Proxmox without a WAN interface, connected to my LAN and two other VLANs. The default gateway is my physical OpenWrt device on the lan interface.
I have tried setting procd_lan_device and procd_wan_interface to every combination I can think of, but none of them make the PBR service start correctly. In all cases, I get the following errors:
... Setting up routing for 'wg0/10.2.0.2' [✗] ... ERROR: ip -4 route add default via 10.50.10.1 dev eth0 proto static table 256 ERROR: Failed to set up 'wg0/10.2.0.2'! ERROR: Failed to set up any gateway!
My goal is to simply have traffic from my VMs and LXCs connected to my pvt VLAN routed through my VPN tunnel, except for traffic bound to private networks (which I will firewall accordingly).
This issue occurs even if no policies are enabled. See full config and logs below.
ubus call system board
{
"kernel": "6.6.86",
"hostname": "pvt-openwrt",
"system": "AMD Ryzen 5 5600 6-Core Processor",
"model": "QEMU Standard PC (Q35 + ICH9, 2009)",
"board_name": "qemu-standard-pc-q35-ich9-2009",
"rootfs_type": "ext4",
"release": {
"distribution": "OpenWrt",
"version": "24.10.1",
"revision": "r28597-0425664679",
"target": "x86/64",
"description": "OpenWrt 24.10.1 r28597-0425664679",
"builddate": "1744562312"
}
}
uci export dhcp
package dhcp
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '0'
option local '/lan/'
option domain 'home.arpa'
option expandhosts '1'
option cachesize '1000'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option localservice '1'
option ednspacket_max '1232'
list server '10.50.10.1'
option dhcpleasemax '500'
option dnsforwardmax '1000'
option logqueries '1'
config dhcp 'lan'
option interface 'lan'
option start '100'
option limit '150'
option leasetime '12h'
option dhcpv4 'server'
option ignore '1'
config dhcp 'wan'
option interface 'wan'
option ignore '1'
config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'
config dhcp 'pvt'
option interface 'pvt'
option start '100'
option limit '150'
option leasetime '1h'
option force '1'
config dhcp 'services'
option interface 'services'
option start '100'
option limit '150'
option leasetime '1h'
option force '1'
config host
list mac 'BC:24:11:DF:14:DF'
option ip '10.0.50.3'
option leasetime '12h'
option dns '1'
uci export network
package network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd9e:c330:2035::/48'
option packet_steering '1'
config interface 'lan'
option device 'eth0'
option proto 'static'
option ipaddr '10.50.60.1'
option netmask '255.255.0.0'
option gateway '10.50.10.1'
list dns '10.50.10.1'
option delegate '0'
config device
option name 'eth0'
option acceptlocal '1'
option ipv6 '0'
config interface 'wg0'
option proto 'wireguard'
option private_key '<redacted>'
list addresses '10.2.0.2/32'
list dns '10.2.0.1'
option delegate '0'
config wireguard_wg0
option public_key '<redacted>'
option endpoint_host '144.48.38.178'
option endpoint_port '51820'
option persistent_keepalive '25'
option description 'Proton AU#297'
list allowed_ips '0.0.0.0/0'
config interface 'pvt'
option proto 'static'
option device 'eth0.60'
list ipaddr '10.0.60.2/24'
list dns '10.2.0.1'
config interface 'services'
option proto 'static'
option device 'eth0.50'
list ipaddr '10.0.50.2/24'
uci export pbr
package pbr
config pbr 'config'
option enabled '1'
option verbosity '2'
option strict_enforcement '1'
option resolver_set 'none'
list resolver_instance '*'
option ipv6_enabled '0'
option boot_timeout '20'
option rule_create_option 'add'
option procd_reload_delay '5'
option webui_show_ignore_target '1'
option nft_rule_counter '1'
option nft_set_auto_merge '1'
option nft_set_counter '1'
option nft_set_flags_interval '1'
option nft_set_flags_timeout '0'
option nft_set_policy 'performance'
list webui_supported_protocol 'all'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list procd_lan_device 'eth0'
list procd_lan_device 'eth0.50'
list procd_lan_device 'eth0.60'
option procd_wan_interface 'lan'
config policy
option name 'Ignore to Local'
option interface 'ignore'
option enabled '0'
option dest_addr '10.0.0.0/8 192.168.0.0/16 172.16.0.0/12'
config policy
option name 'Private Services'
option src_addr '10.0.60.0/24'
option interface 'wg0'
option enabled '0'
config dns_policy
option name 'Private Services DNS WireGuard'
option src_addr '10.0.60.0/24'
option dest_dns 'wg0'
option enabled '0'
/etc/init.d/pbr status [service stopped]
pbr - environment
pbr 1.1.8-r16 running on OpenWrt 24.10.1.
Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
pbr chains - policies
chain pbr_forward { # handle 45
}
chain pbr_input { # handle 46
}
chain pbr_output { # handle 47
}
chain pbr_postrouting { # handle 49
}
chain pbr_prerouting { # handle 48
}
chain pbr_dstnat { # handle 44
}
pbr chains - marking
pbr nft sets
pbr tables & routing
/etc/init.d/pbr reload [service starts]
Using uplink interface (on_start): lan [✓]
Found uplink gateway (on_start): 10.50.10.1 [✓]
Setting up routing for 'wg0/10.2.0.2' [✗]
Installing fw4 nft file [✓]
Setting interface trigger for wg0 [✓]
pbr 1.1.8-r16 monitoring interfaces: wg0
ERROR:
ip -4 route add default via 10.50.10.1 dev eth0 proto static table 256
ERROR: Failed to set up 'wg0/10.2.0.2'!
ERROR: Failed to set up any gateway!
WARNING: Please set 'dhcp.lan.force=1' to speed up service start-up.
/etc/init.d/pbr status [service running]
pbr - environment
pbr 1.1.8-r16 running on OpenWrt 24.10.1.
Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
pbr chains - policies
chain pbr_forward { # handle 45
}
chain pbr_input { # handle 46
}
chain pbr_output { # handle 47
}
chain pbr_postrouting { # handle 49
}
chain pbr_prerouting { # handle 48
}
chain pbr_dstnat { # handle 44
}
pbr chains - marking
chain pbr_mark_0x010000 { # handle 3291
counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 3292
return # handle 3293
}
pbr nft sets
pbr tables & routing
IPv4 table 256 pbr_wg0 route:
default via 10.2.0.2 dev wg0
IPv4 table 256 pbr_wg0 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wg0
Manually entering the command that is shown as the one creating the error (ip -4 route add...), returns RTNETLINK answers: File exists. If I ignore this, and enable the policies anyway,
uci export pbr
root@pvt-openwrt:~# uci export pbr
package pbr
config pbr 'config'
option enabled '1'
option verbosity '2'
option strict_enforcement '1'
option resolver_set 'none'
list resolver_instance '*'
option ipv6_enabled '0'
option boot_timeout '20'
option rule_create_option 'add'
option procd_reload_delay '5'
option webui_show_ignore_target '1'
option nft_rule_counter '1'
option nft_set_auto_merge '1'
option nft_set_counter '1'
option nft_set_flags_interval '1'
option nft_set_flags_timeout '0'
option nft_set_policy 'performance'
list webui_supported_protocol 'all'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list procd_lan_device 'eth0'
list procd_lan_device 'eth0.50'
list procd_lan_device 'eth0.60'
option procd_wan_interface 'lan'
config policy
option name 'Ignore to Local'
option interface 'ignore'
- option enabled '0'
option dest_addr '10.0.0.0/8 192.168.0.0/16 172.16.0.0/12'
config policy
option name 'Private Services'
option src_addr '10.0.60.0/24'
option interface 'wg0'
- option enabled '0'
config dns_policy
option name 'Private Services DNS WireGuard'
option src_addr '10.0.60.0/24'
option dest_dns 'wg0'
- option enabled '0'
/etc/init.d/pbr reload
Using uplink interface (on_start): lan [✓]
Found uplink gateway (on_start): 10.50.10.1 [✓]
Setting up routing for 'wg0/10.2.0.2' [✗]
+ Routing 'Ignore to Local' via ignore [✓]
+ Routing 'Private Services' via wg0 [✓]
+ Routing 'Private Services DNS WireGuard' DNS to wg0 [✓]
Installing fw4 nft file [✓]
Setting interface trigger for wg0 [✓]
pbr 1.1.8-r16 monitoring interfaces: wg0
ERROR:
ip -4 route add default via 10.50.10.1 dev eth0 proto static table 256
ERROR: Failed to set up 'wg0/10.2.0.2'!
ERROR: Failed to set up any gateway!
WARNING: Please set 'dhcp.lan.force=1' to speed up service start-up.
/etc/init.d/pbr status
pbr - environment
pbr 1.1.8-r16 running on OpenWrt 24.10.1.
Dnsmasq version 2.90 Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
pbr fw4 nft file: /usr/share/nftables.d/ruleset-post/30-pbr.nft
add chain inet fw4 pbr_mark_0x010000
add rule inet fw4 pbr_mark_0x010000 counter mark set mark and 0xff00ffff xor 0x010000
add rule inet fw4 pbr_mark_0x010000 return
+ add rule inet fw4 pbr_prerouting ip daddr { 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12 } counter return comment "Ignore to Local"
+ add rule inet fw4 pbr_prerouting ip saddr { 10.0.60.0/24 } counter goto pbr_mark_0x010000 comment "Private Services"
+ add rule inet fw4 pbr_dstnat ip saddr { 10.0.60.0/24 } counter meta nfproto ipv4 tcp dport 53 dnat ip to 10.2.0.1:53 comment "Private Services DNS WireGuard"
+ add rule inet fw4 pbr_dstnat ip saddr { 10.0.60.0/24 } counter meta nfproto ipv4 udp dport 53 dnat ip to 10.2.0.1:53 comment "Private Services DNS WireGuard"
pbr chains - policies
chain pbr_forward { # handle 45
}
chain pbr_input { # handle 46
}
chain pbr_output { # handle 47
}
chain pbr_postrouting { # handle 49
}
chain pbr_prerouting { # handle 48
+ ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 } counter packets 20 bytes 1188 return comment "Ignore to Local" # handle 3377
+ ip saddr 10.0.60.0/24 counter packets 0 bytes 0 goto pbr_mark_0x010000 comment "Private Services" # handle 3378
}
chain pbr_dstnat { # handle 44
+ ip saddr 10.0.60.0/24 counter packets 0 bytes 0 meta nfproto ipv4 tcp dport 53 dnat ip to 10.2.0.1:53 comment "Private Services DNS WireGuard" # handle 3379
+ ip saddr 10.0.60.0/24 counter packets 0 bytes 0 meta nfproto ipv4 udp dport 53 dnat ip to 10.2.0.1:53 comment "Private Services DNS WireGuard" # handle 3380
}
pbr chains - marking
- chain pbr_mark_0x010000 { # handle 3291
+ chain pbr_mark_0x010000 { # handle 3373
- counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 3292
+ counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 3374
- return # handle 3293
+ return # handle 3375
}
pbr nft sets
pbr tables & routing
IPv4 table 256 pbr_wg0 route:
default via 10.2.0.2 dev wg0
IPv4 table 256 pbr_wg0 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_wg0
my test Windows VM still does not get internet access:
ipconfig
Windows IP Configuration
Ethernet adapter Ethernet:
Connection-specific DNS Suffix . : home.arpa
Link-local IPv6 Address . . . . . : fe80::5bbb:e270:9f1f:cdb2%5
IPv4 Address. . . . . . . . . . . : 10.0.60.158
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.0.60.2
ping 1.1.1.1
Pinging 1.1.1.1 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 1.1.1.1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
Changing procd_wan_interface to wg0 does not change anything.
I'm not sure if this is exactly the issue, but the status command says that PBR is setting the default gateway for the table to 10.2.0.2:
IPv4 table 256 pbr_wg0 route:
default via 10.2.0.2 dev wg0
Not only does that conflict with the output of the service start command, which I believe tries to set the default gateway for the same table to my WAN one:
ERROR:
ip -4 route add default via 10.50.10.1 dev eth0 proto static table 256
ERROR: Failed to set up 'wg0/10.2.0.2'!
ERROR: Failed to set up any gateway!
But 10.2.0.2 is the local WireGuard interface's IP address; should it not be 10.2.0.1 (which is provided as the DNS server on Proton VPN)?
Then again, I don't believe the gateway address of a WireGuard interface is ever known (is it even a concept in WireGuard?), all I've ever seen is setting the WireGuard device as the gateway; if I were to set option route_allowed_ips '1' on my WireGuard interface, ip route would show something like: default dev wg0 table default...