Policy-Based-Routing (pbr) package discussion

Community Builds, Projects & Packages
336

thanks, I do apologise I am new to openwrt so not sure how to do logread -epbr or service pbr info but I gave it a shot below while with error message loaded:

logread -epbr:

Blockquote
root@OpenWrt:~# logread -epbr
Wed Mar 1 19:10:16 2023 user.notice pbr: Reloading pbr due to includes of firewall
Wed Mar 1 19:10:16 2023 user.notice pbr: Reload on firewall action aborted: service not running.
Wed Mar 1 19:10:16 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:17 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:18 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:19 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:20 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:21 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:22 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:24 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:25 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:26 2023 user.notice pbr: service waiting for wan gateway...
Wed Mar 1 19:10:27 2023 user.notice pbr: Activating traffic killswitch [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Setting up routing for 'wan/eth1/8.x.x.x' [✓]
Wed Mar 1 19:10:27 2023 daemon.notice procd: /etc/rc.d/S94pbr: Error: argument "257
Wed Mar 1 19:10:27 2023 daemon.notice procd: /etc/rc.d/S94pbr: 258" is wrong: invalid table ID
Wed Mar 1 19:10:27 2023 daemon.notice procd: /etc/rc.d/S94pbr:
Wed Mar 1 19:10:27 2023 user.notice pbr: Setting up routing for 'WG/10.x.x.x' [✗]
Wed Mar 1 19:10:27 2023 user.notice pbr: Setting up routing for 'WGAIRBackup/0.0.0.0' [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Routing 'Virgin-TV-Upstairs' via wan [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Routing 'Virgin-TV-Downstairs' via wan [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Routing 'LGTV' via wan [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: service monitoring interfaces: wan WG WGAIRBackup
Wed Mar 1 19:10:27 2023 user.notice pbr: Reloading pbr due to includes of firewall
Wed Mar 1 19:10:27 2023 user.notice pbr: Activating traffic killswitch [✓]
Wed Mar 1 19:10:27 2023 user.notice pbr: Setting up routing for 'wan/eth1/8.x.x.x' [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Setting up routing for 'WG/10.x.x.x' [✗]
Wed Mar 1 19:10:28 2023 user.notice pbr: Setting up routing for 'WGAIRBackup/0.0.0.0' [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Routing 'Virgin-TV-Upstairs' via wan [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Routing 'Virgin-TV-Downstairs' via wan [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Routing 'LGTV' via wan [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: service monitoring interfaces: wan WG WGAIRBackup
Wed Mar 1 19:10:28 2023 user.notice pbr: Reloading pbr due to includes of firewall
Wed Mar 1 19:10:28 2023 user.notice pbr: Activating traffic killswitch [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Setting up routing for 'wan/eth1/8.x.x.x' [✓]
Wed Mar 1 19:10:28 2023 user.notice pbr: Setting up routing for 'WG/10.x.x.x' [✗]
Wed Mar 1 19:10:29 2023 user.notice pbr: Setting up routing for 'WGAIRBackup/0.0.0.0' [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Routing 'Virgin-TV-Upstairs' via wan [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Routing 'Virgin-TV-Downstairs' via wan [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Routing 'LGTV' via wan [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: service monitoring interfaces: wan WG WGAIRBackup
Wed Mar 1 19:10:29 2023 user.notice pbr: Reloading pbr due to includes of firewall
Wed Mar 1 19:10:29 2023 user.notice pbr: Activating traffic killswitch [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Setting up routing for 'wan/eth1/8.x.x.x' [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Setting up routing for 'WG/10.x.x.x' [✗]
Wed Mar 1 19:10:29 2023 user.notice pbr: Setting up routing for 'WGAIRBackup/0.0.0.0' [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Routing 'Virgin-TV-Upstairs' via wan [✓]
Wed Mar 1 19:10:29 2023 user.notice pbr: Routing 'Virgin-TV-Downstairs' via wan [✓]
Wed Mar 1 19:10:30 2023 user.notice pbr: Routing 'LGTV' via wan [✓]
Wed Mar 1 19:10:30 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Wed Mar 1 19:10:30 2023 user.notice pbr: service monitoring interfaces: wan WG WGAIRBackup

Also service pbr info

Blockquote
{
"pbr": {
"instances": {
"main": {
"running": false,
"command": [
"/bin/true"
],
"term_timeout": 5,
"exit_code": 0,
"data": {
"gateways": [
{
"name": "wan",
"device_ipv4": "eth1",
"gateway_ipv4": "8.x.x.x",
"device_ipv6": "eth1",
"gateway_ipv6": "f::x:x:x:x/64",
"default": false,
"action": "create",
"table_id": "256",
"mark": "0x010000",
"priority": "30000"
},
{
"name": "WGAIRBackup",
"device_ipv4": "",
"gateway_ipv4": "",
"device_ipv6": "",
"gateway_ipv6": "",
"default": false,
"action": "create",
"table_id": "258",
"mark": "0x030000",
"priority": "30002"
}
],
"status": {
"gateways": "wan/eth1/8.x.x.x\nWGAIRBackup/0.0.0.0\n",
"errors": "errorFailedSetup WG/10.x.x.x",
"mode": "strict"
}
}
}
},
"triggers": [
[
"config.change",
[
"if",
[
"eq",
"package",
"openvpn"
],
[
"run_script",
"/etc/init.d/pbr",
"reload"
]
]
],
[
"config.change",
[
"if",
[
"eq",
"package",
"pbr"
],
[
"run_script",
"/etc/init.d/pbr",
"reload"
]
]
],
[
"interface.",
[
"if",
[
"eq",
"interface",
"wan"
],
[
"run_script",
"/etc/init.d/pbr",
"on_interface_reload",
"wan"
]
]
],
[
"interface.",
[
"if",
[
"eq",
"interface",
"WG"
],
[
"run_script",
"/etc/init.d/pbr",
"on_interface_reload",
"WG"
]
]
],
[
"interface.*",
[
"if",
[
"eq",
"interface",
"WGAIRBackup"
],
[
"run_script",
"/etc/init.d/pbr",
"on_interface_reload",
"WGAIRBackup"
]
]
]
],
"validate": [
{
"package": "pbr",
"type": "include",
"rules": {
"enabled": "bool",
"path": "file"
}
},
{
"package": "pbr",
"type": "policy",
"rules": {
"chain": "or(, forward, input, output, prerouting, postrouting, FORWARD, INPUT, OUTPUT, PREROUTING, POSTROUTING)",
"dest_addr": "list(neg(or(host,network,string)))",
"dest_port": "list(neg(or(portrange,string)))",
"enabled": "bool",
"interface": "or(uci(network, @interface),ignore)",
"name": "string",
"proto": "or(string)",
"src_addr": "list(neg(or(host,network,macaddr,string)))",
"src_port": "list(neg(or(portrange,string)))"
}
},
{
"package": "pbr",
"type": "pbr",
"rules": {
"boot_timeout": "integer",
"enabled": "bool",
"fw_mask": "regex('0x[A-x-x-x]{x}')",
"icmp_interface": "or(,ignore, uci(network, @interface))",
"ignored_interface": "list(uci(network, @interface))",
"ipv6_enabled": "bool",
"procd_boot_delay": "integer",
"procd_reload_delay": "integer",
"resolver_set": "or(, none, dnsmasq.ipset, dnsmasq.nftset)",
"rule_create_option": "or(, add, insert)",
"secure_reload": "bool",
"strict_enforcement": "bool",
"supported_interface": "list(uci(network, @interface))",
"verbosity": "range(0,2)",
"wan_ip_rules_priority": "uinteger",
"wan_mark": "regex('0x[A-x-x-x]{x}')",
"webui_supported_protocol": "list(string)"
}
}
]
}

I have removed any ip/mac info from the post. I think I have done a incorrect set up with 2 WG interfaces, I did remove my 2nd back up WG so just left 1 WG interface, rebooting that few times and no pbr gateway error so it works fine with just 1 WG interface.

Since I am new to openwrt I can just stick with 1 WG interface and modify as required might be easier, but if there is a simple fix thanks.

337

Thank you. There's still this

pbr.cfg076ff5.name=DEFAULTVPNI validates as string with true
pbr.cfg076ff5.enabled=0 validates as bool with true
pbr.cfg076ff5.interface=somevpn_wg validates as or("ignore", "tor", uci("network", "@interface")) with true
pbr.cfg076ff5.proto is unset and defaults to or(string) (null)
pbr.cfg076ff5.chain is unset and defaults to or("", "forward", "input", "output", "prerouting", "postrouting", "FORWARD", "INPUT", "OUTPUT", "PREROUTING", "POSTROUTING") prerouting
pbr.cfg076ff5.src_addr=  validates as list(neg(or(host,network,macaddr,string))) with false
pbr.cfg076ff5.src_port is unset and defaults to list(neg(or(portrange,string))) (null)
pbr.cfg076ff5.dest_addr is unset and defaults to list(neg(or(host,network,string))) (null)
pbr.cfg076ff5.dest_port is unset and defaults to list(neg(or(portrange,string))) (null)

TOR interface reappeared in the UI.

logread

...
daemon.err dnsmasq[1]: nftset inet fw4 pbr_tor_4_dst_ip Error: No such file or directory
...

And for this

I provide part of the config for now:

config policy                           
        option src_port '4000'        
        option dest_addr '101.01.010.10'
        option interface 'wan2'        
        option chain 'output'          
        option name 'INIT_SOME_VPN2_WG'
        option dest_port '9000'        
...
config pbr 'config'                                                                                                                                                                                                                               
        option verbosity '2'                                                                                                                                                                                                                      
        option strict_enforcement '1'   
        option boot_timeout '0'                                                                                                                                                                                                                   
        option secure_reload '0'                                                                                                                                                                                                                  
        option procd_reload_delay '0'   
        option webui_sorting '1'                                                                                                                                                                                                                  
        list webui_supported_protocol 'tcp'                                                                                                                                                                                                       
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'                                                                                                                                                                                                   
        list webui_supported_protocol 'icmp'                                                                                                                                                                                                      
        list webui_supported_protocol 'all' 
        option webui_enable_column '1'                                                                                                                                                                                                            
        option webui_protocol_column '1'                                                                                                                                                                                                          
        option webui_chain_column '1'       
        option webui_show_ignore_target '1'                                                                                                                                                                                                       
        option ipv6_enabled '0'                                                                                                                                                                                                                   
        option dest_ipset '1'               
        option src_ipset '1'                                                                                                                                                                                                                      
        list supported_interface 'lan'                                                                                                                                                                                                            
        list supported_interface 'tor'      
        list supported_interface 'wan'                                                                                                                                                                                                            
        list supported_interface 'wan2'                                                                                                                                                                                                           
        list supported_interface 'somevpn_wg'
        list supported_interface 'somevpn2_wg'                                                                                                                                                                                                                                                                                                                                                                                                       
        list ignored_interface 'wgserver'                                                                                                                                                                                                         
        option rule_create_option 'add'                                                        
        option resolver_set 'dnsmasq.nftset'                                                                                                                                                                                                      
        option enabled '1'

And


daemon.err dnsmasq[1]: nftset inet fw4 pbr_somevpn2_wg_4_dst_ip_cfg136ff5 Error: No such file or directory; did you mean set ‘pbr_wan_4_dst_ip_cfg026ff5’ in table inet ‘fw4’?
338

Not sure if you've redacted the value or it's set to nothing. If latter, that's invalid.

How are you testing this?

339

I suspect it's an older/outdated version of pbr, could you please confirm the installed pbr version thru WebUI?

1 Like
340

Realtime graph. There's no traffic going through this specific wan2 interface.

Yes it's not set and invalid. Anyway, the policy is disabled.

network

...
config interface 'wan2'                                                                                            
        option proto 'prototype'   
        option force_link '1'      
        option auth 'none'                                                                                         
        option peerdns '0'         
        list dns '127.0.0.1'       
        option iptype 'ipv4v6'                                                                                     '
        option device '/sys/devices/...'
        option defaultroute '0'    

config interface 'somevpn2_wg'                                                                                     
        option proto 'wireguard'                                           
        list addresses '10.10.101.010/32'                                                                          
        list addresses ':ipv6::::/128'                                                           
        option private_key 'somekey'  
        option force_link '1'                                                                                      
        option listen_port '4000'                                                                                 
        option defaultroute '0'       

config wireguard_somevpn2_wg                                                                                       
        option description 'somedescription'                                
        list allowed_ips '0.0.0.0/0'                                                                               
        list allowed_ips '::0/0'                                                                                   
        option endpoint_host '101.01.010.10'                               
        option endpoint_port '9000'                                                                                
        option public_key 'somekey'

# wan2         
config device                                                                                                      
        option name 'wwan1'   

config device                                                                                                      
        option name 'somevpn2_wg'                                                                                  
...

firewall

...
config include 'pbr'                  
        option fw4_compatible '1'    
        option type 'script'          
        option path '/usr/share/pbr/pbr.firewall.include'
...
341

thanks just checked (version: 1.0.1-16 using nft)

342

Upgraded to 1.1.0-4 .. policies appear to be working correctly but I'm getting these odd 'Unknown Error!' messages under Service Errors

Syslog doesn't show anything out of the ordinary either.

Service Status

Running (version: 1.1.0-4 using nft)

Service Gateways

wan/pppoe-wan/10.11.1.201 ✓
tun0_VPN/tun0/10.7.3.2
tun1_VPN/tun1/10.7.1.6
tun2_VPN/tun2/10.7.3.5
tun3_VPN/tun3/10.15.0.4

Service Errors

Unknown Error!
Unknown Error!
Unknown Error!
Unknown Error!

edit: I see the errors in pbr info

"errors": "errorInvalidOVPNConfig tun0_VPN#errorInvalidOVPNConfig tun1_VPN#errorInvalidOVPNConfig tun2_VPN#errorInvalidOVPNConfig tun3_VPN"
343

Update the luci app and if you have already updated it, clear the browser cache.

Please also post (or PM me) your /etc/config/network and /etc/config/openvpn.

344

Post browser clear the messages have changed to this. Policies seem to be working fine still. I'm DM'ing you those files.


Service Errors

Invalid OpenVPN config for tun0_VPN interface
Invalid OpenVPN config for tun1_VPN interface
Invalid OpenVPN config for tun2_VPN interface
Invalid OpenVPN config for tun3_VPN interface
345

Also started getting those: Invalid OpenVPN config for...
And assuming by that check is_valid_ovpn() @ /etc/init.d/pbr, it fails if the name of the OpenVPN config's entry (dev_ovpn="$(uci -q get "openvpn.${1}.dev")") is not the same as the name of the interface.
Could you please give the idea behind this check? Why are you supposing that it should be named the same as the interface and what are the consequences of pbr assuming that the OpenVPN config is invalid?
Should I rename the OpenVPN config's entries accordingly?

346

@rawd @yxtc934 https://docs.openwrt.melmac.net/pbr/#multiple-openvpn-clients

dev should match between the /etc/config/network and /etc/config/openvpn.

You'll continue receiving these errors until you fix the OpenVPN config.

1 Like
347

Yes, I got it that I would keep receiving them until I rename the configs, but like I've been asking in the original post, what's the idea and what it affects?

348

Dev do match within each.. The ovpn config has them set and I've been using this same set up for various versions.

349

No they don't, the /etc/config/openvpn you PMed me doesn't have dev definitions at all.

350

No need to rename configs.

If OpenVPN is properly configured, the order of the tunnels is guaranteed to persist between restarts and interface down/up events.

351

If dev is set inside of the .ovpn config file does it not satisfy that ?

also if I restart pbr, the messages begin to duplicate if it's any help

Service Errors
Invalid OpenVPN config for tun0_VPN interface
Invalid OpenVPN config for tun1_VPN interface
Invalid OpenVPN config for tun2_VPN interface
Invalid OpenVPN config for tun3_VPN interface
Invalid OpenVPN config for tun0_VPN interface
Invalid OpenVPN config for tun1_VPN interface
Invalid OpenVPN config for tun2_VPN interface
Invalid OpenVPN config for tun3_VPN interface
352

So I'll just happily ignore the error?
By the way that check in pbr is also giving me the error about the wireguard config being invalid, which in no possible way could have been matched by uci -q get "openvpn.${1}.dev"
As a side note, I never had problems with the interface order.

353

I had the same Service errors (Invalid OpenVPN config for tun0 interface | Invalid OpenVPN config for tun1 interface) popping up after updating to 1.1.0-4 using nft even when I'm using Wireguard and not Openvpn, although the name of the 2 WG interfaces were tun0 and tun1. An update to 1.1.0-5 using nft resulted in Unknown Warning! message. I then changed the name of the 2 interfaces to WG0 and WG1 and now the Service error message has disappeared. Strange as it seems PBR is looking at the name of the interface i.s.o. the protocol used?

1 Like
354

If that's the case I'll live with the errors as well. I have my own little naming convention for my tunnels. Everything still works. :+1:

355 
			nft meta nfproto ipv4 udp daddr "@${set_name4}" dport 53 counter redirect to :"$dnsPort" comment "Tor-DNS-UDP-ipv4" || s=1
				nft meta nfproto ipv4 tcp daddr "@${set_name4}" dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-TCP-ipv4" || s=1
				nft meta nfproto ipv4 udp daddr "@${set_name4}" dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-UDP-ipv4" || s=1
				nft meta nfproto ipv4 tcp daddr "@${set_name4}" dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-TCP-ipv4" || s=1
				nft meta nfproto ipv4 udp daddr "@${set_name4}" dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-UDP-ipv4" || s=1
				nft6 meta nfproto ipv6 udp daddr "@${set_name6}" dport 53 counter redirect to :"$dnsPort" comment "Tor-DNS-UDP-ipv6" || s=1
				nft6 meta nfproto ipv6 tcp daddr "@${set_name6}" dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-TCP-ipv6" || s=1
				nft6 meta nfproto ipv6 udp daddr "@${set_name6}" dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-UDP-ipv6" || s=1
				nft6 meta nfproto ipv6 tcp daddr "@${set_name6}" dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-TCP-ipv6" || s=1
				nft6 meta nfproto ipv6 udp daddr "@${set_name6}" dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-UDP-ipv6" || s=1

add rule