I hve been using it without dnsmasq.ipset. Would be happy to test a nftables capable version.
Running OpenWRT on a Raspberry pi 4 if it matters
Great! You'll need to uninstall the previous VPR/pbr versions and install the latest pbr-nftables from my repo: https://repo.openwrt.melmac.net/.
The nft binary needs to be present and the resolver_set set to something else than dnsmasq.ipset for the nft to be used when processing policies.
I have briefly tested an earlier version and it worked well, if you see the counters increasing in the pbr_mark_xxxxx policies it's a good sign.
Let me know how it goes, I've made an attempt to use more of the OpenWrt's fw4 framework in the latest version and I haven't had an opportunity to test that yet.
I understand that vpn-policy-routing is getting absolute in the future. So I thought Iām going to switch from vpn-policy-routing to pbr. But there is no package available.
Is this because the Iptable / nftable issues?
Or can I just get them from your repo and use the iptable version?
Iām still on OpenWrt 21.02.3 r16554-1d4dea6d4f / LuCI openwrt-21.02 branch git-22.213.35964-87836ca.
I've updated the relevant section in the README with:
There are now two packages of pbr available: pbr-iptables if you want to use iptables/ipset/dnsmasq.ipset options and pbr-nftables which supports nft (but because OpenWrt's dnsmasq doesn't support nft sets yet, you can't use dnsmasq to resolve domain names from policies). Please note that both vpn-policy-routing and vpnbypass packages will not be transitioned to nftables and will become obsolete once OpenWrt's dnsmasq package no longer supports ipset.
1 Like
Shorten the name of relevant interface!
Why I get this message now??? Where I'm wrong?
I install pbr-iptables but seems to be installed pbr-nftables
### Service Status [pbr-iptables 0.9.7-3]
Service Status
Running (strict mode)
Service Gateways
wan/192.1x.1x.1
VPN_TUN/tun0/10.x.x.2 ā
wwan/0.0.0.0
The **ā** indicates default gateway. See the [README](https://docs.openwrt.melmac.net/pbr/#a-word-about-default-routing) for details.
Service Errors
Resolver set support (dnsmasq.nftset) is enabled in pbr, but this resolver set is not supported on this system!
nft set name 'pbr_wan_4_dst_ip_cfg076ff5' is longer than allowed 16 characters. Shorten the name of relevant interface!
Nowhere, fixed in 0.9.7-4. There were issues with long ipset names for some interfaces with long names and I've enabled checking of the ipset and nft set names length. I've checked online docs and two sources said the maximum nft set name length is 16 bytes, so that's what I've set in the previous version. OpenWrt supports nft sets with names as long as 255 bytes, so I've updated the check with the new value.
The init script is now the same for iptables and nftables versions. For now, if the nft binary is present and the resolver_set is not set to dnsmasq.ipset, it tries to use nft. Since the nftables version also installs some other files, I'll improve the check for which version is installed which netfilter to use in the future.
Just improved the check wherever to use nft.
Oh, I hope to overcome this in the near future, but as of now, after the installation of pbr-nftables you need to run fw4 reload and then start/restart pbr.
I am using openwrt 22.03 x86. So i have to use pbr-nfttables.
But do you have a package you can suggest to use for the resolver_set functionality since DNSmasq can't be used?
So far using PBR and vpn_bypass but neither has worked reliably on 22.03 for me.
Would getting a supported resolver_set fix the some of my issues?
No, you don't have to use pbr-nftables, you can use pbr with iptables. The init script is the same now, you don't have to reinstall the package, but I'd suggest switching 'resolver_set' to dnsmasq.ipset and thus using current dnsmasq version ipset support.
Sorry, the development outpaced documentation, so before I put it into the README, here's what I've changed recently.
In order to be able to update/bugfix both iptables and nftables supporting versions, I've merged both iptables and nftables init scripts into the same init script (what you actually run when you invoke service pbr ... or /etc/init.d/pbr ...).
Now, either pbr-iptables or pbr-nftables actually install the same init script. In addition to that, the pbr-nftables also installes some fw4-specific files to create custom nft chains for pbr and set things up so that after all the standard fw4 chains there's a jump into the pbr-specific chains.
The pbr-iptables installs the config file which has resolver_set set to dnsmasq.ipset.
The pbr-nftables installs the config file which has resolver_set set to dnsmasq.nftset.
If you already have a /etc/config/pbr file, it will NOT be overwritten during either flavour of pbr install.
So if you're on 22.03 and you install pbr-nftables, you can change the resolver_set to dnsmasq.ipset and the pbr will create iptables, ipsets and use dnsmasq ipset support for domains. If you set the resolver_set to anything else than dnsmasq.ipset (the other recognized options for now are none and non-working dnsmasq.nftset), the pbr-nftables will actually use the nft and nft sets.
You can also install and use pbr-iptables on 22.03.
If you're on 21.02 and earlier, you can only install/use pbr-iptables.
The following are known issues and TODO items for pbr-nftables:
- When the plugins for fw4 installed during the
pbr-nftablesinstall, they are not activated untilfw4 reloadis ran. -
pbr-nftablesdoesn't support TOR as a tunnel. -
pbr-nftablesdoesn't have a sensible output forstatuscommand, it just shows the whole fw4 table.
I just upgraded from rc6 to 22.03.0.
My pbr stopped working. I am running pbr-nftables and luci-app-pbr. Both 0.9.7-4
I did reload fw4 command as you suggested.
Whatever I set my policy interface to (wireguard or other vpn) I either stay on wan or get nothing. I've reset the config file to stock, but that didn't help.
I don't do policies based on domain. Just entire mac to interface (wan/wg0/nordvpn).
cat /etc/config/dhcp
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option localservice '1'
list server '8.8.4.4'
list server '8.8.8.8'
option confdir '/tmp/dnsmasq.d'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option quietdhcp '1'
config dhcp 'lan'
option interface 'lan'
option start '100'
option limit '150'
option dhcpv4 'server'
option dhcpv6 'server'
option ra 'server'
list ra_flags 'managed-config'
list ra_flags 'other-config'
option leasetime '12h'
config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'
cat /etc/config/firewall
config defaults
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
option network 'lan'
config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
option network 'wan wan6'
config forwarding
option src 'lan'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option src_ip 'fc00::/6'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config rule
option src 'lan'
list src_mac 'CC:50:E3:39:14:C3'
option target 'REJECT'
option dest '*'
option name 'Truus-Outbound'
list proto 'all'
config include
option path '/etc/firewall.user'
config zone
option name 'vpnfirewall'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
option network 'nordvpntun'
config forwarding
option src 'lan'
option dest 'vpnfirewall'
config rule
option dest 'wan'
option target 'DROP'
option src 'lan'
list src_mac '54:EF:44:C8:F8:CE'
list src_mac '54:EF:44:C8:FA:CC'
option name 'Aqara-WAN-Outbound'
list proto 'all'
config rule
option src 'lan'
option target 'DROP'
list src_mac '54:EF:44:C8:F8:CE'
list src_mac '54:EF:44:C8:FA:CC'
option name 'Aqara-VPN-Outbound'
list proto 'all'
option dest 'OVPN'
config rule
option name 'Aqara-WAN-Inbound'
option src 'wan'
option dest 'lan'
list dest_ip '10.1.1.147'
list dest_ip '10.1.1.243'
option target 'DROP'
config rule
option name 'Aqara-VPN-Inbound'
option dest 'lan'
list dest_ip '10.1.1.147'
list dest_ip '10.1.1.243'
option target 'DROP'
option src 'OVPN'
config rule
option dest 'wan'
option src 'lan'
list src_mac 'D0:50:99:1A:D2:E1'
option target 'REJECT'
option name 'kodibuntu-WAN-Outbound'
list proto 'all'
config rule
option src 'wan'
option dest 'lan'
list dest_ip '10.1.1.178'
option target 'DROP'
option name 'kodibuntu-WAN-Inbound'
config rule
option name 'EpsonPrinter'
option src 'lan'
option dest 'wan'
option target 'DROP'
list proto 'all'
list src_mac '9C:AE:D3:97:7B:B2'
config zone
option name 'OVPN'
option output 'ACCEPT'
option forward 'REJECT'
list network 'wg0'
option input 'REJECT'
option masq '1'
option mtu_fix '1'
config forwarding
option src 'lan'
option dest 'OVPN'
config include 'miniupnpd'
option type 'script'
option path '/usr/share/miniupnpd/firewall.include'
config include 'bcp38'
option type 'script'
option path '/usr/lib/bcp38/run.sh'
option family 'IPv4'
option reload '1'
config include 'pbr'
option type 'script'
option path '/usr/share/pbr/pbr.firewall.include'
cat /etc/config/network
config interface 'loopback'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option device 'lo'
config globals 'globals'
option packet_steering '1'
config interface 'lan'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option ipaddr '10.1.1.1'
option delegate '0'
list dns '8.8.8.8'
list dns '8.8.4.4'
option igmp_snooping '1'
option device 'br-lan'
config interface 'wan'
option proto 'dhcp'
option peerdns '0'
list dns '8.8.8.8'
list dns '8.8.4.4'
option device 'eth0.100'
config interface 'wan6'
option proto 'dhcpv6'
option device 'eth0.100'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '1 2 3 4 6t'
option vid '1'
config switch_vlan
option device 'switch0'
option vlan '2'
option vid '100'
option ports '0t 5t'
config interface 'nordvpntun'
option proto 'none'
option device 'tun0'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth1.1'
config interface 'wg0'
option proto 'wireguard'
option force_link '1'
option private_key 'keyblabla='
list addresses '10.5.0.2'
option defaultroute '0'
config wireguard_wg0
option route_allowed_ips '1'
option persistent_keepalive '0'
option public_key 'keyblabla='
option description 'nl986.nordvpn.com'
list allowed_ips '0.0.0.0/0'
option endpoint_host 'nl986.nordvpn.com'
cat /etc/config/pbr
config pbr 'config'
option verbosity '2'
option ipv6_enabled '0'
list ignored_interface 'vpnserver'
list ignored_interface 'wgserver'
option boot_timeout '30'
option rule_create_option 'add'
option procd_reload_delay '1'
option webui_enable_column '1'
option webui_protocol_column '1'
option webui_chain_column '1'
option webui_show_ignore_target '1'
option webui_sorting '1'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list webui_supported_protocol 'all'
option resolver_set 'none'
list supported_interface 'wg0'
list supported_interface 'nordvpntun'
option strict_enforcement '1'
option enabled '0'
config include
option path '/usr/share/pbr/pbr.user.aws'
option enabled '0'
config include
option path '/usr/share/pbr/pbr.user.netflix'
option enabled '0'
config policy
option name 'StokbookG2'
option src_addr 'D0:57:7B:9B:FC:AD'
option interface 'wan' (yes, this shows WAN now, wg0, nordvpntun dont work)
/etc/init.d/pbr support
Does not work
/etc/init.d/pbr reload
Activating Traffic Killswitch [ā]
Setting up routing for 'wan/eth0.100/136.143.112.1' [ā]
Setting up routing for 'nordvpntun/tun0/10.7.0.3' [ā]
Setting up routing for 'wg0/10.5.0.2' [ā]
Routing 'StokbookG2' via wan [ā]
Deactivating Traffic Killswitch [ā]
pbr 0.9.7-4 monitoring interfaces: wan nordvpntun wg0
pbr 0.9.7-4 started with gateways:
wan/eth0.100/136.143.112.1 [ā]
nordvpntun/tun0/10.7.0.3
wg0/10.5.0.2
Version 0.9.7-5 shows the nft-relevant output on service pbr status, can you install that and post what it outputs?
During install of pbr-nftables on root from 0.9.7-4 to 0.9.7-5... I get:
Command failed: ubus call service delete { "name": "pbr" } (Not found)
Command failed: ubus call service delete { "name": "pbr" } (Not found)
Collected errors:
* resolve_conffiles: Existing conffile /etc/config/pbr is different from the conffile in the new package. The new conffile will be placed at /etc/config/pbr-opkg.
After installing running service pbr status
============================================================
pbr chains - policies
chain pbr_forward {
}
chain pbr_input {
}
chain pbr_output {
}
chain pbr_prerouting {
ether saddr @pbr_wg0_4_src_mac_cfg046ff5 goto pbr_mark_0x030000 comment "StokbookG2"
}
chain pbr_postrouting {
}
============================================================
pbr chains - marking
chain pbr_mark_0x010000 {
counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
return
}
chain pbr_mark_0x020000 {
counter packets 0 bytes 0 meta mark set meta mark & 0xff02ffff | 0x00020000
return
}
chain pbr_mark_0x030000 {
counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000
return
}
============================================================
pbr nft sets
(my initial post of this I had pbr still disabled, so these were empty)
So the nft set pbr_wg0_4_src_mac_cfg046ff5 wasn't created? Is there an output below pbr nft sets?
Is there an output below
pbr nft sets?
No there is not.
nft list table inet fw4 | sed -n "/set pbr_wg0_4_src_mac_cfg046ff5 {/,/}/p"
?
root@OpenWrt:~# nft list table inet fw4 | sed -n "/set pbr_wg0_4_src_mac_cfg046ff5 {/,/}/p"
set pbr_wg0_4_src_mac_cfg046ff5 {
type ether_addr
flags interval
comment "StokbookG2: D0:57:7B:9B:FC:AD"
elements = { d0:57:7b:9b:fc:ad }
So the set is created (status just wasn't showing it), and the correct nft rule is added to match the set and mark the packets, I don't know why it's not working as intended.
I appreciate you testing and your prompt replies here, maybe switch to iptables (or test with the local IP address instead of MAC) for now.
I did some digging into the whole iptables nftables mess. My build didnt have iptables-nft installed.
It's working now.
See also: Packages: Keep iptables support when adding nftables - #2 by bluewavenet
/edit
still seeinf some odd stuff in the status ( Service Errors
nft add rule inet fw4 pbr_prerouting ip saddr @pbr_wan_4_src_ip_cfg046ff5 goto pbr_mark_0x010000 comment "laptop")
and after reloading fw4 it's (pbr) broken again