As a side note, does anyone have any idea why the dnsmasq-full 2.88 with nft set is still not included with the OpenWrt release?
In 1.1.0-5 the error message about invalid ovpn configs demoted to warning, in 1.1.0-6 fixed issue with WG tunnels being incorrectly identified as OpenVPN tunnels if their names start with tun.
@ENKI, please elaborate on your comment.
Just check if it's not necessary to nft add rule at that point.
Further, I'd suggest not validating disabled rules.
1 Like
Hey, thanks for all your hard work on pbr! I've recently tried using it again after updating an old OpenWRT install, and have hit a wall getting it to finish setting up. I have reviewed the README, and most of the posts in the thread. IP's & Names listed have been redacted.
The gateway error only appears when OpenVPN is connected, otherwise my normal WAN interface is set-up correctly.
OpenWRT: 22.03.03
Resolver: dnsmasq-full 2.89
Version: pbr 1.1.0-5
PBR reload
Activating traffic killswitch [✗]
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
Setting up routing for 'WAN/pppoe-WAN/aaa.bbb.ccc.ddd' [✗]
Setting up routing for 'FVPN/tun0/10.8.0.2' [✗]
Routing 'AppleTV' via FVPN [✓]
Routing 'Bedroom FireTV' via FVPN [✓]
Routing 'iPad' via FVPN [✓]
Deactivating traffic killswitch [✓]
pbr 1.1.0-5 monitoring interfaces: FVPN
ERROR: Failed to set up 'WAN/pppoe-WAN/aaa.bbb.ccc.ddd'!
ERROR: Failed to set up 'FVPN/tun0/10.8.0.2'!
ERROR: Failed to set up any gateway!
Status
============================================================
pbr - environment
pbr 1.1.0-5 running on OpenWrt 22.03.3. WAN (IPv4): WAN/pppoe-WAN/aaa.bbb.ccc.ddd.
============================================================
Dnsmasq version 2.89 Copyright (c) 2000-2022 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
chain pbr_forward {
}
chain pbr_input {
}
chain pbr_output {
}
chain pbr_prerouting {
ip saddr @pbr_FVPN_4_src_ip_cfg016ff5 goto pbr_mark_0x020000 comment "AppleTV"
ip saddr @pbr_FVPN_4_src_ip_cfg056ff5 goto pbr_mark_0x020000 comment "Bedroom FireTV"
ip saddr @pbr_FVPN_4_src_ip_cfg066ff5 goto pbr_mark_0x020000 comment "iPad"
}
chain pbr_postrouting {
}
============================================================
pbr chains - marking
chain pbr_mark_0x010000 {
counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000
return
}
chain pbr_mark_0x020000 {
counter packets 462 bytes 104179 meta mark set meta mark & 0xff02ffff | 0x00020000
return
}
============================================================
pbr nft sets
set pbr_FVPN_4_src_ip_cfg016ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "AppleTV"
elements = { aaa.bbb.ccc.ddd counter packets 101 bytes 11052 }
}
set pbr_FVPN_4_src_ip_cfg056ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "Bedroom FireTV"
elements = { aaa.bbb.ccc.ddd counter packets 247 bytes 77244 }
}
set pbr_FVPN_4_src_ip_cfg066ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "iPad"
elements = { aaa.bbb.ccc.ddd counter packets 114 bytes 15883 }
}
============================================================
IPv4 table 256 route: default via aaa.bbb.ccc.ddd dev pppoe-WAN
IPv4 table 256 rule(s):
30000: from all fwmark 0x10000/0xff0000 lookup pbr_WAN
IPv4 table 257 route: default via 10.8.0.2 dev tun0
IPv4 table 257 rule(s):
30001: from all fwmark 0x20000/0xff0000 lookup pbr_FVPN
PBR Config (N.B. Resolver was set to none for testing here, but same behaviour when set to dnsmasq nft set)
config policy
option name 'AppleTV'
option interface 'FVPN'
option src_addr 'aaa.bbb.ccc.ddd'
config pbr 'config'
option verbosity '2'
option src_ipset '0'
option dest_ipset '0'
option ipv6_enabled '0'
list ignored_interface 'vpnserver wgserver'
option boot_timeout '30'
option procd_reload_delay '1'
option webui_protocol_column '0'
option webui_sorting '1'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
list webui_supported_protocol 'all'
option webui_chain_column '1'
option webui_enable_column '1'
option strict_enforcement '0'
option webui_show_ignore_target '0'
option rule_create_option 'add'
option enabled '1'
option resolver_set 'none'
config include
option path '/etc/pbr.netflix.user'
option enabled '0'
config include
option path '/etc/pbr.aws.user'
option enabled '0'
config policy
option name 'Bedroom FireTV'
option interface 'FVPN'
option src_addr 'aaa.bbb.ccc.ddd'
config policy
option interface 'FVPN'
option name 'iPad'
option src_addr 'aaa.bbb.ccc.ddd'
DHCP
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option expandhosts '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option localservice '0'
option nonwildcard '0'
option logdhcp '1'
option logqueries '1'
option logfacility '/tmp/dnsmasq.log'
option authoritative '1'
option confdir '/tmp/dnsmasq.d'
list server '1.1.1.1'
list server '8.8.8.8'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option domain 'domain.tld'
config dhcp 'lan'
option interface 'lan'
option dhcpv6 'server'
option ra 'server'
option ra_management '1'
option ignore '1'
config dhcp 'wan'
option interface 'wan'
option ignore '1'
config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
config dhcp
option interface 'LAN_Default'
option ignore '1'
config dhcp
option leasetime '12h'
option interface 'LAN'
option start '200'
option limit '250'
list dhcp_option '3,aaa.bbb.ccc.ddd'
list dhcp_option '6,aaa.bbb.ccc.ddd,aaa.bbb.ccc.ddd'
Network
config interface 'loopback'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
option device 'lo'
config globals 'globals'
option ula_prefix 'fd22:4j11:636d::/48'
config interface 'WAN'
option proto 'pppoe'
option username 'username@redacted'
option password 'redacted'
option delegate '0'
option _orig_ifname 'eth1'
option _orig_bridge 'false'
option device 'eth1'
option ipv6 'auto'
option peerdns '0'
list dns '1.1.1.1'
list dns '8.8.8.8'
config interface 'LAN'
option proto 'static'
option ipaddr 'aaa.bbb.ccc.ddd'
option netmask '255.255.255.0'
option _orig_ifname 'eth0'
option _orig_bridge 'false'
option device 'eth0'
config interface 'FVPN'
option proto 'none'
option device 'tun0'
option defaultroute '0'
Any pointers here where I'm going wrong?
@ENKI please try 1.1.0-7. I'm not sure if I can skip validation for disabled rules.
@dakaix no idea what's wrong. Given the amount of changes to your system (both lan and LAN and your wan interface being called WAN) I suspect there could be other settings which prevent pbr from configuring interfaces/routes. I'd recommend flashing vanilla OpenWrt and configuring PPPoE and trying then.
How are you generating the traffic?
Found the reason for the output issue:
chain pbr_output {
...
ip daddr @pbr_wan2_4_dst_ip_cfg066ff5 tcp sport 4000 tcp dport 9000 goto pbr_mark_0x030000 comment "INIT_SOMEVPN2_WG"
Protocol was set - correctly - to All. The rule was set only for TCP traffic.
Tested with proto set to UDP and traffic was passed through wan2.
I'd suggest to fix rule creation for proto "All".
Yes, it's still there.
Further, adding rules for tor still doesn't work correctly.
Should be sth like this
nft add rule inet fw4 pbr_prerouting meta nfproto ipv4 udp dport 53 counter redirect to :"9053" comment "Tor-DNS-UDP-ipv4"
Without daddr since it's redirect. Supposed pbr_prerouting.
At the moment I get
Error: Could not process rule: Not supported
The selected prerouting chain must be of type nat.
1 Like
I tried pbr with my openvpn setup and have a couple suggestions
pbr 1.1.0-7 running on OpenWrt 22.03.3.
diff /etc/init.d/pbr /tmp/pbr.ORIG
229c229
< is_ovpn() { local dev; dev="$(uci -q get "network.${1}.device")"; [ "${dev:0:3}" = "tun" ] || [ "${dev:0:3}" = "tap" ] || [ -f "/sys/devices/virtual/net/${dev}/tun_flags" ]; }
---
> is_ovpn() { local dev; dev="$(uci -q get "network.${1}.dev")"; [ "${dev:0:3}" = "tun" ] || [ "${dev:0:3}" = "tap" ] || [ -f "/sys/devices/virtual/net/${dev}/tun_flags" ]; }
1883c1883
< network_get_physdev dev "$iface"
---
> network_get_device dev "$iface"
the first change allows 22.03 to find the openvpn tun interfaces
the second seems to help find the dev for the tunnel more reliably. If I stopped the openvpn using init.d and restart them "network_get_device" didn't return the dev most of the time but network_get_physdev did. This discussion has more details Lib functions network
1 Like
While I'm not planning on backporting pbr to the 21.01 tree, I'd like to maintain general compatibility if either version is installed on 21.01, so I've had to supplement old code with your suggestions.
Please test pbr 1.1.0-9.
The nftsets provide capability to selectively target traffic based on domain name, hence daddr. Please test
Yeah, thanks for bringing it up, the iptables processing had more elaborate logic where missing proto or proto set to auto/all would mean tcp udp and duplicate iptables entries for both protocols, the same logic was implemented for nft in pbr 1.1.0-10.
ip daddr "@pbr_tor_4_dst_ip" udp
nft add rule inet fw4 dstnat meta nfproto ipv4 ip daddr "@pbr_tor_4_dst_ip" udp dport 53 counter redirect to :"9053" comment "Tor-DNS-UDP-ipv4" || s=1
nft add rule inet fw4 dstnat meta nfproto ipv4 ip daddr "@pbr_tor_4_dst_ip" tcp dport 80 counter redirect to :"9040" comment "Tor-HTTP-TCP-ipv4" || s=1
nft add rule inet fw4 dstnat meta nfproto ipv4 ip daddr "@pbr_tor_4_dst_ip" udp dport 80 counter redirect to :"9040" comment "Tor-HTTP-UDP-ipv4" || s=1
nft add rule inet fw4 dstnat meta nfproto ipv4 ip daddr "@pbr_tor_4_dst_ip" tcp dport 443 counter redirect to :"9040" comment "Tor-HTTPS-TCP-ipv4" || s=1
nft add rule inet fw4 dstnat meta nfproto ipv4 ip daddr "@pbr_tor_4_dst_ip" udp dport 443 counter redirect to :"9040" comment "Tor-HTTPS-UDP-ipv4" || s=1
nft6 add rule inet fw4 dstnat meta nfproto ipv6 ip6 daddr "@pbr_tor_6_dst_ip" udp dport 53 counter redirect to :"9053" comment "Tor-DNS-UDP-ipv6" || s=1
nft6 add rule inet fw4 dstnat meta nfproto ipv6 ip6 daddr "@pbr_tor_6_dst_ip" tcp dport 80 counter redirect to :"9040" comment "Tor-HTTP-TCP-ipv6" || s=1
nft6 add rule inet fw4 dstnat meta nfproto ipv6 ip6 daddr "@pbr_tor_6_dst_ip" udp dport 80 counter redirect to :"9040" comment "Tor-HTTP-UDP-ipv6" || s=1
nft6 add rule inet fw4 dstnat meta nfproto ipv6 ip6 daddr "@pbr_tor_6_dst_ip" tcp dport 443 counter redirect to :"9040" comment "Tor-HTTPS-TCP-ipv6" || s=1
nft6 add rule inet fw4 dstnat meta nfproto ipv6 ip6 daddr "@pbr_tor_6_dst_ip" udp dport 443 counter redirect to :"9040" comment "Tor-HTTPS-UDP-ipv6" || s=1
Creating TOR redirects [✓]
...
Routing 'TOR' via tor [✓]
Choose the correct chain of type nat and replace dstnat if necessary.
Works fine! Thank you.
For pbr 1.1.0-11 I've added the code to insert rules for all chains in the $chainsList for nft, like it's done for iptables. Hopefully it'll work.
ip/ip6 before and tcp/udp after daddr "@${set_name4}"
nft add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv4 ip daddr "@${set_name4}" udp dport 53 counter redirect to :"$dnsPort" comment "Tor-DNS-UDP-ipv4" || s=1
nft add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv4 ip daddr "@${set_name4}" tcp dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-TCP-ipv4" || s=1
nft add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv4 ip daddr "@${set_name4}" udp dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-UDP-ipv4" || s=1
nft add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv4 ip daddr "@${set_name4}" tcp dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-TCP-ipv4" || s=1
nft add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv4 ip daddr "@${set_name4}" udp dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-UDP-ipv4" || s=1
nft6 add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv6 ip6 daddr "@${set_name6}" udp dport 53 counter redirect to :"$dnsPort" comment "Tor-DNS-UDP-ipv6" || s=1
nft6 add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv6 ip6 daddr "@${set_name6}" tcp dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-TCP-ipv6" || s=1
nft6 add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv6 ip6 daddr "@${set_name6}" udp dport 80 counter redirect to :"$trafficPort" comment "Tor-HTTP-UDP-ipv6" || s=1
nft6 add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv6 ip6 daddr "@${set_name6}" tcp dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-TCP-ipv6" || s=1
nft6 add rule inet "$nftTable" "${nftPrefix}_${i}" meta nfproto ipv6 ip6 daddr "@${set_name6}" udp dport 443 counter redirect to :"$trafficPort" comment "Tor-HTTPS-UDP-ipv6" || s=1
For the chain selection, you may follow the protocol settings for the rule in pbr config?
Further:
Note that: redirect only makes sense in prerouting and output chains of NAT type.
But most of all, the chain must be of type nat.
I tried
nft add table inet fw4
nft add chain inet fw4 pbr_prerouting { type nat hook prerouting priority -100 \; }
placed at load_environment() or somewhere before the table and chains are usually 'initiated' and the rules have been created when I proceeded only with pbr_prerouting.
For example, I recently replaced iptables based rules with nftables for Wireguard. Table and chains are added at the beginning.
It will fail if one chain is not of type nat. That's why my suggestion is to follow the pbr config for tor based rules (which is by default prerouting).
I've seen, for example, chains are added explicitly for mark and killswitch.
So you may define the chain/s somewhere at the beginning or right before they're usually initiated.
Anyway, the chain type must be defined for nat and then it'll work.
As of version: 1.1.0-12 using nft still getting Invalid OpenVPN config for a wireguard tunnel
Can you post/pm me your network config?
Also, strangely, renaming the OpenVPN configs to be the same as the interface name only got rid of one warning (for vpn_o2):
Invalid OpenVPN config for vpn_o1 interface
Invalid OpenVPN config for vpn_w interface
I'm failing to see the wireguard tunnels defined in that network file.
Just a small update on my 2 Wg interfaces causing some unknown pbr gateway error while using latest openwrt and pbr (March 2023). Bit more detail on it here
I was advised just to stick with 1 WG interface since you can introduce possible more errors so have done just that, with 1 WG interface no such pbr errors and tried several reboots. I have also created backups with a few other VPN servers loaded so can easily restore which ever configs I require if I need to change WG VPN servers more quickly rather then editing the WG interface settings each time.
Perhaps not the best way and no doubt user set up error but since I am new to openwrt, ill keep it simple and error free while everything runs perfect.
My thanks to stangri/community for creating the pbr package, please keep up the good work.