Policy-Based-Routing (pbr) package discussion

fdauti · January 6, 2025, 9:43pm

Check this video mentioned in the documentation: * DevOdyssey

xiaohei · January 7, 2025, 7:27am

Hi, stangri
Are you considering adding a feature to the DNS policy that allows for specifying DNS resolution for particular domains? Similar to the syntax: server=/copilot.microsoft.com/10.98.0.1

Nankey · January 7, 2025, 10:44am

This can be achieved by PBR's Custom User File.

I have a similar requirement: I want dnsmasq to filter out AAAA records for specific domains. To address this, I created a PBR's Custom User File to manipulate the dnsmasq configuration file.

I’ve adjusted my script to suit your need:

You have to manually compose a domain list file with domains you want for particular DNS providers (/usr/share/pbr/domain_list/domain_list.txt). Each domain should be listed on its own line in the domain_list.txt file
The script (/usr/share/pbr/pbr.user.domain_list) will
- read the domain list file,
- remove duplicated domains (if any),
- remove comment line (if any),
- add server=/ to the prefix and /127.0.0.53 to the suffix of each special domain you provided in the domain list file.
- generate a configure file containing server=/copilot.microsoft.com/127.0.0.53 for dnsmasq, and put the file in /tmp/dnsmasq.d/domain_list.dns.conf.
Note that the DNS server 127.0.0.53 is for demonstration purpose. Change 127.0.0.53 in line 26 of /usr/share/pbr/pbr.user.domain_list to the DNS provider of your choice.

/etc/config/dhcp

config dnsmasq
	...
	option confdir '/tmp/dnsmasq.d' 
	...

/etc/config/pbr

config pbr 'config'
	...
	option resolver_set 'dnsmasq.nftset'
	...

config include
	option enabled '1'
	option path '/usr/share/pbr/pbr.user.domain_list'  #or you can put the script in /etc/pbr/

/usr/share/pbr/pbr.user.domain_list

#!/bin/sh
# set -o errexit
# set -o pipefail

TARGET_DL_DOMAIN_LIST="/tmp/domain_list.txt"
TARGET_DOMAIN_LIST_DNS_CONF="/tmp/domain_list.dns.conf"
_ret=0

# If storing domain list online (such as github), comment out lines 14-15 and uncomment lines 10-11.  Remember to change the url.
#url='https://online_address/domain_list.txt'
#data="$(uclient-fetch --no-check-certificate -qO- "$url" 2>/dev/null | grep -v -e '^[[:space:]]*$' -e '^[[:space:]]*#' | sed 's/\r$//' | sed '/^$/d' )"

# If storing domain list locally
path='/usr/share/pbr/domain_list/domain_list.txt'
data="$(cat "$path" 2>/dev/null | grep -v -e '^[[:space:]]*$' -e '^[[:space:]]*#' | sed 's/\r$//' | sed '/^$/d' )"

get_domain_list_data() {
    echo "$data"
}

get_domain_list_data | sort | uniq | sed 's/\r$//' | sed '/^$/d' > "$TARGET_DL_DOMAIN_LIST"


if [ -s "$TARGET_DL_DOMAIN_LIST" ]; then
        # Generate DNS Conf for dnsmasq.  Change 127.0.0.53 to the dns provider of your choice.
        cat "$TARGET_DL_DOMAIN_LIST" | sed 's/^/server=\//' | sed 's/$/\/127.0.0.53/' > "$TARGET_DOMAIN_LIST_DNS_CONF"

        mv "$TARGET_DOMAIN_LIST_DNS_CONF" /tmp/dnsmasq.d/domain_list.dns.conf
	
        # If you want dnsmasq to reload, uncomment next line. 
	    # service dnsmasq restart

        _ret=0
fi

return $_ret

/usr/share/pbr/domain_list/domain_list.txt

# one domain a line, the script will filter out comment line starting with #
domain1.com
sub.domain2.com
# comment line
domain3.com

xiaohei · January 7, 2025, 1:32pm

Thank you for your reply!
I can use a script to write a file in the format: server=/domain/dns

Because I tried to use file:///path/to/my/proxydomain.txt as src_addr in the DNS policy, and dest_dns as 1.1.1.1 , the DNS policy does not work.

egc · January 7, 2025, 1:43pm

@Nankey thanks for the script

@xiaohei, This is not directly related to the PBR DNS policy it is related to dnsmasq, the script adds server=/domain/ip-address-DNS-server to the DNSMasq config.
Hopefully my notes can add some clarification:

xiaohei · January 7, 2025, 2:17pm

Thanks @egc
I am currently using SmartDNS + PBR.

egc · January 7, 2025, 3:19pm

If you are using SmartDNS as a replacment for DNSMasq then you should know that SmartDNS has a comparable server=/ but the syntax is slightly different

xiaohei · January 7, 2025, 4:01pm

I know. smartdns+pbr is already running normally.
ps:

cat /etc/config/pbr

config policy
option name 'proxylist'
option dest_addr 'file:///etc/smartdns/domain-set/selfuse.txt'
option interface 'vpnclient'

config dns_policy
option enabled '1'
option name 'proxydns'
option src_addr 'file:///etc/smartdns/domain-set/selfuse.txt'
option dest_dns '1.1.1.1'

The domain names in selfuse.txtcannot be resolved.

egc · January 7, 2025, 4:14pm

See: https://docs.openwrt.melmac.net/pbr/

For policies the destination can be a ist of space-separated remote/target IP addresses, CIDRs or hostnames/domain names. Versions 1.1.2 and later allow using URLs to list of addresses. If curl is installed you can use the file:// schema, otherwise you can use ftp://, http:// and https:// schemas. This is obviously not compatible with the secure_reload option.

For DNS policies the source address is a source on the LAN e.g IP address of LAN client or MAC address or LAN subnet

xiaohei · January 7, 2025, 4:32pm

Got it,thanks.

elgatito · January 9, 2025, 1:08pm

Hello,

Asking for a support!

I had a 23.05 based router, that had wireguard interface and all the traffic was going through wg0, except the rules I was setting with pbr.

Few days ago I changed the router and had to re-configure everything, I was taking previous configuration.

I have encountered a situation when nothing was routed outside of my network. And restart of pbr was helping to fix the routing.
I thought that is because of using 24.10rc and so re-installed with stable 23.05, but had same situation few more times.

But this time I was able to see the log before everything stopped working

Before pbr restart / When issue happens

Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /etc/hosts - 12 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 4 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 2 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /etc/hosts - 12 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 4 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 4 names
Thu Jan  9 06:02:36 2025 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Thu Jan  9 06:03:06 2025 daemon.notice netifd: wan (4047): udhcpc: sending renew to server 193.47.158.39
Thu Jan  9 06:03:06 2025 daemon.notice netifd: wan (4047): udhcpc: lease of 89.77.143.159 obtained from 193.47.158.39, lease time 172799
Thu Jan  9 06:03:06 2025 user.notice firewall: Reloading firewall due to ifupdate of wan (eth1)
Thu Jan  9 06:03:06 2025 user.notice pbr [32072]: Reloading routing for 'wan/eth1/89.77.142.1/::/0' [✓]
Thu Jan  9 06:03:06 2025 user.notice pbr: Sending reload signal to pbr due to firewall action: includes
Thu Jan  9 06:03:07 2025 user.notice pbr: Reusing the fw4 nft file.

Then I do restart of pbr and see in the log

After pbr restart

Thu Jan  9 06:08:06 2025 daemon.warn odhcpd[1894]: No default route present, overriding ra_lifetime!
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Removing routing for 'wan/eth1/89.77.142.1/::/0' [✓]
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Removing routing for 'wg0/10.65.106.171/fc00:bbbb:bbbb:bb01::2:6aaa/128' [✓]
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: service (fw4 nft file mode) stopped [✓]
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Using wan interface (on_start): wan
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Found wan gateway (on_start): 89.77.142.1
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Using wan6 interface (on_start): wan6
Thu Jan  9 06:08:10 2025 user.notice pbr [728]: Setting up routing for 'wan/eth1/89.77.142.1/::/0' [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Setting up routing for 'wg0/10.65.106.171/fc00:bbbb:bbbb:bb01::2:6aaa/128' [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-remote' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-local' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'dns-google' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'dns-cloud' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-4a' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-4a-5g' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-mipad' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'direct-citrix' via wan [✓]
Thu Jan  9 06:08:11 2025 daemon.warn odhcpd[1894]: No default route present, overriding ra_lifetime!
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'phone-4a' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'phone-4a-5g' via wan [✓]
Thu Jan  9 06:08:11 2025 user.notice pbr [728]: Routing 'k8-route' via wan [✓]
Thu Jan  9 06:08:12 2025 user.notice pbr [728]: Routing '845-eth-test' via wan [✓]
Thu Jan  9 06:08:12 2025 user.notice pbr [728]: Routing '845-wifi-test' via wan [✓]
Thu Jan  9 06:08:12 2025 user.notice pbr [728]: Installing fw4 nft file [✓]
Thu Jan  9 06:08:12 2025 user.notice pbr [728]: service monitoring interfaces: wan wg0
Thu Jan  9 06:08:12 2025 user.notice pbr [728]: service (fw4 nft file mode) started with gateways: wan/eth1/89.77.142.1/::/0 wg0/10.65.106.171/fc00:bbbb:bbbb:bb01::2:6aaa/128 [✓]
Thu Jan  9 06:08:13 2025 user.notice pbr: Sending reload signal to pbr due to firewall action: includes
Thu Jan  9 06:08:14 2025 user.notice pbr: Reusing the fw4 nft file.
Thu Jan  9 06:08:14 2025 user.notice SQM: Stopping SQM on eth1
Thu Jan  9 06:08:14 2025 user.notice SQM: Starting SQM script: piece_of_cake.qos on eth1, in: 450000 Kbps, out: 25000 Kbps
Thu Jan  9 06:08:15 2025 user.notice SQM: piece_of_cake.qos was started on eth1 successfully

Other times it happened I also see same lines in the log, like pbr is partly restarting but not fully. And on luci it says status is errored but without any information:

Thu Jan  9 06:03:06 2025 daemon.notice netifd: wan (4047): udhcpc: sending renew to server 193.47.158.39
Thu Jan  9 06:03:06 2025 daemon.notice netifd: wan (4047): udhcpc: lease of 89.77.143.159 obtained from 193.47.158.39, lease time 172799
Thu Jan  9 06:03:06 2025 user.notice firewall: Reloading firewall due to ifupdate of wan (eth1)
Thu Jan  9 06:03:06 2025 user.notice pbr [32072]: Reloading routing for 'wan/eth1/89.77.142.1/::/0' [✓]
Thu Jan  9 06:03:06 2025 user.notice pbr: Sending reload signal to pbr due to firewall action: includes
Thu Jan  9 06:03:07 2025 user.notice pbr: Reusing the fw4 nft file.

I don't know where to look for and what can cause it.
Any guidance?

Here is the configuration dump: https://pastebin.com/nqaDdY3A

egc · January 9, 2025, 2:21pm

I took a quick look and the problem is not so much in PBR but in your WireGuard setup and maybe also in DNS

About DNS, you are probably not using DNSMasq as DNS server but something else?

About WireGuard maybe delete the whole WireGuard setup both in Network and in Firewall (you expose your whole network through WireGuard) and start over with WireGuard.

If you want everything routed via WireGuard then enable Route Allowed IPs and make the exceptions with PBR if you want the opposite do not enable Route Allowed IPs.

See: https://openwrt.org/docs/guide-user/services/vpn/wireguard/client
Or my own notes: https://github.com/egc112/OpenWRT-egc-add-on/blob/main/OpenWRT%20Installling%20WireGuard%20client%20with%20Luci.pdf

elgatito · January 9, 2025, 4:13pm

About DNS, you are probably not using DNSMasq as DNS server but something else?

Yes, I am now using AGH listening on port 53, and dnsmasq moved to 54.

If you want everything routed via WireGuard then enable Route Allowed IPs and make the exceptions with PBR if you want the opposite do not enable Route Allowed IPs.

Yes, I want everything to be routed through wg0 and exceptions are defined via PBR. That is why I have allowed_ips set to 0.0.0.0/1.

Any specific suggestions what to change and what is causing the issue?

Here are my current routes:

# ip route
0.0.0.0/1 dev wg0 proto static scope link
default via 89.77.142.1 dev eth1 proto static src 89.77.143.159
89.77.142.0/23 dev eth1 proto kernel scope link src 89.77.143.159
128.0.0.0/1 dev wg0 proto static scope link
185.209.196.76 via 89.77.142.1 dev eth1 proto static
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
193.32.248.70 via 89.77.142.1 dev eth1 proto static

egc · January 9, 2025, 4:50pm

To better answer your question, please make a new thread in the Installing and Using OpenWRT section.

Disable PBR and reboot

Please connect to your OpenWRT device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact keys, passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/dhcp
cat /etc/config/firewall
ip route show
ip route show table all
ip rule show
wg show

elgatito · January 9, 2025, 4:59pm

But the question is why I need to reload PBR, so that it recreate all the routes to get back to life. Or what PBR tries to do on ifupdate that breaks everything (or not able to fix everything). I would even prefer to just restart PBR when that happens. 0.5seconds downtime once a day is fine for me.

fdauti · January 9, 2025, 10:44pm

Is there a limit to the Remote Addresses / Domains added in a policy or something else to it apart from separation with a space. For instance, I have ifconfig.me that shows my VPN IP (the interface according to my policy). Then I also add showmyip.com with a space. Now both website show my WAN IP. I remove showmyip and now ifconfig shows the VPN IP. Or maybe its just that I have to wait for DNS to catch up ... smthg like that

ok , i did a /flushdns and refreshed, and not it show he correct interface IP ... lesson learned

fdauti · January 10, 2025, 1:21am

How can I enforce a specific public DNS server for an interface used with PBR?

For instance, WAN interface is my default service gateway in PBR. TUN0 (openvpn) is the second one. I tried adding 1.1.1.1 to the tun0 interface as custom DNS but it didn't work. I also use "https dns proxy" package, so have set the DNS used by WAN there, but it also takes over the VPN one.

egc · January 10, 2025, 7:05am

Could be related to using an outdated PBR version, latest is 1.1.8-r4

egc · January 10, 2025, 7:06am

See my notes about Split DNS

fdauti · January 10, 2025, 3:28pm

Is it ok to upgrade from 1.1.6 to 1.18 with just "opkg upgrade pbr"?