Policy-Based-Routing (pbr) package discussion

Looks good to have encrypted DNS policy too.
Another privacy concerns though at least when trying to use Split Tunneling.
I've setup a policy for several domains (fast.com and speedtest.net among them) to be accessed via VPN.


But when I visit fast.com and speedtest.net firstly I can see the VPN provider network and VPN IP address detected by both domains, something that probably a layman would expect and at this point he/she thinks everything looks OK (no privacy leak issues, etc.).
Strange for me is that running both speed tests I can achieve my full ISP line speeds. Something that I don't expect as it is 1 Gbps. And the test servers are really close to my real location (very strange as the VPN server is in different country).
Additionally (and my big privacy concern) is that on speedtest.net for example if I look at the test details I can see my public (external) ISP IP address detected (What? How come?).
For me at this point it looks as an absolute privacy data leak and should raise a very big ALERT!
I've setup another policy for a torrent port (I don't have nor use UPnP nor NAT-PMP on the router and both are off in torrent client) and running a torrent client on a PC, it initially shows "Detected external IP. IP: "VPN IP" address" in its log but then there are many messages showing my public ISP IP address.
Looking at WireGuard Status page in Luci there are data fields showing Received/Transmitted Data. But they read a few megabytes (tests used over 1G) counted, making me think that almost 99% of the data flows via WAN interface not the VPN WG one as I expect,

simply run a tracert/traceroute from you client to speedtest.net

This is from my client in switzerland connected to my WG server in the Netherlands, you can see that the first hit is the WG subnet

PS C:\Users\egc> tracert speedtest.net

Tracing route to speedtest.net [151.101.130.219]
over a maximum of 30 hops:

  1    28 ms    30 ms    32 ms  172.21.21.1
  2    32 ms    30 ms    29 ms  192.168.0.1
  3     *        *        *     Request timed out.
  4    40 ms    39 ms    39 ms  tb-rc0001-cr101-et89-201.core.as33915.net [213.51.194.49]
  5     *        *        *     Request timed out.

It could be you are leaking IPv6 or that not all IP addresses are resolved of speedtest,net

1 Like

In my case second hit is the WG subnet, first one is the router. I assume it's OK.
From Linux PC with IPv6 disabled

traceroute to speedtest.net (151.101.2.219), 30 hops max, 60 byte packets
 1  OpenWrt.lan (192.168.1.1)  1.403 ms  1.315 ms  1.233 ms
 2  * * 10.10.0.1 (10.10.0.1)  90.491 ms
 3  * * *
 4  * * *
 5  * * et-0-0-18.edge7.Amsterdam1.Level3.net (213.19.194.193)  92.463 ms
 6  * * *

From Windows PC IPv4 & IPv6

C:\Windows\System32>tracert speedtest.net

Tracing route to speedtest.net [2a04:4e42:400::731]
over a maximum of 30 hops:

  1  Destination net unreachable.

Trace complete.

From Windows PC IPv4 & IPv6 directly to speedtest.net IPv4 address.

C:\Windows\System32>tracert 151.101.66.219

Tracing route to 151.101.66.219 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  OpenWrt.lan [192.168.1.1]
  2    90 ms    45 ms    44 ms  10.10.0.1
  3    44 ms    44 ms    44 ms  93-190-138-187.hosted-by-worldstream.net [93.190.138.187]
  4    45 ms    46 ms    44 ms  109.236.95.226
  5     *       46 ms    46 ms  5-2-3.ear4.Amsterdam1.Level3.net [212.72.41.249]
  6    46 ms    46 ms    46 ms  213.19.200.114
  7    47 ms    46 ms    46 ms  151.101.66.219

Trace complete.

It's the same from Android smartphone connected to WLAN (10.10.0.1 is second hit) but exactly on it I can see my external IPv4 address. As if the speedtest.net redirects the client (smartphone) to another server (domain) using WAN route that next detects my external IP.
So isn't this defeating the purpose (mainly privacy) of the VPN Split Tunneling?

It is quite possible that speedtest.net uses other domains to which the client is directed.
See
https://www.netify.ai/resources/domains/speedtest.net

This is also the problem with netflix, a redirection of netflix.com is not sufficient there is a whole bunch of other netflix domains which need to be redirected

1 Like

help. now i can only use resolver_set is none, others doesn't work.

@egc @stangri

zcat was part of my busybox, but do they differ per device? alternatively:

  • gunzip -ck "$1"
  • tar -zxf- "$1", but this would require altering the download line too

Also used jsonfilter to speed up performance, found it being used in the /usr/bin/wireguard_watchdog. So that would be another dependency.

  • I could revert that to awk

:thinking:

1 Like

Yep, most of them are here.
www.netify.ai/resources/speedtest
I've made more tests and for my purpose it seems that currently it is better to use a policy that uses a local address (192.168.1.51 on my PC) to tunnel all the data (not only several domains) through the VPN.

1 Like

Your guess is as good as mine.

Perhaps you can test for it and if it is not present use awk ?

Hmm, isn't there a list of basic commands/tools/packages around? That would be very helpful with checking scripts compatibility.

I took this approach, don't know if its the right one. I picked the devices with the smallest flash size from: https://openwrt.org/toh/views/toh_extended_all

22.03.7 Motorola WE800G Flash=4MB Memory=16MB
openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs.bin

23.05.4 Lantiq XRX200 Flash=8MB Memory=64MB
openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade.bin

searched for the squashfs magic: hsqs

$ grep -abo hsqs *.bin
openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs.bin:1669128:hsqs
openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade.bin:2949120:hsqs

mounted the images:

$ sudo mount -o loop,offset=1669128 openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs.bin openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs
$ sudo mount -o loop,offset=2949120 openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade.bin openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade

then searched for 'jsonfilter' and 'zcat'

$ find openwrt* -iname 'jsonfilter' -o -iname 'zcat' 2>/dev/null
openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs/bin/zcat
openwrt-22.03.7-bcm47xx-legacy-motorola_we800g-squashfs/usr/bin/jsonfilter
openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade/bin/zcat
openwrt-23.05.4-lantiq-xrx200-netgear_dm200-squashfs-sysupgrade/usr/bin/jsonfilter

I think we're good.

EDIT:
@stangri : submitted PR

1 Like

Just an update. Originally my /etc/pbr.custom.user was only setup to ensure inter vlan routing was still working with pbr working as in the past this wasn't the case. Am I right in saying this isn't required any more as when I disable /etc/pbr.custom.user inter vlan routing still works. Despite this, still having the same problems with pbr not running unless VPN gateway is up.

Another update (not sure if this was obvious). Using nft -c -f /var/run/pbr.nft, I found the failure to install the fw4 nft file is caused by the rules which are the rules pushing traffic to the gateway which is down (NORDVPNTUN). Not sure what can be done about this... Also noticed that the issue occurs when Strict enforcement is set to do not enforce.

Please help me.

Tried to setup pbr with tor without success.

Seems nft rules is broken:

# nft -c -f /var/run/pbr.nft
/var/run/pbr.nft:7:41-44: Error: syntax error, unexpected meta
add rule inet fw4 pbr_prerouting dstnat meta nfproto ipv4 ip daddr @pbr_tor_4_dst_ip_cfg096ff5 udp dport 53 counter redirect to :9053 comment 'Tor-DNS-UDP'
                                        ^^^^
/var/run/pbr.nft:8:41-44: Error: syntax error, unexpected meta
add rule inet fw4 pbr_prerouting dstnat meta nfproto ipv4 ip daddr @pbr_tor_4_dst_ip_cfg096ff5 tcp dport 80 counter redirect to :9040 comment 'Tor-HTTP-TCP'
                                        ^^^^
/var/run/pbr.nft:9:41-44: Error: syntax error, unexpected meta
add rule inet fw4 pbr_prerouting dstnat meta nfproto ipv4 ip daddr @pbr_tor_4_dst_ip_cfg096ff5 udp dport 80 counter redirect to :9040 comment 'Tor-HTTP-UDP'
                                        ^^^^
/var/run/pbr.nft:10:41-44: Error: syntax error, unexpected meta
add rule inet fw4 pbr_prerouting dstnat meta nfproto ipv4 ip daddr @pbr_tor_4_dst_ip_cfg096ff5 tcp dport 443 counter redirect to :9040 comment 'Tor-HTTPS-TCP'
                                        ^^^^
/var/run/pbr.nft:11:41-44: Error: syntax error, unexpected meta
add rule inet fw4 pbr_prerouting dstnat meta nfproto ipv4 ip daddr @pbr_tor_4_dst_ip_cfg096ff5 udp dport 443 counter redirect to :9040 comment 'Tor-HTTPS-UDP'
                                        ^^^^
/etc/config/pbr
config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        list resolver_instance '*'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_boot_delay '0'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        option nft_rule_counter '0'
        option nft_set_auto_merge '1'
        option nft_set_counter '0'
        option nft_set_flags_interval '1'
        option nft_set_flags_timeout '0'
        option nft_set_policy 'performance'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.wg_server_and_client'
        option enabled '0'

config dns_policy
        option name 'Redirect Local IP DNS'
        option src_addr '192.168.1.5'
        option dest_dns '1.1.1.1'
        option enabled '0'

config policy
        option name 'Ignore Local Requests'
        option interface 'ignore'
        option dest_addr '10.0.0.0/24 10.0.1.0/24 192.168.100.0/24 192.168.1.0/24'
        option enabled '0'

config policy
        option name 'Plex/Emby Local Server'
        option interface 'wan'
        option src_port '8096 8920 32400'
        option enabled '0'

config policy
        option name 'Plex/Emby Remote Servers'
        option interface 'wan'
        option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
        option enabled '0'

config policy
        option name 'onion'
        option dest_addr 'onion'
        option interface 'tor'
        option proto 'tcp udp'
/etc/tor/custom
VirtualAddrNetworkIPv4 10.192.0.0/10

AutomapHostsOnResolve 1
AutomapHostsSuffixes .onion,.exit

AvoidDiskWrites 1

SocksPort 9050
TransPort 9040
DNSPort 9053

TOR is ok:

# curl -I --proxy socks5h://127.0.0.1:9050 juhanurmihxlp77nkq76byazcldy2hlmovfu2epvl5ankdibsot4csyd.onion
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Sat, 05 Oct 2024 05:20:23 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 4415
Connection: keep-alive
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin

OpenWrt - 23.05.5
pbr - 1.1.6-20
firewall4 - 2023-09-01-598d9fbb-1
dnsmasq-full - 2.90-2

What am I doing wrong?

My guess, remove the dns_policy.

dns_policy is disabled in given example.

Yes the problem seems with redirection of tor ports.
I am not using tor so cannot help much.

If you disable pbr does tor work?

tor works with and without pbr (see curl listing in my message above).
Problem is pbr can't start with rule with tor interface due incorrect nft syntax.

You are not doing anything wrong, there seems to be a bug in that version for tor policies. I'll try to implement a fix and merge changes within 24-48 hours.

2 Likes

I've just compiled a build with latest PR - PBR 1.1.7-21.
I need a little help with IPv6.
My VPN now supports IPv6. I've added all the settings from VPN provider to the wireguard settings - IPv6 ULA ( IPv6: fd::xx:2/128), IPv6 DNS ( IPv6: fd::xx:1), and allowed addresses (::/0) in WG peer settings.
Then I created a policy rule for IPv6 addresses to go through WG VPN. But it still cannot work as I expect.
I suppose my IPv6 rule is not good. As one client (PC) can have multiple IPv6 addresses at the same time I've tried to list all of them in my rule (even added a rule with PD from my ISP - xxxx:85b8:22e:xxa::/64) but still the connection goes through the default WAN gateway.
How is the split tunneling (policy rule) supposed to work with IPv6 as there is no NAT similar to that used with IPv4.
Do I need to have IPv6 ULA-Prefix in OpnWrt Global network options. I removed it few months ago because I didn't use it.

For Wg ipv6 do the following
Enable masq6 on the wg firewall zone (advanced tab)
For allowed ips use 8000::/1 and ::/1 instead of ::0/0
Disable pbr and restart network and test with iplek.net if ipv6 works

For pbr with source routing of individual clients use the mac address of the client much easier

1 Like

Making this change IPv6 works. But ipleak detects my public IPv6 address.
Using mac address in the rule is really simpler but I'm unable to make it work when the WAN is default gateway.
When I set WG to be default gateway everything works as I expect - both IPv4 and IPv6 detected are the ones of the VPN provider and DNS is VPN one too.