PBR with MulladVPN

Hi,

I'm new to OpenWRT, so first of all, I would like to say hello to everyone. I'm not new to - we can say - routing. Maybe I'm not an expert, but 8 years ago, I had a CCNA certificate. But time flies and everything has changed.

I've decided to start using VPN to enhance my security and privacy at home. However, my real problem is that some government services provided via the Internet are not accessible through the Mullvad VPN server in the country I've selected.

Additionally, there are some Internet websites that I would like to access via WAN.

I read an article about PBR. I have configured OpenVPN and PBR. But finally, all traffic goes through VPN.

Where should I find a solution? What kind of logs can the whole community use to resolve my problem?

uci export network; uci export dhcp; ls -l /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/* ; head -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv.*/*

is

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdc3:82cb:4d29::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config device
        option name 'wan'
        option macaddr '8C:DE:F9:92:E1:A4'

config device
        option name 'phy0-ap0'

config interface 'MULLAD_VPN'
        option proto 'none'
        option device 'tun0'

package dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        list dhcp_option '6,10.8.0.1,193.138.218.74'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config host
        option name 'manwe-storage'
        option dns '1'
        option mac '00:11:32:48:74:E7'
        option ip '192.168.1.100'

config host
        option name 'manwe-storage-lan2'
        option dns '1'
        option mac '00:11:32:48:74:E8'
        option ip '192.168.1.101'

lrwxrwxrwx    1 root     root            16 May 16 20:54 /etc/resolv.conf -> /tmp/resolv.conf
-rw-r--r--    1 root     root            47 May 18 14:07 /tmp/resolv.conf
-rw-r--r--    1 root     root            84 May 18 13:13 /tmp/resolv.conf.d/resolv.conf.auto

/tmp/resolv.conf.d:
-rw-r--r--    1 root     root            84 May 18 13:13 resolv.conf.auto
==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf.d <==
head: /tmp/resolv.conf.d: I/O error

==> /tmp/resolv.conf.d/resolv.conf.auto <==
# Interface wan
nameserver 212.75.96.2
nameserver 212.75.122.2
search supermedia.pl

root@OpenWrt:~# /etc/init.d/pbr restart
Activating traffic killswitch [βœ“]
Removing routing for 'wan/212.180.183.1/::/0' [βœ“]
Removing routing for 'MULLAD_VPN/tun0/10.10.0.12/fdda:d0d0:cafe:1196::100a/64' [βœ“]
Deactivating traffic killswitch [βœ“]
pbr 1.1.1-7 (nft) stopped [βœ“]
Activating traffic killswitch [βœ“]
Setting up routing for 'wan/212.180.183.1/::/0' [βœ“]
Setting up routing for 'MULLAD_VPN/tun0/10.10.0.12/fdda:d0d0:cafe:1196::100a/64' [βœ“]
Routing 'my ip' via wan [βœ“]
Routing 'gmail by wan' via wan [βœ“]
Routing 'speedtest to vpn' via wan [βœ“]
Routing 'Plex/Emby Remote Servers' via wan [βœ“]
Restarting dnsmasq [βœ“]
Deactivating traffic killswitch [βœ“]
pbr 1.1.1-7 monitoring interfaces: wan MULLAD_VPN
pbr 1.1.1-7 (nft) started with gateways:
wan/212.180.183.1/::/0
MULLAD_VPN/tun0/10.10.0.12/fdda:d0d0:cafe:1196::100a/64 [βœ“]
root@OpenWrt:~# cat /etc/config/pbr

config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        option ipv6_enabled '1'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'my ip'
        option dest_addr 'whatismyipaddress.com'
        option interface 'wan'

config policy
        option name 'gmail by wan'
        option dest_addr 'mail.google.com'
        option interface 'wan'

config policy
        option name 'speedtest to vpn'
        option dest_addr 'speedtest.pl'
        option interface 'wan'

config policy
        option name 'Plex/Emby Local Server'
        option interface 'wan'
        option src_port '8096 8920 32400'
        option enabled '0'

config policy
        option name 'Plex/Emby Remote Servers'
        option interface 'wan'
        option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'

I woud like to add that when i add policy (using luci) based on source ip address (i.e. my laptop) everything works fine and traffic goes thru WAN

whatismyip.com in pbr should go by wan using domain name
ipaddress.my in pbr is should go by wan using ip adress

whatismyipaddress.com != whatismyip.com

But also check the pbr README, specifically Getting Help section.

/etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '0'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option noresolv '0'
        option port '54'
        list server '192.168.1.1'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        list dhcp_option '6,192.168.1.1'
        list dhcp_option '3,192.168.1.1'
        list dns 'fdc3:82cb:4d29::1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config host
        option name 'manwe-storage'
        option dns '1'
        option mac '00:11:32:48:74:E7'
        option ip '192.168.1.100'

config host
        option name 'manwe-storage-lan2'
        option dns '1'
        option mac '00:11:32:48:74:E8'
        option ip '192.168.1.101'

config host
        option ip '192.168.1.200'
        option mac 'D4:5E:EC:B0:4E:CA'

/etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'
        list network 'MULLAD_VPN'
        list network 'MVPN'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'
        option enabled '0'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'
        option enabled '0'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'
        option enabled '0'

config forwarding
        option src 'lan'
        option dest 'wan'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/pbr.firewall.include'

/etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdc3:82cb:4d29::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'
        option peerdns '0'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config device
        option name 'wan'
        option macaddr '8C:DE:F9:92:E1:A4'

config device
        option name 'phy0-ap0'

config interface 'MVPN'
        option proto 'none'
        option device 'tun0'

/etc/config/vpn-policy-routing

cat: can't open '/etc/config/vpn-policy-routing': No such file or directory

/etc/init.d/vpn-policy-routing support

-ash: /etc/init.d/vpn-policy-routing: not found

/etc/init.d/vpn-policy-routing reload

-ash: /etc/init.d/vpn-policy-routing: not found
BusyBox v1.36.0 (2023-05-16 20:54:29 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r22900-abec62a542
 -----------------------------------------------------

Yes, you are right. I meant whatismyipaddress.com whole the time.

After night, whatismyipaddress.com show my WAN IP address, other ip checker shows VPN IP adress

routing to 146.59.70.61 should catch traffic to online stream but it doesn't work
routing from 192.168.1.200 is device from which i would like to have exit via WAN. That works expect that online stream should go through MVPN (first policy)

vpn-policy-routing is deprecated

ok, but it is in getting help

A common issue with many OpenVPN providers (not sure if this applies to Mullvad) is that they, instead of overriding the default route in their configs, provide two routes, one for 0.0.0.0/1 and the other for 128.0.0.0/1. Policy-based routing requires a traditional default route.

Watch for the redirect-gateway def1 option, and remove it. If it is pushed, then add route-nopull. Replace it with a manually added default route.

Also, OpenVPN has a habit of deleting the old default gateway and replacing it with its own. I think that any redirect-gateway option, not only def1, is incompatible with PBR.

1 Like

route-nopull stops pulling all routes, if you only want to stop the redirect gateway then consider using:
pull-filter ignore "redirect-gateway"

But I am not sure if of any of this is necessary for PBR, at least when using WireGuard with PBR you can leave the default route for WireGuard in place.

In my setup (using (Snapshot) PBR just makes two extra tables one routing via WireGuard and the other via WAN.

1 Like

The difference is who installs the default route. With WireGuard, it's netifd. With OpenVPN, it's the openvpn daemon.

True, so it might be different for OpenVPN vs WireGuard.

After adding route-nopull to the OpenVPN client, the only effect was that, on Luci-PBR, I could see that the WAN is the default gateway. Now, unfortunately, I can't select the MVPN interface in the policies. This problem persists even if I remove route-nopull and reboot the router.

So now, the situation is worse than before :sweat_smile:

Curious as I am I did a quick test and indeed it looks like the PBR application copies 0.0.0.0/1 and 128.0.0.0/1, which is added by the OpenVPN client, to the pbr_wan table.
So this table should route via the WAN but it actually does not :frowning:

As you already remarked, as a stop-gap add pull-filter ignore "redirect-gateway def1" in the openvpn config.
But then for my OpenVPN provider (Proton) I did not have any default route which is expected, but PBR does not make any route either so i also have to add:
redirect-gateway

So in the end I got it working by adding this to the openvpn.conf:

pull-filter ignore "redirect-gateway def1"
redirect-gateway

After that I needed a reboot.

Maybe it works for others maybe not, no guarantee.

Maybe this is something which could be solved in the PBR code?

2 Likes

SysLog after
/etc/init.d/log restart; /etc/init.d/network restart; /etc/init.d/openvpn restart; sleep 10

openvpn.conf file

client
pull-filter ignore "redirect-gateway def1"
redirect-gateway
dev tun
resolv-retry infinite
nobind
persist-key
persist-tun
verb 3
remote-cert-tls server
ping 10
ping-restart 60
sndbuf 524288
rcvbuf 524288
cipher AES-256-GCM
tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384
proto udp
auth-user-pass /etc/openvpn/mullvad_client.auth
reneg-sec 0
tun-ipv6
fast-io
remote-random
remote XXXX YYY
remote XXXX YYY
<ca>
...
</ca>

what is interesting, is luci-pbr information:

but still i can't select MVPN as interface for any policy.

Seen that too sometimes.
Try with a reboot.
If that does not help then it might be the ipv6.

For a different package than what's discussed here.

How this now sort of works for me is that you just choose the things (source, destination etc) you want to go via the WAN.

Everything else is already going via the VPN, so you do not need to choose the VPN, (it should be possible to choose the VPN, I have an idea why it is not but the pbr code (/etc/init.d/pbr ?) is somewhat above my pay grade).

ipchicken.com is now going via my WAN (browse to ipchicken.com to see the IP address)

The problem was that the MVPN wasn't set to "bring up on boot". This was odd because traffic was still going through the VPN :thinking:

Things have improved somewhat, but not completely.

Here are my current policies:
Here are my current policies:

Now, I'm only having issues with domain names. They're not working correctly. In theory, all "my IP" sites should be showing the WAN IP, but they're showing the VPN IP instead. :frowning:

I have AdGuardHome ipset selected as Use resolver set support for domains

Let me explain.

By default, I prefer to use VPN for enhanced security and privacy. However, there are some sites, such as my banks and government sites, which I prefer to access via WAN.

I also have a MiBox which streams Disney+ and Netflix. These services don't work over VPN due to geolocation restrictions. I don't mind using them over WAN, but I also have TV Mate on the MiBox, which I prefer to use over VPN.

To address this, I've implemented Policy-Based Routing (PBR) to:

  1. Set policies to route traffic to my banks using domain names via WAN.
  2. Set a policy for TV Mate traffic (cdnx1.plus.tv) to go via VPN.
  3. Create a policy for the MiBox (192.168.1.200) to go via WAN. A secondary policy is a special case of the third one.

Finally, I have a work notebook that must go through the WAN.

Thanks to:

I managed to resolve the routing problem.

The issue I have now is with domain name resolution to IP addresses. Single domain sites such as whatismyipaddress.com are not resolving correctly because when I check my IP on these sites, they show the VPN IP instead of the WAN IP.

Try with Use resolver set support for domains : Disabled

That is working for me (provided DNS is working normally for you):

root@DL-WRX36:~# nft list table inet fw4
table inet fw4 {
        set pbr_wan_4_dst_ip_cfg066ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "ipchicken"
                elements = { 104.26.8.109 counter packets 0 bytes 0, 104.26.9.109 counter packets 0 bytes 0,
                             172.67.73.20 counter packets 0 bytes 0 }
        }

P.S. I hacked my way around /etc/init.d/pbr, line 1754 seems what is adding the 0.0.0.0/1 and 128.0.0.0/1 to all routing tables, I adapted it so that it no longer does that, it then works with the default Mullvad (and most other providers) which use redirect-gateway def1.

But I am sure that I am overlooking something and there must be an easy solution for this.
upside I learned a lot today :slight_smile:

P.S. It also seems that pushed DNS servers are not used by default, probably have to make up and down scripts for that?

it doesn't work for me. I think it's because i use AdGuardHome as DNS provider.

problably i found an issue:

||Linia 586: Fri May 19 20:19:19 2023 daemon.err AdGuardHome[5730]: 2023/05/19 20:19:19.275074 [error] ipset: adding host ips: adding mail.google.com[142.251.36.69] to ipset pbr_wan_4_dst_ip_cfg096ff5: netfilter query: netlink receive: no such file or directory|
|---|---|
||Linia 601: Fri May 19 20:19:24 2023 daemon.err AdGuardHome[5730]: 2023/05/19 20:19:24.675642 [error] ipset: adding host ips: adding api.whatismyip.com[34.98.116.180] to ipset pbr_wan_4_dst_ip_cfg086ff5: netfilter query: netlink receive: no such file or directory|

First of all, I've ditched the Use resolver set support for domains from AdGuardHome ipset to avoid having iptables and nftables at once. This might create some issues.

The real fun starts when in Use resolver set support for domains I set Disabled. AdGuardHome won't start because there is an error:
Fri May 19 21:57:41 2023 daemon.err AdGuardHome[16505]: 2023/05/19 21:57:41.068053 [fatal] dnsServer.Prepare: preparing ipset settings: open /var/run/pbr.adguardhome.ipsets: no such file or directory

The problem is solved by:

touch /var/run/pbr.adguardhome.ipsets
chmod 777 /var/run/pbr.adguardhome.ipsets
service adguardhome restart

Probably the permissions for the file are too big and this is dangerous but it works :wink: I don't know what creates the /var/run/pbr.adguardhome.ipsets file, but putting AdGuardHome ipset in luci-pbr as Use resolver set support for domains apparently causes this file to be created and there is no problem with it (but in return there are iptables and nftables).

Anyway, Disabled still won't work, because there is another error Router DNS issue with Adguard installed. The workaround for this is to add a line to /etc/resolv.conf nameserver 1.1.1.1. The comment Router DNS issue with Adguard installed - #12 by mercygroundabyss says it's the doing of peerdns '0'. But not in my case. I have to add nameserver 1.1.1.1 to /etc/resolv.conf. It's possible that AdGuardHome changes this during installation, but in my case, I definitely changed it intentionally. Anyway, this is not the cause, because changing to 1 does not fix the situation. I need to add the nameserver.

Both of the above solutions are not reboot resistant :wink:

After the above commands a pbr restart /etc/init.d/pbr restart helps:

Activating traffic killswitch [βœ“]
Removing routing for 'wan/X.X.X,X' [βœ“]
Removing routing for 'MVPN/tun0/10.10.0.2' [βœ“]
Deactivating traffic killswitch [βœ“]
pbr 1.1.1-7 (nft) stopped [βœ“]
Activating traffic killswitch [βœ“]
Setting up routing for 'wan/X.X.X.X' [βœ“]
Setting up routing for 'MVPN/tun0/10.10.0.2' [βœ“]
Routing 'tv' via MVPN [βœ“]
Routing 'tv1' via MVPN [βœ“]
Routing 'mibox' via wan [βœ“]
Routing 'my ip' via wan [βœ“]
Routing 'my ip 2' via wan [βœ“]
Routing 'gmail' via wan [βœ“]
Deactivating traffic killswitch [βœ“]
pbr 1.1.1-7 monitoring interfaces: wan MVPN
pbr 1.1.1-7 (nft) started with gateways:
wan/X.X.X.X
MVPN/tun0/10.10.0.2 [βœ“]

(I replaced the WAN IP with X.X.X.X)

and we're home. Almost.

nft list table inet fw4: (in fact, tv1 could be removed as it's redundant).

table inet fw4 {
        set pbr_MVPN_4_dst_ip_cfg046ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "tv"
                elements = { X.Y.Z.W counter packets 0 bytes 0 }
        }

        set pbr_MVPN_4_dst_ip_cfg056ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "tv1"
                elements = { X.Y.Z.W counter packets 0 bytes 0 }
        }

        set pbr_wan_4_src_ip_cfg066ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "mibox"
                elements = { 192.168.1.200 counter packets 0 bytes 0 }
        }

        set pbr_wan_4_dst_ip_cfg076ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "my ip"
                elements = { 104.16.154.36 counter packets 0 bytes 0, 104.16.155.36 counter packets 0 bytes 0 }
        }

        set pbr_wan_4_dst_ip_cfg086ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "my ip 2"
                elements = { 104.21.89.158 counter packets 0 bytes 0, 172.67.189.152 counter packets 0 bytes 0 }
        }

        set pbr_wan_4_dst_ip_cfg096ff5 {
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "gmail"
                elements = { 142.251.36.133 counter packets 0 bytes 0 }
        }

pbr_wan_4_dst_ip_cfg086ff5 refers to the website https://www.whatismyip.com/ (its IP is 104.21.89.158). However, there I see that it shows me the IP from the VPN. Probably some underlying scripts are asking from a different address. I added it just for testing anyway.

Now it's time for streaming tests from my.stream.tv pbr_MVPN_4_dst_ip_cfg046ff5. Here is the tricky part. By default, it should go via VPN on the computer. mibox, because of the aforementioned Netflix and Disney+, got the policy to go through the WAN, although above this policy there is a more detailed one regarding my.stream.tv, which should be first.

So, from a computer that is not a mibox, the traffic goes through I don't know what, because in luci Status > RealTime Graph > Traffic both interfaces tun0 and wan are loaded equally:

It's possible that it's counting that it's going through tun0 but by physical interface, so it counts on both?

Now, Netflix from mibox:

(the peak at the beginning is the start of mibox. You can see that it went through WAN)

And streaming from mibox (I marked the start with a red line):

Here, too, you can see that traffic is being counted on two interfaces.

In summary:

  1. iptables and nftables are mixing up. I'm not such a heavy user (actually, I'm dealing with OpenWRT for the third night and configuring it because I take care of the kids during the day) to handle both well.
  2. I have a workaround for the initial problems.

/etc/rc.local with fixes looks:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

echo 'FIX AdGuardHome start.'
touch /var/run/pbr.adguardhome.ipsets
chmod 777 /var/run/pbr.adguardhome.ipsets
service adguardhome restart

echo 'FIX router DNS problem and restart pbr'
echo 'nameserver 1.1.1.1' >> /etc/resolv.conf
/etc/init.d/pbr restart

exit 0