First of all, I've ditched the Use resolver set support for domains from AdGuardHome ipset to avoid having iptables and nftables at once. This might create some issues.
The real fun starts when in Use resolver set support for domains I set Disabled. AdGuardHome won't start because there is an error:
Fri May 19 21:57:41 2023 daemon.err AdGuardHome[16505]: 2023/05/19 21:57:41.068053 [fatal] dnsServer.Prepare: preparing ipset settings: open /var/run/pbr.adguardhome.ipsets: no such file or directory
The problem is solved by:
touch /var/run/pbr.adguardhome.ipsets
chmod 777 /var/run/pbr.adguardhome.ipsets
service adguardhome restart
Probably the permissions for the file are too big and this is dangerous but it works 
I don't know what creates the /var/run/pbr.adguardhome.ipsets file, but putting AdGuardHome ipset in luci-pbr as Use resolver set support for domains apparently causes this file to be created and there is no problem with it (but in return there are iptables and nftables).
Anyway, Disabled still won't work, because there is another error Router DNS issue with Adguard installed. The workaround for this is to add a line to /etc/resolv.conf nameserver 1.1.1.1. The comment Router DNS issue with Adguard installed - #12 by mercygroundabyss says it's the doing of peerdns '0'. But not in my case. I have to add nameserver 1.1.1.1 to /etc/resolv.conf. It's possible that AdGuardHome changes this during installation, but in my case, I definitely changed it intentionally. Anyway, this is not the cause, because changing to 1 does not fix the situation. I need to add the nameserver.
Both of the above solutions are not reboot resistant
After the above commands a pbr restart /etc/init.d/pbr restart helps:
Activating traffic killswitch [✓]
Removing routing for 'wan/X.X.X,X' [✓]
Removing routing for 'MVPN/tun0/10.10.0.2' [✓]
Deactivating traffic killswitch [✓]
pbr 1.1.1-7 (nft) stopped [✓]
Activating traffic killswitch [✓]
Setting up routing for 'wan/X.X.X.X' [✓]
Setting up routing for 'MVPN/tun0/10.10.0.2' [✓]
Routing 'tv' via MVPN [✓]
Routing 'tv1' via MVPN [✓]
Routing 'mibox' via wan [✓]
Routing 'my ip' via wan [✓]
Routing 'my ip 2' via wan [✓]
Routing 'gmail' via wan [✓]
Deactivating traffic killswitch [✓]
pbr 1.1.1-7 monitoring interfaces: wan MVPN
pbr 1.1.1-7 (nft) started with gateways:
wan/X.X.X.X
MVPN/tun0/10.10.0.2 [✓]
(I replaced the WAN IP with X.X.X.X)
and we're home. Almost.
nft list table inet fw4: (in fact, tv1 could be removed as it's redundant).
table inet fw4 {
set pbr_MVPN_4_dst_ip_cfg046ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "tv"
elements = { X.Y.Z.W counter packets 0 bytes 0 }
}
set pbr_MVPN_4_dst_ip_cfg056ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "tv1"
elements = { X.Y.Z.W counter packets 0 bytes 0 }
}
set pbr_wan_4_src_ip_cfg066ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "mibox"
elements = { 192.168.1.200 counter packets 0 bytes 0 }
}
set pbr_wan_4_dst_ip_cfg076ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "my ip"
elements = { 104.16.154.36 counter packets 0 bytes 0, 104.16.155.36 counter packets 0 bytes 0 }
}
set pbr_wan_4_dst_ip_cfg086ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "my ip 2"
elements = { 104.21.89.158 counter packets 0 bytes 0, 172.67.189.152 counter packets 0 bytes 0 }
}
set pbr_wan_4_dst_ip_cfg096ff5 {
type ipv4_addr
flags interval
counter
auto-merge
comment "gmail"
elements = { 142.251.36.133 counter packets 0 bytes 0 }
}
pbr_wan_4_dst_ip_cfg086ff5 refers to the website https://www.whatismyip.com/ (its IP is 104.21.89.158). However, there I see that it shows me the IP from the VPN. Probably some underlying scripts are asking from a different address. I added it just for testing anyway.
Now it's time for streaming tests from my.stream.tv pbr_MVPN_4_dst_ip_cfg046ff5. Here is the tricky part. By default, it should go via VPN on the computer. mibox, because of the aforementioned Netflix and Disney+, got the policy to go through the WAN, although above this policy there is a more detailed one regarding my.stream.tv, which should be first.
So, from a computer that is not a mibox, the traffic goes through I don't know what, because in luci Status > RealTime Graph > Traffic both interfaces tun0 and wan are loaded equally:
It's possible that it's counting that it's going through tun0 but by physical interface, so it counts on both?
Now, Netflix from mibox:
(the peak at the beginning is the start of mibox. You can see that it went through WAN)
And streaming from mibox (I marked the start with a red line):
Here, too, you can see that traffic is being counted on two interfaces.
In summary:
- iptables and nftables are mixing up. I'm not such a heavy user (actually, I'm dealing with OpenWRT for the third night and configuring it because I take care of the kids during the day) to handle both well.
- I have a workaround for the initial problems.
/etc/rc.local with fixes looks:
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
echo 'FIX AdGuardHome start.'
touch /var/run/pbr.adguardhome.ipsets
chmod 777 /var/run/pbr.adguardhome.ipsets
service adguardhome restart
echo 'FIX router DNS problem and restart pbr'
echo 'nameserver 1.1.1.1' >> /etc/resolv.conf
/etc/init.d/pbr restart
exit 0