DNS requests being NAT'd unexpectedly

OpenWrt 23.05.5... My lan subnet is 10.19.76.0/24 (and the not-so-usual use of a bonded interface between the router and my switch):

/etc/config/network:

config device
        option name 'br-lan'
        option type 'bridge'
        option bridge_empty '1'
        list ports 'bond-LACP'
        list ports 'eth0'
        option ipv6 '0'
        option stp '1'
        option priority '32767'

config interface 'lan'
        option proto 'static'
        list ipaddr '10.19.76.1/24'
        option broadcast '10.19.76.255'
        option delegate '0'
        option device 'br-lan.1'
        list dns '10.19.76.13'

config interface 'LACP'
        option proto 'bonding'
        option netmask '255.255.255.0'
        list slaves 'eth1'
        list slaves 'eth2'
        option all_slaves_active '0'
        option bonding_policy '802.3ad'
        option min_links '0'
        option ad_select 'stable'
        option link_monitoring 'mii'
        option miimon '1000'
        option downdelay '2000'
        option updelay '5000'
        option use_carrier '1'
        option ad_actor_sys_prio '8192'
        option lacp_rate 'fast'
        option delegate '0'
        option xmit_hash_policy 'layer3+4'
        option ipaddr '10.19.75.2'

I have pihole+unbound running on a server at VIP 10.19.76.13 (floats in a HA setup, with 10.19.76.12 and 10.19.76.14 the "real" IPs of the hosts).

I have a firewall rule to redirect DNS requests to the pihole VIP, excluding their range of IPs (as well as the zone config):

/etc/config/firewall:

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        list network 'wg_lan'
        list network 'LACP'

config redirect
        option target 'DNAT'
        option src 'lan'
        option src_dport '53'
        option dest 'lan'
        option dest_ip '10.19.76.13'
        option dest_port '53'
        option name 'Nemo-Gateway: Redirect DNS to PiHole'
        option src_ip '!10.19.76.12/30'

config nat
        option name 'Prevents hardcoded DNS Clients from giving unexpected source error after DNS r>        list proto 'tcp'
        list proto 'udp'
        option src 'lan'
        option dest_ip '10.19.76.13'
        option dest_port '53'
        option target 'MASQUERADE'
        option enabled '0'

and dnsmasq set to listen on port 0 for dns (so, disabled), and dhcp hands out the dns server option in it's leases:

/etc/config/dhcp:
config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d'
        option port '0'
        list addnhosts '/etc/ethers'

config dhcp 'lan'
        option interface 'lan'
        option dhcpv4 'server'
        option force '1'
        option netmask '255.255.255.0'
        option start '200'
        option limit '55'
        option leasetime '6h'
        list dhcp_option '6,10.19.76.13'

With this, I would expect dns requests in pihole to show up as coming from the IP of the client. So running nslookup daringfireball.net from my computer at 10.19.76.80 (respecting the dhcp-advertised dns server):

~# nslookup daringfireball.net
Server:         10.19.76.13
Address:        10.19.76.13#53

Non-authoritative answer:
Name:   daringfireball.net
Address: 104.26.5.133
Name:   daringfireball.net
Address: 172.67.74.128
Name:   daringfireball.net
Address: 104.26.4.133

pihole.log shows:

Oct 23 14:11:55: query[A] daringfireball.net from 10.19.76.80
Oct 23 14:11:55: forwarded daringfireball.net to 127.0.0.1#5335
Oct 23 14:11:55: reply daringfireball.net is 104.26.5.133
Oct 23 14:11:55: reply daringfireball.net is 172.67.74.128
Oct 23 14:11:55: reply daringfireball.net is 104.26.4.133

But with the disabled firewall NAT rule to masquerade the redirected requests made to a different dns server, I would expect the same pihole logs, and the client to throw an unexpected source error, but that's not the case:

~# nslookup daringfireball.net 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   daringfireball.net
Address: 104.26.5.133
Name:   daringfireball.net
Address: 172.67.74.128
Name:   daringfireball.net
Address: 104.26.4.133

pihole.log shows the request originating from the router this time (10.19.76.1):

Oct 23 14:15:13: query[A] daringfireball.net from 10.19.76.1
Oct 23 14:15:13: cached daringfireball.net is 104.26.5.133
Oct 23 14:15:13: cached daringfireball.net is 172.67.74.128
Oct 23 14:15:13: cached daringfireball.net is 104.26.4.133

Likewise, for a non-existant domain that has a record made in pihole, I would expect nslookup to return a NXDOMAIN if I specify the dns server to query, but it does not:

~# nslookup piholetest.example.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53

Name:   piholetest.example.com
Address: 123.123.123.123

and pihole shows:

Oct 23 14:18:00: query[A] piholetest.example.com from 10.19.76.1 
Oct 23 14:18:00: /etc/pihole/custom.list piholetest.example.com is 123.123.123.123

I'm trying to figure out why masquerading is happening. I'm expecting the packet flow to go from client (10.19.76.80) to gateway router (10.19.76.1) asking for 8.8.8.8:53, router forwards (but doesn't NAT) the packet to the local dns server (10.19.76.13), local dns server responds to the client with the requested info, client complains that it didn't get that response from the server it originally queried (8.8.8.8).

Take a careful read in that DNAT name - do you see a revealing pattern in last three letters? You have to make clients connect to pihole, like with DHCP option 6.

Do I want to do a traffic rule instead? Something like:

config rule
        option name 'LAN DNS'
        option src 'lan'
        option dest 'lan'
        list dest_ip '10.19.76.13'
        option dest_port '53'
        option target 'ACCEPT'
        list src_ip '!10.19.76.12/30'
        option enabled '0'

DNAT-ed packets leave router with routers address and NAT state is kept to return response.
Firewall rules obviously have to pass direct 53 tcpudp communication from clients to pihole.

Sorry if I'm being super ignorant. It's that, on my guest network, whch is comparably configured, dns queries are forwarded across zones, and the queries are still logged as the client IP, and not any of the router's interface IPs:

config zone
        option output 'ACCEPT'
        option forward 'REJECT'
        option name '40_Guest'
        list network '40_Guest'
        option input 'ACCEPT'

config forwarding
        option src '40_Guest'
        option dest 'wan'

config rule
        option name 'Guest DNS'
        option src '40_Guest'
        list dest_ip '10.19.76.13'
        option target 'ACCEPT'
        option dest 'lan'
        list proto 'tcp'
        list proto 'udp'
        option dest_port '53'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option src_dport '53'
        option dest_ip '10.19.76.13'
        option dest_port '53'
        option name 'Nemo-Gateway: Redirect Guest DNS to PiHole'
        option src '40_Guest'
        list proto 'tcp'
        list proto 'udp'

config nat
        option name 'Guest DNS'
        option family 'ipv4'
        list proto 'tcp'
        list proto 'udp'
        option src '40_Guest'
        option dest_ip '10.19.76.13'
        option dest_port '53'
        option target 'MASQUERADE'
        option enabled '0'

So from a client at 10.1.40.167 (guest subnet is 10.1.40.0/24):

~# nslookup daringfireball.net 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	daringfireball.net
Address: 104.26.5.133
Name:	daringfireball.net
Address: 172.67.74.128
Name:	daringfireball.net
Address: 104.26.4.133
Oct 23 18:26:21: query[A] daringfireball.net from 10.1.40.167
Oct 23 18:26:21: forwarded daringfireball.net to 127.0.0.1#5335
Oct 23 18:26:21: reply daringfireball.net is 104.26.5.133
Oct 23 18:26:21: reply daringfireball.net is 172.67.74.128
Oct 23 18:26:21: reply daringfireball.net is 104.26.4.133

~# nslookup daringfireball.net        
Server:		10.19.76.13
Address:	10.19.76.13#53

Non-authoritative answer:
Name:	daringfireball.net
Address: 104.26.5.133
Name:	daringfireball.net
Address: 172.67.74.128
Name:	daringfireball.net
Address: 104.26.4.133

Oct 23 18:27:17: query[A] daringfireball.net from 10.1.40.167
Oct 23 18:27:17: cached daringfireball.net is 104.26.5.133
Oct 23 18:27:17: cached daringfireball.net is 172.67.74.128
Oct 23 18:27:17: cached daringfireball.net is 104.26.4.133



~# dig piholetest.example.com @8.8.8.8

; <<>> DiG 9.10.6 <<>> piholetest.example.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23224
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;piholetest.example.com.		IN	A

;; ANSWER SECTION:
piholetest.example.com.	0	IN	A	123.123.123.123

;; Query time: 4 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Oct 23 18:28:38 EDT 2024
;; MSG SIZE  rcvd: 67

Oct 23 18:28:38: query[A] piholetest.example.com from 10.1.40.167
Oct 23 18:28:38: /etc/pihole/custom.list piholetest.example.com is 123.123.123.123

I guess I think my issue is, I expect to be able to force the unexpected source error, but can't. The NAT Rule doesn't seem to matter anymore, where in the past, I believe that the port forward rule alone would redirect the requests made to other DNS servers, and clients would throw the error when the reply comes from pihole. The NAT rule was necessary to keep clients happy, if none the wiser. Now it seems that the port forward is NAT'ing (when it didn't before?) and the NAT Rule is unnecessary.

Add to that, the router is masq'ing when the client and pihole are in the same subnet, but not when the client is in a different subnet. This I don't understand. If anything, I would expect a client's request from a different subnet would show as one of the router's interface IPs, and not the client, but that isn't the case.

It is necessary and if it works, the rule was created one way or another.
To help you investigate further, we will need to see the output of the following commands:

for chain in dstnat dstnat_lan srcnat srcnat_lan; do nft list chain inet fw4 $chain; done

This would require an explicit SNAT-ing/Masquerading rule, which in this case is unnecessary, because when the initiator and responder are on different subnets, the replies are always returned through the router.

The port forward and the NAT rule were created a long while ago, running 22.03 probably (and probably following this page, as I named the rules the same way). At the time, yes, the NAT rule was explicitly necessary to prevent the unexpected source errors. But now, the NAT rule being enabled or disabled doesn't look to matter, the output from nslookup/dig and pihole.log are the same either way.

root@Nemo-Gateway:~# for chain in dstnat dstnat_lan srcnat srcnat_lan; do nft list chain inet fw4 $chain; done
table inet fw4 {
        chain dstnat {
                type nat hook prerouting priority dstnat; policy accept;
                iifname { "wg_lan", "wg_lan2", "br-lan.1", "bond-LACP" } jump dstnat_lan comment "!fw4: Handle lan IPv4/IPv6 dstnat traffic"
                iifname { "eth3", "eth4", "mr8300", "re7001" } jump dstnat_wan comment "!fw4: Handle wan IPv4/IPv6 dstnat traffic"
                iifname "br-lan.40" jump dstnat_40_Guest comment "!fw4: Handle 40_Guest IPv4/IPv6 dstnat traffic"
                jump pbr_dstnat comment "Jump into pbr dstnat chain"
        }
}
table inet fw4 {
        chain dstnat_lan {
                ip saddr != 10.19.76.12/30 tcp dport 53 counter packets 38 bytes 2280 dnat ip to 10.19.76.13:53 comment "!fw4: Nemo-Gateway: Redirect DNS to PiHole"
                ip saddr != 10.19.76.12/30 udp dport 53 counter packets 444 bytes 28566 dnat ip to 10.19.76.13:53 comment "!fw4: Nemo-Gateway: Redirect DNS to PiHole"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } tcp dport 5201 dnat ip to 10.19.76.1:5201 comment "!fw4: Nemo-Gateway: iperf3 from RE7000's (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } udp dport 5201 dnat ip to 10.19.76.1:5201 comment "!fw4: Nemo-Gateway: iperf3 from RE7000's (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } tcp dport 2223 dnat ip to 10.19.76.1:2222 comment "!fw4: Nemo-Gateway: SSH_Ext (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } udp dport 51820 dnat ip to 10.19.76.32:51820 comment "!fw4: Nemo-Gateway: WireGuard (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } udp dport 52000 dnat ip to 10.19.1.1:52000 comment "!fw4: Nemo-Gateway: WireGuard2 (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } tcp dport 2233 dnat ip to 10.19.76.152:2222 comment "!fw4: PrePro: SSH_Ext (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } udp dport 51823 dnat ip to 10.19.76.23:51820 comment "!fw4: pi4b2: WireGuard (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } tcp dport 53 dnat ip to 10.19.76.13:53 comment "!fw4: RE7000-1: DNS Requests (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr { 68.x.x.x, 172.16.1.2, 192.168.20.33 } udp dport 53 dnat ip to 10.19.76.13:53 comment "!fw4: RE7000-1: DNS Requests (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 192.168.20.33 tcp dport 22 dnat ip to 10.19.76.22:22 comment "!fw4: MR8300 config backup (reflection)"
        }
}
table inet fw4 {
        chain srcnat {
                type nat hook postrouting priority srcnat; policy accept;
                oifname { "wg_lan", "wg_lan2", "br-lan.1", "bond-LACP" } jump srcnat_lan comment "!fw4: Handle lan IPv4/IPv6 srcnat traffic"
                oifname { "eth3", "eth4", "mr8300", "re7001" } jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
        }
}
table inet fw4 {
        chain srcnat_lan {
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.1 tcp dport 5201 snat ip to 10.19.76.1 comment "!fw4: Nemo-Gateway: iperf3 from RE7000's (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.1 udp dport 5201 snat ip to 10.19.76.1 comment "!fw4: Nemo-Gateway: iperf3 from RE7000's (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.1 tcp dport 2222 snat ip to 10.19.76.1 comment "!fw4: Nemo-Gateway: SSH_Ext (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.32 udp dport 51820 snat ip to 10.19.76.1 comment "!fw4: Nemo-Gateway: WireGuard (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.1.1 udp dport 52000 snat ip to 10.19.1.1 comment "!fw4: Nemo-Gateway: WireGuard2 (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.152 tcp dport 2222 snat ip to 10.19.76.1 comment "!fw4: PrePro: SSH_Ext (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.23 udp dport 51820 snat ip to 10.19.76.1 comment "!fw4: pi4b2: WireGuard (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.13 tcp dport 53 snat ip to 10.19.76.1 comment "!fw4: RE7000-1: DNS Requests (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.13 udp dport 53 snat ip to 10.19.76.1 comment "!fw4: RE7000-1: DNS Requests (reflection)"
                ip saddr { 10.19.1.1, 10.19.75.0-10.19.76.255 } ip daddr 10.19.76.22 tcp dport 22 snat ip to 10.19.76.1 comment "!fw4: MR8300 config backup (reflection)"
        }
}

This looks like automatically created reflection rules for RE7000-1: DNS Requests (whatever that is) and they are currently doing the SNAT.

Add option reflection '0' to that rule and you should be able to break the queries to external DNS servers as you expect.

I have a few devices running openwrt that are remotely deployed (one each using a wg tunnel pair of 172.16.1.1/172.16.1.2, 172.16.2.1/172.16.2.2, and 172.16.3.1/172.16.3.2 (hence the /22 src_ip), allowing traffic from them, back to my router. Maybe I ended up misconfiguring?

This is the rule:

config redirect
        option dest 'lan'
        option target 'DNAT'
        option src 'wan'
        option src_dport '53'
        option dest_ip '10.19.76.13'
        option dest_port '53'
        option name 'RE7000-1: DNS Requests'
        option src_ip '172.16.0.0/22'

I've added the no reflection option to it, I'll test when I get home.

So that got the expected unexpected source errors to return. Re-enabling the NAT rule takes care of that again.

Now.... what about the forwarding rule (with NAT loopback enabled) was making the issue occur? I don't understand.

This is what NAT loopback does. Let's say you have some kind of server on your lan and you want it to be accessible from the wan. You must create a wan=>lan forwarding rule. NAT loopback is enabled by default and the destination zone (lan) in the redirect (DNAT) rule is used as the source zone for the reflection rules. The whole idea is that if you initiate a request from lan to the wan address(es) of the router, you should be redirected to the host (server) located on the lan.

This requires a combination of DNAT + SNAT rules like these:

In your case, the SNAT rule(s) accidentally caused the unexpected "issue".

Honestly, I would have never guessed that you created a redirect rule wan=>lan to port 53...

It's... a slowly evolving method of internetworking the remote endpoints. The forward rule is restricted to origin IPs in the endpoints subnets (the 172.16.0.0/22), so it's not a wide-open all-WAN port 53 service.. They end up having two tunnels connected to my router: one that connects to my lan-zone wireguard server (wg_lan), and another that connects to a wan-zone interface (re7001,re7002, etc). The idea being the tunnel that connects to the wg_lan interface in my router is in a wan-zone on their end, and gets them access to my home, so services that expect to come from here, come from here. And the tunnel that connects from the wan-zone interfaces on my router is on the other end one that is in their lan-zone, such that I can route traffic through them. I have been slow tinkering and probably made a bad change. Ultimately I want bi-directionality through single tunnels, but I haven't done that yet/right.