So, I discovered some really unexpected behavior on my network.
TL;DR: DNS queries from my LAN clients is sent out on a WireGuard VPN interface, which I have never configured it to do, it was meant to be sent on WAN.
I'm simplifying the explanation a little bit to remove anything that is not relevant. Let's say that I had the original network interfaces, LAN and WAN.
LAN interface:
config interface 'LAN'
option proto 'static'
option ipaddr '10.0.10.1'
option netmask '255.255.255.0'
option type 'bridge'
option device 'eth0.10'
In tihs config, "list dns" is not set. So the clients send their DNS requests to the router at 10.0.10.1, which in turn send it to the upstream DNS server over the WAN interface (WAN is a DHCP client). Everything is perfectly normal so far.
The problem comes when I add a WireGuard VPN interface, and another local interface (LAN2, on another VLAN), that is supposed to use the VPN interface instead of WAN.
WireGuard VPN interface:
config interface 'WireGuard_VPN'
option proto 'wireguard'
option private_key 'SuperSecretKey'
list addresses '10.0.69.5/32'
option delegate '0'
WireGuard Peer:
config WireGuard_VPN
option description 'SomePeer'
option public_key 'PublicKeyOfPeer'
option route_allowed_ips '1'
option endpoint_host '50.100.150.200'
option endpoint_port '51820'
list allowed_ips '10.0.69.0/24'
list allowed_ips '1.1.1.1'
LAN2 interface:
config interface 'LAN2'
option proto 'static'
option device 'eth0.20'
option ipaddr '10.0.20.1'
option netmask '255.255.255.0'
option delegate '0'
option defaultroute '0'
list dns '1.1.1.1'
The key thing here is that LAN2 is set to use DNS server 1.1.1.1, which is routed over the WireGuard VPN interface, not on WAN. I set firewall rules to allow LAN to send to WAN, but not WireGuard_VPN. LAN2 is allowed to send to WireGuard_VPN, but not WAN. Everything seems to be working ok.
Then I did some packet capture with tcpdump, and saw DNS queries originating from LAN clients, on the WireGuard_VPN interface. LAN clients are sending queries to 10.0.10.1:53, which I expected to be sent out on WAN, but instead the router are sending some queries from 10.0.69.5, to 1.1.1.1:53 over WireGuard_VPN.
As far as I can tell, some queries are sent over WAN (to the DNS server provided by my ISP), and some over WireGuard_VPN. By no means do I have insight into the inner workings of OpenWrt, but it seems that dnsmasq are just treating these DNS servers as equal, so I get this really unwanted and unexpected behavior.
I have solved the symptoms of this issue, partially by using Pi-hole on LAN, and specifying some public IPv4 addresses as upstream DNS, so OpenWrt is not handling any DNS queries anymore. The details are not that relevant.
I would love to hear your thoughts on this. It may or may not be a bug, but it was certainly very unexpected for me. Is there perhaps something that can be done to avoid users accidentally ending up in this situation?