you can install AGH on your router directly if you have enough space and ram.
Alternately if you want to run any other type of DNS service like AGH or PiHole on another device you must get Dnsmasq to pass option 6 as part of the dhcp request. This will set your downstream clients to use the broadcasted DNS. However, be aware that without dns interception rules, anyone using a hardcoded or alternate set DNS will bypass your filtering.
# DHCP option 6: which DNS (Domain Name Server) to include in the IP configuration for name resolution
uci add_list dhcp.lan.dhcp_option='6,'"<IP of PI>"
using these rules (DNS enforcing), it seems the raspberry and the router blocks itself or at least I don't get more ping or unable to connect? I'm getting a loophole? where I should exempt the dns server (raspberry pi) and the router (openwrt) ? or what I'm doing wrong?
I installed ADH on a raspberry pi, I didn't add any extra rules to the firewall's custom rules, and everything is working fine so far. I think the above firewall custom rules are only valid when ADH is installed on the same device as openwrt. For exam only.
Yes those rules are for same host DNS. However if you are not intercepting DNS queries then anyone setting their own DNS settings behind your router will bypass your DNS entirely. Additionally that rule ONLY intercepts plain DNS queries. To block DOH or other encrypted queries takes additional rules.
(edit) I've added a note to the wiki to highlight that.