use the /etc/hosts file.
You might not have seen that because i edited it in after i realised you were doing this directly from your router.
(edit)
The other way is to do what is called domain interception.
https://openwrt.org/docs/guide-user/base-system/dhcp_configuration#selective_dns_forwarding using this to tell dnsmasq to forward those requests to your AGH instance will work and cover all internal domains requests.
(edit2)
Internally we do this with AGH so downstream clients get searched for first.
What you are wanting to do is have your router be aware DNS wise of its downstream clients. Thus you want selective forwarding of those requests to AGH but everything else go upstream to your external DNS.