AdGuardHome with DoH not working after power cycle - Solved!

Installing and Using OpenWrt
1

I'm wrting this to share my experience and solution of a very weird problem in the hope that this information might help others in the future. I searched on the Internet and here in the forum, and I did not find any mention to this problem nor a solution. This way I'm sharing below my findings and solution. If anyone has a better way to solve this issue please feel free to share!

1. The Context

I installed AdGuardHome (on a NanoPi R4S but I believe this might apply to any other device) and everything was working as expected. I rebooted OpenWrt many times and AdGuardHome came up and life was good.

2. The Problem

Howvever, after a power cycle (power disconnect and reconnect) AdGuard stopped working and the only solution was to remove and reinstall it. It worked well until the next power cycle. Any transient power outage could result in losing internet connectivity due to AdGuardHome being offline after a power cycle.

Looking at AdGuardHome logs there was no clear indication of any problem. AdGuardHome just stopped responding to DNS queries in this situation.

It took me many hours and many different trial-and-error experiments to identify the problem. After this the solution was easy, which I will try to summarize below.

3. The Root Cause

I have AdGuardHome configured to use DoH upstream to NextDNS (
https://dns.nextdns.io/xxxxxx). While AdGuardHome has a configuration of bootstrap DNS to resolve the DoH URL to an IP address for the initial connection, the problem was that the HTTPS connection to the DoH server was failing because of wrong system time (I believe it is related to certificate validation). It took me a while to find this out since there was no clear error about this in the log. Also rebooting OpenWrt does preserve the correct date-time, so I needed an "eureka" moment to understand the issue.

After I realized that the wrong system date/time was the root cause, then I understood the problem. OpenWrt is confiured by default to update the system date/time with NTP servers (ex: 0.openwrt.pool.ntp.org). However since AdGuardHome does not come up, NTP client cannot resolve the NTP server name to update the system date/time (I have the router itself also configured to use AdGuardHome as primary DNS). And with the wrong date/time, AdGuardHome cannot connect to the DoH server (I believe due to failing https certificate validation). So the classic chicken or egg problem.

4. The Solution

After many hours going down in this rabbit hole, the solution (actually a workaround) was very simple: I just replace the default NTP servers in OpenWrt by the IP addressess as follows:

This will break the chicken-or-egg problem (DNS needs correct date time, and NTP needs DNS to update it). With this very simple change I was able to solve the problem. Now even after a power cycle AdGuardHome is starting fine. It can be done either via LuCI or via CLI. To facilitate below are the UCI commands to replace the default NTP servers by their respective IP addresses:

uci -q delete system.ntp.server
uci add_list system.ntp.server="198.199.14.69"
uci add_list system.ntp.server="54.36.152.158"
uci add_list system.ntp.server="194.0.5.123"
uci add_list system.ntp.server="200.189.40.8"
uci commit system
/etc/init.d/sysntpd restart

Another possible solution would be to change the local (OpenWrt) DNS to another DNS server (so OpenWrt could resolve ntp servers hostnames regardless if AdGuard is working). But I want to have OpenWrt using AdGuard to be able to resolve local hostnames in my home network, and also to be protected by te AdGuard filtes.

2 Likes
2

... or set the router to use some other DNS, than the one you configured manually.

2 Likes
3

Yep! I was just editing my post explaining why I did not use this approach (I need OpenWrt to resolve hostnames in my local network). But you are right!

1 Like
4

I do warn about this in my thread. This is why the router is set to use one dns and then the clients use DOH via AGH.

(The reason you setup OpenWrt to use a different unencrypted upstream is so when it brings up your connection it can do NTP and updates as AGH will still be loading. Without NTP setting your router time / date correctly, SSL will fail and thus no encrypted DNS.)

There is also this https://openwrt.org/docs/guide-user/services/dns/adguard-home#bypassing_encrypted_dns_for_ntp for passing NTP unencrypted to the upstream if you really want to use AGH for router DNS and Client DNS. However the router really doesn't need to loop through AGH.

[/pool.ntp.org/]1.1.1.1
[/pool.ntp.org/]1.0.0.1
[/pool.ntp.org/]2606:4700:4700::1111
[/pool.ntp.org/]2606:4700:4700::1001

4 Likes
5

Why? I think your problems are based on this approach.

Fully agreed!

1 Like
6

Since router's DNSmasq is running on port 5353, what do you recommend to resolve local names (the ones assigned and reserved via DHCP, for example) on the router itself (such in an SSH session)?

7

reverse PTR lookups would solve your problem. it tells AGH to look at dnsmasq for the names. You must have missed that step.

Its all in the thread i linked.

Its also on the wiki

1 Like
8

port 5353 is actually assigned for mDNS, DNS-SD (apple) and avachi linux service.

Thus i have now redone my scripts and advice to move dnsmasq to port 54 instead. It officially belongs to Xerox for one of their older networking protocols but it is highly unlikely to be used in todays age of TCP/IP.

1 Like
9

This is my way of implementation:

  1. define a new interface 'agh' with a static address (e.g. 192.168.53.53/24) and direct AGH to listen on port 53 of this interface.

  2. configure dnsmasq to listen on all interfaces, but the interface 'agh' (exclude interface 'agh' for dnsmasq)

  3. in AGH under 'Setting/DNS settings' enter in the field for DNS server the following line:

[/domain-of-your-home-lan/]127.0.0.1:53

  1. in AGH under 'Setting/Private reverse DNS servers' enter in the field the following line:

127.0.0.1:53

I assume your dnsmasq is listening on port 53, otherwise adjust the port number.

The last two steps is just what @mercygroundabyss proposed.

With this construction your router will offer two DNS servers:

  1. an unfiltered DNS server via dnsmasq (192.168.1.1:53)
  2. a filtering DNS server via AGH (192.168.53.53:53)

The file /etc/resolv.conf of your router should contain the following line:

nameserver 127.0.0.1

That way your router will always get unfiltered DNS responses.

Good luck.

10

Seems needlessly overcompllicated.

Far easier to just bump dnsmasq to port 54 and setup ptr lookups and domain interception rules.

That means AGH is now inserted as primary dns on port 53. dnsmasq is now downstream local PTR lookup on port 54 and nothing else needs to change. AGH will serve DNS and any local lookups are diverted to dnsmasq.

The reason for this is dnsmasq forks every request and thus uses more memory. So if you had added AGH behind dnsmasq you not only add an additional DNS "hop" incurring extra latency, you use more memory too.

Now you've looped your router DNS. This runs the risk of you not getting NTP updates.

Your router should have its OWN upstream either provided by your ISPs dhcp or a DNS service you choose. My DNS script changes your upstream router dns for cloudflares. This means your router does unencrypted DNS lookups to them. AGH once you set it up will do encrypted DNS for your downstream clients (everything behind the router) But this means you dont run the risk of a failed NTP update which will break DNS as your date/time will be incorrect. An incorrect date/time means SSL negotiation fails and means no encrypted connections until the correct date/time is set.

Another option is just to disable OpenWrt's dnsmasq and dhcp and use AGH to replace both DNS and DHCP then you dont need PTR lookups because AGH will do it instead. However unless your network is fairly simple i dont recommend this as AGH are still sorting out a few things regarding DHCP.

1 Like
11

Do you have a list of the things that still preclude the use of AGH's DHCP server?

TIA

12

issue is still open.

I was waiting for them to resolve that before i did testing. But my problem is that i actually dont get IPv6 from my ISP so would require someone who does to do some proper testing and report back.

Some have reported using AGH DHCP works but I'm unwilling to recommend unless i've personally tested it.

(edit) additionally OpenWrt's DHCP has been well tested and documented for much more complex network setups and thus is preferred option.

1 Like
13

I was re-reading your post. I believe I have the configurations correct, since AGH is resolving local reserved addresses. But my need is to have this in the local router.

Let me try to explain: I have for example the following entry in /etc/config/dhcp:

config domain
        option name 'apesc'
        option ip '192.168.1.3'

All clients in the network can resolv it fine via AGH. See my laptop for example (it is using AGH as DNS):

C:\>nslookup apesc
Server:  router
Address:  192.168.1.1

Non-authoritative answer:
Name:    apesc.home
Address:  192.168.1.3

However, if I open an SSH session to OperWrt with the recommended configuration in the AGH Wiki this does not work since it is using my WAN DNS (Google):

root@router:~# nslookup apesc
Server:         8.8.8.8
Address:        8.8.8.8:53

** server can't find apesc.home: NXDOMAIN

** server can't find apesc.home: NXDOMAIN

root@router:~#

So the way I use to solve this is to point the lan DNS to the local server (previously dnsmasq and now AGH):

root@router:~# uci add_list network.lan.dns='192.168.1.1'
root@router:~# uci commit network
root@router:~# nslookup apesc
Server:         192.168.1.1
Address:        192.168.1.1:53

Non-authoritative answer:
Name:   apesc.home
Address: 192.168.1.3

Non-authoritative answer:

root@router:~#

And while this last step enables local name resolution in OpenWrt itself, it causes the problem I described in the original post when using AGH with upstream encryption.

Am I missing anything in your recommendation about a better resolution for this situation? I believe your other recommendation for passing NTP unencrypted to the upstream in AGH is in fact a valid alternative instead of using NTP IP addresses in OpenWrt configuration I proposed.

14

(edit) just re-read and realise you are trying this directly from OpenWrt. You are correct. As it uses a separate upstream it will not know about dnsmasq.

Easiest option here is to edit /etc/hosts and insert your entry in there.

192.168.1.3 apesc

(edit - this fixes for downstream clients not for the router itself)
Two ways to solve this for downstream clients.

https://openwrt.org/docs/guide-user/services/dns/adguard-home#lan_domain_interception follow this and add in to your upstream. This will intercept any local lookup request and pass it to dnsmasq.

The other way is a DNS Re-Write.
On the AGH web interface go to Filters > DNS Re-write.

Click Add

image
image623Ă—646 22.9 KB

Fill in like this and any requests for that address will reply 192.168.1.3

1 Like
15

I think I haven't been clearly enough: AGH doesn't sit behind dnsmasq.

Please look at my little picture, it displays my current construction of DNS services in my OpenWRT router.

openwrt-dns-server
openwrt-dns-server1755Ă—1240 106 KB

For non-filtering DNS and DHCP I use dnsmasq as it is installed by OpenWRT. The filtering DNS is a second service beside of dnsmasq, realized by AGH.

I hope, I've expressed myself more clearly.

1 Like
16

Yes you have.

As i have pointed out in my main thread. AGH is all in one encrypted DNS provider. It does NOT need additional packages like unbound. Unbound/Stubby etc was the older way to patch in DOH or other encrypted DNS methods.

AGH should talk directly upstream to your external DNS. Your unbound resolving server is an extra hop that really isn't required. AGH has a resizable cache internally that can be configured.

A default OpenWrt install has Dnsmasq as dns and local host ptr cache. With an AGH install we move dnsmasq to a downstream provider and insert AGH into its place. Now AGH becomes the upstream encypted DNS provider for your downstream clients. And if you do the rDNS config and Domain Interception then it redirects local lookups back to dnsmasq. That is the simplest way to achieve DNS filtering and family controls other than disabling both DNS and DHCP in OpenWrt and replacing them with AGH's services instead.
I also ensure the router itself has a separate unencrypted DNS that it can do NTP updates on boot so SSL and thus encrypted DNS works.

It is as best i can determine the most logical way to correctly install and provide filtering for an OpenWrt router.

17

Sure. But I do not want to rely on an external DNS, who's collecting data about my DNS usage. That's the reason, why I use unbound as a resolving DNS. Therefore I don't need an external DNS and there's also no extra hop. My AGH does nothing else but filtering.

I consider this as your personal opinion and preference and don't want to argue against this, but a German proverb says: viele Wege fĂĽhren nach Rom. :wink:

18

urm... so what is your unbound pointed at to do its resolving? And is it doing encrypted lookups? Also if you are concerned about that... That's why encrypted clients like AGH were invented and privacy respecting DNS services who provide lookups without logging. Cloudflare only logs for issues and then deletes them.

https://blog.cloudflare.com/announcing-the-results-of-the-1-1-1-1-public-dns-resolver-privacy-examination/

  1. Cloudflare will retain only the limited transaction and debug log data (“Public Resolver Logs”) for the legitimate operation of our Public Resolver and research purposes, and Cloudflare will delete the Public Resolver Logs within 25 hours.

chuckle fair enough but it is based on networking principles and best practises. But as always the choice is always the users.

19

chuckle

Obviously you don't know the principles of a resolving DNS server. Unbound (when working as a resolving DNS server) does not need an external DNS server, it does the resolving by itself, starting with the root DNS servers (which are listed in the file root.hints).

Encryption protects the data on the transport way, not at the end point. Your external DNS provider sees all your dns queries, can log them and create user profiles.

You can believe it ... or not. In fact, you have no possibility to verify their promises.

I doubt, that you really know about networking principles and best practises, when you even haven't a fair knowledge about the working principles of a resolving DNS server.

20

well assumptions just make an ass out of you and me.

Making your own root DNS server is vastly overkill for what you are doing and requires significant setting up and properly cascading. In short you are re-creating the phonebook from scratch rather than trusting the phone company to make you one to look numbers up in.

Cloudflare was audited independently if you had read the blog.

Unless you are using TLS or some other encryption then your queries as you follow the root chain can still be read.

There are indeed many roads to rome.