Stubby dns over tls using dnsmasq-full for dnssec & caching

READ ENTIRE GUIDE BEFORE YOU BEGIN
See here for GETDNS AND STUBBY on OPENWRT / LEDE:
https://github.com/openwrt/packages/blob/master/net/stubby/files/README.md

OPENWRT STUBBY DNS OVER TLS USING UNBOUND:


I have written tutorials where DNS OVER TLS setup is focused on deploying UNBOUND STUBBY and GETDNS along with DNSMASQ for DHCP on OPENWRT/LEDE. Thanks to my good friend Specimen ( see his tutorial / guide here : [Tutorial] DNS-over-TLS with dnsmasq and stubby (no need for unbound) - I was able to realize that we can eliminate UNBOUND all together for those who wish to do so for any number of reasons. For those of you who that may have limited memory or storage available on your router and you need more storage and swap memory for your router see here: http://ediy.com.my/index.php/blog/item/118-how-to-increase-storage-on-tp-link-tl-mr3020-with-extroot and here: https://samhobbs.co.uk/2013/11/more-space-for-packages-with-extroot-on-your-openwrt-router -Specimen's tutorial features a method to install STUBBY and GETDNS to RAM very smartly and efficiently as well - that will not be covered here as I have found that DNSMASQ-FULL is a better solution in my opinion.

So - let's get started with no further ado:
1 -
A - opkg update
B - opkg install ca-certificates
C - opkg install stubby ( GETDNS and LIBYAML will be installed as dependencies )

2 - This guide aforementioned at the top of this page: https://github.com/openwrt/packages/blob/master/net/stubby/files/README.md is the one I followed. However, the only issue is that the guide gives one several options as to how to deploy STUBBY and GETDNS with DNSMSQ and / or DNSMSQ-FULL. With the availability of options just may come confusion and mistakes. So, the purpose of this tutorial is to demonstrate and eliminate potential errors during setup of STUBBY DNS OVER TLS USING DNSMASQ-FULL FOR DNSSEC & CACHING as the title asserts.

3 - I chose to use the /etc/stubby/stubby.yml file to configure STUBBY. My reasons for preferring to configure Stubby with the /etc/stubby/stubby.yml file instead of the now default UCI system /etc/config/stubby file are for several reasons. I found that I have more control over the security options which DNS OVER TLS is intended to provide. Like padding - 853 or 443 port and so on. So in order to use /etc/stubby/stubby.yml file, you must change a default setting in the /etc/config/stubby file to allow manual configuration.

4 - To keep this simple - go into default UCI STUBBY file which is /etc/config/stubby by entering nano /etc/config/stubby and then set option manual '1' - if you leave it at default setting of option manual 'o' you will not be able to use the /etc/stubby/stubby.yml file in order to configure STUBBY as before. So, after changing option manual '1' in the /etc/config/stubby file - configure /etc/stubby/stubby.yml as follows in step 5

5 - The next step is to configure /etc/stubby/stubby.yml file in the standard fashion. Note that DNSSEC is not configured in STUBBY as DNSMASQ-FULL will be configured to implement this feature later on in this process. Here is my /etc/stubby/stubby.yml file nano /etc/stubby/stubby.yml
Stubby github config: https://raw.githubusercontent.com/getdnsapi/stubby/develop/stubby.yml.example
VERY IMPORTANT UPDATE:
After checking, rechecking and the triple checking on this website mentioned above : https://www.immuniweb.com/ssl/?id=Su8SeUQ4 I have made some very serious discoveries regarding which DNS Privacy Test Servers to use. The bottom line that I strongly suggest you only choose to deploy servers which support the TLSv1.3 protocol. See here for information and importance of TLSv1.3 : https://kinsta.com/blog/tls-1-3/ 1
I will save you some considerable leg work and post below the best configuration for your stubby.yml file. Here it is:

## All DNS Privacy Servers Below Tested On October 10 2019 With A+ Rating - 
##100%  Perfecto Configuration on website: https://www.immuniweb.com/ssl/?id=Su8SeUQ4n

# Note: by default on OpenWRT stubby configuration is handled via
# the UCI system and the file /etc/config/stubby. If you want to
# use this file to configure stubby, then set "option manual '1'"
# in /etc/config/stubby.
resolution_type: GETDNS_RESOLUTION_STUB
round_robin_upstreams: 1
appdata_dir: "/var/lib/stubby"
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
edns_client_subnet_private: 1
idle_timeout: 60000
listen_addresses:
  - 127.0.0.1@5453
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_connection_retries: 5
tls_backoff_time: 900
timeout: 2000
tls_ca_path: "/etc/ssl/certs/"
upstream_recursive_servers:
### IPV4 Servers ###
### DNS Privacy Test Servers ###
## 1 - The Surfnet/Sinodun DNS TLS Server #1  A
  - address_data: 145.100.185.15
    tls_auth_name: "dnsovertls.sinodun.com"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 62lKu9HsDVbyiPenApnc4sfmSYTHOVfFgL3pyB+cBL4=
## 2 - The Surfnet/Sinodun DNS TLS Server #3  A+
  - address_data: 145.100.185.18
    tls_port: 853
    tls_auth_name: "dnsovertls3.sinodun.com"
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 5SpFz7JEPzF71hditH1v2dBhSErPUMcLPJx1uk2svT8=
### Test servers ###
## 3 - The DNS Warden DNS TLS Server #1  A
  - address_data: 116.203.70.156
    tls_auth_name: "dot1.dnswarden.com"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: aPns02lcGrDxnJQcRSHN8Cfx0XG+IXwqy5ishTQtzR0=
## 4 - The DNS Warden DNS TLS Server #2  A+
  - address_data: 116.203.35.255
    tls_auth_name: "dot2.dnswarden.com"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: aPns02lcGrDxnJQcRSHN8Cfx0XG+IXwqy5ishTQtzR0=
## 5 - The dns.containerpi.com DNS TLS Server  A+
  - address_data: 45.77.180.10
    tls_auth_name: "dns.containerpi.com"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: xz8kGlumwEGkPwJ3QV/XlHRKCVNo2Fae8bM5YqlyvFs=
## 6 - The ibuki.cgnat.net DNS TLS Server  A+
  - address_data: 35.198.2.76
    tls_auth_name: "ibuki.cgnat.net"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: Y2LUHSMw7x+Z1U6uQszwjz16/1jR9JWoU/WhmxUpsE8=
## 7 - The doh.li DNS TLS Server  A+
  - address_data: 46.101.66.244
    tls_auth_name: "doh.li"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: ljpOTyJrq2vh8sdxCohIEf7aBFWlOjJeL0xsZuX4kvc=
## 8 - The Andrews & Arnold DNS TLS Server #1  A+
  - address_data: 217.169.20.23
    tls_auth_name: "dns.aa.net.uk"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: qcahcJ8VhWz83citsQsOXC/qoCK3b20ja58MZMMnAIg=
## 9 - The Andrews & Arnold DNS TLS Server #2  A+
  - address_data: 217.169.20.22
    tls_auth_name: "dns.aa.net.uk"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: MjpiTbfaE5lFDLkL8iMFbmN/OOlcxtmIvxCfWLOPa3c=
## 10 - The dns.cmrg.net DNS TLS Server  A+
  - address_data: 199.58.81.218
    tls_auth_name: "dns.cmrg.net"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 3IOHSS48KOc/zlkKGtI46a9TY9PPKDVGhE3W2ZS4JZo=
## 11 - The BlahDNS German DNS TLS Server  A+
  - address_data: 159.69.198.101
    tls_auth_name: "dot-de.blahdns.com"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: lI/c+XiSmaAm79YulIzRmskcP7MAAD4G4uaD3iLs3Bk=
## 12 - The BlahDNS Japan DNS TLS Server  A+
  - address_data: 108.61.201.119
    tls_auth_name: "dot-jp.blahdns.com"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: psuldEImRyeSkU88b2ORtiNQ2uBdo+RCwAw6SxaJWQ4=
## 13 - The securedns.eu DNS TLS Server  A+
  - address_data: 146.185.167.43
    tls_auth_name: "dot.securedns.eu"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: h3mufC43MEqRD6uE4lz6gAgULZ5/riqH/E+U+jE3H8g=
## 14 - The dns.neutopia.org  DNS TLS Server  A+
  - address_data: 89.234.186.112
    tls_auth_name: "dns.neutopia.org"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: wTeXHM8aczvhRSi0cv2qOXkXInoDU+2C+M8MpRyT3OI=
## 15 - The dns.seby.io - Vultr DNS TLS Server  A+
  - address_data: 139.99.222.72
    tls_auth_name: "dot.seby.io"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 8A/1KQQiN+aFWenQon076nAINhlZjGkB15C4E/qogGw=
## 16 - The dns.seby.io - OVH DNS TLS Server  A+
  - address_data: 45.76.113.31
    tls_auth_name: "doh.seby.io"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: H13Su1659zEn0ZIblEShwjZO+M5gxKK2wXpVKQHgibM=
## 17 - The appliedprivacy.net DNS TLS Server  A+
  - address_data: 37.252.185.232
    tls_auth_name: "dot1.appliedprivacy.net"
    tls_port: 443
    tls_pubkey_pinset:
      - digest: "sha256"
        value: yJ5GuTCv9+gRyR78zryHT38gTJ0lmAcsXZXTH/XVA0Y=
## 18 - The Secure DNS Project by PumpleX DNS TLS Server #1  A+
  - address_data: 51.38.83.141
    tls_auth_name: "dns.oszx.co"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: uXHfOKxBJ4aqMWmVw7+NtXGCkiYLyaeM7WujER0jIkM=
## 19 - The Secure DNS Project by PumpleX DNS TLS Server #2  A+
  - address_data: 51.38.82.198
    tls_auth_name: "dns.pumplex.com"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: GGR/Mugb+WNqMmxTI0pHjRJsY96XjzlVDqVuqQ8Cknw=
## 20 - The dns.digitale-gesellschaft.ch DNS TLS Server #1  A+
  - address_data: 185.95.218.43
    tls_auth_name: "dns.digitale-gesellschaft.ch"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: JnvUziCIRjvSPYAqcTkQu7ZPuWLP3R6R6aPKrDvlzMs=
## 21 - The dns.digitale-gesellschaft.ch DNS TLS Server #2  A+
  - address_data: 185.95.218.42
    tls_auth_name: "dns.digitale-gesellschaft.ch"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: nBRTYH4++qjDTSJAhlzd2wxXf5cBviICH74qg4Qi3uw=
## 22 - The dot.tiar.app DNS TLS Server  A+
  - address_data: 174.138.29.175
    tls_auth_name: "doh.tiar.app"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: p/yGyWNX4qS8rDH+ouR6PgSZXfQVf1tNiaJeEQG+pFA=
## 23 - The dns-nyc.aaflalo.me DNS TLS Server  A+
  - address_data: 168.235.81.167
    tls_auth_name: "dns-nyc.aaflalo.me"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: KqzeDRgYePfKuZrKttwXM8I2Ej4kD6Sayh0kp4NWaJw=
## 24 - The dns.aaflalo.me DNS TLS Server  A+
  - address_data: 176.56.236.175
    tls_auth_name: "dns.aaflalo.me"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: 9QK9j+GK8Vc6HrzAGlwxjKL+dWGe/fpLjleufiKKU6o=
## 25 - The jp.tiar.app DNS TLS Server  A+
  - address_data: 172.104.93.80  
    tls_auth_name: "jp.tiar.app"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: aPcNFv5Cx9Az8IsHzd+Q6fYGAqoDjIWkExfk6fG0fbY=
### Anycast DNS Privacy Public Resolvers ###
## 26 - The security-filter-dns.cleanbrowsing.org DNS TLS Server  A+
  - address_data: 185.228.168.9
    tls_auth_name: "security-filter-dns.cleanbrowsing.org"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: rb2O6hMTZZ/go/vOqyVLY2lATD9DkD6+BkKfJwYYMFw=
## 27 - The DNS.SB DNS TLS Server #1  A+
  - address_data: 185.222.222.222
    tls_auth_name: "dns.sb"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: /qCm+kZoAyouNBtgd1MPMS/cwpN4KLr60bAtajPLt0k=
## 28 - The DNS.SB DNS TLS Server #2  A+
  - address_data: 185.184.222.222
    tls_auth_name: "dns.sb"
    tls_port: 853
    tls_pubkey_pinset:
      - digest: "sha256"
        value: /qCm+kZoAyouNBtgd1MPMS/cwpN4KLr60bAtajPLt0k=

# Set the minimum acceptable TLS version. Works with OpenSSL >= 1.1.1 only.
# This option can also be given per upstream.
tls_min_version: GETDNS_TLS1_2
## For Version OpenSSL 1.1.1  TLSv1.3 and above
# Set the maximum acceptable TLS version. Works with OpenSSL >= 1.1.1 only.
# This option can also be given per upstream.
tls_max_version: GETDNS_TLS1_3
# Set the acceptable ciphers for DNS over TLS.  With OpenSSL 1.1.1 this list is
# for TLS1.2 and older only. Ciphers for TLS1.3 should be set with the
#tls_ciphersuites option. This option can also be given per upstream.
tls_cipher_list: "EECDH+AESGCM:EECDH+CHACHA20"
# Set the acceptable cipher for DNS over TLS1.3. OpenSSL >= 1.1.1 is required
# for this option. This option can also be given per upstream.
tls_ciphersuites: "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256"


PS - Lastly, you can and should take advantage of this new DNS OVER TLS provider.
You need to sign up and use configured settings in order to use it.
NextDNS is a free service - ANYCAST and pretty much cutting edge.
ANYCAST speeds up your DNS - Here it is:
NextDNS https://my.nextdns.io/configuration/19474f/setup

6 - Integration of STUBBY with DNSMASQ
A - Set DNSMASQ to send DNS requests to STUBBY - this is done to allow Localhost ( 127.0.0.1 ) on port 5453 to be the sole resolver used by your router. This forces router to use DNS OVER TLS as STUBBY listens on the default address / port 127.0.0.1#5453 .
There are two methods to do this:

uci add_list dhcp.@dnsmasq[-1].server='127.0.0.1#5453'
uci set dhcp.@dnsmasq[-1].noresolv=1
uci commit

Or edit the /etc/config/dhcp file

nano /etc/config/dhcp
list server '127.0.0.1@5453'
option noresolv '1'

7 - Disable Sending DNS Requests to ISP Provided DNS Servers

uci set network.wan.peerdns='0'
uci set network.wan.dns='127.0.0.1'
uci set network.wan6.peerdns='0' # If you use STUBBY for IPV6
uci set network.wan6.dns='0::1' # If you use STUBBY for IPV6
uci commit

Or

In the Luci Web interface under Network > Interfaces > Edit Wan > Advanced Settings > Remove Check From Box Next To " Use DNS servers advertised by peer " and enter DNS Server 127.0.0.1 -

8 - Now restart DNSMASQ and enable, start and restart STUBBY just to make sure everything is up and running before you proceed. Run the following commands:

/etc/init.d/dnsmasq restart
/etc/init.d/stubby enable
/etc/init.d/stubby start
/etc/init.d/stubby restart

9 - Enabling DNSSEC - We are going to use DNSMASQ-FULL in order to enable this feature. This one command removes DNSMASQ and installs DNSMASQ-FULL. In order to achieve this end, enter this as one command:

A - In the version 18 and later builds of OpenWRT this step has been modified as follows to keep from having two IPv6 DHCP servers running:

opkg install dnsmasq-full --download-only && opkg remove odhcpd-ipv6only && opkg remove dnsmasq && opkg install dnsmasq-full --cache . && rm *.ipk

10 - We are now going to configure STUBBY not to perform DNSSEC validation and configure DNSMASQ-FULL to require DNSSEC validation. We do so by entering the following commands via UCI:

uci set dhcp.@dnsmasq[-1].dnssec=1
uci set dhcp.@dnsmasq[-1].dnsseccheckunsigned=1
uci commit

Or edit the /etc/config/dhcp file

nano /etc/config/dhcp
option dnssec '1'
option dnsseccheckunsigned '1'

To verify DNSSEC trust-anchors, this is how to do it ( 1 ) trust-anchors are here: TRUSTANCHORSFILE="/usr/share/dnsmasq/trust-anchors.conf"
so in SSH shell issue command : cat /usr/share/dnsmasq/trust-anchors.conf - and Voila' - Whoop There It Is !

11 - Now I am used to running UNBOUND so I accustomed its' caching feature. To increase DNSMASQ-FULL cache use one of these two methods:

A - Via UCI (Unified Configuration Interface) - in shell
uci set dhcp.@dnsmasq[0].cachesize=1000
uci commit dhcp

Or edit the /etc/config/dhcp file

nano /etc/config/dhcp
option cachesize '1000'

Now restart DNSMASQ and enable, start and restart STUBBY once again:
/etc/init.d/dnsmasq restart
/etc/init.d/stubby restart

12 - I have found that for whatever reasons it is best to make these entries in startup in order for STUBBY and DNSMASQ-FULL to fire up after a reboot. On boot, in case GETDNS and STUBBY fails to start. This is very likely due to Internet connection not available yet at time of starting DNSMASQ-FULL GETDNS and STUBBY. In such a case, the workaround is to wait for Internet connection to be available before restarting DNSMASQ-FULL GETDNS and STUBBY. The solution is to add the following lines into /etc/rc.local:
You may also enter these additions via Luci menu Startup > Local Startup
nano /etc/rc.local

Note: you must comment out with the symbol ## the BOLD FACED ENTRIES BELOW:
( at the beginning of each line ONLY )

Put your custom commands here that should be executed once
the system init finished. By default this file does nothing.

Wait until Internet connection is available
for i in {1..60}; do ping -c1 -W1 99.192.182.100 &> /dev/null && break; done

Restart DNS Privacy Daemon - Stubby as it requires a successful
time sync for its encryption to work/
/etc/init.d/dnsmasq restart
/etc/init.d/stubby restart
/etc/init.d/openvpn restart #If you run VPN as you should

exit 0

Reboot your router just to make sure everything is running as designed.

13 - Two quick command line tests for you to conduct after rebooting your router:

A - DNS query name minimisation to improve privacy, along with DNS resolution speed and accuracy. The name servers listed I use help to consistently ensure QNAME Minimisation functions as designed within. The idea is to minimise the amount of data sent from the DNS resolver to the authoritative name server. You need to opkg install bind-tools or opkg install bind-dig
command : dig txt qnamemintest.internet.nl +short and / or dig -t txt qnamemintest.internet.nl
The results in any of these scenarios will show either:
"HOORAY - QNAME minimisation is enabled on your resolver :)!”
or β€œNO - QNAME minimisation is NOT enabled on your resolver :(.”
Reference https://discourse.pi-hole.net/t/unbound-and-qname-minimisation/10038/4
You will and should get HOORAY ! - if you used the name servers listed in this guide for your Stubby configuration.

B - DNSSEC TEST - command : dig dnssectest.sidn.nl +dnssec +multi @127.0.0.1
Look at the flags section. You should see : ;; flags: qr rd ra ad; As long as you get ad flag as you should, you now have verified DNSSEC as well.

VERY IMPORTANT TIP:
Please note that right at the top of the main DNS Privacy Test Servers Homepage ( https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Test+Servers ) It Ominously Declares:
DoT servers
The following servers are experimental DNS-over-TLS servers.
Note that they are experimental offerings (mainly by individuals/small organisations) with no guarantees on the lifetime of the service, service level provided. The level of logging may also vary (see the individual websites where available) - the information here about logging has not been verified.Also note that the single SPKI pins published here for many of these servers are subject to change (e.g on Certificate renewal) and should be used with care!!
For these reasons it is most important to check and verify your SPKI pin(s) for TLS authentication manually yourself from time to time. There is are fire methods to make sure that you are using the correct value for any upstream nameserver ( aka tls_pubkey_pinset value ) - Go to https://blahdns.com/ and scroll down to the section to the yellow section entitled What is DNS OVER TLS click on it and it will open up.
When you do it will state some general information, but what you want to pay attention to is this section:
How to get SPKI

Most Simple and Direct Method:
gnutls-cli --print-cert -p 853 159.69.198.101 | grep "pin-sha256" | head -1 
       And / Or With Adjustment For SSL Port and Address Being Tested
gnutls-cli --print-cert -p 443 159.69.198.101 | grep "pin-sha256" | head -1 - where you must opkg install gnutls

OR
echo | openssl s_client -connect '185.49.141.37:853' 2>/dev/null | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

There is also a third option. kdig -d @185.49.141.37 +tls-ca +tls-host=getdnsapi.net example.com - where you must install knot-dig / opkg install knot-dig
This is my personal favorite as the readout from this command will list the certificate specifically like so:
;; DEBUG: #1, CN=getdnsapi.net
;; DEBUG: SHA-256 PIN: foxZRnIh9gZpWnl+zEiKa0EJ2rdCGroMWm02gaxSc9Q=
and let you know that the certificate is valid like so: ;; DEBUG: TLS, The certificate is trusted.
Remember to change port to 443 or port for IPV6 if different than standard 853 where applicable.
To use kdig certificate verification method on an alternate port example: kdig -d @199.58.81.218 -p 443 +tls-ca +tls-host=dns.cmrg.net example.com

Peace Unto All,

directnupe

Parting Thoughts:
I really like this deployment and implementation of DNS OVER TLS. It seems to be very snappy in resolving DNS queries - even a bit more responsive than UNBOUND. It is pretty simple and straight forward to set up and the documentation is very easily understood. Moreover, DNSMASQ is the native resolver for OpenWRT, so this set up minimizes any other components which may bog down your router. In essence, this setup is most clean and elegant in my estimation. Also, DNSMASQ-FULL allows you a more robust resolver than the native install standard DNSMASQ version. DNSMASQ-FULL allows for DNSSEC and QNAME Minimisation. I am using this setup now and I will report back later on; however, for now it is working beautifully.

1 Like

@directnupe
Many thanks for posting this very useful guide!
I managed to follow it, and have set up my router using stubby.
A few questions:

  1. Any reason you set padding size to 256? (dnsprivacy.org seem to recommend 128).
  2. Does using stubby for DNSSEC require dnsmasq-full or is dnsmasq sufficient?
  3. Is there any way to check and /or validate that any of this is working as intended? My setup seems to be working, but I have no way of knowing if e.g. the DNSSEC bit is working. If you could add something to your guide about logging and /or how to check whether things are working it would be much appreciated!
    Thanks to everyone (including yourself) who is working towards dns privacy!

Hello Caveat, I'm not directnupe but since this is based on my guide I think I can answer 2 and 3 better.

But first I should inform that directnupe forgot an essential seeting for DNSSEC to work, he forgot to copy it from my guide: [Tutorial] DNS-over-TLS with dnsmasq and stubby (no need for unbound)

You need this line in stubby.yml:

dnssec_return_status: GETDNS_EXTENSION_TRUE

DNSSEC validation is being done by stubby, dnsmasq is just proxying the DNSSEC information ('proxy-dnssec').
There are a bunch of sites to test DNSSEC on that you can find by just googling 'DNSSEC Test', here's a couple of them, the second one tests many other things related to DNS also:

https://dnssec.vs.uni-due.de/
https://cmdns.dev.dns-oarc.net/

@directnupe

You forgot to copy this line from my guide:

dnssec_return_status: GETDNS_EXTENSION_TRUE
appdata_dir: "/tmp/stubby"

To stubby.yml, this is what enables stubby to do dnssec validation. the appdata_dir is optional, but I recommend it, by default the keys are stored in /root, moving them to /tmp moves them to the RAM only.

From my guide: [Tutorial] DNS-over-TLS with dnsmasq and stubby (no need for unbound)

Explanation:
dnssec_return_status: GETDNS_EXTENSION_TRUE : This line provides DNSSEC validation
appdata_dir: Where the keys for DNSSEC are stored

@Specimen many thanks for the very helpful reply, and also many thanks for the original guide! I actually came across your guide first, and I think I used it and directnupe's together. I had the dnssec_return_status line in my .yml file but not the appdata_dir. You say it is is optional and that by default keys are stored in /root but my /root directory remains empty at all times.
Also many thanks for the DNSSEC test sites. Both test sites say that DNSSEC is not working!
I've added the appdata line and rebooted. It's created the /tmp/stubby directory but the directory itself is empty. Also DNSSEC still not working. Any ideas?

Is it really empty? By default the dir the keys are stored in is hidden (starts with a .), use 'ls -a' command to show these 'hidden' files and folders.

If stubby's DNSSEC validation is working you should see these files:

-rw-------    1 root     root        4.0K Aug 18 10:08 root-anchors.p7s
-rw-------    1 root     root         651 Aug 18 10:08 root-anchors.xml
-rw-------    1 root     root        1.6K Aug 19 19:34 root.key

In whatever is the dir assigned in appdata_dir or in the /root/.<I don't remenber the name> :slight_smile:

As for ideas to why it's not working, recheck all your setup, first you have to guarantee that all the dns requests from clients are going through stubby and that all the DNS servers in stubby support DNSSEC, recheck all settings.

If the dir is empty than it's likely the DNS servers you have in your config do not support DNSSEC, try with just Cloudflare to start with.

You have also to make sure TLS authentication is working, try running stubby manually on the router with the fla '-v6' (verbose mode level 6) it will spit out in the console what is doing if TLS handshakes succeed or not.

You have to manually create a root.key file containing the keys for DNSSEC to work properly. Create such file (i.e., "root.key") in /etc/stubby and include the following lines in it:

. IN DS 19036 8 2 49AAC11D7B6F6446702E54A1607371607A1A41855200FD2CE1CDDE32F24E8FB5
. IN DS 20326 8 2 E06D44B80B8F1D39A95C0B0D7C65D08458E880409BBC683457104237C7F8EC8D

Then, in the configuration file stubby.yml, include the following line:

dnssec_trust_anchors: "/etc/stubby/root.key"

That should solve it.

Cheers.

No, you don't need to do this!

If stubby is working properly it creates those by itself, downloads the trust anchors.

1 Like

Both directories /root and /tmp/stubby are empty (ls -a).
Just for sake of sanity I'm posting my stubby.yml file in case anything is amiss:

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
edns_client_subnet_private : 1
round_robin_upstreams: 1
idle_timeout: 10000
listen_addresses:
  - 127.0.0.1@5453
  -  0::1@5453
# To enable DNSSEC validation when using Stubby:
dnssec_return_status: GETDNS_EXTENSION_TRUE
appdata_dir: "/tmp/stubby"

upstream_recursive_servers:
# IPv4 addresses
# # Cloudflare servers
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
    tls_port: 853

  - address_data: 1.0.0.1
    tls_auth_name: "cloudflare-dns.com"
    tls_port: 853

Running stubby -v6 I get the following output:

[20:23:25.393209] STUBBY: 1.1.1.1                                  : Upstream   : TLS - Resps=    19, Timeouts  =     0, Best_auth =Success
[20:23:25.393405] STUBBY: 1.1.1.1                                  : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      1, Backoffs     =     0
[20:23:25.401388] STUBBY: 1.0.0.1                                  : Upstream   : TLS - Resps=    19, Timeouts  =     0, Best_auth =Success
[20:23:25.401579] STUBBY: 1.0.0.1                                  : Upstream   : TLS - Conns=     1, Conn_fails=     0, Conn_shuts=      1, Backoffs     =     0
[20:23:40.542556] STUBBY: 1.1.1.1                                  : Upstream   : TLS - Resps=    60, Timeouts  =     0, Best_auth =Success

Does this mean TLS authentication is working?

I know that is not the best way to do it but stubby for whatever reason is not creating the keys itself. And these keys, by the way, are not a secret, they are widely available in the internet as you can see here: https://www.icann.org/dns-resolvers-updating-latest-trust-anchor

1 Like

Yes, then it's better to post the link instead.

TLS is working. The config seems fine.

You're both reporting that Stubby is not downloading the keys, I can't understand immediately why this is happening.

Maybe related to the getdns version?
source: https://dnsprivacy.org/wiki/display/DP/Configuring+Stubby

DNSSEC

To enable DNSSEC validation when using Stubby add the following option to the configuration file

dnssec_return_status: GETDNS_EXTENSION_TRUE

A trust anchor is also required for DNSSEC validation.

getdns version 1.2 and later include support for automatic trust anchor management - which will automatically fetch a trust anchor if none is present on the system. See 'Zero configuration DNSSEC' (and below) for the specific details of key management for DNSSEC for this case.

If using a version of getdns earlier than 1.2 then a trust anchor must be manually installed and managed on the system. We recommend using unbound-anchor .

For my architecture, on 18.06.1, getdns is version: 1.4.2

I've just made it work with automatic installation of the keys. Use the following line in the config file stubby.yml:

appdata_dir: "/tmp"

ok many thanks @Specimen for the confirmation that TLS authentication is working at least!
My architecture is 18.06.0, and getdns version like you is 1.4.2
Are you running your "install to ram" method or your "permanent install"? I ask because the big difference is that in your "install to ram" method you use ca-bundle where as in the permanent you use ca-certificates. Could that possibly be the issue here? I installed ca-certificates.

If TLS is working than the certificates are working.
I explain in a post in my tutorial the difference between ca-bundle and ca-certificates and why they are used differently.

We can safely assume that TLS (and certs) is not the problem, it's just DNSSEC.

1 Like

Ok I think I've manage to resolve the issue!
As you see from my stubby.yml file posted above I was using tls_auth_name method and not tls_pubkey_pinset for my cloudfare servers. I had tried adding the pubkeys before but I had gotten the formatting wrong (yaml is very picky!) and it wouldn't work, so I'd just left them out thinking they were not necessary.
But it seems they are necessary for DNSSEC to work, at least on my setup. If I have either:

  1. just the tls_pubkey_pinset, or
  2. both tls_pubkey_pinset and the tls_auth_name
    then DNSSEC validates, but if I have
  3. just the tls_auth_name then DNSSEC fails

Also the 3 files:
root-anchors.p7s root-anchors.xml root.key
are now in the /tmp/stubby directory where they should be!

Many thanks for your help in resolving this issue @Specimen. Much appreciated!

1 Like

I'm glad you got it sorted... But...

It's interesting that you need pubkeys for Cloudflare, in my setup (working) I'm just using the tis_auth_name method.
Also, from your output of running stubby with -v6 the TLS auth was working ("Best_auth =Success"), so I don't quite see how the pubkeys can be related to DNSSEC.

Pairing that with rog's 'solution', of using /tmp instead of /tmp/stubby (it shouldn't make ANY difference, if that's what you actually meant, @rog , if you weren't setting the appdir in the first it means you didn't read my recommendation), I think the problem was just a temporary unavailability of the server it downloads the keys from rather than a specific setup setting.

@Specimen, you may be right. I've removed the keys (so just using tls_auth_name) and DNSSEC is still working. Temporary unavailability of the server it downloads keys from seems like a good explanation.

1 Like

Also the fact that if the keys are already there (if you didn't delete them or rebooted the router) then it just reuses them when it starts.