OpenWrt only advertises the first dns server when using dhcp option 6

justaname · November 9, 2022, 10:14am

Hi

As the title says, I can't get openwrt to advertise two DNS servers using "dhsp option 6" on my LAN interface.

I have two recursive dns resolvers on my lan which are both setup to 1) resolve local domains, 2) DNS sinkhole and 3) recursively resolve external domains:

Server (192.168.24.4): k8s-gateway (local dns) --> Pi-hole (sinkhole) --> technitium (recursive resolver)
Rpi4 (192.168.24.2): Pi-hole (local dns + sinkhole) --> unbound (recursive resolver)

If I advertise either the server or the Rpi4 on the LAN interface using 6,192.168.24.4 and 6,192.168.24.2 respectively, nslookup google.com reveals that the expected dns server is used but when I try to advertise both the server and the Rpi4 simultaneously using 6,192.168.24.4,192.168.24.2 only the first dns server is advertised.

Any ideas?

EDIT: I'm on 22.03 by the way.

frollic · November 9, 2022, 10:53am

how did you verify this ?

justaname · November 9, 2022, 12:01pm

By only adding a local entry to one of the dns servers...

I actually discovered this behavior by a mistake. When I spin up a container on my server, k8s-gateway will automatically add a dns entry but I can only resolve it when the server is specified first in dhcp option 6.

frollic · November 9, 2022, 12:07pm

A client only using one/the 1st of the DNSes provided via DHCP, isn't really proof of it not receiving both.

If you "kill" the primary DNS, will the client then fail over to the 2nd DNS' IP ?

What does the OS on the client(s) say ?

ipconfig /all
/etc/resolv.conf
etc ...

justaname · November 9, 2022, 12:28pm

I'll have a look.

I do want to note however, that it is not only one but three clients (fedora, iPhone and iPad) that cannot resolve the domain in question unless my server is specified first.

trendy · November 9, 2022, 2:08pm

opkg update; opkg install tcpdump; tcpdump -i br-lan -vvvn udp port 67
Adjust the lan interface accordingly if it is not br-lan. Then make a dhcp release/renew at a lan host to capture the packets. Post the output here in preformatted text (the </> button).

This is not correct way to test. The client expects to get the same answers from either nameserver. If you add a local record only on one of them, then you need to configure the other one to find it on the first.

eduperez · November 9, 2022, 2:13pm

Enable dnsmasq's DHCP logs, and check what is answering to the clients.
Check on the client (preferably Linux or Android) what info has received.

trendy · November 9, 2022, 4:05pm

It shows the IP it has leased, not all the options offered.

justaname · November 9, 2022, 6:30pm

Okay so the router is advertising both dns servers.

    ~  resolvectl status                                                                                                                            
Global
       Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.24.2
       DNS Servers: 192.168.24.2 192.168.24.4
        DNS Domain: lan

frollic · November 9, 2022, 6:37pm

Yay!
I guess ?

justaname · November 9, 2022, 6:40pm

opkg update; opkg install tcpdump; tcpdump -i br-lan -vvvn udp port 67

Okay let me look into that.

This is not correct way to test. The client expects to get the same answers...

Yeah I know. As I stated, it wasn't a test hut rather a mistake of me forgetting to add an entry in Pi-hole and wondering why I couldn't resolve the new subdomain as k8s-gateway adds it automatically and my router advertise both the server and the Rpi4. Then when playing around with the settings I found that it works when the server is specified first.

justaname · November 9, 2022, 6:42pm

Sort of I guess. I still would like to know why I can't resolve the subdomain unless the server is specified first.

frollic · November 9, 2022, 6:44pm

it's a client "issue", not router / OpenWRT / DNS / DHCP.

justaname · November 9, 2022, 7:19pm

Okay so tracking packages with tcpdump -i br-lan -vvvn udp port 67 while renewing the dhcp lease with sudo dhclient -v -r and sudo dhclient -v yielded this result:

tcpdump: listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
20:23:10.734733 IP (tos 0x0, ttl 64, id 52750, offset 0, flags [DF], proto UDP (17), length 328)
    192.168.24.155.68 > 192.168.24.1.67: [udp sum ok] BOOTP/DHCP, Request from 60:57:18:94:45:a8, length 300, xid 0xf20da33d, Flags [none] (0x0000)
	 Client-IP 192.168.24.155
	 Client-Ethernet-Address 60:57:18:94:45:a8
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: Release
	   Server-ID Option 54, length 4: 192.168.24.1
	   END Option 255, length 0
	   PAD Option 0, length 0, occurs 50
20:23:14.226935 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 60:57:18:94:45:a8, length 300, xid 0x6d33fd3d, Flags [none] (0x0000)
	 Client-Ethernet-Address 60:57:18:94:45:a8
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: Discover
	   Requested-IP Option 50, length 4: 192.168.24.155
	   Parameter-Request Option 55, length 13: 
	     Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	     Domain-Name, Domain-Name-Server, Hostname, YD
	     YS, NTP, MTU, Option 119
	     Default-Gateway
	   END Option 255, length 0
	   PAD Option 0, length 0, occurs 35
20:23:17.694480 IP (tos 0xc0, ttl 64, id 59244, offset 0, flags [none], proto UDP (17), length 329)
    192.168.24.1.67 > 192.168.24.155.68: [bad udp cksum 0xb333 -> 0xdf41!] BOOTP/DHCP, Reply, length 301, xid 0x6d33fd3d, Flags [none] (0x0000)
	 Your-IP 192.168.24.155
	 Server-IP 192.168.24.1
	 Client-Ethernet-Address 60:57:18:94:45:a8
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: Offer
	   Server-ID Option 54, length 4: 192.168.24.1
	   Lease-Time Option 51, length 4: 3600
	   RN Option 58, length 4: 1800
	   RB Option 59, length 4: 3150
	   Subnet-Mask Option 1, length 4: 255.255.255.0
	   BR Option 28, length 4: 192.168.24.255
	   Default-Gateway Option 3, length 4: 192.168.24.1
	   Domain-Name Option 15, length 3: "lan"
	   Domain-Name-Server Option 6, length 8: 192.168.24.2,192.168.24.4
	   END Option 255, length 0
20:23:17.696190 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 60:57:18:94:45:a8, length 300, xid 0x6d33fd3d, Flags [none] (0x0000)
	 Client-Ethernet-Address 60:57:18:94:45:a8
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: Request
	   Server-ID Option 54, length 4: 192.168.24.1
	   Requested-IP Option 50, length 4: 192.168.24.155
	   Parameter-Request Option 55, length 13: 
	     Subnet-Mask, BR, Time-Zone, Classless-Static-Route
	     Domain-Name, Domain-Name-Server, Hostname, YD
	     YS, NTP, MTU, Option 119
	     Default-Gateway
	   END Option 255, length 0
	   PAD Option 0, length 0, occurs 29
20:23:17.697766 IP (tos 0xc0, ttl 64, id 59245, offset 0, flags [none], proto UDP (17), length 329)
    192.168.24.1.67 > 192.168.24.155.68: [bad udp cksum 0xb333 -> 0xdc41!] BOOTP/DHCP, Reply, length 301, xid 0x6d33fd3d, Flags [none] (0x0000)
	 Your-IP 192.168.24.155
	 Server-IP 192.168.24.1
	 Client-Ethernet-Address 60:57:18:94:45:a8
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: ACK
	   Server-ID Option 54, length 4: 192.168.24.1
	   Lease-Time Option 51, length 4: 3600
	   RN Option 58, length 4: 1800
	   RB Option 59, length 4: 3150
	   Subnet-Mask Option 1, length 4: 255.255.255.0
	   BR Option 28, length 4: 192.168.24.255
	   Default-Gateway Option 3, length 4: 192.168.24.1
	   Domain-Name Option 15, length 3: "lan"
	   Domain-Name-Server Option 6, length 8: 192.168.24.2,192.168.24.4
	   END Option 255, length 0
20:23:23.252390 IP (tos 0x0, ttl 64, id 248, offset 0, flags [DF], proto UDP (17), length 292)
    192.168.24.73.68 > 192.168.24.1.67: [udp sum ok] BOOTP/DHCP, Request from 34:98:b5:9a:51:04, length 264, xid 0xc265403, secs 1800, Flags [none] (0x0000)
	 Client-IP 192.168.24.73
	 Client-Ethernet-Address 34:98:b5:9a:51:04
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   Lease-Time Option 51, length 4: 3600
	   DHCP-Message Option 53, length 1: Request
	   Parameter-Request Option 55, length 3: 
	     Subnet-Mask, Default-Gateway, Lease-Time
	   Client-ID Option 61, length 7: ether 34:98:b5:9a:51:04
	   END Option 255, length 0
20:23:23.253895 IP (tos 0xc0, ttl 64, id 60154, offset 0, flags [none], proto UDP (17), length 328)
    192.168.24.1.67 > 192.168.24.73.68: [bad udp cksum 0xb2e0 -> 0x86de!] BOOTP/DHCP, Reply, length 300, xid 0xc265403, secs 1800, Flags [none] (0x0000)
	 Client-IP 192.168.24.73
	 Your-IP 192.168.24.73
	 Server-IP 192.168.24.1
	 Client-Ethernet-Address 34:98:b5:9a:51:04
	 Vendor-rfc1048 Extensions
	   Magic Cookie 0x63825363
	   DHCP-Message Option 53, length 1: ACK
	   Server-ID Option 54, length 4: 192.168.24.1
	   Lease-Time Option 51, length 4: 3600
	   RN Option 58, length 4: 1687
	   RB Option 59, length 4: 3037
	   Subnet-Mask Option 1, length 4: 255.255.255.0
	   BR Option 28, length 4: 192.168.24.255
	   Default-Gateway Option 3, length 4: 192.168.24.1
	   END Option 255, length 0
	   PAD Option 0, length 0, occurs 14
^C
7 packets captured
7 packets received by filter
0 packets dropped by kernel

In case the relevant the output of sudo dhclient -v -r and sudo dhclient -v was:

    ~  sudo dhclient -v -r
Killed old client process
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/wlp2s0/60:57:18:94:45:a8
Sending on   LPF/wlp2s0/60:57:18:94:45:a8
Sending on   Socket/fallback
DHCPRELEASE of 192.168.24.155 on wlp2s0 to 192.168.24.1 port 67 (xid=0xd62f3a5c)
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory

    ~  sudo dhclient -v
Internet Systems Consortium DHCP Client 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
Listening on LPF/wlp2s0/60:57:18:94:45:a8
Sending on   LPF/wlp2s0/60:57:18:94:45:a8
Sending on   Socket/fallback
DHCPDISCOVER on wlp2s0 to 255.255.255.255 port 67 interval 4 (xid=0x46dd9c4e)
DHCPDISCOVER on wlp2s0 to 255.255.255.255 port 67 interval 10 (xid=0x46dd9c4e)
DHCPOFFER of 192.168.24.155 from 192.168.24.1
DHCPREQUEST for 192.168.24.155 on wlp2s0 to 255.255.255.255 port 67 (xid=0x46dd9c4e)
DHCPACK of 192.168.24.155 from 192.168.24.1 (xid=0x46dd9c4e)
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
/usr/sbin/dhclient-script: line 706: /etc/resolv.conf: Permission denied
bound to 192.168.24.155 -- renewal in 1567 seconds.

EDIT: I just redid the test as I realized that another client also renewed lease while running the test.

krazeh · November 9, 2022, 7:54pm

It's probably worth noting that if the primary (or first) DNS server responds with a valid response, including name not found, then many clients will not check the secondary DNS. That'd only happen if the primary didn't respond at all afaik.

frollic · November 9, 2022, 7:57pm

Yeah, I see this too, on my two piholes.

The client will keep asking the 2nd dns, even if the 1st one comes back up.
Not entirely sure for how long, but by the looks of it, until the next DHCP renewal request is sent.

trendy · November 9, 2022, 9:49pm

Doesn't matter. Any packet would do. And we can see that they are advertised properly a few times.

The sequence that the client will probe multiple upstream nameservers is not standard.

Might try primary and in case of timeout to try secondary.
Might try both simultaneously.
Might persist on the one that replied last time and if it fails to try the other.

We tend to avoid using primary and secondary for failover, for these reasons of unpredictable client behaviour, and there are load balancers in front of a dns server pool.

justaname · November 10, 2022, 7:08am

Okay, so now it is definitely confirmed that the router does advertise both dns servers. Thanks a lot for your help.

Yeah I guess this is infarct the exact behavior I am experiencing .

eduperez · November 10, 2022, 10:25am

It does, when you enable logging:

Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 DHCPREQUEST(br_ext) 192.168.7.138 fe:15:13:3d:23:2c
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 tags: ext, br_ext
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 DHCPACK(br_ext) 192.168.7.138 fe:15:13:3d:23:2c
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 requested options: 1:netmask, 121:classless-static-route, 3:router,
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 requested options: 6:dns-server, 15:domain-name, 108, 114, 119:domain-search,
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 requested options: 252
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 next server: 192.168.7.254
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  1 option: 53 message-type  5
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option: 54 server-identifier  192.168.7.254
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option: 51 lease-time  12h
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option: 58 T1  5h17m32s
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option: 59 T2  9h47m32s
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option:  1 netmask  255.255.255.0
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option: 28 broadcast  192.168.7.255
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option:  3 router  192.168.7.254
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  4 option:  6 dns-server  192.168.7.254
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  8 option: 15 domain-name  intranet
Thu Nov 10 11:24:30 2022 daemon.info dnsmasq-dhcp[1]: 263026743 sent size:  1 option:252   0a

trendy · November 10, 2022, 1:12pm

Even with logging enabled I don't see so much detail

Thu Nov 10 14:05:13 2022 daemon.info dnsmasq-dhcp[4591]: DHCPREQUEST(br-lan) 10.0.2.101 86:8c:40
Thu Nov 10 14:05:13 2022 daemon.info dnsmasq-dhcp[4591]: DHCPACK(br-lan) 10.0.2.101 86:8c:40:6b: