I'm trying to tamp down an issue with dnsmasq's handling of DHCP while running a macvlan interface. DHCP is handled on the router. DNS is via pihole, currently on an ubuntu VM on my homelab server, but I'm trying to make it a docker container on the router.
In my current network, I have
10.19.76.1 gateway (APU2) on openwrt 21.02.1, has the docker bits and kmod-macvlan installed.
10.19.76.2 Linksys EA6350 as an AP (no firewall/dnsmasq/odhcpd/etc) on openwrt 21.02.1
10.19.76.3 Linksys EA8300 as an AP (no firewall/dnsmasq/odhcpd/etc) on openwrt 21.02.1
10.19.76.4 macvlan interface for docker containers
10.19.76.5 docker pihole container
10.19.76.6 managed PoE switch
10.19.76.13 an ubuntu VM with pihole/unbound as the current DNS server for the lan.
The APU2 router is running dnsmasq for DHCP, with DNS listening disabled, handing out DHCP addresses from 10.19.76.128/25. Everything below 10.19.76.128 is via static IP or DHCP reservation configuration.
The pihole VM IP is advertised via DHCP options, with the br-lan/WAN interfaces using it, and firewall rules to keep hardcoded devices from using alternate DNS servers.
I'm trying to consolidate and get pihole running as a docker container on my primary router and decommission the VM since that's much heavier than a container. I've tried following along with two guides:
- This one here on the openwrt forums that's running on an x86 install supposedly,
- This one that's run on a raspberry pi using VLANs to create a LAN and WAN interface off of the single ethernet device.
I can get the 10.19.76.5 docker container up and running, and pingable from the host (aka the gateway router). Routing works, and staticly-configured devices run fine. The issue is DHCP clients, whether they are random-assigned or have a DHCP reservation, cannot renew or acquire an IP from the router.
The phone below has a DHCP reservation IP of 10.19.76.98 in the router, but dnsmasq starts churning through DHCP pool IPs as it and the phone keep offering and declining. The br-lan.20 device is a macvlan interface with a static 10.19.76.4 IP, br-lan as the parent. br-lan has two devices it is bridging: eth1 and eth2 (eth0 is WAN). Nothing is connected to the eth2 port, the core switch is connected to eth1. The AP devices are connected to the switch. In past experiences, devices that were connected to the eth2 interface showed up in pihole queries with the router hostname as the originator, rather than the device's hostname. So everything got moved to a bigger, single switch on eth1.
A snippet of the system log while the br-lan.20 interface was up:
Mon Nov 7 18:46:28 2022 daemon.info dnsmasq-dhcp[31526]: DHCPDISCOVER(br-lan) f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:28 2022 daemon.info dnsmasq-dhcp[31526]: DHCPOFFER(br-lan) 10.19.76.220 f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:28 2022 daemon.warn dnsmasq-dhcp[31526]: not using configured address 10.19.76.98 because it was previously declined
Mon Nov 7 18:46:28 2022 daemon.info dnsmasq-dhcp[31526]: DHCPDISCOVER(br-lan.20) f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:28 2022 daemon.info dnsmasq-dhcp[31526]: DHCPOFFER(br-lan.20) 10.19.76.220 f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:29 2022 daemon.info dnsmasq-dhcp[31526]: DHCPREQUEST(br-lan) 10.19.76.220 f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:29 2022 daemon.info dnsmasq-dhcp[31526]: DHCPACK(br-lan) 10.19.76.220 f8:87:f1:cb:e9:2a iPhone11
Mon Nov 7 18:46:29 2022 daemon.info dnsmasq-dhcp[31526]: DHCPREQUEST(br-lan.20) 10.19.76.220 f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:29 2022 daemon.info dnsmasq-dhcp[31526]: DHCPNAK(br-lan.20) 10.19.76.220 f8:87:f1:cb:e9:2a wrong server-ID
Mon Nov 7 18:46:31 2022 daemon.info dnsmasq-dhcp[31526]: DHCPDECLINE(br-lan) 10.19.76.220 f8:87:f1:cb:e9:2a
Mon Nov 7 18:46:54 2022 daemon.warn dnsmasq-dhcp[31526]: not using configured address 10.19.76.98 because it was previously declined
Here's what the syslog looks like as the macvlan interface is brought down and clients connect again:
Wed Nov 9 12:13:22 2022 daemon.warn dnsmasq-dhcp[27040]: not using configured address 10.19.76.98 because it was previously declined
Wed Nov 9 12:13:22 2022 daemon.info dnsmasq-dhcp[27040]: DHCPDISCOVER(br-lan) f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:22 2022 daemon.info dnsmasq-dhcp[27040]: DHCPOFFER(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:23 2022 daemon.info dnsmasq-dhcp[27040]: DHCPREQUEST(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:23 2022 daemon.info dnsmasq-dhcp[27040]: DHCPACK(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a iPhone11
Wed Nov 9 12:13:48 2022 daemon.warn dnsmasq-dhcp[27040]: not using configured address 10.19.76.98 because it was previously declined
Wed Nov 9 12:13:48 2022 daemon.info dnsmasq-dhcp[27040]: DHCPDISCOVER(br-lan) f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:48 2022 daemon.info dnsmasq-dhcp[27040]: DHCPOFFER(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:49 2022 daemon.info dnsmasq-dhcp[27040]: DHCPREQUEST(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a
Wed Nov 9 12:13:49 2022 daemon.info dnsmasq-dhcp[27040]: DHCPACK(br-lan) 10.19.76.146 f8:87:f1:cb:e9:2a iPhone11
Wed Nov 9 12:13:52 2022 daemon.notice netifd: Interface 'macvlan' is now down
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Withdrawing address record for 10.19.76.4 on br-lan.20.
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Leaving mDNS multicast group on interface br-lan.20.IPv4 with address 10.19.76.4.
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Interface br-lan.20.IPv4 no longer relevant for mDNS.
Wed Nov 9 12:13:52 2022 kern.info kernel: [17089.618720] device br-lan left promiscuous mode
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Interface br-lan.20.IPv6 no longer relevant for mDNS.
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Leaving mDNS multicast group on interface br-lan.20.IPv6 with address fe80::20d:b9ff:fe4a:c397.
Wed Nov 9 12:13:52 2022 daemon.info avahi-daemon[3483]: Withdrawing address record for fe80::20d:b9ff:fe4a:c397 on br-lan.20.
Wed Nov 9 12:13:52 2022 daemon.notice netifd: Interface 'macvlan' is disabled
Wed Nov 9 12:13:52 2022 daemon.notice netifd: macvlan 'br-lan.20' link is down
Wed Nov 9 12:13:52 2022 daemon.notice netifd: Interface 'macvlan' has link connectivity loss
Wed Nov 9 12:13:54 2022 daemon.info dnsmasq-dhcp[27040]: DHCPDISCOVER(br-lan) b6:dc:80:62:c5:04
Wed Nov 9 12:13:54 2022 daemon.info dnsmasq-dhcp[27040]: DHCPOFFER(br-lan) 10.19.76.214 b6:dc:80:62:c5:04
I read a thread here that the EA6350/EA8300 ipq40xx devices have issues with VLANs, but the DHCP issue strikes all clients including hardwired ethernet devices.
So long as a DHCP lease is active, the devices have no connectivity issues. It's when they renew or join fresh that DHCP negotiation fails. In the DHCP server settings, I've tried setting the Listen Interfaces and Exclude Interfaces, but that fails. I've tried removing eth2 from br-lan and making it it's own interface with the macvlan attached to it, but that fails too. Tried setting a manual MAC address for the macvlan interface, same thing. Devices can't negotiate a DHCP lease. As soon as I disable the macvlan interface, devices can connect successfully again.
The host router needs to be able to communicate with the pihole docker container as I do a great deal of ssh SOCKS proxy tunneling from a remote site, and if the router can't talk to the container, DNS resolution for that remote device fails.
spicy bits of /etc/config/network
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth1'
list ports 'eth2'
config interface 'lan'
option device 'br-lan'
option proto 'static'
list dns '10.19.76.13'
list ipaddr '10.19.76.1/24'
config interface 'macvlan'
option proto 'static'
option defaultroute '0'
option netmask '255.255.255.255'
option device 'br-lan.20'
option ipaddr '10.19.76.4'
config device
option type 'macvlan'
option ifname 'br-lan'
option mode 'bridge'
option name 'br-lan.20'
option acceptlocal '1'
option macaddr '00:0D:B9:4A:C3:97'
config route
option interface 'macvlan'
option target '10.19.76.5'
option netmask '255.255.255.255'
relevant /etc/config/dhcp
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option localservice '1'
option ednspacket_max '1232'
option confdir '/tmp/dnsmasq.d'
list server '10.19.76.13'
option port '0'
list addnhosts '/etc/ethers'
list interface 'br-lan'
list notinterface 'br-lan.20'
config dhcp 'lan'
option interface 'lan'
option leasetime '12h'
option dhcpv4 'server'
list dhcp_option '6,10.19.76.13'
list dhcp_option '4,10.19.76.1'
list dhcp_option '42,10.19.76.1'
option start '128'
option limit '127'
list ra_flags 'none'
...
config host
option name 'iPhone11'
option dns '1'
option mac 'F8:87:F1:CB:E9:2A'
option ip '10.19.76.98'
option leasetime '12:00:00'
the docker-compose.yml for creating the pihole container:
root@Gateway:~# cat docker-compose.yml
version: "3.3"
# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
hostname: gatewaypihole.lan
environment:
TZ: 'America/New York'
WEBPASSWORD: '123456'
# Volumes store your data between container upgrades
volumes:
- './pihole/etc-pihole/:/etc/pihole/'
- './pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/'
- './pihole/var-log/:/var/log'
- './pihole/etc-cont-init.d/10-fixroutes.sh:/etc/cont-init.d/10-fixroutes.sh'
# Recommended but not required (DHCP needs NET_ADMIN)
# https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
cap_add:
- NET_ADMIN
restart: unless-stopped
networks:
internal:
lan:
ipv4_address: 10.19.76.5
networks:
internal:
lan:
name: lan
driver: macvlan
driver_opts:
parent: br-lan.20
ipam:
config:
- subnet: 10.19.76.0/24
).