Hi folks, just a PSA here in the hopes that it saves someone the weeks of effort I went through trying to figure this out:
TL;DR: My ISP did "upgrades" which included refusing DHCP to any device with the U/L bit set in its MAC address. By default, my OpenWrt OS changed my WAN MAC from the device's original hardware
14:xx:xx:xx:xx:xx address to a "locally administered"
16:xx:xx:xx:xx:xx address, and caused very weird DHCP behavior until I figured it out. Is there a reason this bit is set on the WAN MAC address by default on a fresh OpenWrt installation? Is it normal/expected for my ISP to refuse service to such an address? What's the best fix here in folks' opinion?
I have a Linksys WRT1200AC that I originally loaded with 22.03.5, then successfully upgraded to 23.05.0 with no issues. About a month ago, my ISP (Xfinity/Comcast in northern New England) sent out an email that we should expect outages as they "perform upgrades."
The outage happened, I turned off my router and modem (Netgear CM600) for a while to wait it out, and when I turned them back on (the modem first, waiting for all the LEDs, then the router, as usual, that will become important) I could not connect to the internet through the router. Now the weird part is that if I connected my laptop directly to the modem via ethernet cable, it connected to the internet with no issues.
So, I hooked the router back up to the modem, power cycle them as before, and log in to LuCI, only to find that my router WAN has no IP address assigned... So I download tcpdump to take a closer look at things only to find that my router is sending DHCP Discover packets, but none are being returned by the modem/ISP, so then I hook the laptop up to the modem again and use wireshark to find that, as expected, the whole DHCP exchange is working perfectly, so I figure there must be something wrong with the router DCHP Discover packets.
<tangent> Now, where this gets really weird is that, by some miracle, if I turn the router on before the modem, everything just works, and I get my router WAN IP assigned via DHCP just fine. </tangent>
Well anyway, this reduced the urgency of finding a fix, but I couldn't let it go, and after weeks of tweaking the router DHCP settings to match the laptop DHCP settings and getting nowhere, I replayed a bunch of raw DHCP packets from my laptop and started tweaking bits to see what would get my modem to send a DHCP Offer request, and as the title suggests, it wasn't the DHCP Discover request at all, it was the fact that the seventh most significant bit (the U/L bit) of my router's MAC address was set, and so my ISP was just refusing to continue the DHCP negotiation until I manually unset it.
<tangent-explanantion> But what about the case when I turned the router on before the modem? Why was it working then? Well, I honestly have no idea, but after packet-sniffing that scenario, it seems that it has something to do with the modem... So this modem, like many others, allows you to go to 192.168.100.1 in your browser in order to manage it via HTTP. In order to do this, if you are connected to it as it boots, it will answer a DHCP negotiation by offering an address of 192.168.100.10 to the host connected to it, so that the modem can be reached by that host's browser. Anyhow, for some reason, when this modem DHCP negotiation succeeds, the ISP DHCP negotiation also succeeds, and everything just works. Maybe the modem passes on its MAC address to the ISP in this case to get a DHCP IP assigned? Who knows. </tangent-explanation>
The more important question to me at this point is, what's the best fix here? Should OpenWrt not be setting the U/L bit on the WAN MAC address by default? Does anyone know where or why this happens? Or should I call my ISP and see if I can get an explanation from them, and convince them to respond to DHCP negotiations from "locally administered" MAC addresses?
Thanks in advance for any insight, and sorry for the long read!
EDIT: It seems that the issue wasn't the U/L bit, but instead the fact that the WAN MAC was at all different than the LAN MACs, see @psherman's post below.