WiFi connected, but can't even access AP IP

Using OpenWrt for the first time to try and improve roaming on my home wifi. I have 2 TP-Link Archers C6 v2 (not optimal, I know, but I got them super cheap to try this out), both setup as APs (neither of them is my main router). In order to fit both wpad-full and DAWN on the 8MB flash these routers have, I built a custom image of OpenWrt 23.05.5, where I removed dnmasq, replaced the "kmod-ath10k-ct" and "ath10k-firmware-qca9888-ct" with "non-ct" variants as suggested on the page for this router, replaced wpad-basic with wpad-full and added DAWN. Everything seemed really good, the roaming was fantastic, speeds were impressive. But then I came across a random issue, where I'd still be connected to wifi, but unable to connect anywhere, I couldn't even ping the AP I was connected to. And this seemed to sometimes only affect a single band, like only the 5GHz would stop working, while the 2,4GHz would still be fine, while other times nothing would work. Sometimes I just needed to restart the interface, sometimes I had to turn off the AP using the physical button and turn it back on again (I couldn't access the web interface at all even using a cable). It doesn't seem like there's any kind of a pattern in it stopping working and it's happening too often to ignore it - like 2-3 times a day. The usage doesn't seem to go over 50% and I even tried fully disabling DAWN with no effect. I'll be very thankful for any ideas what might be causing these issues and how to solve them.