2025-April-20: Thanks to @aparcar and @efahl, ASU/OWUT and customized images via the firmware selector should now be working properly. Please report any issues here.
2025-April-19: There is a known issue that affects customized image building via the firmware selector and Attended Sysupgrade (ASU) / OpenWrt Upgrade Tool (OWUT). The system will queue build requests, but does not appear to be executing. As such, the queue has become very long and obviously nobody is able to obtain a customized image.
Please do not create new threads about this issue since it is already known and being discussed here and here.
I believe this issue is completely resolved. @aparcar worked on a bunch of stuff to clear the issue, and I've been running tests for a couple of hours without issue. We're putting some stuff into place to eliminate (fingers crossed) the problem in future.
Root cause appears to be server disk filling with build artifacts under high load, like when 24.10.1 was released (we've seen this before and thought it was under control, oops). The first pass has reduced the time-to-live values for builds (that change is live now). We're looking at ways to implement a more effective janitorial service to clear out old junk as soon as it has expired, rather than just doing a once-per-day sweep. There are also some prophylaxis bits to limit the frustration when things do go wrong (hey, it's software, which means there will always be bugs).
(If anyone is curious, just ask, I can point out PRs/code changes and describe what they accomplish.)
Thanks for your work @efahl and @aparcar ! I've amended the banner post and I'll keep it live for a day or two just in case any additional issues remain/arise.