For a little while now (I’d say a few years at least), everytime I’m trying to upgrade OpenWRT in-place (either via sysupgrade or recently owut), the system hangs at the first startup and I’m forced to reinstall from scratch and restore configuration afterwards.
Subsequent reboots does not affect the system at all, it just works. It’s just the very first boot after an upgrade that systematically hangs.
The only messages displayed on the serial console make me think of an issue with /etc/urandom.seed or RNG in general. I’m currently using OpenWRT on an Alix apu1d4 with default packages for random (urngd, urandom-seed-3, etc…), I don’t know if it’s relevant or not.
Have you had this issue? Have you a solution to suggest?
Debugging boot issues is always a nightmare. I'm digging through https://github.com/openwrt/procd/blob/main/initd/init.c (procd init) for the first time, the only thing I can see is that maybe one of the kmodloader packages is hanging???
If the random seed cannot not be made then the boot process hangs making a random seed on low end routers can take some time dependant on the way it is made and can be the most time consuming part of the boot process so it could be the problem in your case.
So first of all give the boot process some more time (e.g. 10-20 minutes) but it might be something in your config which is missing to make the random seed.
I think the random seed is made at first boot so that could explain it
Is there any way to monitor this eventually? Like can I open another terminal and check what processes are running at boot time to check this?
Isn’t the random.seed from previous version not preserved when upgrading? That’s what puzzling me the most: not issue on clean install, issue on upgrade, this does not make any sense to me
No, it is explicitly removed from the backup. (My guess is that it's best practice to do so, in case the device was compromised and the value was somehow poisoned, but that seems like an extremely roundabout compromise...)
In any case, you could experiment with this as a root cause by doing two things:
echo "/etc/urandom.seed" >> /etc/sysupgrade.conf
Edit /sbin/sysupgrade and remove that line "just in case" (this only matters with certain sysupgrade options, so is really optional).