Boot hangs in grub on x86 with no serial (Openwrt 19.07.3)

[pdecat]

Hi,

after each sysupgrade, the boot of my x86 system with a USB keyboard and VGA display, but no serial console, hangs in grub's selection menu. Getting the system to boot requires physically pressing the Enter key on the keyboard.

Each time, I have to apply the work-around from [Solved] Boot "hangs" on 'LEDE' item in grub menu on x86 (no grub 'timeout'), that is to comment the first two lines in /boot/grub/grub.cfg:

cat /boot/grub/grub.cfg
#serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
#terminal_input console serial; terminal_output console serial

set default="0"
set timeout="5"
set root='(hd0,msdos1)'

menuentry "OpenWrt" {
        linux /boot/vmlinuz root=PARTUUID=339542e9-02 rootfstype=ext4 rootwait console=tty0 console=ttyS0,115200n8 noinitrd
}
menuentry "OpenWrt (failsafe)" {
        linux /boot/vmlinuz failsafe=true root=PARTUUID=339542e9-02 rootfstype=ext4 rootwait console=tty0 console=ttyS0,115200n8 noinitrd
}

This is with a setup freshly upgraded from Openwrt 19.07.2 to 19.07.3, but I had the same issue with previous releases too.

Last comment from the above mentioned thread stated that this could be investigated on current releases so here I am

Is there any thing else to check or should I just report a bug in the issue tracker?

Note: the other work-around to set the timeout to 0 also works.

[pdecat]

Still current with Openwrt 19.07.4, same work-around works.

[lleachii]

@pdecat, welcome to the community!

I'm not sure why you consider this a workaround or a bug. You clearly understand why you have to disable serial on a machine without it...but yes, feel free to make a bug report. It doesn't hurt.

[pdecat]

Hi @lleachii, thanks!

Well, I feel this is a bug because it requires physical access to the machine after each sysupgrade and this can't the nominal behavior. I've got 6 other devices of several brands/models that do not expose that behavior.

And I consider the solution here a work-around because it just avoids physical access after subsequent reboots, but does not help during upgrades.

Or is there a way to disable serial in /boot/grub/grub.cfg after a sysupgrade but before reboot?

The only way I can think of right now would be to add /boot/grub/grub.cfg to the list of files to restore in /etc/sysupgrade.conf but that feels risky, isn't it?

[lleachii]

@pdecat

Well, I feel this is a bug because it requires physical access to the machine after each sysupgrade and this can't the nominal behavior.

Of course, since you don't have a serial port (you're making it seem like you don't understand that).

@pdecat

Or is there a way to disable serial in /boot/grub/grub.cfg after a sysupgrade but before reboot?

Ummmm...yea!

Simply make the image with the proper config for you! Then you can Sysupgrade all day remotely (which is not advised anyways, physical access is always suggested)!

@pdecat

The only way I can think of right now would be to add /boot/grub/grub.cfg to the list of files to restore in /etc/sysupgrade.conf but that feels risky, isn't it?

Not possible...well it is an x86, so I would say...not suggested, but why not try!

It should work with the ext4 image... ....oops...my apologies...you cannot sysupgrade that nope...you can't edit the files in the image.

[anon50098793]

grep 'sed' /lib/upgrade/platform.sh

[pdecat]

Hi @anon50098793, I've checked and this command does not return anything on 6 of my 7 devices:

• PC Intel Core2 Duo (x86/64)
• Netgear R7800 (ipq806x/generic)
• TP-Link TL-WDR4900 (mpc85xx/generic)
• TP-Link Archer C7 v5 (ath79/generic)
• Netgear WN2500RP (brcm47xx/mips74k)
• TP-Link EAP225-Outdoor (ath79/generic)

The only device for which it returns something is my Linksys EA8300 (ipq40xx/generic):

# grep 'sed' /lib/upgrade/platform.sh
                part="$(awk -F 'ubi.mtd=' '{printf $2}' /proc/cmdline | sed -e 's/ .*$//')"

Not sure how that's supposed to help.

Are you suggesting there are some platforms for which there are hooks to perform changes before reboot?

[pdecat]

@lleachii

Of course, since you don't have a serial port (you're making it seem like you don't understand that).

I know I don't have a serial console, I'm just surprised it is assumed all x86-64 setups have one.
Maybe it could be checked during the sysupgrade process to adapt the grub.cfg accordingly. Hence the usefulness of a bug report. I'll open one if I do not find any existing solution my issue.

@lleachii

Simply make the image with the proper config for you! Then you can Sysupgrade all day remotely (which is not advised anyways, physical access is always suggested)!

That's actually my long term plan, and I'm eyeing on https://github.com/aparcar/asu for that.

[vgaetera]

@pdecat

#serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
#terminal_input console serial; terminal_output console serial

if serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
then terminal_input console serial; terminal_output console serial
fi

How about this?

[pdecat]

I came up with the following dirty hack in /etc/rc.local that will patch /lib/upgrade/platform.sh on the firstboot after a sysupgrade. Then the next sysupgrade will in turn patch /boot/grub/grub.cfg before rebooting:

if [ ! -f /root/firstboot ] ; then
  # Patch /lib/upgrade/platform.sh
  cp -a /lib/upgrade/platform.sh /lib/upgrade/platform.sh.orig
  sed -i "/^\t\tmount -t ext4 -o rw,noatime \"\/dev\/\$partdev\" \/mnt/a \\\t\tsed -i 's\/^serial\/#serial\/g' \/mnt\/boot\/grub\/grub.cfg\n\t\tsed -i 's\/^terminal_input\/#terminal_input\/g' \/mnt\/boot\/grub\/grub.cfg" /lib/upgrade/platform.sh

  # Mark first boot complete
  touch /root/firstboot
fi

The resulting patch:

diff -u /lib/upgrade/platform.sh.orig /lib/upgrade/platform.sh
--- /lib/upgrade/platform.sh.orig       2020-09-06 16:19:39.000000000 +0000
+++ /lib/upgrade/platform.sh    2020-09-27 08:45:53.299999573 +0000
@@ -39,6 +39,8 @@

        if export_partdevice partdev 1; then
                mount -t ext4 -o rw,noatime "/dev/$partdev" /mnt
+               sed -i 's/^serial/#serial/g' /mnt/boot/grub/grub.cfg
+               sed -i 's/^terminal_input/#terminal_input/g' /mnt/boot/grub/grub.cfg
                cp -af "$UPGRADE_BACKUP" "/mnt/$BACKUP_FILE"
                umount /mnt
        fi

Also, /etc/rc.local is added to /etc/sysupgrade.conf to be restored after sysupgrades. (edit: not needed, /etc/rc.local is restored by default on sysupgrades).

[pdecat]

@vgaetera

How about this?

That looks cleaner and if it works in both situations, i.e. with and without a serial console, maybe that could be implemented in openwrt images for x86 platforms.

[pdecat]

@vgaetera

if serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
then terminal_input console serial; terminal_output console serial
fi

Just tried that on my setup without a serial console and it did not work, boot was hung on grub menu like before.

[vgaetera]

@pdecat

sed -i 's/^serial/#serial/g' /mnt/boot/grub/grub.cfg
sed -i 's/^terminal_input/#terminal_input/g' /mnt/boot/grub/grub.cfg

sed -i -r -e "/^(serial|terminal_input)\s/s/^/#/" /mnt/boot/grub/grub.cfg

Note that one-time scripts are simpler to implement with uci-defaults:
https://openwrt.org/docs/guide-developer/uci-defaults

[pdecat]

Good to know, but it won't help for my use case as the scripts in /etc/uci-defaults are deleted upon success or re-executed on next boot until success. I need the script to be executed only once at each first boot after sysupgrade, but not deleted.