within the past two weeks I upgraded from 23->24->25 using attending sysupgrade in LuCI.
Each time I've had the system reboot into a read-only filesystem (ext4). I don't really understand what is causing this and each time requires me to usb boot a sys-rescue iso and e2fsck the root filesystem.
I'd really like a way to just filesystem check on boot but in other threads I've found on the forums the two techniques I've seen described, using tune2fs to schedule regular checks, and modifying fstab, don't seem to work for people trying it.
What is the best way to solve this problem going forward ?
maybe some way of using the failsafe boot image or grub?
I have been wanting to build a image for my x86 systems that has 2 file system data partitions, with a main and alternate "image" that you can swap between when upgrading or also optionally at boot time, with the boot/efi partition, that has both kernels on it. I know several people were talking about doing this and were attempting to make it a build option or something but I do not know if it ever happened. Probably are some nuances I am not thinking of right now
To answer your original question, your options are to:
Build a custom image with initramfs support. This requires building from source (make menuconfig → enable initramfs support and fsck tools) then add the applicable modules and hooks. Could probably copy a hook script from Ubuntu or something.
You could maybe try to add fsck.mode=force to the grub.cfg kernel command line but I have no clue if this will work.
I always ended up just doing a manual install of a new image on x86 if I wanted to do a major version upgrade or something. Usually I would temporarily take the system offline and do a full disk write of the new combined-image to the hard disk, although you could download/build the separate components (kernel, filesystem, etc...) and write the new filesystem to the correct partition and the new kernel file in the boot partition. Save any config files beforehand. Openwrt is made primarily to run on a read only squashfs filesystems with a writable overlay on embedded devices. I still much prefer to use it on x86 systems in some use cases because it runs very fast and can handle stuff like SQM on gigabit connections without trouble. However, I am not sure how much time has been put into makiong sure everyting works as well on a ext4 writible filesystem image.
<blah>
set default="0"
set timeout="5"
search -l kernel -s root
menuentry "OpenWrt 1a" {
linux /boot/vmlinuz root=PARTUUID=whatever1 rootwait console=tty0 console=ttyS0,115200n8 noinitrd microcode.force_minrev=0
}
menuentry "OpenWrt 1b" {
linux /boot/vmlinuz2 root=PARTUUID=whatever2 rootwait console=tty0 console=ttyS0,115200n8 noinitrd microcode.force_minrev=0
}
I use ImageBuilder because image has too many packages to fit in owut builds.
owut list on router to get list of packages
use list on a downloaded ImageBuilder (in my case extending root partition to 2GB)
If owut builds work for you, use download instead of upgrade and use the image if you were to follow steps below.
To upgrade:
save settings from current router
check grub.conf to remember which partition is in use (the set default="0" line)
dd the new image to the partition not in use
resize the filesystem (can't remember the command... one liner involving a loop device) & e2fsck.
mount the partition to /mnt/new (or wherever). Untar the saved settings to the new filesystem. Unmount.
copy the new kernel to /boot overwriting the one not in use
update grub.conf so set default="1" given what it has now
cross fingers and reboot
Once router comes back up, I add: /usr/sbin/e2fsck -f -v -y /dev/sda2 (or sda3)
to /lib/preinit/80_mount_root just before the last line which invokes do_mount_root.
This because we've had crashes due to power outages which left the filesystem in RO state. Internet access requires a couple docker containers running, which won't on a RO filesystem. Your post reminded me of this so there goes the flame magnet.
Overall simple to upgrade. Would be great if owut would handle this but no biggie as is.
Same, and it's not a big deal except you probably need some way of getting console/direct access to the machine when it first boots in order to get the saved config back onto it. This is mostly an issue for me with virtual machines, but these days it's actually pretty easy to get a quick vnc server up on the host you can look at arch linux headless example as an example you can just 'pacman -S wayvnc labwc foot', and then you just might need to replace the ip address on the last line with the hosts. Then you can just vnc in, right click on the blackness to start a terminal and run the graphic app to manage, eg virt-manager. Once that's done you can use commands to get access eg. if you need to access it through a particular vlan
ip link del br-lan
ip link add link eth0 temp0 type vlan id 5
ip addr add 192.168.1.3/24 dev temp0
ip link set temp0 up
nft flush ruleset
Then you can just browse to 192.168.1.3 as an example and upload the config.
You could, but kind of chicken-and-egg thing. How do you get that line in /lib/preinit/80_mount_root, and have that file be in the new image?
Only way I can think of is adding that to your current install, and adding /lib/preinit/80_mount_root to the list of files saved as part of configuration. I don't like that; that file is part of the OpennWrt distro and while I think odds of changes are low, if there are any, this approach would overwrite them, potentially leaving the router unbootable. But meh, if that is a risk (probability very low) you are willing to take, sure will work.
That's my red line. In our case the router is a headless industrial mini-pc. Just getting to it is painful. Getting a console involves serial. Our upgrades need to be 100% reliable and 100% remote. In our (both) cases this seems not to be the issue.