Alright, remounting in read-only mode did the trick; thank you!
Somehow i thought of ro-mode as not useful and write-protected also for fsck. But fsck and tune2fs can modify the filesystem in ro-mode.
Now i will combine it with another tip to put the check before rw-remount in:
/lib/preinit/80_mount_root
or i will wait for boot-completion and do the check with a few remounts.
So while this is possible to have the running system fsck'ed i wonder why the process of checking and fixing is not supported by LEDE/OpenWRT on its own? Sudden powerloss accidents do happen from time to time ...
I'm not sure which exact command @conner2 used but I had success on the Raspberry Pi (running 17.01 based system) by modifying the /lib/preinit/80_mount_root file adding this new line (you need to have the e2fsprogs package installed): /usr/sbin/e2fsck -y -v -f /dev/mmcblk0p2
before this existing line: [ "$INITRAMFS" = "1" ] || boot_hook_add preinit_main do_mount_root
Also I tested the in-built "check filesystem" option under the menu System->Mount Points but it didn't appear to work and in most cases gave a 'Bad Gateway' error after which the only recovery path was a reboot/power cycle.
Anyway it would be great if there was working support for filesystem checks in OpenWRT/LEDE as some storage systems and set ups aren't quite so reliable.
On desktop systems, it behaves like this...on OpenWRT/LEDE it does NOT!
As stated in the initial post, no method whatsoever will enable this at boot for the root partition.
My solution is to drive the system in read-only mode for all filesystems that reside on the sd-card. I additionally mount an USB-stick with ext4 to write stuff. There it behaves like you would expect and properly recovers from power failures and such incidents.
When a file-system has gone to read-only state, you can fsck and repair it from within a running running system...even if it is the root partition. And then you can force it to stay read-only as i have explained above. Or you need to take out the sd-card, check and repair it on a desktop system and then prevent it from going to write-mode ever again if you want to protect it.
can you translate that to simpler terms?. The reference link is way over my head. I have a USB3 external drive and would like to have the fsck run on boot. Where exactly is this line placed (in what config file)? Thanks.
Oh, wow, that was easy. Under the "System", "Mount Points" following down the list of topics there is a section "Mount Points" with the USB drive is in the list and is checked if it is enabled. If you click the "Edit" option, then on the "Advanced settings" is a check box for "Run system file check" (on boot). Excellent! Thanks!!
I don't see the prints I have added in the /lib/preinit/80_mount_root file.
However, to check I installed tune2fs, using that I see that:
tune2fs -l /dev/sda1 | grep Last
Last mounted on:
Last mount time: Tue Jul 25 23:54:51 2023
Last write time: Tue Jul 25 23:54:51 2023
Last checked: Tue Jul 25 22:08:39 2023
For some reason, Last checked is not updating on every reboot.
After I installed tune2fs, just to get the "Last checked" stats, I notice that "tune2fs -c 1" is required to execute the block of code given above in /lib/preinit/80_mount_root file.
In summary, below worked for me.
Have the check using /usr/sbin/fsck.ext4 -y /dev/sda1 in /lib/preinit/80_mount_root as suggested.
tune2fs -c 1 /dev/sda1 ; tune2fs -c 1 /dev/sda2
Confirm disck check happening at boot using command tune2fs -l /dev/sda1 | grep Last
I appreciate if some expert vets my steps are correct.
I am a complete noob on filesystems.
Preinit is run before any mounts and before the system logging is enabled. Thus "logger" likely fails. (kernel log is already active).
As it is run so early, no change from overlay can be used, and the script change needs to be in the original rootfs in the compiled/flashed firmware. (at least with a normal NAND flash with overlay. Not sure about SD cards, eMMC etc,)
Thank you for the comment. I got the part that logger is a bit too late, didn't understand other parts of your post.
So anyway, the below output tells me filesystem check is happening at every reboot, and I am all set?
reboot time is matching with Last checked time.
~# tune2fs -l /dev/sda1 | grep Last
Last mounted on: <not available>
Last mount time: Wed Jul 26 00:25:44 2023
Last write time: Wed Jul 26 00:25:44 2023
Last checked: Wed Jul 26 00:25:44 2023
This doesn't really fix errors in the hard disk, tune2fs does say it was checked but I'm still getting these type of errors on unclean shutdown and requires me to run fsck.ext4 -y manually to fix disk errors, any help?
do_mount_root() {
# disk check - START
if [ -x '/usr/sbin/fsck.ext4' ]; then
if [ -e '/dev/sda1' ]; then
echo "Checking disk /dev/sda1...."
/usr/sbin/fsck.ext4 -y /dev/sda1
fi
if [ -e '/dev/sda2' ]; then
echo "Checking disk /dev/sda2..."
/usr/sbin/fsck.ext4 -y /dev/sda2
fi
fi
# disk check - END
mount_root
boot_run_hook preinit_mount_root
[ -f /sysupgrade.tgz -o -f /tmp/sysupgrade.tar ] && {
echo "- config restore -"
sorry, it has been too long.
I cannot understand even my own comments in this thread.
I will come back to it shortly though, as I am facing a suspect file system corruption issue.