E2fsck warning in dmesg

Hello:

I have OpenWRT installed on a single drive WD My Book Live and at the start had a great many issues originated in a bad shutdown.

I solved them with two things:

  1. the hardware issue
    I rigged a cable to the 12v barrel socket and to a SPST switch at the back of the unit along with a scrip to shut down the drive that executes sync, stops the HDD with hdparm -Y and then executes halt.

The trick here is to flip the switch less than 20s after the front LED turns blue.
If not done in time, openwrt/watchdog will reboot the system and flipping the switch will result in a hard shutdown.

But gone was the pull the plug to switch off the NAS routine, my HDD was happy and I was happy.

  1. the software issue

Of course, what I did above will not work under certain circumstances ie: a hard shutdown due to a power cut, a rare but worrying/possible prospect which will cause some problem to the HDD.

I found the solution in a post by forum member iwman from back in 11/2019, which consists in editing /lib/preinit/80_mount_root and adding this bit of code between lines do_mount_root() and mount_root:

 if [ -x '/usr/sbin/fsck.ext4' ]; then
                if [ -e '/dev/sda1' ]; then
                        echo "Checking disk /dev/sda1...."
                        /usr/sbin/fsck.ext4 -y /dev/sda1
                fi
                if [ -e '/dev/sda2' ]; then
                        echo "Checking disk /dev/sds2..."
                        /usr/sbin/fsck.ext4 -y /dev/sda2
                fi
        fi

This way, every time the system boots, it will check and repair the disk, even after a hard shutdown.

Or so I thought.

A couple of days ago I had to resort to a hard shutdown ie: flipping the switch to reboot after the system locked up completely while attempting to troubleshoot the ethernet speed configuration with ethtool.

I checked dmesg to see if everything was allright, and there it was again:

root@OpenWrt:~# dmesg | grep e2fsck
[    0.885063] EXT4-fs (sda2): warning: mounting unchecked fs, running e2fsck is recommended
root@OpenWrt:~#

I know the script works because after I installed it, the warnings about the number of boots limit were gone.

Searching for the dmesg warning, I found it is referenced quite a bit on the web but most times not as a problem to be solved but as part of the post which includes the dmesg printout.

Besides printing some strange output, tune2fs says that both /dev/sda1 and /dev/sda2 are not clean:

root@OpenWrt:~# tune2fs -l /dev/sda1 | grep -i "filesystem\|mount"
Filesystem volume name:   <none>
Last mounted on:          /boot
Filesystem UUID:          e61124a4-e786-44d9-bb0a-73d676fe5a61
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      filetype
Default mount options:    (none)
Filesystem state:         not clean
Filesystem OS type:       Linux
Last mount time:          Wed Dec 31 21:00:05 1969   # <----------- ?
Mount count:              1
Maximum mount count:      20
root@OpenWrt:~#
root@OpenWrt:~# tune2fs -l /dev/sda2 | grep -i "filesystem\|mount"
Filesystem volume name:   rootfs
Last mounted on:          /
Filesystem UUID:          ff313567-e9f1-5a5d-9895-3ba130b4a864
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode filetype extent sparse_super large_file uninit_bg
Filesystem flags:         unsigned_directory_hash 
Default mount options:    (none)
Filesystem state:         not clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Last mount time:          n/a
Mount count:              67
Maximum mount count:      -1
root@OpenWrt:~#

Any suggestions?

Thanks in advance,

D.

Hello:

e2fsck cannot be run on /dev/sda2 because it holds the root filesystem and as such is mounted.

root@OpenWrt:~# e2fsck /dev/sda2
e2fsck 1.45.6 (20-Mar-2020)
/dev/sda2 is mounted.
e2fsck: Cannot continue, aborting.
root@OpenWrt:~#

I have read that I can mount / as ro with mount / -o remount,ro, which would in turn allow me to run e2fsck /mnt/sda2 and fix the problem described above:

root@OpenWrt:~# mount / -o remount,ro
root@OpenWrt:~# e2fsck /dev/sda2
e2fsck 1.45.6 (20-Mar-2020)
rootfs was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Padding at end of inode bitmap is not set. Fix<y>? no

rootfs: ********** WARNING: Filesystem still has errors **********

rootfs: 1634/6656 files (0.1% non-contiguous), 8472/26624 blocks
root@OpenWrt:~#

But I am weary of doing it lest I screw up something.

I say this because I have repeatedly tried setting the apropiate Options in the Mount Points UI page ie: Check filesystems before mount both globally and individually but the dmesg warning about needing to run e2fsck on /dev/sda2 remains at every boot.

Which begs the question: why doesen't the UI setting fix whatever is wrong with /dev/sda2?

It is a long time issue and as I have mentioned, the fix I posted above does not work.

I have found nothing in the system/kernel logs indicating any e2fsck activity resulting from editing /lib/preinit/80_mount_root as indicated.

The warning remains and running e2fsck on the ro root filessytem prints a warning indicating it has errors.

So ...
While file system errors in /dev/sda1 and /dev/sda3 can be fixed easily enough, those in /dev/sda2 require remounting the partition ro.

How can the these errors detected by the system in the root filesystem be fixed?
Is the procedure described above safe?

I'd appreciate comments on this as I am not sure that going ahead ie: typing <y> would not screw up something and have to go through the setup again.

Thanks in advance,

PCL

Hello:

With no replies (makes sense, who uses a MBL these days?), I decided to risk it and went ahead with e2fsck on the unmounted / filesystem.

But not without first taking a series of precautions.

For those would find themselves in such a pickle, here is some well written and instructive reading from stackexchange which helped me solve the issue:

What is it that e2fsck does not say?
[https://unix.stackexchange.com/questions/494256/what-is-it-that-e2fsck-does-not-say/498151#498151]

What does fsck -p (preen) do on ext4?
[https://unix.stackexchange.com/questions/18526/what-does-fsck-p-preen-do-on-ext4/18551#18551]

I highly recommend reading both threads.

Keep in mind that this is about a WD 1TB My Book Live running on OpenWRT the 21.02.2-factory version for this hardware.

:~$ uname -v
#0 Wed Feb 16 20:29:10 2022
:~$
:~$ uname -a
Linux OpenWrt 5.4.179 #0 Wed Feb 16 20:29:10 2022 ppc GNU/Linux
:~$ 

Thanks for that. 8^D
Without OpenWRT the MBL is, more than anything, a paperweight.

Please take into account that not all systems will have the same error generated by hard/bad shutdowns or behave the same way as mine.

Not having done this before, I took the safest path to fix the dmesg warnings.
It is has far less risk just mounting the HDD on another running system.

But that entails opening up the case, disassembling and assembling it again which when done enough times at some point will screw up the plastic tabs, connectors, etc.

ie: caveat emptor applies to this post from this point on.

Here is the terminal printout of the process from start to finish.

--- >process start

  1. ssh into the MBL

  2. as root remount / read only:

root@OpenWrt:~# mount / -o remount,ro
  1. check that / is indeed read only:
root@OpenWrt:~# mount
/dev/sda2 on / type ext4 (ro,noatime)
--- snip ---
root@OpenWrt:~#
  1. run e2fsck with the -n (read only) and -v (verbose) options with echo $? at the end to get the e2fsck exit code at the end which is important.
    See EXIT CODE in e2fsck manpage for a complete table.
root@OpenWrt:~# e2fsck -nv /dev/sda2; echo $?
e2fsck 1.45.6 (20-Mar-2020)
rootfs was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Padding at end of inode bitmap is not set. Fix? no

rootfs: ********** WARNING: Filesystem still has errors **********

        1640 inodes used (24.64%, out of 6656)
           0 non-contiguous files (0.0%)
           1 non-contiguous directory (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 1368
        8540 blocks used (32.08%, out of 26624)
           0 bad blocks
           0 large files

        1217 regular files
         157 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
         257 symbolic links (257 fast symbolic links)
           0 sockets
------------
        1631 files
`4`      <- # exit code `4` -> File system errors left uncorrected
  1. run e2fsck with the -v (verbose) option
root@OpenWrt:~# e2fsck -v /dev/sda2; echo $?
e2fsck 1.45.6 (20-Mar-2020)
rootfs was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Padding at end of inode bitmap is not set. Fix<y>? yes

rootfs: ***** FILE SYSTEM WAS MODIFIED *****
rootfs: ***** REBOOT SYSTEM *****

        1640 inodes used (24.64%, out of 6656)
           0 non-contiguous files (0.0%)
           1 non-contiguous directory (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 1368
        8540 blocks used (32.08%, out of 26624)
           0 bad blocks
           0 large files

        1217 regular files
         157 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
         257 symbolic links (257 fast symbolic links)
           0 sockets
------------
        1631 files
`3`      # <- exit code `3` = 1+2 -> File system errors corrected, system should
                                     be rebooted
root@OpenWrt:~#
  1. reboot the system

  2. check dmesg output for file system errors:

:~$ sudo dmesg | grep -i ext4-fs
[    0.896299] EXT4-fs (sda2): mounted filesystem without journal. Opts: (null)
[    5.451533] EXT4-fs (sda2): re-mounted. Opts: (null)
[    5.884694] EXT4-fs (sda1): mounted filesystem without journal. Opts: (null)
[    6.627084] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: 
:~$ 
  1. check the state of all the mounted file systems
:~$ mount
/dev/sda2 on / type ext4 (rw,noatime)    # <------ rootfs is rw -> | OK |
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/sda1 on /boot type ext4 (rw,noatime,errors=remount-ro)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
/dev/sda3 on /mnt/sda3 type ext4 (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
:~$

process end <---

That's about it.
Do read the e2fsck man page and the threads linked above before attempting this.

Best,

PCL

2 Likes

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.