[ 3231.229079] usb 3-1: new SuperSpeed Gen 1 USB device number 3 using xhci-hcd
[ 3231.260007] usb-storage 3-1:1.0: USB Mass Storage device detected
[ 3231.266273] usb-storage 3-1:1.0: Quirks match for vid 152d pid 0567: 5000000
[ 3231.273430] scsi host2: usb-storage 3-1:1.0
[ 3232.349724] scsi 2:0:0:0: Direct-Access H/W RAID 5 0523 PQ: 0 ANSI: 6
[ 3232.358518] sd 2:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[ 3232.365984] sd 2:0:0:0: [sda] 23441768448 512-byte logical blocks: (12.0 TB/10.9 TiB)
[ 3232.373863] sd 2:0:0:0: [sda] 4096-byte physical blocks
[ 3232.379249] sd 2:0:0:0: [sda] Write Protect is off
[ 3232.384094] sd 2:0:0:0: [sda] Mode Sense: 67 00 10 08
[ 3232.389326] sd 2:0:0:0: [sda] No Caching mode page found
[ 3232.394672] sd 2:0:0:0: [sda] Assuming drive cache: write through
[ 3232.552578] sda: sda1
[ 3232.556188] sd 2:0:0:0: [sda] Attached SCSI disk
[ 3246.774299] JBD2: journal transaction 873181 on sda1-8 is corrupt.
[ 3246.780518] EXT4-fs (sda1): error loading journal
The disk is a raid 5 array connected over USB. A power outage that extended past my 45 minute backup battery did this to my drive. I assume an unclean umount, but this is more serious than fsck can address
Ok, solved. I had to manually run fsck even though it runs it on mount
root@OpenWrt:/mnt# fsck.ext4 /dev/sda1
e2fsck 1.45.6 (20-Mar-2020)
BigMedia: recovering journal
Journal transaction 873181 was corrupt, replay was aborted.
BigMedia contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 260048371 extent tree (at level 2) could be narrower. Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
BigMedia: ***** FILE SYSTEM WAS MODIFIED *****
BigMedia: 682466/366280704 files (7.3% non-contiguous), 862340243/2930220795 blocks
root@OpenWrt:/mnt# block mount
root@OpenWrt:/mnt# cd BigMedia/
root@OpenWrt:/mnt/BigMedia# ls
GooglePhotos Pictures db lost+found torrents
Music Videos home-backup transmission
root@OpenWrt:/mnt/BigMedia#
It seems to be an actual hardware issue. These entries appear to be communication errors that cause the journal to go sour. It can only be one of three things: The Linksys eSATA port, the cable, or the RAID enclosure. I've ordered a new RAID box. I used to think my boot order was wrong in NFSd was coming up before block devices, but a deep read of syslog says this isn't the case. I've had this communication problem since the beginning and now it happens randomly and is much worse.
[19508.155051] ata2.00: failed to read SCR 1 (Emask=0x40)
[19508.160298] ata2.01: failed to read SCR 1 (Emask=0x40)
[19508.165478] ata2.02: failed to read SCR 1 (Emask=0x40)
[19508.170638] ata2.03: failed to read SCR 1 (Emask=0x40)
[19508.175803] ata2.04: failed to read SCR 1 (Emask=0x40)
[19508.180965] ata2.00: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
[19508.188235] ata2.00: cmd 35/00:28:10:3b:84/00:00:ba:02:00/e0 tag 12 dma 20480 out
[19508.188235] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[19516.587999] ata2.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19516.594746] ata2.00: hard resetting link
[19516.930306] ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[19518.068542] ata2.01: failed to resume link (SControl 0)
[19518.074036] ata2.01: SATA link down (SStatus 0 SControl 0)
[19519.188936] ata2.02: failed to resume link (SControl 0)
[19519.194429] ata2.02: SATA link down (SStatus 0 SControl 0)
[19520.329346] ata2.03: failed to resume link (SControl 0)
[19520.334830] ata2.03: SATA link down (SStatus 0 SControl 0)
[19521.439728] ata2.04: failed to resume link (SControl 0)
[19521.445205] ata2.04: SATA link down (SStatus 0 SControl 0)
[19526.511484] ata2.00: qc timeout (cmd 0xec)
[19526.515604] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19526.521737] ata2.00: revalidation failed (errno=-5)
[19526.526638] ata2.15: hard resetting link
[19534.964449] ata2.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19534.971166] ata2.00: hard resetting link
[19535.306738] ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[19545.868278] ata2.00: qc timeout (cmd 0xec)
[19545.872398] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19545.878531] ata2.00: revalidation failed (errno=-5)
[19545.883432] ata2.15: hard resetting link
[19554.321243] ata2.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19554.327962] ata2.00: hard resetting link
[19554.673526] ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[19555.791810] ata2.01: failed to resume link (SControl 0)
[19555.797295] ata2.01: SATA link down (SStatus 0 SControl 0)
[19556.932177] ata2.02: failed to resume link (SControl 0)
[19556.937663] ata2.02: SATA link down (SStatus 0 SControl 0)
[19558.072579] ata2.03: failed to resume link (SControl 0)
[19558.078054] ata2.03: SATA link down (SStatus 0 SControl 0)
[19559.192973] ata2.04: failed to resume link (SControl 0)
[19559.198461] ata2.04: SATA link down (SStatus 0 SControl 0)
[19589.403549] ata2.00: qc timeout (cmd 0xec)
[19589.407670] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[19589.413801] ata2.00: revalidation failed (errno=-5)
[19589.418703] ata2.00: failed to recover link after 3 tries, disabling
[19589.425095] ata2.00: disabled
[19589.428077] ata2.15: hard resetting link
[19597.836499] ata2.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[19598.168660] ata2.00: device reported invalid CHS sector 0
[19598.174098] sd 1:0:0:0: [sda] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[19598.182410] sd 1:0:0:0: [sda] tag#12 Sense Key : 0x5 [current]
[19598.186621] sd 1:0:0:0: rejecting I/O to offline device
[19598.188365] sd 1:0:0:0: [sda] tag#12 ASC=0x21 ASCQ=0x4
[19598.188370] sd 1:0:0:0: [sda] tag#12 CDB: opcode=0x8a 8a 00 00 00 00 02 ba 84 3b 10 00 00 00 28 00 00
[19598.188374] blk_update_request: I/O error, dev sda, sector 11719162640 op 0x1:(WRITE) flags 0x800 phys_seg 5 prio class 0
[19598.188397] ata2: EH complete
[19598.193672] sd 1:0:0:0: rejecting I/O to offline device
[19598.198893] Aborting journal on device sda1-8.
[19598.208145] blk_update_request: I/O error, dev sda, sector 10821306624 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 0
[19598.208150] Buffer I/O error on dev sda1, logical block 1352663072, lost async page write
[19598.219183] ata2.00: detaching (SCSI 1:0:0:0)
[19598.222155] sd 1:0:0:0: rejecting I/O to offline device
[19598.261033] blk_update_request: I/O error, dev sda, sector 11052026160 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[19598.272059] blk_update_request: I/O error, dev sda, sector 11719149568 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[19598.272119] EXT4-fs warning (device sda1): htree_dirblock_to_tree:997: inode #172687428: lblock 0: comm nfsd: error -5 reading directory block
[19598.283072] Buffer I/O error on dev sda1, logical block 1464893440, lost sync page write
[19598.283279] EXT4-fs warning (device sda1): htree_dirblock_to_tree:997: inode #172687428: lblock 0: comm nfsd: error -5 reading directory block
[19598.296404] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[19598.304369] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[19598.334801] Buffer I/O error on dev sda1, logical block 0, lost sync page write
[19598.342488] EXT4-fs (sda1): I/O error while writing superblock
[19598.348535] EXT4-fs error (device sda1): ext4_journal_check_start:61: Detected aborted journal
[19598.358180] EXT4-fs (sda1): Remounting filesystem read-only
[19598.364528] blk_update_request: I/O error, dev sda, sector 2048 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[19598.375223] Buffer I/O error on dev sda1, logical block 0, lost sync page write
[19598.382859] EXT4-fs (sda1): I/O error while writing superblock
[19598.496285] JBD2: Error while async write back metadata bh 1352663072.
[19598.513580] sd 1:0:0:0: [sda] Stopping disk
[19598.524634] sd 1:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00