Are I/O errors on mtdblock concerning?

[joaoabs]

Hi,

Just got a brand new Linksys EA8500 and the first thing I did was to flash the latest stable OpenWRT!

I'm still configuring it, but whenever it boots I can see some I/O error messages.

[   10.326413] print_req_error: I/O error, dev mtdblock0, sector 0
[   10.332545] print_req_error: I/O error, dev mtdblock0, sector 8
[   10.333077] print_req_error: I/O error, dev mtdblock0, sector 16
[   10.337791] print_req_error: I/O error, dev mtdblock0, sector 24
[   10.344175] print_req_error: I/O error, dev mtdblock0, sector 0
[   10.349507] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   10.356403] print_req_error: I/O error, dev mtdblock1, sector 0
[   10.363161] print_req_error: I/O error, dev mtdblock1, sector 8
[   10.368986] print_req_error: I/O error, dev mtdblock1, sector 16
[   10.374987] print_req_error: I/O error, dev mtdblock1, sector 24
[   10.381138] print_req_error: I/O error, dev mtdblock1, sector 0
[   10.386480] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   10.489760] Buffer I/O error on dev mtdblock2, logical block 0, async page read
[   10.492019] Buffer I/O error on dev mtdblock3, logical block 0, async page read
[   10.498983] Buffer I/O error on dev mtdblock4, logical block 0, async page read
[   10.514562] Buffer I/O error on dev mtdblock5, logical block 3, async page read
[   10.516939] Buffer I/O error on dev mtdblock6, logical block 0, async page read
[   10.523062] Buffer I/O error on dev mtdblock7, logical block 0, async page read
[   10.554320] Buffer I/O error on dev mtdblock9, logical block 0, async page read

I have a SSD disk connencted to the router usb3 port, but that is /dev/sda. These mtdblock errors seem to be from the router itself (internal ROM, flash?).

This is a brand-new router. Would this be related with premature corruption in the router, or would it be normal in the current release?

OpenWrt 18.06.2, r7676-cddd7b4c77

Should I flash the OEM factory image back and return the router?

Any hint will be appreciated,
JoaoABS

[tmomas]

Same situation with same sectors on R7800:

Build for Netgear R7800

Thank you slh. I believe hnyman has addressed these errors earlier, but wondering if your getting these errors too. [10386.583652] print_req_error: 2 callbacks suppressed [10386.583658] print_req_error: I/O error, dev mtdblock0, sector 0 [10386.587825] print_req_error: I/O error, dev mtdblock0, sector 8 [10386.593807] print_req_error: I/O error, dev mtdblock0, sector 16 [10386.599746] print_req_error: I/O error, dev mtdblock0, sector 24 [10386.606316] print_req_error: I/O error, dev mtdblock0,…

Seems normal for NAND:

Build for Netgear R7800

Similar error messages are there usually at boot, so i think those are rather notrmal. No idea about the actual reason.

Netgear R7500v2 (ipq806x) 18.06.1 experience and issues

The TFTP recovery flash mode is pretty common for many Netgear routers. So, likely you do not need any serial adapter, but you just need to trigger the TFTP mode during early boot. Those dmesg mtd log lines are rather normal, also in R7800. I have never bothered to look into the origins of those, as everything seems to work.

[joaoabs]

Just some more findings

While configuring the router, it was getting unstable.
SSH wasn't working all the time and from a point there was no switching between WAN and LAN. Lost a few hours trying to understand what was it.

The presinte boot process (had it saved for reference) had something like:

[   22.475034] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   22.476071] dwmac1000: Master AXI performs any burst length
[   22.484025] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[   22.489464] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
[   22.498644] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   22.510531] br-lan: port 1(eth0.1) entered blocking state
[   22.511103] br-lan: port 1(eth0.1) entered disabled state
[   22.516978] device eth0.1 entered promiscuous mode
[   22.521950] device eth0 entered promiscuous mode
[   22.533508] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   22.549591] IPv6: ADDRCONF(NETDEV_UP): eth0.2: link is not ready
[   23.514114] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

And with this instability become this:

[   18.760475] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   18.862636] ipq806x-gmac-dwmac 37200000.ethernet: Failed to reset the dma
[   18.862667] ipq806x-gmac-dwmac 37200000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
[   18.868409] ipq806x-gmac-dwmac 37200000.ethernet eth0: stmmac_open: Hw setup failed
[   18.890308] stmmac_ethtool_get_link_ksettings: eth0: PHY is not registered

And then there were a lot of messages like

stmmac_ethtool_get_link_ksettings: eth0: PHY is not registered

The router was simply unusable, so in one of the many reboots I reflashed the OpenWRT image (this time the upgrade image), and at the moment I don't have none of these (neither the initial ones).

Maybe the lastest errors (Switch) came from the initial ones (I/O errors)? Don't know....

So, I guess reflashing the router could be a solution for these I/O errors and avoiding future issues.

I'll re-configure everything again and will post here if any related error arises.

[joaoabs]

Some more updates:

My configuration requires USB support so I can use a 3G modem for sending SMS and an SSD.
Therefore, I have to install a lot of packages to support it.

After installing it with

opkg install libusb-1.0 kmod-usb-storage kmod-fs-ext4 e2fsprogs fdisk kmod-usb-storage-extras usbutils blkid nfs-utils kmod-fs-nfs kmod-fs-nfs-common coreutils-timeout coreutils-stty bash curl php7 php7-cgi block-mount htop 

the I/O error appeared again (immediately and also after reboots):

[    9.718193] mount_root: loading kmods from internal overlay
[    9.827949] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    9.960126] usbcore: registered new interface driver usb-storage
[    9.960223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    9.987213] print_req_error: I/O error, dev mtdblock0, sector 0
[    9.987814] print_req_error: I/O error, dev mtdblock0, sector 8
[    9.992585] print_req_error: I/O error, dev mtdblock0, sector 16
[    9.998404] print_req_error: I/O error, dev mtdblock0, sector 24
[   10.004808] print_req_error: I/O error, dev mtdblock0, sector 0
[   10.010083] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   10.016465] print_req_error: I/O error, dev mtdblock1, sector 0
[   10.023503] print_req_error: I/O error, dev mtdblock1, sector 8
[   10.029345] print_req_error: I/O error, dev mtdblock1, sector 16
[   10.035308] print_req_error: I/O error, dev mtdblock1, sector 24
[   10.041534] print_req_error: I/O error, dev mtdblock1, sector 0
[   10.047059] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   10.149949] Buffer I/O error on dev mtdblock2, logical block 0, async page read
[   10.152191] Buffer I/O error on dev mtdblock3, logical block 0, async page read
[   10.159136] Buffer I/O error on dev mtdblock4, logical block 0, async page read
[   10.174850] Buffer I/O error on dev mtdblock5, logical block 3, async page read
[   10.177220] Buffer I/O error on dev mtdblock6, logical block 0, async page read
[   10.183463] Buffer I/O error on dev mtdblock7, logical block 0, async page read
[   10.244304] Buffer I/O error on dev mtdblock9, logical block 0, async page read

I then uninstalled these packages and rebooted the router. I don't have the error anymore.
Therefore, I guess it would be safe to assume that its related with one of the packages I have installed.

Any hint what may be?

[slh]

I've also seen those messages in a vanilla 18.06.2 VM for x86_64 (squashfs, no user-installed packages), ruling out actual hardware trouble.

[joaoabs]

With a fresh install (firmware reflashed) the issue didn't happen again until I've installed kmod-usb-storage and kmod-fs-ext4.

So, even if not really a HW trouble, I guess something could be improved in one of these packages so they don't have this behaviour.

[dr123]

One year on and now at version 19.07.1 the issue is still present on my EA4500 router.

.
[ 14.696143] __nand_correct_data: uncorrectable ECC error
[ 14.701527] __nand_correct_data: uncorrectable ECC error
[ 14.706869] print_req_error: I/O error, dev mtdblock0, sector 0
[ 14.712911] __nand_correct_data: uncorrectable ECC error
[ 14.718254] print_req_error: I/O error, dev mtdblock0, sector 8
[ 14.724291] __nand_correct_data: uncorrectable ECC error
[ 14.729629] __nand_correct_data: uncorrectable ECC error
[ 14.734977] print_req_error: I/O error, dev mtdblock0, sector 16
[ 14.741085] __nand_correct_data: uncorrectable ECC error
[ 14.746434] print_req_error: I/O error, dev mtdblock0, sector 24
[ 14.841490] __nand_correct_data: uncorrectable ECC error
[ 14.846835] __nand_correct_data: uncorrectable ECC error
[ 14.852208] print_req_error: I/O error, dev mtdblock0, sector 0
[ 14.858157] Buffer I/O error on dev mtdblock0, logical block 0, async page read

[darksky]

I have been seeing these sorts of errors on my R7800 for years. Probably normal as @hnyman said.

 ...
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.241601] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.276004] print_req_error: I/O error, dev mtdblock0, sector 0
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.276450] print_req_error: I/O error, dev mtdblock0, sector 8
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.281141] print_req_error: I/O error, dev mtdblock0, sector 16
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.287107] print_req_error: I/O error, dev mtdblock0, sector 24
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.293338] print_req_error: I/O error, dev mtdblock0, sector 0
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.298864] Buffer I/O error on dev mtdblock0, logical block 0, async page read
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.314205] print_req_error: I/O error, dev mtdblock1, sector 0
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.314673] print_req_error: I/O error, dev mtdblock1, sector 8
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.334177] print_req_error: I/O error, dev mtdblock1, sector 16
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.334726] print_req_error: I/O error, dev mtdblock1, sector 24
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.339680] print_req_error: I/O error, dev mtdblock1, sector 0
Sat Mar  7 06:07:01 2020 kern.err kernel: [   10.345226] Buffer I/O error on dev mtdblock1, logical block 0, async page read
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.440216] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.444439] block: extroot: not configured
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.498118] mount_root: switching to ubifs overlay
Sat Mar  7 06:07:01 2020 user.warn kernel: [   10.533318] urandom-seed: Seeding with /etc/urandom.seed
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.637668] procd: - early -
Sat Mar  7 06:07:01 2020 user.info kernel: [   10.637859] procd: - watchdog -

[karloff]

I am quite "happy" to see that others have this issue too, I was a bit afraid that my NAND is slowly start to dying.

root@sagan:~# dmesg | grep "I/O error"
[   14.043712] print_req_error: I/O error, dev mtdblock0, sector 0
[   14.063686] print_req_error: I/O error, dev mtdblock0, sector 8
[   14.064133] print_req_error: I/O error, dev mtdblock0, sector 16
[   14.068792] print_req_error: I/O error, dev mtdblock0, sector 24
[   14.075148] print_req_error: I/O error, dev mtdblock0, sector 0
[   14.080641] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   14.185514] print_req_error: I/O error, dev mtdblock0, sector 0
[   14.185550] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   14.190944] print_req_error: I/O error, dev mtdblock1, sector 0
[   14.198236] print_req_error: I/O error, dev mtdblock1, sector 8
[   14.204103] print_req_error: I/O error, dev mtdblock1, sector 16
[   14.209922] print_req_error: I/O error, dev mtdblock1, sector 24
[   14.216309] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   14.223953] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   14.994921] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   15.000224] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   15.007452] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   15.014787] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   41.086892] print_req_error: I/O error, dev mtdblock0, sector 0
[   41.099466] print_req_error: I/O error, dev mtdblock0, sector 8
[   41.099894] print_req_error: I/O error, dev mtdblock0, sector 16
[   41.104642] print_req_error: I/O error, dev mtdblock0, sector 24
[   41.132680] print_req_error: I/O error, dev mtdblock0, sector 0
[   41.132704] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   41.148892] print_req_error: I/O error, dev mtdblock0, sector 0
[   41.148917] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   41.174308] print_req_error: I/O error, dev mtdblock1, sector 0
[   41.174715] print_req_error: I/O error, dev mtdblock1, sector 8
[   41.179435] print_req_error: I/O error, dev mtdblock1, sector 16
[   41.185391] print_req_error: I/O error, dev mtdblock1, sector 24
[   41.196486] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   41.199496] Buffer I/O error on dev mtdblock1, logical block 0, async page read

[pwned]

You can try to verify with (if smartmontools available and working for mtd block devices?):

smartctl -a /dev/mtdblock1

The tool will tell you what kind of error it is. Google will tell you you if it is reapairable and if it is harmfull or not. Usually those errors are not harmfull and just deleting the sector will fix it. debugfs will tell you if the sector is empty and safe to delete e. g.: debugfs /dev/mtdblock1. You can delete/zero with dd the affected sectors.

[hnyman]

Flash memory

FS#1871 - Kernel :Ipq806x MTD ECC ERROR

Closed by Jonas Gorski
21.01.2019 13:35
Reason for closing: Not a bug
Additional comments about closing:

The first few blocks of a NAND flash are guaranteed good to ensure that a bootloader stored there can never get corrupted, so it will get written without valid ECC data (the SoC won't check the ECC anyway).

When block-mount scans all block devices, it will try to read from those blocks, which are exposed as partitions, and the NAND driver will report failed ECC checks (the I/O errors in the log).

There is nothing wrong here in either way, and nothing we can really do to prevent it.