.config Changes Cause `ubi_attach: valid VID header but corrupted EC header`

"My kernel is too small to boot"

Any idea why a smaller kernel should "corrupt" the UBIFS?

        KERNEL_SIZE := 3145728    # 0x5800000 - 0x5500000
        IMAGE_SIZE := 89128960    # 0x5800000
        BLOCKSIZE := 128k
        PAGESIZE := 2048
        IMAGES := sysupgrade.bin factory.bin
        IMAGE/factory.bin  := append-kernel | pad-to $$$${KERNEL_SIZE} | append-ubi | linksys-image type=EA8300

(linksys-image adds a "signature" at the end of the image -- see scripts/linksys-image.sh)

(All images flashed through OEM U-Boot)



Started cleaning up my kernel config on an IPQ4019-based Linksys EA3800 by removing "development" kernel features such as various traces. All of a sudden, I'm seeing boot failures and, in the logs, just before failure to mount root

[    1.917737] ubi0: attaching mtd11
[    1.960673] ubi0 warning: ubi_attach: valid VID header but corrupted EC header at PEB 39
[    2.322537] ubi0: scanning is finished
[    2.329871] ubi0 error: ubi_read_volume_table: bad attaching information, error 1

or similar.

OK, working from the other partition

root@OpenWrt:/# nandtest /dev/mtd11
ECC corrections: 0
ECC failures   : 0
Bad blocks     : 0
BBT blocks     : 0
054e0000: checking...of 4)...
Finished pass 1 successfully

hopefully rules out premature NAND failure.

I start replacing the removed packages, and it isn't until I re-enable ftrace that is starts to boot again. Mess around there with a few combinations and it seems to be the issue. To confirm that ftrace is the key, I move to a very minimal config and it fails to boot. Add in ftrace and, guess what, it fails to boot.

So I start going for kernel bloat. Every language, all kinds of USB drivers I don't have, you name it. Throw it all in. Took a while, and noticed that

The PEB identified starts at 23 and starts going up to 40 (or perhaps more).

Eventually I hit the point where it boots again. I'm poking at it slowly now, trying to find the tipping point in case that size is meaningful.

Any clues as to how to chase this down and resolve it?

Just a shot in the dark, but maybe the signature script (or the validation code within uboot) doesn’t expect /know how to handle an image smaller than some value (something going negative )

E: I’m on mobile so can’t see the script currently

It's especially puzzling as

append-kernel | pad-to $$$${KERNEL_SIZE} | append-ubi

should have the location and creation of the UBI file system being completely independent of the size of the kernel, as long as the kernel doesn't exceed 3 MB (which it doesn't).

The "signature" is only for the OEM web interface. It is not checked by U-Boot in any way.

At least in one quick check, flashing a "combined" image from append-kernel | pad-to $$$${KERNEL_SIZE} | append-ubi worked, whereas the factory image from the exact same compile run failed.

-       IMAGES := sysupgrade.bin factory.bin
+       IMAGES := sysupgrade.bin factory.bin combined.bin
        IMAGE/factory.bin  := append-kernel | pad-to $$$${KERNEL_SIZE} | append-ubi | linksys-image type=EA8300
+       IMAGE/combined.bin  := append-kernel | pad-to $$$${KERNEL_SIZE} | append-ubi

Looking into things with hexdump quickly this morning gets even stranger as it looks like

  • The more bloated kernel that works is actually smaller (0x00225840) than the one that fails (0x00226040)

  • The repetitive UBI pattern is different between "combined" (left) and "factory" (right) is different from the same build. I would have assumed that the UBI file system would have been identical during the same build, with the exception being the trailing signature.

$ diff --side --suppress-common-lines combined.hex factory.hex | less
00780010  00 00 08 00 00 00 10 00  76 17 42 6d 00 00 00 00  | | 00780010  00 00 08 00 00 00 10 00  5a 78 f5 e3 00 00 00 00  |
00780030  00 00 00 00 00 00 00 00  00 00 00 00 75 c0 15 67  | | 00780030  00 00 00 00 00 00 00 00  00 00 00 00 53 10 da 7a  |
007a0010  00 00 08 00 00 00 10 00  76 17 42 6d 00 00 00 00  | | 007a0010  00 00 08 00 00 00 10 00  5a 78 f5 e3 00 00 00 00  |
007a0030  00 00 00 00 00 00 00 00  00 00 00 00 75 c0 15 67  | | 007a0030  00 00 00 00 00 00 00 00  00 00 00 00 53 10 da 7a  |
007c0010  00 00 08 00 00 00 10 00  76 17 42 6d 00 00 00 00  | | 007c0010  00 00 08 00 00 00 10 00  5a 78 f5 e3 00 00 00 00  |
007c0030  00 00 00 00 00 00 00 00  00 00 00 00 75 c0 15 67  | | 007c0030  00 00 00 00 00 00 00 00  00 00 00 00 53 10 da 7a  |
007e0000                                                      | 007e0000  2e 4c 49 4e 4b 53 59 53  2e 30 31 30 30 30 34 30  |
                                                              > 007e0010  39 45 41 38 33 30 30 20  20 20 20 20 20 20 20 20  |
                                                              > 007e0020  39 44 39 31 33 44 34 35  20 20 20 20 20 20 20 4b  |
                                                              > 007e0030  30 30 30 30 30 30 30 46  30 32 34 36 34 33 34 00  |
                                                              > 007e0040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |
                                                              > *
                                                              > 007e00f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0a  |
                                                              > 007e0100

for completeness, "unsuppressed" output at the end of the image

007d24a0  00 00 00 01 59 5a 0a cf  46 00 00 00 00 00 98 d3  |....YZ..F.......   007d24a0  00 00 00 01 59 5a 0a cf  46 00 00 00 00 00 98 d3  |....YZ..F.......
007d24b0  46 00 00 00 00 00 04 80  00 00 00 00 b6 d4 46 00  |F.............F.   007d24b0  46 00 00 00 00 00 04 80  00 00 00 00 b6 d4 46 00  |F.............F.
007d24c0  00 00 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |................   007d24c0  00 00 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  |................
007d24d0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................   007d24d0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................
*                                                                               *
007e0000                                                                      | 007e0000  2e 4c 49 4e 4b 53 59 53  2e 30 31 30 30 30 34 30  |.LINKSYS.0100040
                                                                              > 007e0010  39 45 41 38 33 30 30 20  20 20 20 20 20 20 20 20  |9EA8300         
                                                                              > 007e0020  39 44 39 31 33 44 34 35  20 20 20 20 20 20 20 4b  |9D913D45       K
                                                                              > 007e0030  30 30 30 30 30 30 30 46  30 32 34 36 34 33 34 00  |0000000F0246434.
                                                                              > 007e0040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
                                                                              > *
                                                                              > 007e00f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 0a  |................
                                                                              > 007e0100

Both confirmed to start with UBI# at 0x0300000 as expected, though subtle differences in the header can be seen

00226040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................   00226040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
*                                                                               *
00300000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............   00300000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 00  |UBI#............
00300010  00 00 08 00 00 00 10 00  76 17 42 6d 00 00 00 00  |........v.Bm.... | 00300010  00 00 08 00 00 00 10 00  5a 78 f5 e3 00 00 00 00  |........Zx......
00300020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................   00300020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
00300030  00 00 00 00 00 00 00 00  00 00 00 00 75 c0 15 67  |............u..g | 00300030  00 00 00 00 00 00 00 00  00 00 00 00 53 10 da 7a  |............S..z
00300040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................   00300040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................
*                                                                               *
00300800  55 42 49 21 01 01 00 05  7f ff ef ff 00 00 00 00  |UBI!............   00300800  55 42 49 21 01 01 00 05  7f ff ef ff 00 00 00 00  |UBI!............
00300810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................   00300810  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
*                                                                               *
00300830  00 00 00 00 00 00 00 00  00 00 00 00 b8 25 64 a8  |.............%d.   00300830  00 00 00 00 00 00 00 00  00 00 00 00 b8 25 64 a8  |.............%d.
00300840  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................   00300840  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................
*                                                                               *
00301000  00 00 00 25 00 00 00 01  00 00 00 00 01 00 00 06  |...%............   00301000  00 00 00 25 00 00 00 01  00 00 00 00 01 00 00 06  |...%............
00301010  72 6f 6f 74 66 73 00 00  00 00 00 00 00 00 00 00  |rootfs..........   00301010  72 6f 6f 74 66 73 00 00  00 00 00 00 00 00 00 00  |rootfs..........
00301020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................   00301020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
*                                                                               *
003010a0  00 00 00 00 00 00 00 00  fc 74 98 25 00 00 00 09  |.........t.%....   003010a0  00 00 00 00 00 00 00 00  fc 74 98 25 00 00 00 09  |.........t.%....
003010b0  00 00 00 01 00 00 00 00  01 00 00 0b 72 6f 6f 74  |............root   003010b0  00 00 00 01 00 00 00 00  01 00 00 0b 72 6f 6f 74  |............root
003010c0  66 73 5f 64 61 74 61 00  00 00 00 00 00 00 00 00  |fs_data.........   003010c0  66 73 5f 64 61 74 61 00  00 00 00 00 00 00 00 00  |fs_data.........
003010d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................   003010d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................
*                                                                               *