Weird that it's even trying to mount ubi, when you select "-n". It should just wipe the partition/create a new ubi on boot, right?
Maybe because initramfs isn't booted from ram but from flash?
Until then:
@opensesame could you provide a full boot log please of the device booting right after flashing it from Asus's firmware via ssh and mtd-write? (the regular installation method you used before)
I've added two more builds from today, one that removes the nmbm exclusion range and one without nmbm at all, can you try them to see if any of them works for you? (same thing applies, first use the initramfs from that respective build, either flash it via restoration tool or boot it from u-boot, since you have serial now).
I tried the no-nmbm-exclusion build first. It was completely successful!
root@OpenWrt:/tmp# sysupgrade -v -n openwrt-ramips-mt7621-asus_rt-ax54-squashfs-sysupgrade.bin
Fri Apr 28 03:25:09 UTC 2023 upgrade: Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
Watchdog does not have CARDRESET support
Fri Apr 28 03:25:10 UTC 2023 upgrade: Sending TERM to remaining processes ...
Fri Apr 28 03:25:14 UTC 2023 upgrade: Sending KILL to remaining processes ...
[ 117.986277] stage2 (3413): drop_caches: 3
Fri Apr 28 03:25:20 UTC 2023 upgrade: Switching to ramdisk...
Fri Apr 28 03:25:23 UTC 2023 upgrade: Performing system upgrade...
verifying sysupgrade tar file integrity
Unlocking kernel ...
Writing from <stdin> to kernel ...
[ 121.359826] ubi0: attaching mtd7
[ 121.465547] UBI: EOF marker found, PEBs from 31 will be erased
[ 122.115942] ubi0: scanning is finished
[ 122.119759] ubi0 error: ubi_read_volume_table: the layout volume was not found
[ 122.127229] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd7, error -22
ubiattach: error!: cannot attach mtd7
error 22 (Invalid argument)
ubiformat: mtd7 (nand), size 48234496 bytes (46.0 MiB), 368 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning eraseblock 364 -- 50 % complete
ubiforma[ 124.667772] ubi0: attaching mtd7% complete
ubiformat: formatting eraseblock 367 -- 100 % c
[ 125.850820] ubi0: scanning is finished
[ 125.870930] ubi0: attached mtd7 (name "ubi", size 46 MiB)
[ 125.876352] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[ 125.883228] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[ 125.890007] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[ 125.896956] ubi0: good PEBs: 368, bad PEBs: 0, corrupted PEBs: 0
[ 125.902938] ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128
[ 125.910140] ubi0: max/mean erase counter: 0/0, WL threshold: 4096, image sequence number: 2097559741
[ 125.919262] ubi0: available PEBs: 345, total reserved PEBs: 23, PEBs reserved for bad PEB handling: 19
[ 125.928576] ubi0: background thread "ubi_bgt0d" started, PID 4116
UBI device number 0, total 368 LEBs (46727168 bytes, 44.5 MiB), available 345 LEBs (43806720 bytes, 41.7 MiB), LEB size 126976 bytes (124.0 KiB)
Volume ID 0, size 32 LEBs (4063232 bytes, 3.8 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs", alignment 1
Set volume size to 39743488
Volume ID 1, size 313 LEBs (39743488 bytes, 37.9 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs_data", alignment 1
Unlocking kernel ...
Writing from <stdin> to kernel ...
sysupgrade successful
I then tried the no-nmbm version. It failed:
root@OpenWrt:/tmp# sysupgrade -n -v openwrt-ramips-mt7621-asus_rt-ax54-squashfs-sysupgrade.bin
Fri Apr 28 03:37:41 UTC 2023 upgrade: Commencing upgrade. Closing all shell sessions.
Watchdog handover: fd=3
- watchdog -
Watchdog does not have CARDRESET support
Fri Apr 28 03:37:41 UTC 2023 upgrade: Sending TERM to remaining processes ...
Fri Apr 28 03:37:46 UTC 2023 upgrade: Sending KILL to remaining processes ...
[ 148.338095] stage2 (3296): drop_caches: 3
Fri Apr 28 03:37:52 UTC 2023 upgrade: Switching to ramdisk...
Fri Apr 28 03:37:54 UTC 2023 upgrade: Performing system upgrade...
verifying sysupgrade tar file integrity
Unlocking kernel ...
Writing from <stdin> to kernel ...
[ 151.686761] ubi0: attaching mtd7
[ 152.700497] ubi0 error: ubi_io_is_bad: error -77 while checking if PEB 315 is bad
[ 152.708231] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd7, error -77
ubiattach: error!: cannot attach mtd7
error 77 (Bad message)
ubiformat: mtd7 (nand), size 48234496 bytes (46.0 MiB), 368 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes
libscan: scanning [ 153.559272] ubi0: attaching mtd7
libscan: scanning eraseblock 31libmtd: error!: [ 154.573435] ubi0 error: ubi_io_is_bad: error -77 while checking if PEB 315 is bad
[ 154.581160] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd7, error -77
ubiattach: error!: cannot attach mtd7
error 77 (Bad message)
cannot attach ubi mtd partition ubi
sysupgrade failed
Next I reverted to stock firmware and attempted to upgrade via mtd-write, intending to use the serial console to capture the errors as requested by @Djfe. I used the latest snapshot build from the Firmware Selector (2023-04-27, r22652-9a26669510).
To my shock, it worked perfectly.
admin@RT-AX1800S-B1C8:/tmp# mtd-write -i openwrt-ramips-mt7621-asus_rt-ax54-squashfs-factory.bin -d Kernel
2450: check_action 0: 2180(firmware_check_)
2450: set_action 2
file len=0x880000
mtd size=3200000, erasesize=20000, writesize=800, type=4
freeram=8c4f000 unit_len=20000 filelen=880000 mi.erasesize=20000 mi.writesize=800, skip_bytes=0
ofs=0 n=20000/20000 ei.start=0 ei.length=20000
ofs=20000 n=20000/20000 ei.start=20000 ei.length=20000
ofs=40000 n=20000/20000 ei.start=40000 ei.length=20000
ofs=60000 n=20000/20000 ei.start=60000 ei.length=20000
ofs=80000 n=20000/20000 ei.start=80000 ei.length=20000
ofs=a0000 n=20000/20000 ei.start=a0000 ei.length=20000
ofs=c0000 n=20000/20000 ei.start=c0000 ei.length=20000
ofs=e0000 n=20000/20000 ei.start=e0000 ei.length=20000
ofs=100000 n=20000/20000 ei.start=100000 ei.length=20000
ofs=120000 n=20000/20000 ei.start=120000 ei.length=20000
ofs=140000 n=20000/20000 ei.start=140000 ei.length=20000
ofs=160000 n=20000/20000 ei.start=160000 ei.length=20000
ofs=180000 n=20000/20000 ei.start=180000 ei.length=20000
ofs=1a0000 n=20000/20000 ei.start=1a0000 ei.length=20000
ofs=1c0000 n=20000/20000 ei.start=1c0000 ei.length=20000
ofs=1e0000 n=20000/20000 ei.start=1e0000 ei.length=20000
ofs=200000 n=20000/20000 ei.start=200000 ei.length=20000
ofs=220000 n=20000/20000 ei.start=220000 ei.length=20000
ofs=240000 n=20000/20000 ei.start=240000 ei.length=20000
ofs=260000 n=20000/20000 ei.start=260000 ei.length=20000
ofs=280000 n=20000/20000 ei.start=280000 ei.length=20000
ofs=2a0000 n=20000/20000 ei.start=2a0000 ei.length=20000
ofs=2c0000 n=20000/20000 ei.start=2c0000 ei.length=20000
ofs=2e0000 n=20000/20000 ei.start=2e0000 ei.length=20000
ofs=300000 n=20000/20000 ei.start=300000 ei.length=20000
ofs=320000 n=20000/20000 ei.start=320000 ei.length=20000
ofs=340000 n=20000/20000 ei.start=340000 ei.length=20000
ofs=360000 n=20000/20000 ei.start=360000 ei.length=20000
ofs=380000 n=20000/20000 ei.start=380000 ei.length=20000
ofs=3a0000 n=20000/20000 ei.start=3a0000 ei.length=20000
ofs=3c0000 n=20000/20000 ei.start=3c0000 ei.length=20000
ofs=3e0000 n=20000/20000 ei.start=3e0000 ei.length=20000
ofs=400000 n=20000/20000 ei.start=400000 ei.length=20000
ofs=420000 n=20000/20000 ei.start=420000 ei.length=20000
ofs=440000 n=20000/20000 ei.start=440000 ei.length=20000
ofs=460000 n=20000/20000 ei.start=460000 ei.length=20000
ofs=480000 n=20000/20000 ei.start=480000 ei.length=20000
ofs=4a0000 n=20000/20000 ei.start=4a0000 ei.length=20000
ofs=4c0000 n=20000/20000 ei.start=4c0000 ei.length=20000
ofs=4e0000 n=20000/20000 ei.start=4e0000 ei.length=20000
ofs=500000 n=20000/20000 ei.start=500000 ei.length=20000
ofs=520000 n=20000/20000 ei.start=520000 ei.length=20000
ofs=540000 n=20000/20000 ei.start=540000 ei.length=20000
ofs=560000 n=20000/20000 ei.start=560000 ei.length=20000
ofs=580000 n=20000/20000 ei.start=580000 ei.length=20000
ofs=5a0000 n=20000/20000 ei.start=5a0000 ei.length=20000
ofs=5c0000 n=20000/20000 ei.start=5c0000 ei.length=20000
ofs=5e0000 n=20000/20000 ei.start=5e0000 ei.length=20000
ofs=600000 n=20000/20000 ei.start=600000 ei.length=20000
ofs=620000 n=20000/20000 ei.start=620000 ei.length=20000
ofs=640000 n=20000/20000 ei.start=640000 ei.length=20000
ofs=660000 n=20000/20000 ei.start=660000 ei.length=20000
ofs=680000 n=20000/20000 ei.start=680000 ei.length=20000
ofs=6a0000 n=20000/20000 ei.start=6a0000 ei.length=20000
ofs=6c0000 n=20000/20000 ei.start=6c0000 ei.length=20000
ofs=6e0000 n=20000/20000 ei.start=6e0000 ei.length=20000
ofs=700000 n=20000/20000 ei.start=700000 ei.length=20000
ofs=720000 n=20000/20000 ei.start=720000 ei.length=20000
ofs=740000 n=20000/20000 ei.start=740000 ei.length=20000
ofs=760000 n=20000/20000 ei.start=760000 ei.length=20000
ofs=780000 n=20000/20000 ei.start=780000 ei.length=20000
ofs=7a0000 n=20000/20000 ei.start=7a0000 ei.length=20000
ofs=7c0000 n=20000/20000 ei.start=7c0000 ei.length=20000
ofs=7e0000 n=20000/20000 ei.start=7e0000 ei.length=20000
ofs=800000 n=20000/20000 ei.start=800000 ei.length=20000
ofs=820000 n=20000/20000 ei.start=820000 ei.length=20000
ofs=840000 n=20000/20000 ei.start=840000 ei.length=20000
ofs=860000 n=20000/20000 ei.start=860000 ei.length=20000
2450: set_action 0
Image successfully flashed
Following a reboot, I am successfully booted into snapshot OpenWRT. No errors.
So, perhaps the no-nmbm-exclusion build fixed something somewhere in flash that "healed" the router and now the normal install method works again?
I cannot explain this. I'm happy that it works now but it is unsatisfying to not know why.
I think that if you go into u-boot console again and check nmbm state, you'll find a bad sector.
Just a guess.
I checked and you are exactly right!
=> nmbm nmbm0 state
Physical blocks:
Legends:
- Good data block
+ Good management block
B Bad block
I Main info table
i Backup info table
M Remapped spare block
S Signature block
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------B-----
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
Logical blocks:
Legends:
- Good block
+ Initially remapped block
M Remapped block
B Bad/Unmapped block
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------M-----
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
Also, I was too hasty in my assessment that everything is working fine now.
After rebooting to check nmbm nmbm0 state, the router no longer boots successfully.
The symptoms are exactly the same as previous -- the device boot loops. But now with the serial console I can see why:
[ 3.748764] mt7530 mdio-bus:1f lan1 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7530 PHY] (irq=26)
[ 3.761208] DSA: tree 0 setup
[ 3.767847] UBI: auto-attach mtd6
[ 3.771273] ubi0: attaching mtd6
[ 5.428779] ubi0 error: ubi_io_is_bad: error -77 while checking if PEB 516 is bad
[ 5.436552] ubi0 error: ubi_attach_mtd_dev: failed to attach mtd6, error -77
[ 5.443640] UBI error: cannot attach mtd6
[ 5.449232] /dev/root: Can't open blockdev
[ 5.453378] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[ 5.460865] Please append a correct "root=" boot option; here are the available partitions:
[ 5.469189] 1f00 512 mtdblock0
[ 5.469201] (driver?)
[ 5.475730] 1f01 384 mtdblock1
[ 5.475741] (driver?)
[ 5.482259] 1f02 1024 mtdblock2
[ 5.482270] (driver?)
[ 5.488768] 1f03 1024 mtdblock3
[ 5.488777] (driver?)
[ 5.495296] 1f04 1024 mtdblock4
[ 5.495308] (driver?)
[ 5.501824] 1f05 4096 mtdblock5
[ 5.501835] (driver?)
[ 5.508335] 1f06 114816 mtdblock6
[ 5.508343] (driver?)
[ 5.514869] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.523110] Rebooting in 1 seconds..
The stock snapshot build (which you flashed last?) still has the range used by ubifs excluded from nmbm.
I'll build a recent snapshot for you in a few minutes with the change included.
But I'm worried about something else now, I don't see the last 64 blocks with management etc.
LE: sorry, it takes a while, I'm testing the changes on my AX53U first. Would be easier if you'd join the IRC channel.
Do you have the latest bootlogs by any chance?
I'm interested in the lines between these ones:
[ 1.459468] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xf1
[....]
[ 1.540435] Creating 7 MTD partitions on "mt7621-nand":
Should be something NMBM related.
You can fix it by writting initramfs-kernel image into nand flash and then execute sysupgrade via OpenWrt SSH. OpenWrt factory image may fail on nmbm devices if there are bad blocks on ubi partition.
That's exactly what failed, flashing initramfs, boot initramfs, flash sysupgrade failed.
What seemed to work are the builds with no remap range defined.
But I'm worried about the changed last 64 blocks, management etc doesn't seem to be there anymore.
I saw his logs. Just need another small step. execute mtd erase ubi before sysupgrade.
1 Like
interesting idea 
but that's not the full command, so @opensesame should be careful to choose the correct mtd partition (named ubi) to erase before running sysupgrade
anyways I was asking for full boot logs for right after you executed mtd-write before 
In general it would be helpful for us if you could post full logs here and not excerpts. right now there are a few variables I'm wondering about because we never saw a full bootlog if your Asus, yet. you can post them on pastebin.com and link them here
I don't need the mtd-write log right now, try out erasing ubi partition and then running sysupgrade (on a build with nmbm range imo, but maybe wait for what Znevna recommends. it doesn't help if we confuse you by recommending 10 things at once.
but whatever you do, we need full bootlogs here
There's only one ubi partition.
When you have time, please join IRC so we can do this in a little more real time https://openwrt.org/irc
But considering that the device developed a bad block in the ubi range while already flashed with OpenWrt and this led to an unbootable device, to avoid something like this maybe we don't want to exclude ubi from nmbm in the future?
I think it is not necessary. ubifs can handle bad block during the writing process. However, if the block is damaged during the read process, neither nmbm nor ubifs can rescue it.
Still, not having the ubi range excluded from nmbm helped flashing without any other intervention, and nmbm somehow marked that block as bad.
I don't know.
Here are the full logs from yesterday when I installed the Snapshot build.
Just prior to this log, I flashed the Snapshot initramfs image via the mtd-write tool (I seem to have lost the log where I did this, sadly). This log is after booting into that Snapshot initramfs environment and try sysupgrade to the sysupgrade image. Which seemed to boot cleanly.
I then re-flashed the router to stock Asus firmware to try mtd-write with the factory image...
This log is where I execute the mtd-write to write the factory OpenWRT build:
This is the log where I rebooted after the factory flash, and everything came up nicely:
This is the log where I rebooted again to check nmbm info, then the boot kernel panics:
I am signed into the IRC channel now, probably for the next two hours or so. I am not currently set up to linger in there for long periods
I'm not sure why the last 64 blocks weren't detected in nmbm nmbm0 state yesterday. Perhaps it was still somehow using the "no nmbm" settings when I ran it? Though I had definitely rebooted the router by then.
It shows properly now, however:
=> nmbm nmbm0 state
Physical blocks:
Legends:
- Good data block
+ Good management block
B Bad block
I Main info table
i Backup info table
M Remapped spare block
S Signature block
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------B-----
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
I++i++++++++++++++++++++++++++++++++++++++++++++++++++++++++++MS
Logical blocks:
Legends:
- Good block
+ Initially remapped block
M Remapped block
B Bad/Unmapped block
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------M-----
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
----------------------------------------------------------------
=> nmbm nmbm0 bad
Physical blocks:
378 [0x02f40000] - Bad
Logical blocks:
I tried today the suggestion of doing the mtd erase ubi. It did not end well!
Here is what I did. I used the current Snapshot build, flashed initramfs image. The following log picks up from this point. I execute the mtd erase ubi, then perform the sysupgrade. This fails, leaving the router in an unusable state. I have re-flashed it to the Snapshot initramfs for the time being.
It was 5AM for me when you joined IRC, sorry. Timezones ^^
Ok, I'm glad that the management blocks are there.
So on what build are you on now? stock Snapshot? and it works fine?
Understandable!
I am using the plain snapshot initramfs, flashed into the router. It "works", in that it routes packets, but since it runs from RAM only, I am not able to save any settings. When I try to sysupgrade, it fails due to the same ubi errors as previous. Still searching for a stable way to have a working ubifs.
Here is a log of me trying to sysupgrade out of the initramfs (without explicitly doing the mtd erase ubi step, as that caused the router to be even worse off than before). The sysupgrade step fails.