Cautionary tale - bricked OpenWRT One on first day

cpswan · October 18, 2024, 4:33pm

TL;DR I didn't find 'OpenWrt One - Setup, Install, and Discussion' and particularly the OpenWRT One - Howto (pdf) before doing some stupid stuff.

I got my OpenWRT One yesterday, and since I'd just built a LuCI app I wanted to try that out. So I needed to install LuCI, but when I tried that it wanted a newer kernel, so I had to update my firmware.

My One arrived switched to NOR, and I didn't have a good understanding of NOR vs NAND, which led to some frustration when any attempt at sysupdate -v openwrt-mediatek-filogic-openwrt_one-squashfs-sysupgrade.itb didn't work.

Through the magic of the boot menu, I managed to update the various NOR components to the latest snapshot

So then on to NAND... by using TFTP to upload the snapshot initramfs I was able to boot into the latest kernel, and install LuCI. Progress, I though. I used LuCI to perform a firmware update with the snapshot sysupgrade image.

On rebooting:

F0: 102B 0000
FA: 1040 0000
FA: 1040 0000 [0200]
F9: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 2400 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [1000]
T0: 0000 0244 [010F]
Jump to BL

NOTICE:  BL2: v2.9.0(release):OpenWrt v2023.10.13~0ea67d76-1 (mt7981-spim-nand-ubi-ddr4)
NOTICE:  BL2: Built : 09:35:38, May 21 2024
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 1024MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  CPU: MT7981 (1300MHz)
NOTICE:  SPI_NAND parses attributes from parameter page.
NOTICE:  SPI_NAND Detected ID 0xef
NOTICE:  Page size 2048, Block size 131072, size 268435456
NOTICE:  UBI: scanning [0x100000 - 0x10000000] ...
NOTICE:  UBI: scanning is finished
NOTICE:  UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
NOTICE:  UBI: VID header offset: 2048 (aligned 2048), data offset: 4096
ERROR:   UBI error: No volume named fip could be found
NOTICE:  UBI: scanning [0x100000 - 0x10000000] ...
NOTICE:  UBI: scanning is finished
NOTICE:  UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
NOTICE:  UBI: VID header offset: 2048 (aligned 2048), data offset: 4096
ERROR:   UBI error: No volume named fip could be found
ERROR:   BL2: Failed to load image id 3 (-2)

Oops. At least the NOR was still working. I removed the jumper to ensure it wouldn't be overwritten.

At some stage (trying to follow various Banana Pi and other MT7981 guides) it seems that I did an mtd erase of the NOR (whilst trying to write stuff to NAND), and the jumper not being present didn't save me

F0: 102B 0000
FA: 5100 0000
FA: 5100 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 102C 0000
F5: 1026 0000
00: 1005 0000
FA: 5100 0000
FA: 5100 0000 [0200]
F9: 103F 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 102C 0000
01: 102A 0001
02: 1005 0000
BP: 2000 00C0 [0001]
EC: 0000 0000 [0000]
T0: 0000 00D6 [010F]
System halt!

and then I finally found the docs I should have been following, but it's too late as I've managed to break the 'failsafe recovery'.

I have a JTAG/SWD programmer ordered, which hopefully arrives tomorrow. I'll be following Steps to load u-boot via JTAG to try to get things working again.

blogic · October 19, 2024, 3:20am

that is unfortunate. could you tell me what exactly you did inside NOR boot that broke the device ?
the bootmenu should provide evevrything you need without having to manually erase stuff.

cpswan · October 19, 2024, 4:14am

@blogic this deserves two explanations:

Going from memory (I foolishly lost my console logs by restarting PuTTY). I think the command that bricked the NOR was mtd erase /dev/mtd0. At the time I ran that I thought there was no way of damaging the NOR as the RW jumper was removed.
If the instructions from the Howto pdf were in the One wiki page none of this would have happened. I'd have used a USB to do the NAND update, and assuming it worked, happy days...

For what I was trying to do, the bootmenu didn't have everything I needed. The NOR menu provides means to update NOR, but nothing that touches NAND. Likewise the NAND menu only dealt with NAND stuff (though I only saw it once before breaking NAND boot).

Reflecting back, I think there are a couple of things that contributed to this:

Booting into NAND from TFTP may have put the device into a state where flashing the firmware to NAND appeared to work, but didn't address the partitions correctly.
The role of the NOR RW jumper is unclear. I had too much faith that removing the jumper would ensure no changes to NOR (having made changes to NOR with the jumper in place).

I'm reminded of the saying "you might think your thing is idiot proof, but never underestimate the ingenuity of idiots".

I might take a swing at updating the wiki with the instructions from the how-to once it's daylight in my time zone. Meanwhile any guidance on how to use openocd to flash the factory .bin files would be gratefully received.

blogic · October 19, 2024, 4:48am

are you able to compile a rust tool ? if so you can use mtkuartboot to recover mtd0 using the serial port. I assume you are on windows as you mentioned putty.

cpswan · October 19, 2024, 5:24am

@blogic I'm happy to compile Rust stuff. Windows and PuTTY were just what I had to hand with the laptop I attached to the LAN port (as WAN is hooked up to my home LAN for now).

I was expecting to use a Linux VM to compile and run a patched OpenOCD.

Is this the mtk_uartboot you're referring to, and can you provide an example command line for restoring the NOR? Also does the router need to be in a particular state to use the tool?

blogic · October 19, 2024, 6:08am

@daniel knows the tool better than me

Lynx · October 19, 2024, 6:50am

So how come NOR could be erased notwithstanding jumper removed?

cpswan · October 19, 2024, 8:45am

@blogic (and @daniel if you can help) I'm not having much luck with mtk_uartboot. Whatever I try in terms of the .bin file I'm consistently seeing:

mtk_uartboot - 0.1.1
Using serial port: /dev/ttyACM0
Handshake...
hw code: 0x7981
hw sub code: 0x8a00
hw ver: 0xca00
sw ver: 0x1
Baud rate set to 460800
sending payload to 0x201000...
thread 'main' panicked at src/bootrom.rs:47:13:
returned data isn't the same. Tx: [215] Rx: [254]

Here's the Rust backtrace (not that it seems to shed much light):

stack backtrace:
   0: rust_begin_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   1: core::panicking::panic_fmt
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
   2: mtk_uartboot::bootrom::BootROM::echo
             at /home/chris/git/github.com/981213/mtk_uartboot/src/bootrom.rs:47:13
   3: mtk_uartboot::bootrom::BootROM::send_da
             at /home/chris/git/github.com/981213/mtk_uartboot/src/bootrom.rs:99:9
   4: mtk_uartboot::load_bl2
             at /home/chris/git/github.com/981213/mtk_uartboot/src/main.rs:70:20
   5: mtk_uartboot::main
             at /home/chris/git/github.com/981213/mtk_uartboot/src/main.rs:151:16
   6: core::ops::function::FnOnce::call_once
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ops/function.rs:250:5

In order to get mtk_uartboot past Handshake... I'm powering up the router. It doesn't seem to do anything with power already applied.

For what it's worth these are the lines I think I need (though I've tried other variations with exactly the same error as above):

For NAND:

sudo ./mtk_uartboot --aarch64 -s /dev/ttyACM0 -p openwrt-mediatek-filogic-openwrt_one-snand-preloader.bin -f openwrt-mediatek-filogic-openwrt_one-snand-bl31-uboot.fip

For NOR:

sudo ./mtk_uartboot --aarch64 -s /dev/ttyACM0 -p openwrt-mediatek-filogic-openwrt_one-nor-preloader.bin -f openwrt-mediatek-filogic-openwrt_one-nor-bl31-uboot.fip

blogic · October 19, 2024, 9:10am

@cpswan you need to use the RAM version of BL2
https://downloads.openwrt.org/snapshots/targets/mediatek/filogic/mt7981-ram-ddr4-bl2.bin

blogic · October 19, 2024, 9:16am

@Lynx the WP bits in the NOR were probably not set. lets get the device booting again first and then worry about the WP part.
spi-nor behaves different to patallel-nor in regards the the /WP pin

daniel · October 19, 2024, 10:18am

You may need to reduce baudrate to 115200, ie. --brom-load-baudrate 115200 --bl2-load-baudrate 115200.

cpswan · October 19, 2024, 10:50am

Thanks @blogic. bl2-ddr4.bin was one of the bins I'd already tried... but... trying again with a lower baud rate seems to do something. But from NOR it's still booting to System halt!. So I tried also adding the .fip

sudo ./mtk_uartboot --aarch64 --brom-load-baudrate 115200 -s /dev/ttyACM0 -p bl2-ddr4.bin -f openwrt-mediatek-filogic-openwrt_one-nor-bl31-uboot.fip

which gets Timeout waiting for specified message.:

mtk_uartboot - 0.1.1
Using serial port: /dev/ttyACM0
Handshake...
hw code: 0x7981
hw sub code: 0x8a00
hw ver: 0xca00
sw ver: 0x1
Baud rate set to 115200
sending payload to 0x201000...
Checksum: 0xec1a
Setting baudrate back to 115200
Jumping to 0x201000 in aarch64...
Waiting for BL2. Message below:
==================================
NOTICE:  BL2: v2.8(release):0911d48b2
NOTICE:  BL2: Built : 10:36:22, Mar  7 2023
NOTICE:  WDT: disabled
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 1024MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  CPU: MT7981 (1300MHz)
NOTICE:  Waiting for FIP data and signal from debugger
==================================
Timeout waiting for specified message.

daniel · October 19, 2024, 10:55am

NOTICE: BL2: v2.8(release):0911d48b2

This looks like the bl2-ddr4.bin file from mtk-openwrt-feeds for use with JTAG.
It is NOT the same as the mt7981-ram-ddr4-bl2.bin which is for use with mtk_uartboot.
Also note that you will need to set both, --brom-load-baudrate 115200 as well as --bl2-load-baudrate 115200.

blogic · October 19, 2024, 12:08pm

and in my notes I also have a -a in front of the -f

blogic · October 19, 2024, 12:10pm

ah you already have --aarch64

cpswan · October 19, 2024, 12:48pm

Thanks @daniel that seems to be providing some progress.

Running: sudo ./mtk_uartboot --aarch64 --brom-load-baudrate 115200 --bl2-load-baudrate 115200 -s /dev/ttyACM0 -p mt7981-ram-ddr4-bl2.bin -f openwrt-mediatek-filogic-openwrt_one-nor-bl31-uboot.fip

Gets me this, which looks like it's doing its thing:

mtk_uartboot - 0.1.1
Using serial port: /dev/ttyACM0
Handshake...
hw code: 0x7981
hw sub code: 0x8a00
hw ver: 0xca00
sw ver: 0x1
Baud rate set to 115200
sending payload to 0x201000...
Checksum: 0x3afe
Setting baudrate back to 115200
Jumping to 0x201000 in aarch64...
Waiting for BL2. Message below:
==================================
NOTICE:  BL2: v2.10.0   (release):OpenWrt v2024.01.17~bacca82a-3 (mt7981-ram-ddr4)
NOTICE:  BL2: Built : 11:38:21, Oct 18 2024
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  EMI: Using DDR4 settings
NOTICE:  EMI: Detected DRAM size: 1024MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  CPU: MT7981 (1300MHz)
NOTICE:  Starting UART download handshake ...
==================================
BL2 UART DL version: 0x10
Baudrate set to: 115200
FIP sent.
==================================
NOTICE:  Received FIP 0x558a4 @ 0x40400000 ...
==================================

But... it still boots to System halt!

I also tried this, with the switch set to NAND sudo ./mtk_uartboot --aarch64 --brom-load-baudrate 115200 --bl2-load-baudrate 115200 -s /dev/ttyACM0 -p mt7981-ram-ddr4-bl2.bin -f openwrt-mediatek-filogic-openwrt_one-snand-bl31-uboot.fip

Which gets: NOTICE: Received FIP 0xea8b1 @ 0x40400000 ...

But when I reboot the console logs still run to:

ERROR:   UBI error: No volume named fip could be found
NOTICE:  UBI: scanning [0x100000 - 0x10000000] ...
NOTICE:  UBI: scanning is finished
NOTICE:  UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
NOTICE:  UBI: VID header offset: 2048 (aligned 2048), data offset: 4096
ERROR:   UBI error: No volume named fip could be found
ERROR:   BL2: Failed to load image id 3 (-2)

I'd note that the BL2 version at the start of that boot is NOTICE: BL2: v2.9.0(release):OpenWrt v2023.10.13~0ea67d76-1 (mt7981-spim-nand-ubi-ddr4), which is different from what the UART programmer logs: NOTICE: BL2: v2.10.0 (release):OpenWrt v2024.01.17~bacca82a-3 (mt7981-ram-ddr4)

So I feel like I'm getting close here, but not quite there yet.

cpswan · October 19, 2024, 1:30pm

I should add that I have a USB drive inserted with all the .bin files from One Downloads on the off chance that part of this process needs them.

There are two things I'm possibly missing:

Should the SPI NOR WP jumper be on or off? I've tried both, but what should it be?
What should be happening after the mtk_uartboot tool has completed. I see some activity on the front LEDs, with boot flashing and then OK eventually lighting up. But nothing happening in the console.

I've also tried pressing both buttons (individually and together). But I'm not sure if that's helping.

daniel · October 19, 2024, 1:08pm

Good, that indeed looks much better.

But... it still boots to System halt!

That's the expected result, because what mtk_uartboot does is just load
things into the RAM. It does not write anything to the flash, it just
helps you to boot even though the device has a broken bootloader.
You still have to re-write the bootloader yourself, as you seem to be
using Linux, the best way is to directly start screen once
mtk_uartboot as completed, ie:

./mtk_uartboot --aarch64 --brom-load-baudrate 115200 --bl2-load-baudrate 115200 -s /dev/ttyACM0 -p mt7981-ram-ddr4-bl2.bin -f openwrt-mediatek-filogic-openwrt_one-nor-bl31-uboot.fip ; screen /dev/ttyACM0 115200

Hint: If you add you user to the 'uucp' group you won't need sudo
(after logout and subsequent login).

Now you can use a TFTP server to re-flash the bootloader on the NOR
flash using the bootmenu.

Once you have completed that, you can follow the instructions in the PDF
document and re-flash the NAND bootloader.

Sorry about the slightly messy situation, only the first 50 boards were
affected by this issue because the size of the flash chip was increased
last minute...

cpswan · October 19, 2024, 4:48pm

Thanks @daniel (and @blogic) using screen got me to see the boot menu in a way that simply wasn't happening with minicom.

There was one very counterintuitive part to the process - I had to select the option to lock the NOR. Once I did that it took much longer to write the various pieces from TFTP, which I took as a good sign that it was actually working.

With the NOR restored I was then able to use the recovery from USB to get NAND back in shape (and then up to date). The one hiccup there is it seems a little fussy about USB drives. The first one I tried got me:

usb_new_device: Cannot read configuration, skipping device 058f:6387
1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
** Bad device specification usb 0 **
Couldn't find partition usb 0:1
Can't set block device

So I pulled another (maybe better quality) one out of the sorting hat, and that worked

Another oddity is that it wouldn't play nicely with the Ethernet port on my Lenovo X13 (possibly due to lacking auto MDX at both ends?) whilst it worked fine with my HP Omen 15 (that I'd been using yesterday for TFPT etc.).

I'll try to write up a succinct version of the process in the Wiki - tomorrow - I need a break from this for a bit...

Thanks again for all the help, and good luck to anybody else finding this thread.

blogic · October 19, 2024, 5:06pm

awesome, its alive again ...

u-boot only supports MBR+FAT32 ... possibly the stick had something different on it. hence you see the -> ' Couldn't find partition usb 0:1'

I will write a bit more info about the spi-nor locking tomorrow.