Thanks. Yes, I definitely ran into that driver issue on Windows. Which is one of the reasons I moved over to the Linux machine for a lot of this testing. That plus the delay related to putty.exe versus screen to make the serial capture more reliable after mtk_uarboot (at least up until now).
I will plan on adding a CH341 based adapter to my debugging items in the future. Thanks for the guidance.
Great work guys -- I can now upgrade my E8450 without the fear of OKD. Current device has been working fine with last October trunk build after (serendipitous) 1.0.2 UBI upgrade. I wanted to move to newer kernel for a while now but you guys had me worried.
Does this work for 1.0.3 version ?
So I guess other than pushing the patch upstream, what else can we do? What's the best plan of action for owrt-ubi-installer?
EDIT: Oops.. Didn't see @grauerfuchs response above. Sounds like you have things well in hand..
OKD fixed thanks to @daniel!
The below files apply his not-yet-merged patch to the TF-A 2.10 bootloader:
For flashing to the device: https://github.com/grauerfuchs/owrt_device_support/raw/main/bl2-2.10-mediatek-mt7622-linksys_e8450-snfi_fix.bin
For booting via mtk_uartboot (using FIP on flash): https://github.com/grauerfuchs/owrt_device_support/raw/main/bl2-via-mtk_uartboot-mediatek-mt7622-linksys_e8450-snfi_fix.bin
For flashing to the device: https://github.com/grauerfuchs/owrt_device_support/raw/main/bl2-2.10-mediatek-mt7622-linksys_e8450-fip_in_ubi-snfi_fix.bin
For booting via mtk_uartboot (using FIP on flash): https://github.com/grauerfuchs/owrt_device_support/raw/main/bl2-via-mtk_uartboot-mediatek-mt7622-linksys_e8450-fip_in_ubi-snfi_fix.bin
No doubt he's taking a well-deserved breather before updating and publishing new installer versions with the patch in place.
fantastic work all! esp @daniel for his initial vision and contnued brilliance.
i look forward to resurrection on a more stable frameowrk.
As follow-up - I tried my Windows system and it worked fine. Boots the router up and still connected to serial console in putty. Tried several times and worked every time.
And there is more output now, versus the other system screen output that I posted before, including that the network interface really did go to 1Gbps like I would expect. This is just later in the router boot-up process and was cutoff in the other output.
There must be some strange incompatibility with my serial adapter and that Linux box. I will get a new serial interface ordered.
@daniel - Thanks so much for finding the root cause!!!
And Thanks to everyone else involved in this debug and fix!!
I updated the bl2 on my router that was supposed to have a deadly OKD on my next reboot. This didn't happen with the new version.
Thank you daniel and all the developers for the excellent job.
For those of us who used the 1.0.3 installer, can we just patch the bl2 in the next few days and be good?
If so, can it be done on a running instance (e.g. overwrite the partition somehow)?
The preferred method of updating is via tftp and the U-Boot console, because updating bl2 is an item in the boot menu and it will take care of everything for you.
If you don't have access to that, it should be possible to do so with the kmod-mtd-rw module enabled. However, we're seeing a lot of bitflips taking place on the flash of these units, and the typical 'overwrite via dd' method only writes a single copy of the bootloader vs. the U-Boot method, which writes the primary copy and three backups. Therefore, I highly recommend the U-Boot method.
Thanks. It's probably a good idea to put out a step by step guide (e.g. on the wiki next to the existing OKD recovery steps) as I'm sure there will be lots of people asking how to do it (like me!)
edit: I assume you are talking about this u-boot menu item?
8. Load BL2 preloader via TFTP then write to flash.
I recommend to just downgrade bl2 to https://archive.openwrt.org/releases/22.03.0/targets/mediatek/mt7622/openwrt-22.03.0-mediatek-mt7622-linksys_e8450-ubi-preloader.bin
You can do that either in U-Boot menu using a TFTP server (you have to rename the file to openwrt-mediatek-mt7622-linksys_e8450-ubi-preloader.bin or by installing kmod-mtd-rw when inside OpenWrt/Linux.
^^ You heard it from the man in charge. The version for 22.03.x is the official version, so there's no need to worry about unknown binaries produced by someone you don't really know, and who knows what other bugs might be present in 2.10 that we haven't yet found? TF-A 2.4 is more than adequate for booting the 23.x stable firmware, and it's well tested and solid.
As for the option on the boot menu, yes; That's the one.
Are there different steps (and consequences) to replace the bootloader if we're on SNAPSHOT or 23.x?
Will this change affect u-boot then?
Edit:
Usually in the Linux driver world, negative return code means errors. Success is usually 0.
So far, we have only tested the fix on FIP stored in MTD partition? Do we have any successful test of the fix for FIP stored in UBI volumes?
I've spent the last 2 hours doing exactly that: Gradually increasing the number of bitflips on the LEDs covering the fip volume in UBI.
Results:
bitflip_threshold has been reachedbitflip_threshold is initialized to the ECC strength for SPI-NAND, which seems a bit late, hence https://patchwork.ozlabs.org/project/linux-mtd/patch/2117e387260b0a96f95b8e1652ff79e0e2d71d53.1723427450.git.daniel@makrotopia.org/All together I've taken care of it there:
If there are no objections within a short time tomorrow I would merge it.
Excellent!
Now that OKD is resolved, we can start playing with snapshot builds safely, once a new installer has been released that is ... heh heh.