R8000 Seemingly Bricked after 22.03.5 -> 23.05.0 Upgrade

My R8000 seems to have semi-bricked itself after the upgrade. I told it to preserve settings, flashed it... and it never came back. Well, it never came back on the LAN at least.

WiFi was up and I could see the IP. I could connect but it was not issuing IP addresses via DHCP. I couldn't get the connection to stay up in that way, even when I set my phone to static IP on that connection, it would disconnect. So at least it doesn't look like it completely bricked itself at that point.

I did a factory reset by long-pressing the reset button on it, and now no WiFi comes up, but I still can't get to it on the LAN ports, either the one I had it set to before nor the default 192.168.1.1 (I set my laptop to 192.168.1.11/24 for troubleshooting purposes.

Power light eventually goes white. Port link lights do light up when I plug in a cable, my laptop sees the link is up, but I can't ping 192.168.1.1 where I would expect the R8000 to be after a factory reset.

Is there a way to troubleshoot this further without additional hardware? I have 3 more R8000 units on 22.03.5, but I obviously have serious concerns about flashing any others until I can establish the root cause and resolve this problem with the first one.

Based on this:

failsafe mode is inaccessible on R8000? Is that really the case?

After a bit more troubleshooting it looks like if I time the reset press to between when the power light turns orange and the link light for the ethernet port comes up - I can get 3 pings back from 192.168.1.1. And then it goes away again and seemingly carries on with the boot and remains completely inaccessible.

Edit: No, it looks like 192.168.1.1 becomes responsive for about 3 seconds regardless of whether I press the reset button. And then goes away forever again. Is there anything I can do in that 3 second window to try to make it work again? It's almost like it enters the failsafe mode but then proceeds out of it into regular mode that doesn't work.

Keep hammering that button like your life depends on it. Or keep it pressed from the moment you power it on a couple of minutes.

One of those usually works🤣

1 Like

OK, I capitulated on the reset button and got a serial console onto it.
I ran firstboot and rebooted it.
It boots. It sets itself the IP address on br-lan of 192.168.1.1.
My laptop plugged into any of the LAN ports and set to 192.168.1.11 cannot ping it.
It cannot ping my laptop on 192.168.1.11.

So I figured OK, no network for whatever reason, I can't load anything over tftp.
But it turns out default install image doesn't contain usb-storage driver.
How exactly do I reflash this thing with no network and non usb storage driver? That seems like a massive, asinine oversight.

Am I really going to have to paste the image in and save it from stdin?

As another additional factoid - bootloader initialized the network and briefly appears on 192.168.1.1.
That goes away as soon as the kernel begins to load.
Is there a way to use the bootloader to tftp down the flash image somewhere where it will be accessible from the linux console once the kernel boots?

Yup, I actually went there in the end - copy paste the image file through serial terminal. I'll document the procedure later, but the main point here is that 23.05.0 bricks at least some R8000 routers by rendering all wired networking inoperable. It should be removed from the download list for R8000 until that is rectified because the recovery procedure is non-trivial and requires a serial console cable.

Decoding a base64 encoded file (for copy-pasting purposes through serial console) using an awk program (because base image comes with awk but not with base64 decoder) takes almost as long as copying an 8MB file over 115200 bps serial connection.

1 Like

Netgear routers can usually be revived using tftp by holding down the reset button when u turn it on and keep holding it down until the lights flash. This is what I do on my R7800.

You can also use nmprflash. It does basically the same thing as tftp, but ymmv.

I tried holding down the reset button. Network IP of the router never becomes accessible once the bootloader starts booting the kernel. I got the serial console on it, and the IP is set but it cannot be pinged from the same subnet, nor can it ping out on the same subnet. Switch works, but the bridge never connects.

Failsave mode makes no difference (triggered via the serial console).

Unless tftp can be initiated from the bootloader itself, I don't see how any recovery method that requires networking to work can be successful.

Network is initiated by he bootloader itself with a static ip of 192.168.1.1 (afaik this is the default ip on all netgear routers)

Yes, but this is only up until it starts loading the kernel. Once it starts loading the kernel bootloader configured networking goes away. So I guess I could have tftp-ed a working kernel from a working image to boot that instead of 22.05.0, but that would have been more complex to follow to completion compared to just pasting in a working image and flashing that.

Not following your arguments here. The point of invoking tftp flash with the bootloader is flash your router before it boots the kernel.

And no, it’s not more complex than opening up your router, connecting the serial port, booting up OpenWrt only to find that the network doesn’t work, then transfering a new firmware through the console. Lol :slight_smile:

It is arguably more complex because you have to setup a tftp server and the kernel and initramfs to load over tftp. It is at the very least comparable.

No, you just: tftp -i 192.168.1.1 put «image»

Either the openwrt image or the oem image. (No kernel/initramfs).

It literally takes a few seconds.

Huh, fair enough. I always assumed it's like normal linux tftp boot that requires kernel and initramfs. I guess this is different if it just takes one single big blob. Still, it was a fun exercise.

2 Likes

@gordan-bobic
Thank you for this post, Sir.
I just got into the same problem and was completely lost until I found this thread.

Please, someone, remove the image from the firmware finder until it resolved.
And, please, share the unbrick via awk instructions when you have a chance as I'm not that advanced and can't spent any more time trying to figure out it myself.

@theMan
I tried legacy recovery via TFTP multiple times, but it doesn't work as it used to. I have no clue why, as it used to work just fine previously. Even when I manually try to enable tftpd in CFE it won't take an image...

I appreciate you all a lot. Much love.

There is an additional snag, I found that after I super-clean-flashed things back to 22.03.5, wifi wouldn't work at all. No Wireless menu in Luci, nothing. I suspect that flashing everything super clean like that actually nuked some system partitions containing wifi firmware that aren't replaced / brought along by OpenWRT.

So to unbrick wifi after this episode, I actually had to flash back to Netgear's own firmware, get that working, then flash back to 22.03.5, and only then did my wifi radios work again.

Which then begs the question - is there a way to update this potentially hidden firmware with OpenWRT without flashing completely back to Netgear firmware, by just extracting the blobs from the Netgear firmware and somehow putting them onto one of the MTD partitions?

Either way, my R8000 is now finally fully unbricked and working again.

Not too sure the oem firmware includes «blobs» since those are supposed to be specific for every router.
But Broadcom (or the bootloader aka CFE) use nvram to store certain variables, such as boot_wait and other stuff. Maybe these were messed up rendering tftp and/or wifi/wired network unusable.

DD-Wrt has a wiki on 30/30/30 reset, other methods of resetting, and nvram. I don’t how much of this is relevant for R8000 though.

https://wiki.dd-wrt.com/wiki/index.php/Hard_reset_or_30/30/30

As promised, here are the instructions for how to paste in and decode a firmware image over serial console.

I hope this helps, apologies for taking so long to get back to this, work time pressures have been very high.

2 Likes