EAP600 Bricked

lightsong57 · June 23, 2022, 2:30am

I have an Engenius EAP600 that was running an OpenWRT snapshot from August 2021. Not knowing about the Senao Failsafe issue, I pushed an update to the latest version through LUCI (not keeping settings). After waiting a while, it was still not accessible at 192.168.1.1 (or its old IP address).
There is a chance that we switched from POE to powered during the update.

Connecting with serial and watching it start up, I very quickly get the output below, but then it hangs indefinitely.

U-Boot 1.1.4 (Jan 10 2013 - 11:04:51)

EAP600 (ar934x) U-boot
DRAM:
sri
Wasp 1.2
wasp_ddr_initial_config(276): Wasp (16bit) ddr1 init
wasp_ddr_initial_config(426): Wasp ddr init done
Tap value selected = 0x8 [0xaa55aa55 - 0x0]
Setting 0xb8116290 to 0x10202d0f
 4 MB
Top of RAM usable for U-Boot at: 80400000
Reserving 230k for U-Boot at: 803c4000
Reserving 192k for malloc() at: 80394000
Reserving 44 Bytes for Board Info at: 80393fd4
Reserving 36 Bytes for Global Data at: 80393fb0
Reserving 128k for boot params() at: 80373fb0
Stack Pointer at: 80373f98

I saw from this forum post that the AP may hang at this point if you boot it with the UART connected. However, booting with RX disconnected hangs at the same spot. Booting with USB disconnected, then connecting ~10 seconds later yields only a blank and non-responsive screen.

The ethernet activity light shows blue and blinks with activity when connected to my computer or router. But pings to 192.168.1.1 time out, and no new devices show up in my router's list.

What else can I try to recover this AP? Is it totally gone?

tassapan · June 23, 2022, 9:53am

Try to force failsafe mode by interrupt boot process with "F" key on serial console.
Then you can flash new OpenWRT firmware with webui at 192.168.1.1.
This is as far as I can recall.

key2029 · June 23, 2022, 9:59am

If you can still be able to ping 192.168.1.1 try the following:
use Putty
root@192.168.1.1
fw_setenv rootfs_checksum 0
reboot

web 192.168.1.1 (If Chrome is failed, try Firefox)
Failsafe mode: Choose File & Upgrade

If you cannot ping to 192.168.1.1 you need USB to serial console cable

Remove 4 screws holding the EAP600 lid
Remove 2 screws holding the board and flip it, you will see 4 pin serial port.

157181166×2592 291 KB
You will see white arrow which is on the left in the picture that is power, attach it with RED cable.
The next one will be ground, attach it with BLACK cable.
Then WHITE and GREEN cable, in some manual it says green then white but it was not usable, for me, white and green worked.
in TTY, set serial to baudrate 115200, 8 data bits, no parity, stop bits 1, flow control: off.
Power on EAP600 to boot
In boot sequence it will ask to press f to get in to failsafe mode, you have to quickly press f (or repeatedly press it since power on) to get into failsafe mode.
exit TTY
pray for it to work and get into web 192.168.1.1

lightsong57 · June 23, 2022, 8:50pm

I am still not able to ping or SSH at 192.168.1.1.
Previously, I was not connecting the red wire, and I had Flow Control set to Xon/Xoff.
However, changing both of those to your recommended settings, it still hangs at Stack Pointer at: 80373f98 and does not proceed from there. I tried pressing F at different times, continuously on boot up, etc, and have not made any more headway.

tassapan · June 23, 2022, 11:45pm

You can try this;

Connect usb to serial as advice above.
Connect RJ45 to your PC.
Power off your device and use putty or kitty and set it as advice.
While power on your device, keep tapping "F", your device will boot again in failsafe mode.
Use webui to inject new firmware.

Maybe this will work;
While power off, keep pressing reset button then power on device. It may take you 10-15 seconds.
You can try this method without serial cable.
You may get in to OpenWRT failsafe mode, not Engenius failsafe mode. Do not sysupgrade current 21.02 or 22.03RC version. Use your prior 21.02-SNAPSHOT, not the current one.
If you successfully unbrick your device, do not upgrade with normal method.
Put your device in failsafe mode by issue command [fw_setenv rootfs_checksum 0] with ssh, then you can directly upgrade to 21.02+ via webui.

Make sure that your PC's subnet must be 192.168.1.0/24.

key2029 · June 24, 2022, 5:37am

Can you copy all the text during boot sequence? We are not sure that you boot passed the failsafe notification or not.

lightsong57 · June 24, 2022, 11:54am

No change in results following the two methods above. The reset button didn't seem to have any effect.

Here is the complete serial output (it has been the same for every test):

U-Boot 1.1.4 (Jan 10 2013 - 11:04:51)

EAP600 (ar934x) U-boot
DRAM:
sri
Wasp 1.2
wasp_ddr_initial_config(276): Wasp (16bit) ddr1 init
wasp_ddr_initial_config(426): Wasp ddr init done
Tap value selected = 0x8 [0xaa55aa55 - 0x0]
Setting 0xb8116290 to 0x10202d0f
 4 MB
Top of RAM usable for U-Boot at: 80400000
Reserving 230k for U-Boot at: 803c4000
Reserving 192k for malloc() at: 80394000
Reserving 44 Bytes for Board Info at: 80393fd4
Reserving 36 Bytes for Global Data at: 80393fb0
Reserving 128k for boot params() at: 80373fb0
Stack Pointer at: 80373f98

tassapan · June 24, 2022, 11:58am

Method 2 of this guide;
git.openwrt.org Git - openwrt/openwrt.git/commitdiff
And you may already known about this;
Upgrade to OpenWrt 21.02-RC3 Bricked EAP600 - need help unbricking - #4 by mpratt14 - Installing and Using OpenWrt - OpenWrt Forum

I had unbrick this device once, but I can't actually recall how to do it;

lightsong57 · June 24, 2022, 12:41pm

I have tried the information from both of those posts. The problem seems to be that it hangs on boot up prior to accepting the failsafe signal.

@mpratt14, any thoughts?

mk24 · June 24, 2022, 1:02pm

This looks quite bad. The bootloader is crashing before the bootloader even fully starts. This could be due to corruption of the bootloader partition(s) of flash, which would take a flash programmer to restore. I don't know if that model had TP-Link's two stage bootloader or not.

This has nothing to do with failsafe in OpenWrt, since the OpenWrt kernel never even tries to boot.

lightsong57 · June 24, 2022, 9:33pm

Can I flash the chip through USB-to-TTL on the UART pins, or will I need a SIOC16 test clip and programmer (probably a Raspberry Pi)?

frollic · June 25, 2022, 7:56am

Can't flash it using the serial port.

mpratt14 · August 27, 2022, 3:34am

Sorry...I'm very late to the party

@lightsong57

it looks like the RAM is toasted, which can be caused by several reasons, most of them power related. It could be bad capacitors or water damage or who knows what. If you ever had POE and a power cable connected together, that maybe could cause something like this.

bootloader appears to detect and allocate only 4 MB out of the 128 MB total RAM that's supposed to be available

I found one of my EAP600 and got the serial log to compare

U-Boot 1.1.4 (Jan 10 2013 - 11:04:51)

EAP600 (ar934x) U-boot
DRAM:
sri
Wasp 1.2
wasp_ddr_initial_config(249): (32bit) ddr2 init
wasp_ddr_initial_config(426): Wasp ddr init done
Tap value selected = 0xf [0x0 - 0x1f]
Setting 0xb8116290 to 0x10202d0f
128 MB
Top of RAM usable for U-Boot at: 88000000
Reserving 230k for U-Boot at: 87fc4000
Reserving 192k for malloc() at: 87f94000
Reserving 44 Bytes for Board Info at: 87f93fd4
Reserving 36 Bytes for Global Data at: 87f93fb0
Reserving 128k for boot params() at: 87f73fb0
Stack Pointer at: 87f73f98
Now running in RAM - U-Boot at: 87fc4000
...
...
...

note that the line right after the "Stack Pointer" print informs you that the bootloader is now running in the RAM, and the address for the stack has a much greater offset. The fact that it's failing there suggests to me the RAM is dead. Earlier you can see that all of these offsets in memory are much different. This line also stands out

wasp_ddr_initial_config(276): Wasp (16bit) ddr1 init

compared to mine

wasp_ddr_initial_config(249): (32bit) ddr2 init

First thing to try is to clean the entire board with alcohol, this includes (CAREFULLY) prying open all of the metal covers (radio frequency shields) with a small flathead screwdriver

after cleaning with alcohol and a brush let it dry for 24 hours and try again

If it's still dead, you would have to find a donor board that has compatible RAM and transfer it with hot air soldering (with flux)

I use a butane torch with hot air accessory, but that's the cheap way and its not very fun
youtube is your friend but you may be able to find a repair shop to do it

I think the bootloader in flash is OK, because it's more likely there would be no output on UART at all, or just garbage, but who knows, its possible there's multiple things going on.

mpratt14 · August 27, 2022, 3:39am

it could be much more simple, that some surface mount component on the board failed or shorted, and that is causing the RAM to not get enough voltage for the chip to operate.

that could be diagnosed by anything that looks burnt on the board, or any part of the board (other than the SOC) that is very hot compared to the rest. Pro repair shops use thermal cameras to pinpoint this, but you could put gloves on and touch the board directly while it is powered

mpratt14 · August 27, 2022, 3:54am

Highly recommend never connecting UART power, if the USB adapter uses 5V instead of 3.3V it can ruin the router.

I actually bend the power pin with pliers when the board is off, at a 45 degree angle, as a reminder to not connect anything to it, and to help me remember the pinouts without having to look at the board closely.

The reason it exists is for developers to test the SOC before a power rail exists on the board, but when you have a finished product, the power rails are connected to that pin, and the board components have to work, otherwise you would have to isolate the power rails from the SOC by removing something from the board in order to properly power it through that pin. Otherwise its possible that you would be powering everything on the board, and the power would hit linear regulators on the output side (i don't even know if it would step it up or not), and the amps could easily exceed the capacity of the board traces, whether there is a short somewhere or not.

(I havent really studied this, im just speculating, but its not safe regardless )

If anything, you can use that pin as a power source. and BTW, you should make sure that pin has 3.3V when you power the board, and if not, then you know there is a power problem for sure (but again, you would see no output on UART if that was the case)

lightsong57 · August 27, 2022, 2:50pm

Hi @mpratt14,

Thank you for the detailed information!
Thinking back, I am fairly confident we plugged in AC power while PoE was still connected, during the update. Based on your explanation I almost certainly fried the RAM.

I'll shelve it until I have either time to figure out the soldering and replacement components, or budget to buy something better.

mpratt14 · August 27, 2022, 9:24pm

it could potentially be a very easy fix or a very difficult one, but needing the right tools regardless.
at the very least, you now have a donor board that can be used to fix another similar board. on Ebay some people sell their non-working devices, or maybe you can find a donor board to fix this one.

screenshot from FCC photos for reference.
SOC is the larger "Atheros" chip in the middle
RAM are the 2 smaller "NANYA" chips on the left (unfortunately they are BGA)
and this is all under the metal RF shields which are removed like a lid around the edges

like I said, there's no way for me to tell you whether the RAM is bad or if it is not getting the right voltage to operate from something else being bad. Apparently, that voltage is 1.8 V

mpratt14 · September 1, 2022, 3:10am

you can try your luck with the most similar RAM chips available

https://www.arrow.com/en/products/w971gg6nb-25/winbond-electronics
https://www.arrow.com/en/products/w9751g6nb-25/winbond-electronics