Sophos APX 120 bricked ?!

h0schi · July 2, 2021, 2:42pm

Hello Comm'

i got two bricked Sophos APX 120 access points.
The Sophos Flash Tool can not help me at my problem:
https://support.sophos.com/support/s/article/KB-000039314?language=en_US

I plugged in an UART-cable to check what the problem is.
I can not stop at the point "Hit any key to stop autoboot: 0" for a TFTP-boot.

Is there any possibility to reflash / unbrick the access points or are the APs totally dead ?

Format: Log Type - Time(microsec) - Message - Optional Info
Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.BF.3.1.1-00120
S - IMAGE_VARIANT_STRING=DAABANAZA
S - OEM_IMAGE_VERSION_STRING=CRM
S - Boot Config, 0x00000020
S - Reset status Config, 0x00000010
S - Core 0 Frequency, 0 MHz
B -       261 - PBL, Start
B -      1338 - bootable_media_detect_entry, Start
B -      1677 - bootable_media_detect_success, Start
B -      1691 - elf_loader_entry, Start
B -      5068 - auth_hash_seg_entry, Start
B -      7210 - auth_hash_seg_exit, Start
B -    577087 - elf_segs_hash_verify_entry, Start
B -    694249 - PBL, End
B -    694273 - SBL1, Start
B -    785279 - pm_device_init, Start
D -         7 - pm_device_init, Delta
B -    786725 - boot_flash_init, Start
D -     52831 - boot_flash_init, Delta
B -    843802 - boot_config_data_table_init, Start
D -      3833 - boot_config_data_table_init, Delta - (419 Bytes)
B -    851006 - clock_init, Start
D -      7586 - clock_init, Delta
B -    863064 - CDT version:2,Platform ID:8,Major ID:1,Minor ID:1,Subtype:0
B -    866477 - sbl1_ddr_set_params, Start
B -    871573 - cpr_init, Start
D -         2 - cpr_init, Delta
B -    875955 - Pre_DDR_clock_init, Start
D -         4 - Pre_DDR_clock_init, Delta
D -     13175 - sbl1_ddr_set_params, Delta
B -    889693 - pm_driver_init, Start
D -         2 - pm_driver_init, Delta
B -    959596 - sbl1_wait_for_ddr_training, Start
D -        27 - sbl1_wait_for_ddr_training, Delta
B -    975198 - Image Load, Start
D -    152195 - QSEE Image Loaded, Delta - (297752 Bytes)
B -   1127822 - Image Load, Start
D -      1445 - SEC Image Loaded, Delta - (2048 Bytes)
B -   1138225 - Image Load, Start
D -    223896 - APPSBL Image Loaded, Delta - (458523 Bytes)
B -   1362518 - QSEE Execution, Start
D -        60 - QSEE Execution, Delta
B -   1368739 - SBL1, End
D -    676574 - SBL1, Delta
S - Flash Throughput, 2010 KB/s  (758742 Bytes,  377342 us)
S - DDR Frequency, 537 MHz


U-Boot 2012.07 [Chaos Calmer unknown,unknown] (Nov 02 2018 - 08:13:09)

smem ram ptable found: ver: 1 len: 3
DRAM:  256 MiB
@machid : 0x8010100
NAND:  spi_nand: spi_nand_flash_probe SF NAND ID 0:ef:ab:21
SF: Detected W25M02GV with page size 2 KiB, total 256 MiB
SF: Detected MX25L1605D with page size 4 KiB, total 2 MiB
ipq_spi: page_size: 0x100, sector_size: 0x1000, size: 0x200000
258 MiB
MMC:
In:    serial
Out:   serial
Err:   serial
machid: 8010100
flash_type: 0
Hit any key to stop autoboot:  0
Creating 1 MTD partitions on "nand1":
0x000000000000-0x000010000000 : "mtd=0"
UBI: attaching mtd2 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI error: ubi_read_volume_table: the layout volume was not found
UBI error: ubi_init: cannot attach mtd2
UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
UBI init error 22
Error, no UBI device/partition selected!
Creating 1 MTD partitions on "nand1":
0x000000000000-0x000010000000 : "mtd=0"
UBI: attaching mtd2 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI error: ubi_read_volume_table: the layout volume was not found
UBI error: ubi_init: cannot attach mtd2
UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
UBI init error 22
Error, no UBI device/partition selected!
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Creating 1 MTD partitions on "nand1":
0x000000000000-0x000010000000 : "mtd=0"
UBI: attaching mtd2 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI error: ubi_read_volume_table: the layout volume was not found
UBI error: ubi_init: cannot attach mtd2
UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
UBI init error 22
Error, no UBI device/partition selected!
Wrong Image Format for bootm command
ERROR: can't get kernel image!
resetting ...

Hardware-Picture:
https://h0schi.cloud/apps/files_sharing/publicpreview/pQHigQyPsDSw2CD?x=2548&y=947&a=true&file=21-06-30%2021-52-47%205106.jpg&scalingup=0

frollic · July 4, 2021, 9:35am

Tried keeping any physical button pressed, to force a TFTP boot?

h0schi · July 4, 2021, 6:43pm

Yes, already tried.
I see that the AP recognized the press of the Reset-button (during COM-connection), but the boot will not stop.

att12 · July 11, 2021, 4:05pm

Hi,

are you shure that you connected both (RX and TX) correctly? Because if the device only sends you the boot output it would not listen to your inputs at all if Rx on the device is not correctly connected.

I have also a bricked APX 120 lying around, but I didn't do anything yet with it because of too few spare time....
Could you send some pictures of your cabling though?

Cheers

h0schi · July 13, 2021, 6:49am

Hi, @att12

here a pic of the cabling:
https://h0schi.cloud/s/kYiatLmJb32gZ2B/preview

I tried some combination, but these combination is working and i have a screen output via putty.

Should i try another UART-adapter ?

att12 · July 13, 2021, 10:43am

If you have a different one, you could try that. sometimes it helps. I remember my Aruba AP was also very picky about the Converter.

BTW: Is it a 5 V Converter? You should use a 3.3V one, maybe its already "fried" if you bombard it with 5V

Cheers

att12

h0schi · July 13, 2021, 11:05am

Do you have a prefered UART-Adapter ?
Can you send me a link ?

frollic · July 13, 2021, 12:24pm

https://www.amazon.com/dp/B07WX2DSVB

Works very well.

h0schi · July 18, 2021, 8:20pm

I received the new UART-adapter but it's not working - it seems likte that the counter-value of "press any key" is 0
Maybe a RJ45 to Serial-cable can help here ?

I have no clue anymore.

frollic · July 18, 2021, 8:50pm

Try pressing any key before you actually see the counter, proactively, one might say.

What does the physical format of the cable have to do with the 0 sec timeout?
Get a new mouse mat, it's just as relevant ,)

h0schi · July 18, 2021, 9:08pm

That exactly what i do

I click on the Putty-windows and press any keys on my keyboard, all the time - i tried it about 15 minutes.

Are you from germany, @frollic ?

h0schi · August 13, 2021, 7:17pm

Anyone another idea ?

JuergenB · July 25, 2022, 1:19pm

Hi,
i was able to debrick 3 APX120 devices.
2 APX120 models hat autoboot set 5 seconds.

I only needed to format the NAND kernelfs and tftpboot the APX.uimage from network.
after a reboot the NAND was formated automatically and the APX came up and connected to Sophos XG.

The third APX had Autoboot set to 0 and i was unable to enter uBoot.

So i compared the NOR Flash (2MBit) from the other two APX120.
The only diff i found was the MAC and serial numbers.

So i flashed the third devices with a NOR Flash, where i set the autoboot to 10s before powering off.

I think there must be a different way, i think i need to get the NAND Chip unreadable by pulling some signals from NAND to GND and uBoot will exit to CLI.

trellix has this information, that might help ..

The glitching method consists of connecting a jumper wire to ground
on one side and poking at one of the I/O pins with the other end of the
jumper cable, while the filesystem loads. Grounding the IO pin will cause
misreads and panic the bootloader

dirk_k · August 4, 2022, 3:44pm

Hi JuergenB,
how do you stop the autoboot ?
i see the following, but are unable to break the boot proccess:
Hit any key to stop autoboot: 5 4 3 2 1 0

JuergenB · August 4, 2022, 5:50pm

Hi,
i stop autoboot by pressing the space bar or any key.

But, i had some problems with a USB-C / RS232 Dongle, that i attached at my Thunderbolt 3 Dock. I could watch uBoot but couldn´t stop.

After i plugged the dongle direct at my Notebook, i was able to press the space key.

after stopping i tried several uImages, starting from early alpha releases to some OpenWRT releases.

at last i loaded the latest uImage from a sophos XG 19.0

eviljin · October 5, 2022, 10:11pm

Thanks God I found this thread.

@JuergenB or someone can share more details on how to unbrick this AP? I have 2 with solid red and desperately trying to get it to work again.Thx.

JuergenB · October 6, 2022, 7:04am

Hi eviljin,
have you checked the console log?
You can connect a RS232 cable to pin 2,3,4 (TX,GND,RX or RX,GND,TX) and see what the APX does. (image is not perfect , yellow and red are TX/RX and black is GND. Pin 1 has 3.3V

i had 2 errors in the past

corrupted mtd2 partition
hardware error (i think NAND or memory chip is faulty)

mainly it´s about breaking uboot (Autoboot=0 is a problem) and you need to modify the flash chip.
Another way would be a boot glicht, but i was not able to find a way.
here one would corrupt the NAND chip during boot and uBoot would fall back to cli.

after you get access to uBoot, you can download new apx.uimage with tftpboot

So please check the boot message

eviljin · October 6, 2022, 12:29pm

Thank you so much for the diagram, saving me time poking around
I'm picking up USB to serial adapter today, I'll update once I get to the console.

eviljin · October 6, 2022, 8:39pm

UBI: attaching mtd2 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI error: ubi_read_volume_table: the layout volume was not found
UBI error: ubi_init: cannot attach mtd2
UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
UBI init error 22
Error, no UBI device/partition selected!
Wrong Image Format for bootm command
ERROR: can't get kernel image!
resetting ...

Bad news is autoboot: 0.

JuergenB · October 7, 2022, 1:40pm

Hi, this i bad news.

you would need to break autoboot and enter uBoot cli.
I do not know if there is any hidden keystroke to enter uBoot.

I used a working APX120, connected to a XG 115, and used SSH from the XG cli.
With access to the APX fom the XG shell, you can change autoboot to 5 sec.

Next i used a REVELTRONICS REVELPROG-IS serial programmer for EEPROM, FLASH, FRAM and a Pomona 5250 SOIC-8 Clip.

With this Clip i was able to read the smaller flash chip U13 from a the working APX and flash the content to the broken one.

At the next boot, autoboot was set to 5s, i changed the MAC and Serial back to the broken one (read from the case). Then i formated the mtd partition and loaded a new apx.uimage file.

after a few reboots, the APX was connecting to the XG firewall.

Another way would be to interrupt the read process from the bigger flash chip U26.
With a old Flash Chip (one with a lot´s of address lines) you could shorten somone address lines and uboot would fail and enter emergency mode.

But this chip is serial one and i have not worked out, how to disturb the load process of code from U26.