Recovering Linkit MT7688 with JTAG

Hello All,

tldr - I want to use jtag for an mt7688 and need some help.

I have been running a custom OpenWrt build on my LinkIt smart board which has a mt7688 processor. It was working fine for quite a while; however, I can no longer connect over ssh and when I try to access it over the serial UART it appears to be booting, but it will output "failed to execut" (note there is no 'e') no matter what input I give.

I can still access the bootloader, but it is a very old uboot that does not have the functionality I need. What I would like to do is use JTAG to copy the current NOR flash and try to examine the problem from a working OS. Additionally, I think this is just a good exercise since I have not used JTAG before.

I'm trying to use OpenOCD with my jlink edu mini. There was not support for the mt7688 in the build of OpenOCD that I pulled, so I've tried using the files/cfg from both https://review.openocd.org/c/openocd/+/6053 (note that it says the config for mt7628 should work for the mt7688), and https://github.com/Angelic47/MT7688-OpenOCD.

With the mt7628 config I am able to see the TAP on the processor and it passes the IR validation scan; however, then it has some "unexpected writes" before it starts spamming "Reset Detected." Additionally, if I try to give it a reset command through the telnet, it will fail as if it was totally unable to access the TAP on the processor. This is where I'm stuck.

I would really appreciate any insights into what this could be.

Output from starting OpenOCD with debugging flag
Open On-Chip Debugger 0.11.0+dev-00687-g7d2ea186c (2022-05-18-11:40)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
User : 3 2 options.c:63 configuration_output_handler(): debug_level: 3
User : 4 2 options.c:63 configuration_output_handler(): 
Debug: 5 2 options.c:244 add_default_dirs(): bindir=/usr/local/bin
Debug: 6 2 options.c:245 add_default_dirs(): pkgdatadir=/usr/local/share/openocd
Debug: 7 2 options.c:246 add_default_dirs(): exepath=/usr/local/bin
Debug: 8 2 options.c:247 add_default_dirs(): bin2data=../share/openocd
Debug: 9 2 configuration.c:44 add_script_search_dir(): adding /home/jfutrell/.config/openocd
Debug: 10 2 configuration.c:44 add_script_search_dir(): adding /home/jfutrell/.openocd
Debug: 11 2 configuration.c:44 add_script_search_dir(): adding /usr/local/bin/../share/openocd/site
Debug: 12 2 configuration.c:44 add_script_search_dir(): adding /usr/local/bin/../share/openocd/scripts
Debug: 13 2 command.c:166 script_debug(): command - ocd_find interface/my_jlink.cfg
Debug: 14 2 configuration.c:99 find_file(): found /usr/local/bin/../share/openocd/scripts/interface/my_jlink.cfg
Debug: 15 2 command.c:166 script_debug(): command - adapter driver jlink
Debug: 16 2 command.c:166 script_debug(): command - adapter speed 500
Debug: 17 2 adapter.c:180 adapter_config_khz(): handle adapter khz
Debug: 18 2 adapter.c:144 adapter_khz_to_speed(): convert khz to adapter specific speed value
Debug: 19 3 adapter.c:144 adapter_khz_to_speed(): convert khz to adapter specific speed value
User : 20 3 options.c:63 configuration_output_handler(): adapter speed: 500 kHz
User : 21 3 options.c:63 configuration_output_handler(): 
Debug: 22 3 command.c:166 script_debug(): command - transport select jtag
User : 23 3 options.c:63 configuration_output_handler(): jtagUser : 24 3 options.c:63 configuration_output_handler(): 
Debug: 25 3 command.c:166 script_debug(): command - ocd_find target/mediatek/mt7628.cfg
Debug: 26 3 configuration.c:99 find_file(): found /usr/local/bin/../share/openocd/scripts/target/mediatek/mt7628.cfg
Debug: 27 3 command.c:166 script_debug(): command - ocd_find target/mediatek/mmio.tcl
Debug: 28 3 configuration.c:99 find_file(): found /usr/local/bin/../share/openocd/scripts/target/mediatek/mmio.tcl
Debug: 29 3 command.c:166 script_debug(): command - ocd_find target/mediatek/memc.tcl
Debug: 30 3 configuration.c:99 find_file(): found /usr/local/bin/../share/openocd/scripts/target/mediatek/memc.tcl
Debug: 31 3 command.c:166 script_debug(): command - ocd_find target/mediatek/mmio.tcl
Debug: 32 3 configuration.c:99 find_file(): found /usr/local/bin/../share/openocd/scripts/target/mediatek/mmio.tcl
Debug: 33 3 command.c:166 script_debug(): command - jtag newtap mt7628 cpu -irlen 5 -ircapture 0x1 -irmask 0x1f
Debug: 34 3 tcl.c:569 jim_newtap_cmd(): Creating New Tap, Chip: mt7628, Tap: cpu, Dotted: mt7628.cpu, 6 params
Debug: 35 3 tcl.c:593 jim_newtap_cmd(): Processing option: -irlen
Debug: 36 4 tcl.c:593 jim_newtap_cmd(): Processing option: -ircapture
Debug: 37 4 tcl.c:593 jim_newtap_cmd(): Processing option: -irmask
Debug: 38 4 core.c:1472 jtag_tap_init(): Created Tap: mt7628.cpu @ abs position 0, irlen 5, capture: 0x1 mask: 0x1f
Debug: 39 4 command.c:166 script_debug(): command - target create mt7628.cpu0 mips_m4k -endian little -chain-position mt7628.cpu
Debug: 40 4 target.c:2204 target_free_all_working_areas_restore(): freeing all working areas
User : 41 5 options.c:63 configuration_output_handler(): dram_initUser : 42 5 options.c:63 configuration_output_handler(): 
Debug: 43 5 command.c:166 script_debug(): command - init
Debug: 44 5 command.c:166 script_debug(): command - target init
Debug: 45 5 command.c:166 script_debug(): command - target names
Debug: 46 5 command.c:166 script_debug(): command - mt7628.cpu0 cget -event gdb-flash-erase-start
Debug: 47 5 command.c:166 script_debug(): command - mt7628.cpu0 configure -event gdb-flash-erase-start reset init
Debug: 48 5 command.c:166 script_debug(): command - mt7628.cpu0 cget -event gdb-flash-write-end
Debug: 49 5 command.c:166 script_debug(): command - mt7628.cpu0 configure -event gdb-flash-write-end reset halt
Debug: 50 5 command.c:166 script_debug(): command - mt7628.cpu0 cget -event gdb-attach
Debug: 51 5 command.c:166 script_debug(): command - mt7628.cpu0 configure -event gdb-attach halt 1000
Debug: 52 5 target.c:1661 handle_target_init_command(): Initializing targets...
Debug: 53 5 jlink.c:647 jlink_init(): Using libjaylink 0.2.0 (compiled with 0.2.0)
Debug: 54 18 jlink.c:525 jaylink_log_handler(): Found device (VID:PID = 1366:0101, bus:address = 002:062).
Warn : 55 1918 jlink.c:525 jaylink_log_handler(): Failed to retrieve serial number: LIBUSB_ERROR_TIMEOUT.
Debug: 56 1918 jlink.c:525 jaylink_log_handler(): Device: USB address = 0.
Debug: 57 1918 jlink.c:525 jaylink_log_handler(): Device: Serial number = N/A.
Debug: 58 1918 jlink.c:525 jaylink_log_handler(): Allocating new device instance.
Debug: 59 1918 jlink.c:525 jaylink_log_handler(): Found 1 USB device(s).
Debug: 60 1918 jlink.c:525 jaylink_log_handler(): Trying to open device (bus:address = 002:062).
Debug: 61 1918 jlink.c:525 jaylink_log_handler(): Using endpoint 81 (IN) and 02 (OUT).
Debug: 62 1918 jlink.c:525 jaylink_log_handler(): Device opened successfully.
Info : 70 2923 jlink.c:718 jlink_init(): J-Link EDU Mini V1 compiled May  2 2022 09:00:59
Info : 83 2932 jlink.c:759 jlink_init(): Hardware version: 1.00
Info : 92 2945 jlink.c:801 jlink_init(): VTarget = 3.267 V
Debug: 98 2949 jlink.c:525 jaylink_log_handler(): Last read operation left 16 bytes in the buffer.
Debug: 108 2957 jlink.c:953 jlink_reset(): TRST: 0, SRST: 0
Debug: 113 2963 log.c:433 gdb_timeout_warning(): keep_alive() was not invoked in the 1000 ms timelimit (2963 ms). This may cause trouble with GDB connections.
Debug: 127 2971 adapter.c:144 adapter_khz_to_speed(): convert khz to adapter specific speed value
Debug: 128 2971 adapter.c:148 adapter_khz_to_speed(): have adapter set up
Debug: 135 2976 adapter.c:144 adapter_khz_to_speed(): convert khz to adapter specific speed value
Debug: 136 2976 adapter.c:148 adapter_khz_to_speed(): have adapter set up
Info : 137 2977 adapter.c:108 adapter_init(): clock speed 500 kHz
Debug: 138 2977 openocd.c:143 handle_init_command(): Debug Adapter init complete
Debug: 139 2977 command.c:166 script_debug(): command - transport init
Debug: 140 2977 transport.c:230 handle_transport_init(): handle_transport_init
Debug: 141 2977 jlink.c:953 jlink_reset(): TRST: 0, SRST: 0
Debug: 146 2979 core.c:824 jtag_add_reset(): SRST line released
Debug: 147 2979 core.c:849 jtag_add_reset(): TRST line released
Debug: 148 2979 core.c:322 jtag_call_event_callbacks(): jtag event: TAP reset
Debug: 149 2979 command.c:166 script_debug(): command - jtag arp_init
Debug: 150 2979 core.c:1503 jtag_init_inner(): Init JTAG chain
Debug: 151 2979 core.c:322 jtag_call_event_callbacks(): jtag event: TAP reset
Debug: 159 2982 core.c:1228 jtag_examine_chain(): DR scan interrogation for IDCODE/BYPASS
Debug: 160 2982 core.c:322 jtag_call_event_callbacks(): jtag event: TAP reset
Info : 168 2986 core.c:1127 jtag_examine_chain_display(): JTAG tap: mt7628.cpu tap/device found: 0x1762824f (mfg: 0x127 (MIPS Technologies), part: 0x7628, ver: 0x1)
Debug: 169 2986 core.c:1358 jtag_validate_ircapture(): IR capture validation scan
Debug: 177 2988 core.c:1416 jtag_validate_ircapture(): mt7628.cpu: IR capture 0x01
Debug: 178 2988 command.c:166 script_debug(): command - dap init
Debug: 179 2988 arm_dap.c:107 dap_init_all(): Initializing all DAPs ...
Debug: 180 2988 openocd.c:160 handle_init_command(): Examining targets...
Debug: 181 2988 target.c:1849 target_call_event_callbacks(): target event 19 (examine-start) for core mt7628.cpu0
Debug: 196 3000 mips_ejtag.c:390 mips_ejtag_init(): EJTAG: Version 3.1 Detected
Debug: 197 3000 mips_ejtag.c:350 ejtag_main_print_imp(): EJTAG main: features: ASID_8 MIPS16 noDMA MIPS32
Debug: 198 3000 mips_ejtag.c:340 ejtag_v26_print_imp(): EJTAG v2.6: features: R4k
Debug: 199 3000 target.c:1849 target_call_event_callbacks(): target event 21 (examine-end) for core mt7628.cpu0
Debug: 200 3000 command.c:166 script_debug(): command - flash init
Debug: 215 3007 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 216 3007 tcl.c:1386 handle_flash_init_command(): Initializing flash devices...
Debug: 217 3007 command.c:166 script_debug(): command - nand init
Debug: 232 3012 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 247 3019 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 311 3049 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 326 3058 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 390 3089 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 405 3097 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 469 3129 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 484 3136 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 499 3142 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 563 3174 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 578 3180 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 642 3214 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 657 3221 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 721 3253 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 736 3259 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 737 3259 mips_m4k.c:1021 mips_m4k_read_memory(): address: 0xff300000, size: 0x00000004, count: 0x00000001
Debug: 752 3268 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 816 3301 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 831 3307 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 895 3340 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 910 3346 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 974 3380 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 989 3387 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 990 3388 target.c:2637 target_read_u32(): address: 0xff300000 failed
Debug: 1005 3394 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 1069 3424 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 1084 3431 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 1148 3467 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 1163 3473 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Debug: 1227 3504 mips32_pracc.c:188 mips32_pracc_exec(): restarting code
Debug: 1242 3512 mips32_pracc.c:199 mips32_pracc_exec(): unexpected write at address ffffffff
Error: 1243 3512 mips32.c:717 mips32_read_config_regs(): isa info not available, failed to read cp0 config register: 0
Debug: 1244 3512 mips_m4k.c:123 mips_m4k_debug_entry(): entered debug state at PC 0x0, target->state: halted
Debug: 1245 3512 target.c:1849 target_call_event_callbacks(): target event 0 (gdb-halt) for core mt7628.cpu0
Debug: 1246 3512 target.c:1849 target_call_event_callbacks(): target event 1 (halted) for core mt7628.cpu0
Debug: 1247 3512 tcl.c:498 handle_nand_init_command(): Initializing NAND devices...
Debug: 1248 3512 command.c:166 script_debug(): command - pld init
Debug: 1263 3519 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 1264 3520 pld.c:205 handle_pld_init_command(): Initializing PLDs...
Debug: 1265 3520 command.c:166 script_debug(): command - tpiu init
Debug: 1280 3537 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Info : 1281 3537 gdb_server.c:3788 gdb_target_start(): starting gdb server for mt7628.cpu0 on 3333
Info : 1282 3537 server.c:309 add_service(): Listening on port 3333 for gdb connections
Debug: 1283 3537 command.c:166 script_debug(): command - halt
Debug: 1298 3543 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 1299 3544 target.c:3308 handle_halt_command(): -
Debug: 1300 3544 mips_m4k.c:267 mips_m4k_halt(): target->state: halted
Debug: 1301 3544 mips_m4k.c:270 mips_m4k_halt(): target was already halted
Debug: 1316 3551 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Info : 1317 3551 server.c:309 add_service(): Listening on port 6666 for tcl connections
Info : 1318 3551 server.c:309 add_service(): Listening on port 4444 for telnet connections
Debug: 1319 3551 command.c:166 script_debug(): command - init
Debug: 1334 3558 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 1349 3564 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 1364 3665 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
Debug: 1379 3765 mips_m4k.c:210 mips_m4k_poll(): Reset Detected
.
.
.
Output when i try to reset from the telnet session
> reset init
JTAG scan chain interrogation failed: all ones
Check JTAG interface, timings, target power, etc.
Trying to use configured scan chain anyway...
mt7628.cpu: IR capture error; saw 0x1f not 0x01
Bypassing JTAG setup events due to errors
isa info not available, failed to read cp0 config register: 0
target halted in MIPS32 mode due to debug-request, pc: 0x00000000
Output when i try to resume from the telnet session
> resume
isa info not available, failed to read cp0 config register: 0
target halted in MIPS32 mode due to target-not-halted, pc: 0x00000000

I guess there is a hardware watch dog in MT7688 and it may reset the cpu. You may try disabling it and see if this issue will happen again. Normally, need to issue one command from OpenOCD to disable it. But I don't know what the exact command in this platform.

I had thought it may be a watchdog issue based on what I have read, but I do not know where to start with disabling it. This post mentions a similar issue to mine, but does not go into detail on how they disabled it.

I've tried manually halting through the telnet and that does not seem to help. Especially because it seems that by the time I get in, the mt7688 is already in a halted state.