Setup remote debugger - crash using step-by-step

For Developers
1

Hi,

I like to contribute to a OpenWRT plugin and setup a build environment. It works fine but I have an issue with remote debugging.

I setup gdbserver and likt to connect it from VSCode. The breakpoints are working fine but I cannot use step-by-step debugging using C++ Extension or with "Native Debug"-Extension. Any ideas how to get it working? It would be easier to debug line by line even if breakpoints are working fine.

image
image1915Ă—1192 186 KB

launch.json

{
    "configurations": [
        {
            "type": "gdb",
            "request": "attach",
            "name": "native Debugger - Attach to gdbserver",
            "executable": "/home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/usteerd",
            "target": "openwrt.muc.lan:7777",
            "remote": true,
            "cwd": "${workspaceRoot}",
            "valuesFormatting": "parseText",
            "gdbpath": "/home/nils/openwrt/build_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-13.3.0_musl_eabi/gdb-14.1/gdb/gdb"
        },
        {
            "name": "GDB for OpenWRT Remote Attach",
            "type": "cppdbg",
            "request": "launch",
            "externalConsole": false,
            "stopAtEntry": true,
            "program": "/home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/usteerd",
            "MIMode": "gdb",
            "cwd": "${workspaceFolder}",
            "miDebuggerPath": "/home/nils/openwrt/build_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-13.3.0_musl_eabi/gdb-14.1/gdb/gdb",
            "miDebuggerServerAddress": "openwrt.muc.lan:7777",
            "miDebuggerArgs": " -ex 'handle all print nostop noignore'",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        },
        {
            "type": "by-gdb",
            "request": "launch",
            "name": "By GDB - Launch(gdb)",
            "program": "/home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/usteerd",
            "cwd": "${workspaceRoot}",
            "remote": {
                "enabled": true,
                "address": "openwrt.muc.lan:7777",
                "mode": "extended-remote",
                "execfile": "${fileBasenameNoExtension}"
            }
        }
    ]
}

Output from "Native Debug" Extension

Reading symbols from /home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/usteerd...

Warning: 'set target-async', an alias for the command 'set mi-async', is deprecated.

Use 'set mi-async'.

Reading /lib/libubox.so.20240329 from remote target...

warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.

Reading /lib/libubus.so.20231128 from remote target...

Reading /lib/libblobmsg_json.so.20240329 from remote target...

Reading /usr/lib/libjson-c.so.5 from remote target...

Reading /usr/lib/libnl-tiny.so.1 from remote target...

Reading /lib/libgcc_s.so.1 from remote target...

2

Reading /lib/ld-musl-armhf.so.1 from remote target...

Reading /home/nils/openwrt/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-13.3.0_musl_eabi/lib/debug/.build-id/c9/17cf6fca830a355530586ac39c5f54c08f71f0.debug from remote target...

0xb6f3fe14 in ?? ()

No source file named /home/nils/usteer/ubus.c.

Breakpoint 1, usteer_dump_node (buf=0x1f450 <b>, node=0xb6ec8818) at /home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/ubus.c:313

313 blobmsg_add_u32(buf, "freq", node->freq);

Breakpoint 4, usteer_dump_node (buf=0x1f450 <b>, node=0xb6ec8818) at /home/nils/openwrt/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usteer-2024-10-13/ubus.c:316

316 blobmsg_add_u32(buf, "load", node->load);

Warning:

Cannot insert breakpoint 0.

Cannot access memory at address 0x4e244

Cheers,
Nils

2

What's the target here? Cortex A15 something?

ubus call system board

?

Try x86 as sanity check of whether this is CPU specific issue? Try a simpler C program first? We talking software or hardware break points?
I have no experience with WSL nor vscode.

Can we simplify the debug chain?

Try local debugging rather than remote debugging? I like GDB TUI.

I'd also try rather than trying to attach to a running process, start it with GDB?

3 
    "kernel": "6.6.54",
    "hostname": "OpenWrt",
    "system": "ARMv7 Processor rev 0 (v7l)",
    "model": "Zyxel NBG6817",
    "board_name": "zyxel,nbg6817",
    "rootfs_type": "squashfs",
    "release": {
            "distribution": "OpenWrt",
            "version": "SNAPSHOT",
            "revision": "r27700-170ecbecfc",
            "target": "ipq806x/generic",
            "description": "OpenWrt SNAPSHOT r27700-170ecbecfc"

Software-Breakpoints. Starting it with GDB will run into the same issue. I will try the other steps but if somebody knows this error please reply.

1 Like
4

OK i've got nothing other than try an x86 VM to replicate the issue =P

If I get around to it maybe i'll try to replicate your problem.

5

Have the same problem, I could fix it if I disable binary stripping in menuconfig but this increases a lot the image size. The way to only disable stripping for a single package did not work for me. It's for sure related to stripping If all is build without it works, but if I try to do it for only one package the one debugged it does not work.

https://openwrt.org/docs/guide-developer/debugging
You have to disable sstrip to keep debug information.

Note, if you just want to build a single package unstripped, try, “make package/foo/compile STRIP=true” Also, unstripped binaries are placed in “staging_dir/target-/root-/” where your host side tools can use them. (Remember, gdbserver can attach to the stripped binary, while gdb loads the unstripped binary) see gd

It is also possible that in fact the binary is not stripped so that parameter works but maybe a shared library used is stripped and that causes the problem, and this would explain why building everything with no stripping works.

1 Like
6

I found how it can be fixed. You need to build the application with -static flag (all libraries must be static build). If not only the executable if build with debug symbols will have them, the libs do not this is why the debugger crashes. So either everything is build with debug symbols (not sure yet if the strip is also required to be disabled or is disabled anyway if you set the debug symbols), or you need static build.

1 Like