Cross Compiled binary fails to run

Strive · March 30, 2020, 12:24pm

Hi,

I am trying to run an USP Agent (https://github.com/BroadbandForum/obuspa/blob/master/QUICK_START_GUIDE.md) in OpenWRT

As expected, the binary built on my local laptop did not work in OpenWRT.

I followed the cross compilation instructions, and build a new binary. I copied the cross compiled binary to OpenWRT VM. When i try to run the binary, i am getting message as -ash: ./obuspa: not found

How to run the cross compiled binary in OpenWRT VM?

Am i missing something.

Thanks,
Strive

eduperez · March 30, 2020, 12:31pm

If you followed the cross-compilation instructions, the you did the right thing. Perhaps you should give more details about how you did it.

Strive · March 30, 2020, 12:47pm

Thanks @eduperez for your response.

Here are the steps that i followed.

Downloaded the obuspa ZIP from github
I build a custom OpenWRT with SDK
(Reason: My first exercise was to build a OpenWRT image where i can create LXC containers, and etc. For that i have a build setup which produces .vdi file. I used the same setup and included SDK option as well from menuconfig)
export STAGING_DIR=/path/to/sdk/staging_dir/
export TOOLCHAIN_DIR=$STAGING_DIR/toolchain-x86_64_gcc-7.5.0_musl
export LDCFLAGS=$TOOLCHAIN_DIR/usr/lib
export LD_LIBRARY_PATH=$TOOLCHAIN_DIR/usr/lib
export PATH=$PATH:$TOOLCHAIN_DIR/bin
cd to the unpacked directory whose code has to be cross compiled
./config.guess to know the build
./configure --build=x86_64-pc-linux-gnu --host=x86_64-openwrt-linux-uclibc
Checked the output and ensure that 'checking whether we are cross compiling… yes ' is yes.
make
find -iname *obuspa*
file obuspa
Copied the file to the OpenWRT VM

mk24 · March 30, 2020, 2:26pm

-ash: ./obuspa: not found

This means that a dynamically linked runtime library (.so) could not be loaded. Yes it would be nice if the message told you which one. Determine which libraries the program depends on and make sure they are all installed.

eduperez · March 30, 2020, 3:45pm

At first sight, that looks fine to me...

As @mk24 commented, perhaps there is a missing library. Could you please execute "ldd ./obuspa" inside the VM?

Strive · March 30, 2020, 3:53pm

Here is the output

/lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x7f30f10d6000)
libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x7f30f0e0f000)
libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x7f30f0d2f000)
libcurl.so.4 => /usr/lib/libcurl.so.4 (0x7f30f0cec000)
libz.so.1 => /usr/lib/libz.so.1 (0x7f30f0cd8000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f30f1202000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x7f30f0cc4000)
libmbedtls.so.12 => /usr/lib/libmbedtls.so.12 (0x7f30f0caa000)
libmbedx509.so.0 => /usr/lib/libmbedx509.so.0 (0x7f30f0c9a000)
libmbedcrypto.so.3 => /usr/lib/libmbedcrypto.so.3 (0x7f30f0c56000)

Error relocating ./obuspa: mallinfo: symbol not found
Error relocating ./obuspa: backtrace: symbol not found

HectoPascal · March 30, 2020, 4:44pm

Maybe a wrong rpath. Does '/lib64/ld-linux-x86-64.so.2' actually exist?
Do you get any output with this:
objdump -x obuspa |grep RPATH

Strive · March 30, 2020, 4:49pm

There is no /lib64/ld-linux-x86-64.so.2

There is /lib64/ld-musl-x86_64.so.1

No output with objdump -x obuspa |grep RPATH

HectoPascal · March 30, 2020, 4:55pm

As a quick test try:
ln -s /lib64/ld-musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
and run the binary again.

Don't forget to delete the symlink afterwards:
rm /lib64/ld-linux-x86-64.so.2

Strive · March 30, 2020, 5:17pm

@HectoPascal, thanks for your response

After creating symlink, I get

Error relocating ./obuspa: mallinfo: symbol not found
Error relocating ./obuspa: backtrace: symbol not found

HectoPascal · March 30, 2020, 6:27pm

I guess your buildroot and your image are not the same. Your binary expects ulibc/glibc and your image has musl only. And you're missing some other libs. You could try to install the missing dependencies or compile a static binary.

Strive · March 31, 2020, 9:30am

@HectoPascal, thanks for your answers

I built a new OpenWRT image with glibc.

I also cross compiled the binary using new SDK

Now, i am getting the following error when i run ./obuspa --help

./obuspa: /lib/libm.so.6: version `GLIBC_2.29' not found (required by ./obuspa)
./obuspa: /usr/lib/libcurl.so.4: no version information available (required by ./obuspa)

ls -l /usr/lib/libcurl*
lrwxrwxrwx    /usr/lib/libcurl.so.4 -> libcurl.so.4.6.0
-rwxr-xr-x    /usr/lib/libcurl.so.4.6.0

ls -l /lib/libm*
-rwxr-xr-x    /lib/libm-2.27.so
lrwxrwxrwx    /lib/libm.so.6 -> libm-2.27.so

Looks like i need libm-2.29 I will check libm error.

Why libcurl error is there?

HectoPascal · April 1, 2020, 12:12am

I think the cross-compile wiki article is somewhat outdated indeed. The triplet should be rather musl than uclibc nowadays i guess. On musl its not compileable anyway due to execinfo.h that is glibc only. Can probably be patched out though as its a crashhandler only. But if you can go for glibc its probably the better approach.
What are your current build commands? Still the same as above?
Not sure if i get the wiki right, but STAGING_DIR should point to staging_dir/toolchain* folder (toolchain dir).

Strive · April 1, 2020, 4:19am

Hi @HectoPascal,

Yes, the build commands are the same as above.

I will point STAGING_DIR to toolchain and try once

Thanks,
Strive

Strive · April 1, 2020, 5:04am

@HectoPascal

I can confirm that build steps are the same. In fact i had a separate environment variable which was pointing to TOOLCHAIN_DIR that was pointing to /staging_dir/toolchain... as you mentioned.

I am still facing same issue.

HectoPascal · April 1, 2020, 9:08am

Can you post the 'configure' output?
Did you try to enable the missing dependencies in 'make menuconfig'?

Strive · April 1, 2020, 9:15am

I did enable.

In fact, libcurl is present. But the error message says no version information available

My C compiler is glibc.

I tried configure with two commands (only difference is glibc and ulibc in the end). One resulted in an error.

First command:

./configure --build=x86_64-pc-linux-gnu --host=x86_64-openwrt-linux-glibc
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for x86_64-openwrt-linux-glibc-strip... no
checking for strip... strip
configure: WARNING: using cross tools not prefixed with host triplet
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports the include directive... yes (GNU style)
checking for x86_64-openwrt-linux-glibc-gcc... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... Invalid configuration `x86_64-openwrt-linux-glibc': machine `x86_64-openwrt-linux' not recognized
configure: error: /bin/bash ./config.sub x86_64-openwrt-linux-glibc failed

Second command:

./configure --build=x86_64-pc-linux-gnu --host=x86_64-openwrt-linux-uclibc
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for x86_64-openwrt-linux-uclibc-strip... no
checking for strip... strip
configure: WARNING: using cross tools not prefixed with host triplet
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports the include directive... yes (GNU style)
checking for x86_64-openwrt-linux-uclibc-gcc... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-openwrt-linux-uclibc
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... no
checking for x86_64-openwrt-linux-uclibc-dumpbin... no
checking for x86_64-openwrt-linux-uclibc-link... no
checking for dumpbin... no
checking for link... link -dump
checking the name lister (nm) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-openwrt-linux-uclibc format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for x86_64-openwrt-linux-uclibc-objdump... no
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for x86_64-openwrt-linux-uclibc-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for x86_64-openwrt-linux-uclibc-ar... no
checking for ar... ar
checking for archiver @FILE support... @
checking for x86_64-openwrt-linux-uclibc-strip... strip
checking for x86_64-openwrt-linux-uclibc-ranlib... no
checking for ranlib... ranlib
checking command to parse nm output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for x86_64-openwrt-linux-uclibc-mt... no
checking for mt... mt
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking for x86_64-openwrt-linux-uclibc-gcc... gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking for x86_64-openwrt-linux-uclibc-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for openssl... yes
checking for sqlite3... yes
checking for libcurl... yes
checking for libcares... yes
checking for zlib... yes
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: executing depfiles commands
config.status: executing libtool commands

HectoPascal · April 1, 2020, 1:00pm

Omit the '--build=' option and use the '--host=' triplet string as used in the toolchain/bin folder. Look for the binaries in the lower part of the folder that are not symlinked. Just remove the '-gcc' or '-strip' part on the end. Then look out in the 'configure' output that the compiler is found properly. Your failed log shows that it couldn't find the right binary due to wrong 'build' triplet.

dl12345 · April 1, 2020, 4:03pm

Try this. I test compiled it on my system and it compiles.

This package you're compiling requires the c-ares (libcares) package in order to build and run, as well as a bunch of other dependencies, so you'll need to install them otherwise it won't work. You can see the packages it needs. If you use opkg to install the generated ipk, it should install the dependencies if they're available

I have not tested it as I don't want to install a bunch of dependencies on my live system, but it should have all the things it needs to run

Install this file below as package/utils/obuspa/Makefile. Then do a make menuconfig and go to the Utilities section and select obuspa

Then run make package/obuspa/compile V=s

include $(TOPDIR)/rules.mk

PKG_NAME:=obuspa
PKG_VERSION:=1.0
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/BroadbandForum/obuspa.git
PKG_SOURCE_VERSION:=f701a6fbfad5aba705df42efe6874c491f2550be
PKG_SOURCE_DATE:=2029-04-01
PKG_MIRROR_HASH:=skip

PKG_BUILD_DEPENDS:=c-ares
PKG_INSTALL:=1
PKG_FIXUP:=autoreconf

include $(INCLUDE_DIR)/package.mk

define Package/obuspa
  SECTION:=utils
  CATEGORY:=Utilities
  TITLE:=Open Broadband-User Services Platform-Agent
  DEPENDS:=+zlib +libcares +libcurl +libopenssl +libsqlite3
endef

define Package/obuspa/description
Open Broadband-User Services Platform-Agent (OB-USP-Agent) is an open source project that
is focused on creating a reference implementation of the User Services Platform (USP)
specification from an "Agent" perspective. 
endef

define Package/obuspa/install
    $(INSTALL_DIR) $(1)/usr/bin
    $(CP) $(PKG_INSTALL_DIR)/usr/bin/obuspa $(1)/usr/bin
endef

$(eval $(call BuildPackage,obuspa))

Note that in the Package/obuspa/install section that these are TAB before the commands. If you just put spaces you'll probably get a missing separator error.

[EDIT: there's a typo in the PKG_SOURCE_DATE, but that's not really important. Just change it to 2020]

Strive · April 1, 2020, 6:56pm

Hi @dl12345,

Thanks for your response.

I followed the steps that you mentioned.

The .ipk file is getting generated. I installed it using the command: opkg install obuspa_1.0-1_x86_64.ipk

I am getting the following error when i run the command: obusp --help
obuspa: error while loading shared libraries: libc.so: cannot open shared object file: No such file or directory

In /lib64/ directory, i have this for libc

-rwxr-xr-x    1 root   root  1763104 Feb 27 21:05 libc-2.27.so
lrwxrwxrwx    1 root   root       12 Feb 27 21:05 libc.so.6 -> libc-2.27.so