Cross-complling C++ program with both dynamic and static linking library failed

For Developers
1

Hi all, I am trying to cross-compile my C++ program for Openwrt using SDK (openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64 for my Linksys WRT 1900ACS). The program is compiled with both dynamic-linked library (libpcap, libcurl and libpthread) and static-linked library( libPacket++,libPcap+ and libCommon++ from pcapplusplus project; libfaup_static from faup project). (I static link these libeary because they don't exsit in openwrt.) The compliation errors seems to suggest the static-linking isn't working. I wonder how could I properly do dynamic and static linking here? thanks!

Specifically, the compliation error is

make[1]: Entering directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64'
make[2]: Entering directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64'
make[3]: Entering directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/package/toolchain'
Makefile:802: WARNING: skipping libgomp -- package has no install section
echo "libc" >> /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/pkginfo/toolchain.default.install
echo "libgcc" >> /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/pkginfo/toolchain.default.install
echo "libpthread" >> /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/pkginfo/toolchain.default.install
touch -r /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/toolchain/.built /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/toolchain/.autoremove 2>/dev/null >/dev/null
find /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/toolchain -mindepth 1 -maxdepth 1 -not '(' -type f -and -name '.*' -and -size 0 ')' -and -not -name '.pkgdir' | xargs -r rm -rf
make[3]: Leaving directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/package/toolchain'
time: package/toolchain/compile#0.24#0.10#0.29
make[3]: Entering directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/package/iotsteed'
rm -f /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed/.built
touch /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed/.built_check
CFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -ffile-prefix-map=/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed=iotsteed -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro  " LDFLAGS="" LIBS="-Wl, -Bdynamic -lpcap -lcurl -lpthread -Wl, -Bstatic -lPacket++ -lPcap++ -lCommon++ -lfaup_static" make -C /home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed AR="arm-openwrt-linux-muslgnueabi-gcc-ar" AS="arm-openwrt-linux-muslgnueabi-gcc -c -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -ffile-prefix-map=/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed=iotsteed -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=arm-openwrt-linux-muslgnueabi-ld NM="arm-openwrt-linux-muslgnueabi-gcc-nm" CC="arm-openwrt-linux-muslgnueabi-gcc" GCC="arm-openwrt-linux-muslgnueabi-gcc" CXX="arm-openwrt-linux-muslgnueabi-g++" RANLIB="arm-openwrt-linux-muslgnueabi-gcc-ranlib" STRIP=arm-openwrt-linux-muslgnueabi-strip OBJCOPY=arm-openwrt-linux-muslgnueabi-objcopy OBJDUMP=arm-openwrt-linux-muslgnueabi-objdump SIZE=arm-openwrt-linux-muslgnueabi-size CROSS="arm-openwrt-linux-muslgnueabi-" CXXFLAGS="-Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -ffile-prefix-map=/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed=iotsteed -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro  " ARCH="arm" iotsteed;
make[4]: Entering directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed'
arm-openwrt-linux-muslgnueabi-g++ -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=hard -ffile-prefix-map=/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed=iotsteed -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro     -c -o main.o main.cc
In file included from tests.h:5,
                 from main.cc:4:
iotsteed.h:11:10: fatal error: pcapplusplus/Packet.h: No such file or directory
 #include <pcapplusplus/Packet.h>
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
<builtin>: recipe for target 'main.o' failed
make[4]: *** [main.o] Error 1
make[4]: Leaving directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed'
Makefile:47: recipe for target '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed/.built' failed
make[3]: *** [/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/iotsteed/.built] Error 2
make[3]: Leaving directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/package/iotsteed'
time: package/iotsteed/compile#0.24#0.11#0.30
package/Makefile:111: recipe for target 'package/iotsteed/compile' failed
make[2]: *** [package/iotsteed/compile] Error 2
make[2]: Leaving directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64'
package/Makefile:107: recipe for target '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/stamp/.package_compile' failed
make[1]: *** [/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a9+vfpv3_musl_eabi/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64'
/home/hangguo/openwrt-sdk-mvebu-cortexa9_gcc-8.3.0_musl_eabi.Linux-x86_64/include/toplevel.mk:218: recipe for target 'world' failed
make: *** [world] Error 2

my package manifest file is

include $(TOPDIR)/rules.mk
  
PKG_NAME:=iotsteed
PKG_RELEASE:=1

PKG_BUILD_DIR := $(BUILD_DIR)/$(PKG_NAME)

include $(INCLUDE_DIR)/package.mk

define Package/iotsteed
        SECTION:=utils
        CATEGORY:=Utilities
        TITLE:= IoTSTEED: IoT traffic monitor
        DEPENDS:=+libstdcpp +libgcc +libpcap +libcurl +libpthread
endef

define Build/Prepare
        mkdir -p $(PKG_BUILD_DIR)
        $(CP) ./src/* $(PKG_BUILD_DIR)/
endef

TARGET_LIBS = -Wl, -Bdynamic -lpcap -lcurl -lpthread \
              -Wl, -Bstatic -lPacket++ -lPcap++ -lCommon++ -lfaup_static

define Build/Compile
        CFLAGS="$(TARGET_CFLAGS) $(EXTRA_CPPFLAGS) " \
        LDFLAGS="$(EXTRA_LDFLAGS)" \
        LIBS="$(TARGET_LIBS)" \
        $(MAKE) -C $(PKG_BUILD_DIR) \
                $(TARGET_CONFIGURE_OPTS) \
                CROSS="$(TARGET_CROSS)" \
                CXXFLAGS="$(TARGET_CFLAGS) $(EXTRA_CPPFLAGS) " \
                ARCH="$(ARCH)" \
                $(1);
endef

define Package/iotsteed/install
        $(INSTALL_DIR) $(1)/bin
        $(INSTALL_BIN) $(PKG_BUILD_DIR)/iotsteed $(1)/bin/
endef

$(eval $(call BuildPackage,iotsteed))

and my src/Makefile is

 SRCS = main.cc iotsteed.cc tests.cc det.cc sln.cc
    OBJS = $(SRCS:.cc=.o)
    
    .PHONY : clean
    
    iotsteed: $(OBJS)
            $(CXX) $(CXXFLAGS) -o iotsteed $(OBJS) $(LIBS)
    
    main.o: dev.h dg_node.h det.h tests.h iotsteed.h version.h debug.h
    
    iotsteed.o: dg_node.h det.h iotsteed.h dev.h sln.h debug.h
    
    tests.o: dev.h dg_node.h det.h tests.h iotsteed.h svr.h debug.h
    
    det.o: dg_node.h det.h debug.h
    
    sln.o: svr.h sln.h debug.h
    
    clean :
            rm -f iotsteed $(OBJS)
2

It's not linking that's failing. The compiler can't find the header file for Pcap++ because it's not installed in the staging_dir. You haven't gotten to the linking part yet, but that will assuredly fail too, for the same reasons.

You need to create a separate package that will download and compile Pcap++ from its github repository and then install it in the staging_dir (read up about Build/InstallDev).

Then you need to add it to the DEPENDS line for your package along with a PKG_BUILD_DEPENDS directive as well that will ensure it gets built and installed before your package tries to build. You will have to do the same for the other library you mention.

Once these steps are complete, the header files and the libraries will be available to the compiler and you won't get the error below

iotsteed.h:11:10: fatal error: pcapplusplus/Packet.h: No such file or directory
#include <pcapplusplus/Packet.h>

3

This should get you started: I created a package Makefile for you for pcapplusplus. This will download, compile and install the libraries and headers in the staging area.

You'll need to create a similar one for your other library from the faup project. Make sure you have
DEPENDS:=+libpcapplusplus in your main package Makeilfe

#
# Copyright (C) 2006-2020 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

include $(TOPDIR)/rules.mk

PKG_NAME:=libpcapplusplus
PKG_VERSION:=1.0
PKG_RELEASE:=1

PKG_SOURCE_URL:=https://github.com/seladb/PcapPlusPlus/archive/
PKG_SOURCE_VERSION:=19.12
PKG_SOURCE:=v$(PKG_SOURCE_VERSION).tar.gz
PKG_HASH:=9bebe2972a6678b8fb80f93b92a3caf9babae346137f2171e6941f35b56f88bb

PKG_LICENSE:=UNLICENSE

PKG_BUILD_DIR:=$(BUILD_DIR)/PcapPlusPlus-$(PKG_SOURCE_VERSION)
PKG_BUILD_PARALLEL:=1
PKG_INSTALL:=1

include $(INCLUDE_DIR)/package.mk

define Package/libpcapplusplus
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=C++ wrapper for libpcap
  DEPENDS:=+libpcap
  URL:=https://pcapplusplus.github.io/
endef

define Package/libpcapplusplus/description
PcapPlusPlus enables capturing and sending network packets 
through easy-to-use C++ wrappers for the most popular packet 
processing engines such as libpcap, WinPcap, Npcap, DPDK and PF_RING
endef

CONFIGURE_CMD  = ./configure-linux.sh
CONFIGURE_ARGS = --default --install-dir=$(PKG_INSTALL_DIR)

# The configure script throws an error if the install-dir does not exist

define Build/Configure
	mkdir -p $(PKG_INSTALL_DIR)
	$(call Build/Configure/Default)
endef

define Build/InstallDev
	$(INSTALL_DIR) $(1)/usr/include
	$(CP) $(PKG_INSTALL_DIR)/include/pcapplusplus $(1)/usr/include/
	$(INSTALL_DIR) $(1)/usr/lib/
	$(CP) $(PKG_INSTALL_DIR)/lib/lib{Common++,Packet++,Pcap++}.a $(1)/usr/lib/
endef

define Package/libpcapplusplus/install
endef

$(eval $(call BuildPackage,libpcapplusplus))
4

Thanks for your reply! It is very helpful and really help me get the big picture here!

I followed your suggestions and try compliling libpcapplusplus based on your manifest file. Howerver the complier complains about missing some GNU C header files, as quoted below:

make[3]: Entering directory '/home/hangguo/Dropbox/openwrt-sdk/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/PcapPlusPlus-19.12'
make[4]: Entering directory '/home/hangguo/Dropbox/openwrt-sdk/build_dir/target-arm_cortex-a9+vfpv3_musl_eabi/PcapPlusPlus-19.12/Common++'
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
==> Building target: Common++
Building file: src/IpAddress.cpp
In file included from ./header/IpUtils.h:6,
                 from src/IpAddress.cpp:5:
/usr/include/netinet/in.h:22:10: fatal error: bits/stdint-uintn.h: No such file or directory
 #include <bits/stdint-uintn.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:49: recipe for target 'Obj/IpAddress.o' failed

I have made sure I selected all avalable C/C++ library in "make menuconfig" (libc, libstdcpp, libcxx and uclibcxx) and installed them in staging_dir with "make". But as shown, I still don't see the bits/stdint-uintn.h that compiler complains about in my staging_dir.

openwrt-sdk/staging_dir$ find ./ -name stdint*.h
./toolchain-arm_cortex-a9+vfpv3_gcc-8.3.0_musl_eabi/lib/gcc/arm-openwrt-linux-muslgnueabi/8.3.0/include/stdint-gcc.h
./toolchain-arm_cortex-a9+vfpv3_gcc-8.3.0_musl_eabi/lib/gcc/arm-openwrt-linux-muslgnueabi/8.3.0/include/stdint.h
./toolchain-arm_cortex-a9+vfpv3_gcc-8.3.0_musl_eabi/arm-openwrt-linux-muslgnueabi/include/c++/8.3.0/tr1/stdint.h
./toolchain-arm_cortex-a9+vfpv3_gcc-8.3.0_musl_eabi/include/stdint.h
./toolchain-arm_cortex-a9+vfpv3_gcc-8.3.0_musl_eabi/include/bits/stdint.h
./target-arm_cortex-a9+vfpv3_musl_eabi/usr/include/c++/v1/stdint.h

Copying bits/stdint-uintn.h from my linux /usr/include doesn't seem to work: the compiler keep complaining about more missing headers.

I wonder do you know any cross-complied C library that I could use to provide bits/stdint-uintn.h?

5

I tested that Makefile and compiled it on my system and it works fine.

It looks like your build environment is broken. I suggest using the full buildroot system instead of the SDK.

You can't copy header files from your live system into a cross-compilation environment. That will just break things further.

The musl header file netinet/in.h does not include bits/stdint-uintn.h. This is clearly the glibc header that is being included and not the musl netinet/in.h header file, which is wrong.

Your build environment is picking up header files from your live system and not from the cross-compilation environment. You need to fix that first, otherwise nothing will work. Clone the entire openwrt repository from github/openwrt.org and compile using that instead.

6

You marked your own question above as the solution. So did you get it working?

7

Oops. I thought I marked yours. Just re-didi it. I end up getting it working. I found /user/include//netinet is hard-coded in a couple places in Pcapplusplus make file such as mk/PcapPlusPlus.mk.linux. Re-pointing these to header directory in staging_dir solve my header not found issue.

Thanks for your help! I should've posted this earlier.

8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.