How to compile VEL21RIPN ndpi-netfilter2 and using for lede/openwrt

hisham2630 · January 5, 2018, 5:50pm

Hi
Can some one point me how to compile NDPI to use with iptables.
i see only lib-ndpi in packages,but i don't know what is the usage of it.
i found that this is the latest ndpi version here : https://github.com/betolj/ndpi-netfilter.

anon57995562 · January 5, 2018, 10:28pm

See this...

github.com

akmaker/openwrt-cn/blob/master/english/ndpi-quick-start-guide.md


### Introduction

nDPI is a DPI library based on OpenDPI and currently maintained by ntop.

In addition to Unix, we also support Windows, in order to provide you a cross-platform DPI 
experience. Furthermore, we have modified nDPI to be more suitable for traffic monitoring 
applications, by disabling specific features that slow down the DPI engine while making 
them unnecessary for network traffic monitoring. 

nDPI allows application-layer detection of protocols, regardless of the port being used. 
This means that it is possible to both detect known protocols on non-standard ports (e.g. 
detect http non ports other than 80), and also the opposite (e.g. detect Skype traffic on port 
80). This is because nowadays the concept of **port=application** no longer holds.

Over the past few months we have added several features including:

* Enhancement of the demo **pcapReader** application both in terms of speed/features 
and encapsulations supported (for instance you can now analyse GTP tunnelled traffic).
* Ability to compile nDPI inside the Linux kernel so that you can use it for developing

This file has been truncated. show original

lantis1008 · January 5, 2018, 11:36pm

Here’s a Makefile you can use with Openwrt.
Note that in my testing it works (and really well!) until it doesn’t. There’s some instability in there.

github.com

lantis1008/gargoyle/blob/9396aee80f7543e940c8e3f1e16f35e9252de685/package/ndpi-netfilter/Makefile

include $(TOPDIR)/rules.mk
include $(INCLUDE_DIR)/kernel.mk

PKG_NAME:=ndpi-netfilter
PKG_VERSION:=3.0
PKG_RELEASE:=1
PKG_REV:=052718d2cf39491d56b3e65d553bad2b8d7ae132

PKG_SOURCE_PROTO:=git
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.bz2
PKG_SOURCE_SUBDIR:=$(PKG_NAME)-$(PKG_VERSION)
PKG_SOURCE_URL:=https://github.com/betolj/ndpi-netfilter
PKG_SOURCE_VERSION:=$(PKG_REV)

PKG_BUILD_DIR:=$(KERNEL_BUILD_DIR)/$(PKG_NAME)-$(PKG_VERSION)

include $(INCLUDE_DIR)/package.mk

define Package/iptables-mod-ndpi
  SUBMENU:=Firewall

This file has been truncated. show original

Betolj’s repo is not “latest” either. Not by a long shot.
Vel21ripn (might have that spelt wrong) has a much more up to date repo but the Makefile would need to be reworked for it.

hisham2630 · January 6, 2018, 1:47pm

thanks guys for ndpi makefile, but i don't know how to compile and where should i put that makefile, also should i use Betolj's code.
i have download the latest openwrt trunk, then i use ./feeds update -a to update packages.
i compile a dnscrypt proxy with plugins before,but i'm not familiar with cross compile.
on internet i see a tutorial for how to compile hello world code, but i think for this ndpi is different.
i'm using Ubuntu in vmware on windows 7.
any help is appreciated.

hisham2630 · January 6, 2018, 4:56pm

Can some one rework makefile for https://github.com/vel21ripn/nDPI
so we can use the latest ndpi.
now should i put the ndpi source code and makefile in package dir inside the openwrt sdk?

hisham2630 · January 7, 2018, 5:40pm

it's compiling now,i'm waiting to see if it will successful.
i found this : https://github.com/ntop/nDPI and it seem's to be the latest ndpi.
is there someway to port it to lede(openwrt) ?

hisham2630 · January 9, 2018, 7:25am

compile fail on Ubuntu with error 1,and toplevel.mk line 209.

hisham2630 · January 11, 2018, 9:02am

finally,compiled the ndpi on Debian 9.3,but i'd like to port the https://github.com/vel21ripn/nDPI version ?

hisham2630 · January 11, 2018, 6:47pm

i found this makefile for openwrt https://github.com/openwrt-develop/ndpi-netfilter
this is based on vel21ripn ndpi.

lantis1008 · January 11, 2018, 7:43pm

Does that one compile?

hisham2630 · January 11, 2018, 7:45pm

i have a slow internet so make download will take a long time,when it's finish i will post back.

hisham2630 · January 12, 2018, 12:59pm

this one for ndpi reader it's compile and working https://github.com/ninglvfeihong/nDPI/blob/master/Makefile

this one for vel21ripn ,build failed
ndpi-netfilter https://github.com/openwrt-develop/ndpi-netfilter/blob/master/Makefile

this is the error :

make[3]: Entering directory '/home/hisham/Desktop/openwrt/source/feeds/packages/net/ndpi-netfilter'
rm -f /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/.built
touch /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/.built_check
CFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c:ndpi-netfilter2-85f459c -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro  -I/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/usr/include -I/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/include -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/usr/include -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/include/fortify -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/include " CXXFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c:ndpi-netfilter2-85f459c -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro  -I/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/usr/include -I/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/include -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/usr/include -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/include/fortify -I/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/include " LDFLAGS="-L/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/usr/lib -L/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/lib -L/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/usr/lib -L/home/hisham/Desktop/openwrt/source/staging_dir/toolchain-mips_24kc_gcc-5.5.0_musl/lib -znow -zrelro " make -j1 -C /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter AR="mips-openwrt-linux-musl-gcc-ar" AS="mips-openwrt-linux-musl-gcc -c -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -iremap/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c:ndpi-netfilter2-85f459c -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=mips-openwrt-linux-musl-ld NM="mips-openwrt-linux-musl-gcc-nm" CC="mips-openwrt-linux-musl-gcc" GCC="mips-openwrt-linux-musl-gcc" CXX="mips-openwrt-linux-musl-g++" RANLIB="mips-openwrt-linux-musl-gcc-ranlib" STRIP=mips-openwrt-linux-musl-strip OBJCOPY=mips-openwrt-linux-musl-objcopy OBJDUMP=mips-openwrt-linux-musl-objdump SIZE=mips-openwrt-linux-musl-size CROSS="mips-openwrt-linux-musl-" ARCH="mips" KERNEL_DIR="/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.9.76" ARCH="mips" NDPI_PATH=/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter CROSS_COMPILE="mips-openwrt-linux-musl-" all ;
make[4]: Entering directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter'
make -C ipt
make[5]: Entering directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/ipt'
make[5]: Nothing to be done for 'all'.
make[5]: Leaving directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/ipt'
make -C src
make[5]: Entering directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src'
make  -C /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.9.76 M=$PWD modules -w;
make[6]: Entering directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.9.76'
  CC [M]  /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.o
In file included from /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/../../src/include/ndpi_main.h:28:0,
                 from /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:47:
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/../../src/include/ndpi_define.h:61:21: warning: "__LITTLE_ENDIAN" is not defined [-Wundef]
 #if __BYTE_ORDER == __LITTLE_ENDIAN
                     ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:119:2: error: #error NF_CONNTRACK_LABELS not defined
 #error NF_CONNTRACK_LABELS not defined
  ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c: In function 'ndpi_net_exit':
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2455:9: error: 'struct netns_ct' has no member named 'labels_used'
  net->ct.labels_used--;
         ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c: In function 'ndpi_net_init':
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2511:18: error: passing argument 1 of 'set_ndpi_malloc' from incompatible pointer type [-Werror=incompatible-pointer-types]
  set_ndpi_malloc(malloc_wrapper);
                  ^
In file included from /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/../../src/include/ndpi_main.h:32:0,
                 from /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:47:
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/../../src/include/ndpi_api.h:621:8: note: expected 'void * (*)(size_t) {aka void * (*)(unsigned int)}' but argument is of type 'void * (*)(long unsigned int)'
   void set_ndpi_malloc(void* (*__ndpi_malloc)(size_t size));
        ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2604:10: error: 'struct netns_ct' has no member named 'labels_used'
   net->ct.labels_used++;
          ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c: In function 'ndpi_mt_init':
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2703:22: error: 'NF_CT_EXT_LABELS' undeclared (first use in this function)
  nf_ct_ext_id_ndpi = NF_CT_EXT_LABELS;
                      ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2703:22: note: each undeclared identifier is reported only once for each function it appears in
In file included from ./include/linux/printk.h:6:0,
                 from ./include/linux/kernel.h:13,
                 from /home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:23:
./include/linux/kern_levels.h:4:18: warning: format '%zu' expects argument of type 'size_t', but argument 6 has type 'long unsigned int' [-Wformat=]
 #define KERN_SOH "\001"  /* ASCII Start Of Header */
                  ^
./include/linux/kern_levels.h:13:19: note: in expansion of macro 'KERN_SOH'
 #define KERN_INFO KERN_SOH "6" /* informational */
                   ^
./include/linux/printk.h:284:9: note: in expansion of macro 'KERN_INFO'
  printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2760:2: note: in expansion of macro 'pr_info'
  pr_info("xt_ndpi v1.2 ndpi %s %s\n"
  ^
./include/linux/kern_levels.h:4:18: warning: format '%zu' expects argument of type 'size_t', but argument 7 has type 'long unsigned int' [-Wformat=]
 #define KERN_SOH "\001"  /* ASCII Start Of Header */
                  ^
./include/linux/kern_levels.h:13:19: note: in expansion of macro 'KERN_SOH'
 #define KERN_INFO KERN_SOH "6" /* informational */
                   ^
./include/linux/printk.h:284:9: note: in expansion of macro 'KERN_INFO'
  printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2760:2: note: in expansion of macro 'pr_info'
  pr_info("xt_ndpi v1.2 ndpi %s %s\n"
  ^
./include/linux/kern_levels.h:4:18: warning: format '%zu' expects argument of type 'size_t', but argument 9 has type 'long unsigned int' [-Wformat=]
 #define KERN_SOH "\001"  /* ASCII Start Of Header */
                  ^
./include/linux/kern_levels.h:13:19: note: in expansion of macro 'KERN_SOH'
 #define KERN_INFO KERN_SOH "6" /* informational */
                   ^
./include/linux/printk.h:284:9: note: in expansion of macro 'KERN_INFO'
  printk(KERN_INFO pr_fmt(fmt), ##__VA_ARGS__)
         ^
/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.c:2760:2: note: in expansion of macro 'pr_info'
  pr_info("xt_ndpi v1.2 ndpi %s %s\n"
  ^
cc1: some warnings being treated as errors
scripts/Makefile.build:293: recipe for target '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.o' failed
make[7]: *** [/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src/main.o] Error 1
Makefile:1495: recipe for target '_module_/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src' failed
make[6]: *** [_module_/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src] Error 2
make[6]: Leaving directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/linux-4.9.76'
Makefile:170: recipe for target 'modules' failed
make[5]: *** [modules] Error 2
make[5]: Leaving directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter/src'
Makefile:2: recipe for target 'all' failed
make[4]: *** [all] Error 2
make[4]: Leaving directory '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/ndpi-netfilter'
Makefile:66: recipe for target '/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/.built' failed
make[3]: *** [/home/hisham/Desktop/openwrt/source/build_dir/target-mips_24kc_musl/linux-lantiq_xrx200/ndpi-netfilter2-85f459c/.built] Error 2
make[3]: Leaving directory '/home/hisham/Desktop/openwrt/source/feeds/packages/net/ndpi-netfilter'
package/Makefile:109: recipe for target 'package/feeds/packages/ndpi-netfilter/compile' failed
make[2]: *** [package/feeds/packages/ndpi-netfilter/compile] Error 2
make[2]: Leaving directory '/home/hisham/Desktop/openwrt/source'
package/Makefile:105: recipe for target '/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/stamp/.package_compile' failed
make[1]: *** [/home/hisham/Desktop/openwrt/source/staging_dir/target-mips_24kc_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/hisham/Desktop/openwrt/source'
/home/hisham/Desktop/openwrt/source/include/toplevel.mk:216: recipe for target 'world' failed

hisham2630 · January 12, 2018, 6:31pm

i see that nf_conntrack_labels is missing or not compiled from openwrt.

panda · January 16, 2018, 3:39am

you need select nf_conntrack_labels in kernel configure.

hisham2630 · January 16, 2018, 10:14am

thanks now i have this problem:

ccache_cc -fPIC -I../include -I../lib -I../src -DOPENDPI_NETFILTER_MODULE -O2 -Wall   -c -o libxt_ndpi.o libxt_ndpi.c
libxt_ndpi.c:34:26: fatal error: ../../config.h: No such file or directory
compilation terminated.

also is this right in the makefile

CONFIGURE_CMD=./autogen.sh; ./configure --with-pic --target=$(GNU_TARGET_NAME) --host=$(GNU_TARGET_NAME) --host-alias=$(GNU_TARGET_NAME) --build=$(GNU_HOST_NAME)

panda · January 19, 2018, 8:07am

Please use the latest version of https://github.com/openwrt-develop/ndpi-netfilter

lantis1008 · January 19, 2018, 8:54am

If this works, god bless. I’ll test later tonight.

hisham2630 · January 19, 2018, 1:58pm

thanks,but this is not the latest version.
because you're using 054bb8f commit and the latest is 0670f78.
i will change and see.
also kmod-ipt-compat-xtables is not available in packages this happened about 3 days ago.
also if i compile your Makefile it will show many errors and won't compile.
i have a Makefile that will compile latest version for openwrt,but need some improvements.

hisham2630 · January 20, 2018, 2:02pm

my MAkefile will successfully compile latest version of vel21ripn ndpi-netfilter, but i have only one problem, which is iptables-mod-ndpi need libc.so.6 and it will fail to produce that package,but it will create kmod-ipt-ndpi package with no problems.
dose someone know how to fix that libc.so.6 problem?

alex3594 · January 23, 2018, 2:54pm

Hello!
Did you have any success compiling iptables ndpi module?
I'm getting this error when trying to compile module from https://github.com/openwrt-develop/ndpi-netfilter:

checking whether we are cross compiling... configure: error: in `/home/openwrt/build_dir/target-i386_pentium4_musl/linux-x86_generic/ndpi-netfilter-0670f78':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details