Split tcpdump in files per packet / editcap wanted

For Developers
1

I am in the process of replacing the pfSense with a new LEDE box (tp-link ac2600).

Currently I have a script running on my old pfSense box that captures packets with tcpdump, splits them to 1 packet per file and then processes those files. It's only a few small packets per minute.

The splitting is done with editcap, part of wireshark: > editcap -c 1 input output
I understand that it is not easy/possible to install wireshark on LEDE.

Is there another simple way to get that same result with what is normally available on/for LEDE?
(afaik, tcpdump can only split based on filesize, not on packets)

If this is not possible(?), perhaps anyone can help me how to build just that 'editcap' binary for LEDE?

2

How about using the -c count option in tcpdump and calling it in a loop from a script? (-c count Exit after receiving count packets.)

If count = 1 the dump file always contains one packet, and might work as long as you don't have a high flow of your special packets.

Might be necessary to use a different name for the dumps, but adding a simple auto-incremented number should be enough (maybe reset after it hits like 9999).

If that isn't enough, keep reading.

It seems some asian guy has uploaded and keeps updated a wireshark package makefile in his github repository https://github.com/liudonghua/wireshark-openwrt. It says only x86 but I think it should compile fine in other architectures too.
(makefile = instructions used by the build system to make a OpenWRT/LEDE package)

if you open the "Makefile" text file you will see a line
--disable-editcap \
that must obviously be removed if you want to have editcap in there.
And below (in the "define Package/wireshark/install" paragraph) you need to add a line
$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/editcap $(1)/usr/bin/
so the editcap binary is loaded in the package after it was compiled.

since you are only interested in editcap, you can delete the lines about other tools you don't need, but you must keep the lines to install the library files.
The build system will compile these binaries but they won't be loaded in the package. You probably don't have space issues in that device though.

here an example of what you want in that paragraph:

define Package/wireshark/install
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/editcap $(1)/usr/bin/
	$(INSTALL_DIR) $(1)/usr/lib
	$(CP) $(PKG_INSTALL_DIR)/usr/lib/lib*.so* $(1)/usr/lib
endef

You can use the LEDE SDK to build it for your device (you need a 64-bit linux system to use that), see the OpenWRT docs about it https://wiki.openwrt.org/doc/howto/obtain.firmware.sdk

You should add a link to the local folder with your new package (the folder where you cloned that github repository and made your modifications) in the SDK's package feed list as explained in "Package Feeds" paragraph, there are example links inside that configuration file to guide you.

The lede SDK for your device is found in the same download folder where you download the flashable firmware images from, but its name starts with "lede-sdk".

3

Thanks, your post crossed my own work. :slight_smile:
I am currently trying to cross-compile only the editcap binary. I created a build environment and got the original wireshark source stuff there. With configure, I seem to be able to remove all except editcap.

Now what I cannot find is what I need to specify as '--host='. Most examples are too generic.

I have a tp-link c2600 and I found that I need something like --host=arm-openwrt-linux-muslgnueabi but that is not exactly the correct name. How can I find what I need there? Or can anyone just tell me what it needs to be?

EDIT: btw, -c for count will not be good, sometimes 2 or 3 packets come in very fast after eachother and I want all 3 of them and then split it into 3 separate. With -c it terminates so I need to start it again. By the time it started again, I missed the other packet(s).

4

you can rotate based on:

-G time
-C size

what are you doing that can only process one packet per file?

5

[quote="supersjimmie, post:3, topic:341, full:true"]Now what I cannot find is what I need to specify as '--host='. Most examples are too generic.

I have a tp-link c2600 and I found that I need something like --host=arm-openwrt-linux-muslgnueabi but that is not exactly the correct name. How can I find what I need there? Or can anyone just tell me what it needs to be?[/quote]I don't know you are exactly doing here but the LEDE table says your device is target ipq806x and pulls packages from the arm_cortex-a15_neon-vfpv4 package repository (LEDE has re-arranged the packages so they don't compile the same packages for each and every target, but only for a group of architectures shared by multiple targets).

https://wiki.lede-project.org/toh/hwdata/tp-link/tp-link_archer_c2600

If you use the SDK, device-specific options should be loaded automatically, btw.

[quote]EDIT: btw, -c for count will not be good, sometimes 2 or 3 packets come in very fast after eachother and I want all 3 of them and then split it into 3 separate. With -c it terminates so I need to start it again. By the time it started again, I missed the other packet(s).[/quote]Yeah, I was suspecting that.

6

I started with this: http://telecnatron.com/articles/Cross-Compiling-For-OpenWRT-On-Linux/
Which is a more clear version of this: https://wiki.openwrt.org/doc/devel/crosscompile which says:
"For GNU configure, use --build=architecture-unknown-linux-gnu --host=architecture-openwrt-linux-uclibc (for example: ./configure --build=x86_64-unknown-linux-gnu –host=mips-openwrt-linux-uclibc)"

What I want to know is how/where I can see what I need for the build= and host=.
When I use --host=arm-openwrt-linux-uclibc and --build=openwrt-linux-gnu, the response is:
configure: WARNING: using cross tools not prefixed with host triplet
And it says Cross compiling no.

@dlang: my solar inverter sends data to a central manufacturer server each 5 minutes. I want to intercept those, get some data from it and send it to pvoutput. If for some reason the inverter misses one time, it sends out 2 packets the next time. That's why I need to be able to split them. The whole scripts works fine under pfsense and for some other people on a rasp-pi.

7

Download the SDK and see what arguments it is using for your device.

8

I am working based on that. The whole build environment that i am using, was created with that.
As specified there, I used make menuconfig and all other stuff to create the entire environment.

After having that, I don't need to create an entire package, because I just want to cross-compile, which is like explained in the links I mentioned. I see CONFIG_TARGET_ARCH_PACKAGES="arm-cortex-a15_neon-vpfv4", ectetera.

9

And, for LEDE I have the source from https://github.com/lede-project because that contains the Qualcomm Atheros IPQ806X for my AC2600.

10

[quote="supersjimmie, post:8, topic:341, full:true"]After having that, I don't need to create an entire package, because I just want to cross-compile, which is like explained in the links I mentioned. I see CONFIG_TARGET_ARCH_PACKAGES="arm-cortex-a15_neon-vpfv4", ectetera.
[/quote]Try to launch a compilation of existing packages (selected through menuconfig) with make -j1 V=s and see what is the full command line given to the compiler.

Then you can mimic it.

Btw, if you just link the package to the SDK and select it as I suggested above, it would have been all over in like 5 minutes and you'd get also an installable package instead of a binary.
(assuming it compiles fine anyway)

11

I am sorry that I spoiled all of your time and energy, but I feel this is just over my head.
While I thought it would be doable for me with some linux and developer knowledge, I guess I will need much more deep knowledge of compilers, makefiles, packages, and lede than I have.

I'm new to openwrt and lede, because I want to migrate from pfsense.
Also my dev experience is on stuff like arduino and a bit windows (cpp) and sometimes a new kernel for linux.
All those are relatively well documented becasue the targets are well-known.

When I do make -j1 V=s within my LEDE build environment I see sooo many lines (entering, leaving, target many directories, but no command line for the compiler) and many times that arm_cortex thing passing by, but I don't see how to translate anything to the build or host parameter.

12

When I just compile bzip2 and use -j1 -V=s it still does not show the commandline that contains --host or --build things.
What I do see is that in the Makefile, there is a line that points to $(STAGING_DIR)/host (which is just ./staging_dir/host) but I don't see how that relates to the parameter that I need.

13

I followed a different approach, following the sdk as much as possible.
I took the wireshark-openwrt from liudonghuaand placed it in ./package/
Then I did all that's in the sdk guide: make menuconfig, make package/wireshark-openwrt/download, prepare, compile, clean, index build.
There are no errors (also not when I use V99 or V=s), but there is also no resulting .ipk anywhere.

14

Heh, I'm not a developer either, more like a long-time user. That's why I know more how to use the build system than how it works internally.

Anyway, I just tried in my PC (OpenSUSE leap 42.2), and I compiled the package successfully.

I write how I did as the wiki seems to not be clear enough (will have to import that page in LEDE wiki later so I can rewrite it):

I created this folder:
/run/media/alby/data_xeon_btrfs/LEDE_make_wireshark and Downloaded the SDK from here https://downloads.lede-project.org/snapshots/targets/ipq806x/generic/
and extracted it in a folder with same name in it

I created this folder:
/run/media/alby/data_xeon_btrfs/LEDE_make_wireshark/newpackages/
and then I git cloned the wireshark-openwrt folder in it.

I added this line in feeds.conf.default I found in the folder I extracted the LEDE SDK in :
src-link custom /run/media/alby/data_xeon_btrfs/LEDE_make_wireshark/newpackages/

(all folder names here are arbitrary, btw)

I then opened a terminal in the SDK's folder wrote a ./scripts/feeds update -a

It downloaded things, and the last step also showed it loaded wireshark from feed "custom".

I wrote ./scripts/feeds install wireshark

It said it installed wireshark and some other stuff (build dependencies of it, these will also be downloaded/installed automatically by opkg when you install this custom package in your system, another reason why a package is better than compiling this manually)

I wrote make menuconfig and made sure that the package wireshark was selected. (it should be showing M, as "module", which means it will be built as a package).

Saved and closed menuconfig.

started compile with make -j8 (to speed up compilation in my processor that has 4 cores and 8 threads, change number for yours as needed)

When everything has finished, I find the wireshark package in /run/media/alby/data_xeon_btrfs/LEDE_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64/bin/packages/arm_cortex-a15_neon-vfpv4/custom/

It is 12 MiB in size, but I compiled with the upstream config (I just git cloned it, made no modifications to the makefile as I was just testing the compile procedure), so it is a bit fat by LEDE standards.

1 Like
15

Thank you VERY much so far!
I made it nearly to the end, but then the make fails at wireshark:

make[3]: Leaving directory '/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64/feeds/base/package/libs/gettext-full'
make[2]: Leaving directory '/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64'
package/Makefile:124: recipe for target '/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl-1.1.15_eabi/stamp/.package_compile' failed
make[1]: *** [/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl-1.1.15_eabi/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64'
/home/jimmy/lede_make_wireshark/lede-sdk-ipq806x_gcc-5.4.0_musl-1.1.15_eabi.Linux-x86_64/include/toplevel.mk:187: recipe for target 'world' failed
make: *** [world] Error 2

16

The actual error message is likely just before the log extract you copied here. The log extract above only shows the end stages of recursive make stopping after an error, but says nothing what was the actual error.

You may need to add "V=s" option to the make command so that you see a detailed log.

17

(removed this one, because I found a relevant error)

18

I stripped make menuconfig to let it do as little as possible. Now I found this error:

checking whether to use libpcap for packet capture... yes
checking for arm-openwrt-linux-pcap-config... (cached) no
checking for pcap-config... /usr/bin/pcap-config
checking for broken pcap-config... no
checking pcap.h usability... no
checking pcap.h presence... yes
configure: WARNING: pcap.h: present but cannot be compiled
configure: WARNING: pcap.h: check for missing prerequisite headers?
configure: WARNING: pcap.h: see the Autoconf documentation
configure: WARNING: pcap.h: section "Present But Cannot Be Compiled"
configure: WARNING: pcap.h: proceeding with the compiler's result
configure: WARNING: ## ----------------------------------------- ##
configure: WARNING: ## Report this to http://bugs.wireshark.org/ ##
configure: WARNING: ## ----------------------------------------- ##
checking for pcap.h... no
configure: error: Header file pcap.h not found; if you installed libpcap
from source, did you also do "make install-incl", and if you installed a
binary package of libpcap, is there also a developer's package of libpcap,
and did you also install that package?

Which is strange, bacause I already installed libpcap and libpcap-dev. As far as I understood, with those I don't need that make install-incl anymore (if so, how would that work?).

Also libpcap is in the make menuconfig as a marked, so shouldn't that fix it then?

19

I found this:

mkdir nembuild
cd nembuild
wget http://www.tcpdump.org/release/libpcap-1.5.3.tar.gz
tar -xf libpcap-1.5.3.tar.gz
cd libpcap-1.5.3
./configure
make && make install

Now it compiled and created the wireshark_2.2.1-5_arm_cortex-a15_neon-vfpv4.ipk :slight_smile:
Next will be to modify wireshark's Makefile to create just editcap.

20

Haha, good you figured that out on your own as I would have no clue. :grinning:

Let me guess, is that an Ubuntu system?

I've seen people on Ubuntu that installed libselinux/-dev and the build was still failing (in a f2fs-tools bug where it suddenly needed such libraries), here on Opensuse I had that installed already and build was running fine.

Next will be to modify wireshark's Makefile to create just editcap.

Might be also worth trying to remove dependencies from the line here so you can reduce the footprint, as these are getting installed in the system when you install the package.

DEPENDS:=+librt +libpcap +glib2 +tcpdump +libc

The battle-tested method of "just remove stuff, do a make clean, then build and see if the compiler complains about missing libraries on compile time" should work.

(btw, it seems the asian guy has just updated his package to remove lua support and dependency)

EDIT: Please copy the makefile in this thread when you're done so people looking for similar functionality in the future can use it too.