How to implement Traffic monitoring (with iptables, port mirroring, etc)

Installing and Using OpenWrt
1

I want to monitor everything passing through the WAN port (i.e. internet traffic) by duplicating all WAN traffic to a dedicated switch port.

I have already set up a dedicated port on the router and connected a Wireshark machine directly to it:

config interface 'monitor'
        option ifname 'eth0.2'
        option proto 'none'
        option ipv6 0
        option auto '1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '5 6t'

Wireshark is directly connected to port 5 and port 6 is the router cpu port. Without the ipv6 0 I get unwanted ipv6 broadcasts showing in Wireshark (but at least that helps prove that part of the setup works). I've tried to set it up without an IP address or anything else so it isn't used for anything apart from the monitoring traffic.

I expect I should be able to do this with an iptables rule so I started by making a zone for the monitoring interface:

config zone
        option name             monitor
        list   network          'monitor'
        option input            ACCEPT
        option output           ACCEPT
        option forward          ACCEPT

However, I don't know how to get frames to/from the wan zone duplicated onto this monitor zone.

I have also tried using the port-mirroring (installed using opkg) with some config like this:

config 'port-mirroring'
        option 'target' 'eth0.2'
        option 'source_ports' 'pppoa-wan'
        option 'filter' ''
        option 'protocol' 'TZSP'
        option promiscuous '1'

Which I start like this

 root@gateway:/etc/config# port-mirroring --debug
2017-11-22 13:58:42[info] port-mirroring::main, mirroring_type:[interface][TZSP], mirroring_source_num:[1], target:[eth0.2], filter:[], opt_promiscuous:[1].
2017-11-22 13:58:42[info] port-mirroring::reopenSendHandle eth0.2 success.

I've also tried the TEE protocol. I've also tried using another machine on the LAN as a port-mirroring target (i.e. target '192,168.x.y').

I sometimes see data in Wireshark and sometimes see nothing. What I do see is meaningless. Lots of output showing unfamiliar mac addresses, zeroes, meaningless numeric protocol numbers, "Ethernet II" or "Bogus IPv4" messages. It looks like garbage, like something isn't configured correctly.

In case it's relevant, the interface on the Wireshark end is brought up like this:

# echo 1 > /proc/sys/net/ipv6/conf/enp3s0/disable_ipv6
# ip link set enp3s0 promisc on
# ip link set enp3s0 up

I have also tried to use iptables-mod-tee without success. Various internet searches suggest commands like

 $ iptables -t mangle -A PREROUTING -j TEE -gw 192.168.a.b
 $ iptables -t mangle -A PREROUTING -j TEE -gw 192.168.a.b

(where 192.168.a.b is the monitoring station running Wireshark)

However these just give errors like iptables v1.4.21: multiple -j flags not allowed. Further searching suggest this refers to a non-standard extension?

I did notice, when installing iptables-mod-tee:

# opkg install iptables-mod-tee
Installing iptables-mod-tee (1.4.21-2) to root...
Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/iptables-mod-tee_1.4.21-2_mips_24kc.ipk
Installing kmod-ipt-tee (4.4.71-1) to root...
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/lantiq/xrx200/packages/kmod-ipt-tee_4.4.71-1_mips_24kc.ipk
Configuring kmod-ipt-tee.
failed to find a module named nf_tee
Configuring iptables-mod-tee.

I don't know whether failed to find a module named nf_tee is an issue; the package did appear to install:

root@gateway:/etc/config# opkg list-installed | grep "mod-tee" 
iptables-mod-tee - 1.4.21-2

So I've tried a few ways to achieve this, only to end up hitting a wall.

Ideally I'd like to do this just with iptables but, if not, then with port-mirroring or some other way...

Any guidance appreciated. I'd like to be able to get it working and then document it here.

2

I'm not sure if you simply need a copy of the traffic, you can also install softflowd to simply collect data on the WAN interface - such as: IP, protocol, port, duration, etc. Then, you don't have to keep a copy of all packets.

See: https://code.google.com/archive/p/softflowd/

3

Thanks @lleachii I am going to look into that also, along with ntop to use the netflow data. I was unaware of this until your reply. Definately worth a look, but I still would like to do packet capture as well.

1 Like
4

Also check out: http://nfsen.sourceforge.net/

5

Well I have softflowd and nfsen up and running. I'm going to let it run for a while to collect some data. Meanwhile I have also installed nlbwmon and YAMon which I'll test for a while.

1 Like
6

If you want to play with Netflow, then you should check out my blog postings about running netflow collectors in Docker images. Using Docker makes it way easier to set up.

My intro post is Netflow Collectors for Home Networks and it discusses my experiments with various tools. I got softflowd with wvnetflow and nfsen working well enough to see that they actually function, and to understand some of their flaws.

BTW I actually like YAMon a great deal, and have started using it heavily in my home routers. I even made a Paypal contribution! (And to keep this note a little closer to on-topic, YAMon uses iptables, etc. to collect the data.)

1 Like
7

Hello @richb-hanover thanks, I had already read your blog postings as it happens and found them very useful. I didn't go the docker route but got it running nonetheless (just because it was the path of least effort on my part). I have softflowd working and nfcapd collecting data and being displayed in my browser with nfsen. I have had a little play but not taken it very far. First I need to set it up on something that's always-on.

Have you looked at SiLK and FlowBat at all? I havent' yet but thought that might be more comprehensive than nfSen which feels a little like 1990 :slight_smile:

I have to agree that YAMon is very likeable too, although I have a little reservation about having something running on the inner sanctum of my primary network gateway "phoning-home" to grab code on a regular basis. But I am running it and so far I like it best out of the things I've tried.

I still have the gap of how actively to capture packets though--Something that I would like the ability to do whenever I feel the need to "drill down" into a particular live flow for whatever reason I might have... Not being able to do this is a gap I'd really like to close.

8

SiLK is the underlying netflow collection facility beneath FlowViewer, FlowBAT and other GUIs.

I made a Docker container for FlowViewer but never warmed to it enough to bother writing it up.

FlowBAT sounds cool - a Javascript product. But it's based on Meteor 0.9.1 (ancient) and isn't being maintained. I forked the FlowBAT repo to force the installer to use Meteor 0.9.1 so that I could make it work. Again, it wasn't sufficiently compelling to it to continue working on it.

To complete the list in my blog posting I looked at DDWarden, but never attempted to run it here.

And there's a stunt that you can use to tell YAMon not to phone home for JS/CSS files. It's mentioned at http://usage-monitoring.com/help/?t=no-internet But this doesn't address files such as jQuery and Google Visualization API files.

9

Well you gave me reason to try Docker. I have been meaning to do it for ages I've been well embedded in LXC for years and haven't had good reason to diverge from that. Until now. And, wow that was easy!!!!!

I am now running your wvnetflow-dockerized image. It's only been up for about half an hour as I write this but it has collected some data. I think it will take some time to understand how it works - I will read the documentation.

I did a very quick test to see if I could identify a big flow displayed in YAMon - I tried to use the ad-hoc query tool to report on flows from an internal IP but it only seems to know about my external IP. But it's early days... once I RTFM I am sure I will understand more.

Actually, I wonder if that's because softflowd is only sending WAN traffic ?

config softflowd
    option enabled        '1'
    option interface      'pppoa-wan'
    option host_port      '10.0.200.1:2055'
    option export_version '9'
    option track_ipv6     '1'

One negative off the bat is that it doesn't support ipv6.

Thanks so much for the work you've already done in this area; it's saved me a heap of time and research :slight_smile:

1 Like
10

Hi ,

was kind of trying one of the approaches to monitor traffic on my wifi access point from here, I

believe is the TEE protocol:

rebuild openwrt with kmod-ipt-tee didnt notice any error during make.

Before I flash the router I have a question about

Set traffic mirror rules to capture all traffic of TEST_DEVICE_IP_ADDRESS to MONITORING_COMPUTER_IP_ADDRESS. Add iptables rules to mirror upstream and downstream traffic.

  1. iptables -A PREROUTING -t mangle -i br-lan ! -d <TEST_DEVICE_IP_ADDRESS> -j TEE --gateway <MONITORING_WORKSTATION_IP_ADDRESS>

  2. iptables -A POSTROUTING -t mangle -o br-lan ! -s <TEST_DEVICE_IP_ADDRESS> -j TEE --gateway <MONITORING_WORKSTATION_IP_ADDRESS>

Is there a way to mirror the traffic for every device that connect to the access point (ie. a range of IP addresses) to be forwarded ??

Other than playing with iptables (the're kind out of my reach right know) would be possible to envisage another approaches in which the AP traffic is mirrored to an ethernet port instead using other ways
and then use a PC ethernet card connected to the router lan port as the wireshark monitor interface ?

Thanks

P.

11

You could monitor the VLAN on a Switch port too. No iptables rules needed.

12

Hi Ileachi, just missing how to implement vlan on AP or SSD or WIFI interface

meaning vlan on wlan that I presume is not under a switch and mirror the data on LAN. My router

has got a switch capable soc but no ports connected to it

just eth0 WAN, eth1 LAN and wlan0 radio.

Please point me to how to mirror accross wlan <-> LAN, or build a vlan containing both LAN and WLAN.

Or correct me if I wrote something stupid still confusing interfaces with wireless firewalls zone routes

ports physical and virtual.

Cheers

P.

13

My suggestion may not be possible then, apologies.

14

Hey to all, best wishes for the new year to everybody.

@Pippo did you find a solution to your problem?
I just found this topic as I'm in search also for a way to monitor all the traffic for my home network.
I also found the TEE topic you posted in an earlier post what I did and work for every device in the network even for Wi-Fi connected devices ( ex. mobile phones )

was to put in the <TEST_DEVICE_IP_ADDRESS> the ip of my router.
so let's say for example my router has an ip of 192.168.1.1 and the dedicated wireshark device has an ip of 192.168.1.30 then the command would be.

iptables -A PREROUTING -t mangle -i br-lan ! -d 192.168.1.1 -j TEE --gateway 192.168.1.30
iptables -A POSTROUTING -t mangle -o br-lan ! -s 192.168.1.1 -j TEE --gateway 192.168.1.30

Then only think you have to do is to open wireshark and start to monitor based on the device you like with the above command. ex. a network device with ip 192.168.1.55
(ip.src == 192.168.1.55) || (ip.dst == 192.168.1.55)

Now I also have questions regarding this matter.
First of all, which command should I run in order to save those two iptables commands and survive after a reboot?
I tried iptables-save but after a reboot I must again run those commands in order to be able to capture with wireshark.

Is there a better way to do this?
For example.
I read about softflowd /nfsen combination but I cannot understand how to set it up.
Basically what I'm trying to figure out is, if there is a way to be able to monitor all the traffic of my WAN.
and to be able to access it from a web interface with nice graphs, also it would be awesome if from there I could do a check/block of the ip that I don't like.

To be honest, I can't understand how a package that will give a fully customizable fw3/iptables ui in luci with all the info that we need from the router without the need of seperate packages/wireshark app etc does not exist already.

15

Hi all. Is there any way to implement monitoring using a flow daemon and a subsample of packets? I have very high bandwidth and don't want to dedicate much CPU, so I'd like to sample randomly say 1% of packets and still analyze the flows so that I can go back and look at patterns like when bandwidth is used, how much, what the dscp tags were, etc. It's fine if I miss some small flows entirely.

16

I use softflowd for that (see above).

17

How do you create the subsample? is it within softflowd or are you able to subsample before it hits softflowd and thereby reduce load on softflowd?

18

What do you mean by "sub" sample?

softflowd is not resource intensive. I easily sample 100+ Mbps without much load on a processor.

19

I mean I want to send randomly only 2/100 packets to softflowd. I have gigabit fiber so I imagine sending all packets to softflowd would be intensive. Certainly running ntopng on my router dropped my speed to about 200Mbps :wink: so if I have an issue I engage it but can't run it all the time. perhaps softflowd is less intensive though?

20

No, I have interfaces with 200+ Mbps traffic, it shouldn't be an issue.

Whoa, I am not suggesting that you run the collector and database on the router!

Only the netflow agent, which is not that resource intensive.