How to implement Traffic monitoring (with iptables, port mirroring, etc)

Gotcha! now I understand. Don't really have a device that's always on and particularly capable so maybe I'll double-purpose the new espressobin which I'm setting up as the hot-spare router for when I break the main router or the power goes out and I need something that can last multiple hours on UPS.

Cool. Also, this is what I use as the collector/database:

Hi @kuriSu-kUN

A while ago I tried Tee on my glinet ar150 modified with a second usb WiFi adapter using relayd.
Don’t remember the settings I used but was kind on faulty like wireshark wasn’t picking up all the packets ! I didn’t investigate more think I assumed it was about not enough computing power of the device. Maybe I was trying to forward br-lan traffic to Ethernet cable, while second antenna was in wireless-client mode or was the built-in one I don’t remember. Need to say that the second adapter was plugged into a usb hub (powered one) toghether with a usb memory stick and a third antenna but I believe this one was off. Sorry :neutral_face: can’t help more; locked away my router related stuff for holyday and still in the cupboard.

Hey @Pippo, no worries :slight_smile:

I will try also today to setup softflowd/nfsen to see how it will go...

1 Like

I tried running softflowd on my x86 router, and during a dslreports speedtest ~ 600Mbps it used about 1 full core and completely destroyed my QoS. Is there a way to set it to sub-sample the packets randomly? Like for example 1/20 of the packets. That'd probably be totally acceptable.

I want to monitor the router traffic on another machine as well. This machine is running v3.8.1 of the community version of ntopng. Since you need a pay license for nprobe now, I just used the iptable rules of

iptables -A POSTROUTING -t mangle -o br-lan ! -s -j TEE --gateway
iptables -A PREROUTING  -t mangle -i br-lan ! -d -j TEE --gateway

(were .189 is the ntop vm) to forward copies of data to this machine. This works great, except, on my WRT1900AC my throughput drops from 38MB download to around 18MB down, and (sometimes worse). My question is, is it supposed to cause such a large impact? Is there any way to make it better? Would softflowd work better? And if it did, how do you get it to ntopng without nprobe?)? I actually tried nfsen but did not find any prebuilt packages and trying to build/make it always failed. I like ntop better anyways and would love if there was a way to 'make it work'

Could try port mirroring to see if it is any less resource intensive.

I am having a similar problem, did you ever find a solution (to lower the impact?) I am not using softflowd but iptables but having a similar problem.

if you mean mirror on a port on the router switch, the problem with that is the router and the esxi server are on opposite ends of the house, it would be very difficult, (which is why I opted for iptables). But searching online, lots of articles talk about iptables/mangle/TEE for this purpose, but no one mentions an impact on throughput, I am partly wondering if what I am experiencing is "normal" ?

Looking at articles like something like net.core.netdev_max_backlog sounds interesting? or anything like net.core.somaxconn? Does anyone have any suggestions? I'm curious where I can look to see what is happening (why the throughput is dropping (dropped packets cause of backlog etc?) )

iptables-tee copys the hole packet and modifies a part hence the (high) resource consumption.
the mentioned solutions try to only extract the relevant bits.

there are multiple encapsulation methods available to solve this, vlans probably beeing the most common, but unneeded if you opt for something like netflow or pcap based.

afaik these only affect localy terminated (not routed) connections.

I initially dropped it to only tcp, and now trying to use limits. Something I noticed, if the machine is off, there is no download delay; does that suggest that the router can handle copying/manipulation of the packets and its a problem on the ntop machine, OR, does it drop/not do the work if it knows it can't get to --gateway

Doesn't the replication of the traffic to port/vlan create about the same amount of work for the router?

When the rules are in place, and downloading CPU1 is only 80% and CPU2 is 40%, what is the limiting/bottleneck piece (network I/O)? There seems to be CPU room and memory is about 130Megs free? With the rules in place, the download starts pretty high, (high 20's) but then starts to fall, like it can't keep up with the workload but I'm not sure what can't keep up?

Finally ... has anyone used these commands and NOT had any throughput problems?

Update: probably will change to hashlimit (testing...)


iptables -A POSTROUTING -t mangle -o br-lan ! -s -p tcp  -m limit --limit 5/second -j TEE --gateway
iptables -A PREROUTING  -t mangle -i br-lan ! -d -p tcp  -m limit --limit 5/second -j TEE --gateway

Just as a final update, (incase it helps anyone else), I seem to have it working (by basically dropping the packets to the copy server (nTop) when the count from a given source is high). The numbers might not be ideal, and need tweaking, but the principle seems to work.

iptables -A POSTROUTING -t mangle -o br-lan ! -s -m hashlimit --hashlimit-mode srcip --hashlimit-upto 500/sec --hashlimit-burst 100 --hashlimit-name nTopCopyLimit -j TEE --gateway

iptables -A PREROUTING  -t mangle -i br-lan ! -d -m hashlimit --hashlimit-mode srcip --hashlimit-upto 500/sec --hashlimit-burst 100 --hashlimit-name nTopCopyLimit -j TEE --gateway