Identify nature of packets reported by collectd

I'm having a very strange periodic packet drop reported by collectd on eth0 (LAN), but only when I have a particular device online. I'm trying to understand better what this reported packet drop is, but I can't find any DROP or REJECTED type messages in the log (neither via logread nor syslog). You can see below it's very regular (the gaps are when I was testing connecting/disconnecting the device) - the spike is exactly 15s, so I'm assuming the device is sending a packet that gets dropped. Probably harmless, but I want to understand what it is.

I'd do a tcpdump and look at it in wireshark, but I honestly have no idea what to look for. Would a tcpdump on the OpenWRT device even catch this or would it be dropped before it hit the tcpdump filter? Might need to capture from another device on the LAN side?

Any ideas?
Thanks & cheers.

Did you enable logging?

I think so? Under Firewall, I clicked Edit for the LAN zone -> Advanced Settings, Enable logging on this zone is checked. Maybe that's not all I need to do?

Hi,
I don't know much about the OpenWrt firewall implementation but I have a background in networking.
It looks like you are graphing ethernet stats and ethernet errors/drops likely won't show at the firewall layer. I wouldn't expect any malformed ethernet frames to get passed up the next layer but depending on ethernet nic firmware and kernel driver implementation there may be more info than a basic error counter and tcpdump might have a low enough hook to see the bad or incomplete frame. You can also look at : > ifconfig eth0 to possibly see what type of error is incrementing:

 eth0      Link encap:Ethernet  HWaddr <Redacted>  
           inet addr:192.168.1.XX  Bcast:192.168.1.255  Mask:255.255.255.0
           inet6 addr: <Redacted>/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:124233725 errors:0 dropped:0 overruns:0 frame:0
           TX packets:86873835 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:177340081542 (165.1 GiB)  TX bytes:7784884662 (7.2 GiB)
           Interrupt:35 

Look to see which of errors, dropped, overruns, frame or carrier are incrementing.
Ethertool may be more useful to query the NIC for the error info. You may need to install it though.
There may be more specific counters you can query with collectd but I don't know how that is setup on your system.
Edit: dmesg may show different info than logread (it does on my system) so looking at that might show something useful.

Good luck.

It looks like the dropped error is what's incrementing (unsurprising, in hindsight):

# while true; do echo -n "$(date) "; ifconfig eth0 | grep 'RX packets' | awk '{print $3 $4 $5 $6}'; sleep 1; done
Mon Jun 27 13:31:33 CDT 2022 errors:0 dropped:52470 overruns:0 frame:0
Mon Jun 27 13:31:34 CDT 2022 errors:0 dropped:52470 overruns:0 frame:0
Mon Jun 27 13:31:35 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:36 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:37 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:38 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:39 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:40 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:41 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:42 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:43 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:44 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:45 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:46 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:47 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:48 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:49 CDT 2022 errors:0 dropped:52471 overruns:0 frame:0
Mon Jun 27 13:31:50 CDT 2022 errors:0 dropped:52472 overruns:0 frame:0
Mon Jun 27 13:31:51 CDT 2022 errors:0 dropped:52472 overruns:0 frame:0
Mon Jun 27 13:31:52 CDT 2022 errors:0 dropped:52472 overruns:0 frame:0
Mon Jun 27 13:31:53 CDT 2022 errors:0 dropped:52472 overruns:0 frame:0
Mon Jun 27 13:31:54 CDT 2022 errors:0 dropped:52472 overruns:0 frame:0
1 Like

Interesting, I found an article about dropwatch, but it seems like nobody's been able to get it to run on OpenWRT :cry:

Edit: Here's the dropped-packets article

Wait...are you saying you want to log all general Zone Drops and Rejects?

:bulb: I thought you made a specific drop rule at the end of a chain (i.e. the bottom of the config) and set that rule to log.

Try doing that.

Some more thoughts...

If the suspect device is a switch or even an endpoint device with a Realtek nic phy you might be seeing RRCP frames or associated protocols like their proprietary loop detection. I don't recall what I was seeing to prompt me to investigated on my network when I first added a Netgear switch but I was seeing those ether types in tcpdump before flashing it with OpenWrt. https://en.wikipedia.org/wiki/Realtek_Remote_Control_Protocol#Loop_Detect_packet

If you haven't tried yet, tcpdump might show the dropped frames but it could be useful to do ithe capture with the network very quiet. If you are not familiar with tcpdump, you can just capture some header info and show it on the terminal. You don't have to capture to a file and analyze later unless you want/need a more in-depth analysis. You could try various capture filters like the mac address of the suspect device to limit capture output.

If there is a switch between the suspected source and your OpenWrt device and other hosts are connected, do the other hosts or stats on the switch show the drops?

Is the suspect device a *nix system with a rich set of admin tools? You might be able to see what process is connecting to the network interface/driver with lsof type tools.