How to check collectd network plugin and prometheus import is working?

[dennys]

I want to see the network traffic by client and I use iptmon, it works.
I can see the chart like https://github.com/oofnikj/iptmon

And then I want to export the data into Prometheus/Grafana, the following is my steps:

1. I install Collectd Exporter (https://github.com/prometheus/collectd_exporter) in my PC (192.168.1.236), it listens UDP on port 25826.

C:\> docker run -p 9103:9103 -p 25826:25826/udp prom/collectd-exporter --collectd.listen-address=":25826"
level=info ts=2022-07-26T04:36:50.837Z caller=main.go:321 msg="Starting collectd_exporter" version="(version=, branch=, revision=)"
level=info ts=2022-07-26T04:36:50.837Z caller=main.go:322 msg="Build context" context="(go=go1.14.2, user=, date=)"
level=info ts=2022-07-26T04:36:50.837Z caller=main.go:344 msg="Listening on address" address=:9103

2. I add network plugin in my collectd.conf.

LoadPlugin network
<Plugin network>
        Server "192.168.1.236" "25826"
        ReportStats true
</Plugin>

3. I restart collectd and it seems work because the log show successfully. (But it's strange, even I change the IP to an invalid IP address, it still shows successfully.)

[2022-07-26 22:13:07] Exiting normally.
[2022-07-26 22:13:07] collectd: Stopping 2 read threads.
[2022-07-26 22:13:07] collectd: Stopping 5 write threads.
[2022-07-26 22:13:07] rrdtool plugin: Shutting down the queue thread.
[2022-07-26 22:13:07] plugin_load: plugin "network" successfully loaded.
[2022-07-26 22:13:07] plugin_load: plugin "iwinfo" successfully loaded.
[2022-07-26 22:13:07] plugin_load: plugin "memory" successfully loaded.
[2022-07-26 22:13:07] plugin_load: plugin "cpu" successfully loaded.
[2022-07-26 22:13:07] plugin_load: plugin "rrdtool" successfully loaded.
[2022-07-26 22:13:07] rrdtool plugin: RRASingle = true: creating only AVERAGE RRAs
[2022-07-26 22:13:07] plugin_load: plugin "load" successfully loaded.
[2022-07-26 22:13:07] plugin_load: plugin "interface" successfully loaded.
[2022-07-26 22:13:07] Initialization complete, entering read-loop.

4. I already have a Prometheus and I use it to see the metrics, but I only see the 2 metrics. (I only check collectd_*** metrics, go_*** and process_*** are ignored)

collectd_exporter_build_info{branch="",goversion="go1.14.2",revision="",version=""} 1
collectd_last_push_timestamp_seconds 0

Because the data flow is long...

collectd -> collectd network plugin -> Prometheus collectd_exporter -> Prometheus server -> Grafana

I'm not sure what's the problem. I only know collectd should be ok (because I can see the graph in luci-app-statistics and RRD files are generated in /var/rrd/OpenWrt
). But Prometheus server doesn't get the data.

I guess it's due to collectd network doesn't send data, but I cannot confirm it. Any suggestion to debug it?

[hnyman]

Verify if it is

• a collectd problem,
• a firewall problem, or
• a prometheus problem

You could setup collectd also into another router and configure data to flow from the router to there. That would enable you to see if the problem is on the collectd networking in the router.

Other thought is that the exporter docs also mention write_http plugin for collectd and the docker example also uses port 9103. The docs leave me uncertain if that plugin is also required (or is that just an example how to transfer data in the same host, when proper networking between devices is not actually needed).

Third idea is Windows firewall. You run the prometheus in docker in Windows, I think. Make sure that you have opened UDP ports 25826 and 9103 in the PC's firewall.

(Ps. I have used the collectd networking between two OpenWrt routers and the data has flowed ok, so that the other router's data can be shown in LuCI.)

[dennys]

@hnyman Thanks, it's firewall problem. I fixed it.
But, because collectd_exporter uses UDP, is there any solution to debug?
If it's a TCP, I can use telnet IP port to check .

[slh]

nc/ netcat?

[dennys]

@slh Thanks, nc is ok. I test it to an invalid port and it refused.

root@OpenWrt:/etc# echo -n 0 | nc -u -w1 192.168.1.236 25825
read(net): Connection refused
root@OpenWrt:/etc# echo -n 0 | nc -u -w1 192.168.1.236 25826