SQM Reporting?

bahtsiz_bedevi · May 9, 2020, 9:44pm

root@OpenWrt:~# ps | grep collectd
 3275 root      1212 S    grep collectd

root@OpenWrt:~# nano /tmp/collectd.conf
LoadPlugin exec
<Plugin exec>
        Exec "nobody:nogroup" "/usr/libexec/collectd/sqm_collectd.sh" "eth0.2"
</Plugin>

I don't know what you mean.

bahtsiz_bedevi · May 9, 2020, 9:53pm

whole /etc/collectd.conf

BaseDir "/var/run/collectd"
Include "/etc/collectd/conf.d"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 30
ReadThreads 2


LoadPlugin rrdtool
<Plugin rrdtool>
        RRASingle false
</Plugin>

LoadPlugin exec
<Plugin exec>
        Exec "nobody:nogroup" "/usr/libexec/collectd/sqm_collectd.sh" "eth0.2"
</Plugin>

LoadPlugin thermal
<Plugin thermal>
        IgnoreSelected false
</Plugin>

bahtsiz_bedevi · May 9, 2020, 10:01pm

I see these in the system log

Sun May 10 00:50:11 2020 daemon.err collectd[3523]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:11 2020 daemon.err collectd[3523]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:11 2020 daemon.err collectd[3523]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:11 2020 daemon.err collectd[3523]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:11 2020 daemon.err collectd[3523]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:11 2020 daemon.err collectd[3523]: Error: Parsing the config file failed!
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:16 2020 daemon.err collectd[3541]: Error: Parsing the config file failed!
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:21 2020 daemon.err collectd[3572]: Error: Parsing the config file failed!
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:26 2020 daemon.err collectd[3575]: Error: Parsing the config file failed!
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:31 2020 daemon.err collectd[3578]: Error: Parsing the config file failed!
Sun May 10 00:50:36 2020 daemon.err collectd[3582]: configfile: stat (/etc/collectd/conf.d) failed: No such file or directory
Sun May 10 00:50:37 2020 daemon.err collectd[3582]: plugin_load: plugin "rrdtool" successfully loaded.
Sun May 10 00:50:37 2020 daemon.err collectd[3582]: plugin_load: plugin "exec" successfully loaded.
Sun May 10 00:50:37 2020 daemon.err collectd[3582]: plugin_load: Could not find plugin "thermal" in /usr/lib/collectd
Sun May 10 00:50:37 2020 daemon.err collectd[3582]: There is configuration for the `thermal' plugin, but the plugin isn't loaded. Please check your configuration.
Sun May 10 00:50:37 2020 daemon.err collectd[3582]: Error: Parsing the config file failed!

ldir · May 10, 2020, 2:22pm

I'm really not sure what's going on. You need to look in /etc/config/luci_statistics and make sure only relevant modules are enabled. For sqm stats that would just be 'collectd_exec'.

It's slightly confusing because the collectd service does the collection, but luci_statistics does the setup of /etc/collect.conf. luci_stats overwrites the default config by placing a symbolic link to its new config /etc/collectd.conf -> /var/etc/collectd.conf

JonP · May 10, 2020, 9:39pm

Very interesting... Couple of questions, I assume that this is not in the 5.11.0-1 collectd upgrades I see from the 5.10.xx I have now?

And wondering what the potential for creating a UIO (Unexpected Internet Outage) if I dropped this in my 19.7.1 router box, resulting in unrest and threats of violence in the currently calm home... I'm thinking "this particular bleeding edge thing shouldn't cause problems on the home router". I've said those words before....

ldir · May 11, 2020, 8:18am

I can't really advise at all on the package aspect. I build my own version builds from scratch and select what I want in the image and flash the entire thing so I never really get involved with the updates & how they work.

jow · May 11, 2020, 10:25am

The rrdtool output plugin is not configured, so collectd will not create *.rrd files.

anon50098793 · May 15, 2020, 10:44am

rpi4 45000/17500 peiceof thankyou @ldir !!!

Knomax · May 16, 2020, 3:27pm

Awesome to have this....but can someone provide us a step by step guide how to etc install all this staff?

hnyman · May 16, 2020, 3:55pm

That is in post #49 of this thread.

The installations requires some package updates and config editing, so the advice there assumes that you are familiar with installing packages and editing config files.

Bartvz · May 17, 2020, 2:04pm

Followed the instructions, everything necessary is installed and configuration has been done, but I don't get the two tabs. I get an unknown qdisc type in system log:

daemon.err collectd[10306]: exec plugin: exec_read_one: error = Unknown qdisc type

Using SQM with cake. What am I doing wrong?

Sparks · May 27, 2020, 11:44am

I’ve been following closely this topic with keen interest. Is there any news on this one? Is there a chance it might become available under 19.07.3 or would it have to wait for 20.x ?

hnyman · May 27, 2020, 1:18pm

The data collection part (collectd modules) would work in 19.07 already now (and I have backported it for my own R7800 19.07 build), but the LuCI graphs would need adjustments, as the master has javascript based LuCI statistics app, while 19.07 still has the lua based one. So the LuCI part can not be cherry-picked.

Easiest would be to wait for 20.0x.

anon50098793 · June 2, 2020, 8:50am

[ "$jsn" = "[]" ] && return

rickk · June 3, 2020, 8:13pm

I've cobbled together a Grafana dashboard that takes the SQM collectd data from InfluxDB.
Could be useful if you happen to have this (collectd->influxdb->grafana) stack working already.
Almost works out-of-the-box with the owrt1907-r11086-ab7e9754df-20200525 hnyman build, you only need to configure the collectd exec plugin(s) for your interface(s).

Thanks @ldir and @hnyman!

dtaht · June 6, 2020, 2:28am

Getting cooler by the day on this front, everyone!

I've been out of openwrt dev so long that I can't figure out how to fix this:

WARNING: Makefile 'package/feeds/packages/collectd/Makefile' has a dependency on 'libiptc', which does not exist

And then I dug a hole for myself by trying to compile with gcc 9...

hnyman · June 6, 2020, 9:26am

Drop iptables plugin from your collectd build (or edit the Makefile or fix your repo structure).

It has been fixed in master, as libiptc has been removed from iptables 1.8.4.
https://git.openwrt.org/?p=feed/packages.git;a=commitdiff;h=d5717f3930aa3ce8d824c6f6f294f37f5af111ba;hp=da288e1f28151ec2101da68558def5e553afcf81

I wonder from which branch you are building: the SQM cake reporting exists only in master, while that error might only surface in the ancient 19.07 stable branch. And in the 19.07 branch, iptables is still 1.8.3 with that libiptc still defined, so you should not run into that error. So, I really wonder how you can end up with that error.

At the first glance I think that you should only see that error if you have somehow mixed repo of mismatched openwrt core sources plus 19.07 packages (which would not include SQM stats monitoring) ????

If you want SQM reporting (= you build from master), you should not run into that error.

dtaht · June 6, 2020, 2:19pm

So far as I knew I was pulling from head (most recent commit was the wireguard thing from ldir). Perhaps I'm not pulling from head on the packages?

src-git packages https://git.openwrt.org/feed/packages.git^99efce0cd27adfcc53384fba93f37e5ee2e517de
src-git luci https://git.openwrt.org/project/luci.git^13dd17fca148965d38f0d4e578b19679a7c4daa2
src-git routing https://git.openwrt.org/feed/routing.git^efa6e5445adda9c6545f551808829ec927cbade8
src-git telephony https://git.openwrt.org/feed/telephony.git^6f95d6ab3f359ee2ce81c20522700937424d1591

I also do not have anything but

CONFIG_PACKAGE_collectd=m

set in the .config file
~

bahtsiz_bedevi · June 6, 2020, 2:39pm

I got stuck with 19.07.3 because 20.0x for MT7621 doesn't have kmod-sched-cake-oot and waiting this to be merged for 15 days...
Unbelievable

hnyman · June 6, 2020, 2:47pm

Well, as you are explicitly pulling from a pinpointed ancient commit, you are not pulling from the HEAD.
"99efce0cd27a" is a commit in the 19.07 branch committed in February...
Looks like you have locked your package feeds' source def into the 19.07.2 release. This is what was set as fixed for 19.07.2:
https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=33732f4a9c17921b782167a0dcaba9703d4e6753

Somehow you have 19.07 stuff in the package feed definitions your main openwrt sources, while the SQM reporting only exists in the master branch.

Likely you have modified your feeds.conf.default, and you should git checkout a normal version of that file. The current master has this:
https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=feeds.conf.default;h=da6c6656244f4ae41a05ad91d7031e1fe78cc711;hb=HEAD

   src-git packages https://git.openwrt.org/feed/packages.git
   src-git luci https://git.openwrt.org/project/luci.git
   src-git routing https://git.openwrt.org/feed/routing.git
   src-git telephony https://git.openwrt.org/feed/telephony.git