Update rrdtool up to 1.4 or higher

buitre-buitaker · October 13, 2022, 10:18am

Hi!

I have seem a topic on version 1.2, but it is 3 years old, and an older version, and the author didn't publish his results.

I need to update rrdtool up to 1.4 to get the --json data dump in rrdttool.

I would like to use openwrt SDK, but from 1.0.50 version there are a lot of changes in the dependencies.

Any experience updating rrdtool up to 1.4 or higher version?

Any guide, tutorial, support or similar process with other openwrt package to follow it?

Thank you!

slh · October 13, 2022, 11:51am

Work on the source and build the whole firmware image including your changes from source. The SDK only works (reasonably) for (modified) leaf packages, not libraries used by multiple other packages.

hnyman · October 13, 2022, 7:35pm

I did the rrdtool upgrade a year ago for testing purposes, but did not keep it, as newer rrdtool versions require several nonstandard graphics/font libraries.

We have kept using the really ancient rrdtool 1.0.0.50 from year 2003/2005 as it is small and suitable for limited flash space.

(I added the cairo and pango libraries manually, but nowadays they exist also in the nonstandard video feed.)

Edit:
I upgraded to 1.7.2, and one example graph is here:

I could try digging up the source changes on weekend, when I am back home.

buitre-buitaker · October 14, 2022, 3:43am

I have been checking the forum post. My target system is a misp 32 bits router, but I could try to readapt the code.

I really appreciate if you could find the source code for the rrdtool package with your changes.

Thank you @hnyman !

hnyman · October 16, 2022, 3:30pm

Btw, do you intend to use rrdtool as a separate tool, or to integrate it to LuCI statistics and collectd?

I am currently trying to compile the new version, and as I want to seamlessly integrate it into the LuCI statistics, there are a few more steps than for a pure separate tool installation.

Also note that the old rrdtool 1.0.50 has been patched to ignore the 64 bit time_t in 32 systems and to handle time data as native 32bit in 32bit systems.

I am not going to patch 1.7.2 that way, as it should recognise the time_t type better in any case. But the drawback from that is that the earlier databases from earlier rrdtool versions are not compatible (at least for some tools). That should not bother you if you use rrdtool separately, but could hurt you if you aim to use it to export old LuCI databases.

buitre-buitaker · October 16, 2022, 5:26pm

Hi @hnyman,

OK, I will describe my current setup and how I expect to use rrdtool. I think that I am going to be in troubles.

I am using opemwrt collectd and the mqtt plugin versions to collect sensors and home devices data into rrd databases.

I would like to use rrdtool to fetch data from these rrd databases an dump it using json format, with the rrddrv server. That way I will be able to draw data in Grafana.

I think, if I understand properly what you explained, that I am not going to be able to use rrdtool 1.7 to fetch data from the old format rrd databases generated by collectd plugin.

Is this the problem?

Thank you

hnyman · October 16, 2022, 5:47pm

Yes, it is.
You would need to patch 1.7.2 similarly as I finally did for 1.0.50 in the commit below, in order to be able to read from the old databases. 1.7.2 properly evaluates the size on time_t and recognises the 64bit time_t (which needs to be avoided)

github.com/openwrt/packages

rrdtool: Keep time as 'long' despite 64bit time_t in musl

committed 05:58PM - 10 Oct 21 UTC

hnyman

+631 -1

Adjust to the 64 bit time_t in musl 1.2.2 by keeping time internally in RRD stil…l as 'long', so that old databases created during musl 1.1.x still work in 32 bit targets. That also helps with several interal functions that use 'long' for calculations and printf formatting. * Define internal type TIME_INT as 'long', and use that internally in calculations * Use time_t vs. TIME_INT casting when necessary * Provide time_t variable pointers for functions expecting a time_t pointer. Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>

I originally thought to go to the other direction, patching the ancient rrdtool to properly use 64bit time_t also in 32bit systems, but after discussing with jow, I went with the approach of keeping 32bit time_t in 32bit systems so that old databases are compatible.

Ps.
I compiled the 1.7.2 but it does not work, yet.
I think that the reason is that this time I used cairo, pango and mesa from the video feed, and they enable more features than necessary for the simple graphs. That leads into some runtime library linking problem. (and also caused my firmware image to grow by 4 MB)

Sadly I do not have the cairo & pango patches from the 2021 trial, when I created simple versions of the pango and cairo packages myself.

hnyman · November 20, 2022, 9:25pm

FYI, although you might not want this any more (due to the possible incompatibility with existing databases), I document here that I re-implemented support for new rrdtool versions.

Now there is already rrdtool 1.8.0.

I have only tested this in my mt7622/RT3200, which is a 64bit system, so the old databases created by 1.0.50 are compatible with the new rrdtool 1.8.0.

But with 32 bit systems, the time_t size is different and the old databases are incompatible. (but I haven't tested that yet with my R7800. Nor have I tried to create a compatibility layer like I did for 1.0.50)

Both luci-app-statistics and collectd have a new option to select the rrdtool version.
Both options need to be selected if the new rrdtool is used. They default to the old one.

Below are links to my Github repos, where there are rrdtool180 branches in packages and luci:

3 commits in the packages repo
https://github.com/hnyman/packages/commits/rrdtool180

Importing the needed graph libraries (fribidi, harfbuzz, cairo, pango)
(I didn't use the versions in the video feed, as they seemed to be too beefed up for this purpose)
rrdtool 1.8.0
collectd modifications

1 commit for LuCI:
https://github.com/hnyman/luci/commits/rrdtool180

select the correct rrdtool version for luci-app-statistics

The LuCI part is not needed if you are using collectd separately from LuCI stats.

Note that this is just a proof-of-concept, not really a final version.

hnyman · December 7, 2022, 4:53pm

Brief summary of my thoughts:

github.com/openwrt/luci

RFC: creating an enhanced version of LuCI statistics graphs?

opened 04:49PM - 07 Dec 22 UTC

hnyman

**Creating an alternative version of LuCI statistics with enhanced graphics?** … We are currently using the year 2003 version of rrdtool 1.0.50 in LuCI statistics. The benefit is that the RRD library 1.0.50 is really small, compact and has no external dependencies. It is really efficient on flash space. The drawback is that we miss all the developments for RRDtool in the last 19 years and the graphs are rather basic and show their age. In practice there is also no support for the current RRDtool for other needs than LuCI statistics. The graphs from the current RRDtool 1.8.0 are more polished and crisp. The 1.8.0 is also aware of the time_t size and can adjust to 64bit time in 32bit systems. One year ago I briefly tested with new RRDtool, when looking at the musl 64bit time_t issues. See * rejected proposal to accept 64bit time_t and modify the ancient RRDtool for that: https://github.com/openwrt/packages/pull/16703 * current modification of keeping 32 bit time in 32 bit CPUs: https://github.com/openwrt/packages/pull/16835 We ended up patching the ancient RRDtool version to use 32bit time with 32bit CPUs despite musl always has 64bit time_t. That is a short-term kludge, but offers no compatibility with the current RRDtool versions and is not future-proof. (However, that enables the old databases to be compatible with the current musl. That is useful to those (few) that have preserved stats data in sysupgrades.) Triggered by the recent forum discussion in https://forum.openwrt.org/t/update-rrdtool-up-to-1-4-or-higher/139473 I have now implemented a new upgrade to RRDtool 1.8.0, and I am considering the possible roadmap scenarios for LuCI stats. The crucial aspect of RRDtool 1.8.0 is that RRDtool has in 2005-2008 moved from the ancient self-contained lib to more complex graphics libraries cairo, pango, freetype etc. (requiring also glib2). So, the size of the required libraries is approx. 3 MB larger (as measured from a compressed sysupgrade image for RT3200). That is a lot of flash space :-( Due to the size increase, I see no feasibility in wholly replacing the old rrdtool 1.0.50 with the new version. Instead, there might be two alternative versions of rrdtool and LuCI statistics. **For that purpose, I have created the patch set that** https://github.com/hnyman/packages/tree/rrdtool180 * creates the necessary graphics libs in feeds/packages: harfbuzz, fribidi, pango, cairo * creates an alternative new "rrdtool" next to the old "rrdtool1" * modifies collectd to select between the old rrdtool 1.0.50 and the new 1.8.0 at build time https://github.com/hnyman/luci/tree/rrdtool180 * modifies LuCI statistics to select between rrdtool 1.0.50 and 1.8.0 (Both luci-app-statistics and collectd need to have the enhanced version selected manually, so this is just a proof-of-concept at this point. But it works.) The non-standard video feed contains versions of pango, cairo, harfbuzz and fribidi authored by @dangowrt , but they have too many features enabled (and caused compile errors), so I created new minimal versions of the same packages into the packages feed. **So, regarding a possible future roadmap I am considering:** * renaming the old rrdtool1 as rrdtool-mini (or rrdtool-old ???) * renaming the old collectd-mod-rrdtool to collect-mod-rrdtool-mini (or -old ???) * introducing "-mini" versions of pango, cairo, harfbuzz and fribidi into the packages feed. The video feed might have more full version of them. * introducing new rrdtool (the current 1.8.0) (and also keep the old rrdtool1 as rrdtool-mini) * introducing new collectd-mod-rrdtool that links to the new rrdtool 1.8.0. That requires patching collectd to have a two rrdtool plugins. Having two separate collectd-mod-rrdtool variants would enable LuCI to have one master option about the normal or enhanced LuCI stats, instead of separately defining the underlying lib also for collectd-mod-rrdtool at the build time. * modifying LuCI to either * have a similar stats build-time config option that I use now, or alternatively * offer two statistics variants, "normal" and "enhanced" Sounds feasible? **Specific questions:** * I have now been testing with RT3200 that has 64bit CPU, so the RRD database is compatible between 1.0.50 and 1.8.0. I do not see much point in patching the 1.8.0 so that it would always still use 32bit time with 32bit CPUs. I would prefer to keep it in sync with the upstream RRDtool. So I propose that the new 1.8.0 would be the current default without patching it to use 32bit time. That would more easily enable it to be used for other purposes than just LuCI stats. (It would break compatibility to old 32bit databases, but those can be manually converted. I tested that a year ago) Ok? * Does introducing -mini versions of pango, cairo etc. in feeds/packages sound feasible? * Renaming rrdtool1 to rrdtool-mini? @jow- @dangowrt