Prevent luci-app-statistics from killing my USB drive

Hey there.

Lately I'm playing around with luci-app-statistics.

I'd love to have my data collection interval set to its default of 30 seconds, but I'd rather not have the rrd file written to my USB stick very 30 seconds. Writing the rrd file every 30 minutes is fine, loosing half an hour of data in case I reboot doesn't matter.

What would be the best way of decoupling the collection interval from the persistance interval?

I thought about setting the dara directory to its default on /tmp, create a cronjob copying the rrd files every 30 minutes to my USB drive and copying the rrd file structure back from USB to /tmp when USB mounts.

Is simply copying the files a good idea or is there something like vnstat --exportdb but for the statistics package?

I found an existing script and modified it a bit. My version creates a new filename every day when it bundles and compresses RRD files so it does not always overwrite the previous files. The file just gets overwritten on one day, then a new file is created. This sort of simulates wear leveling.

Original: https://gist.github.com/tmkasun/2ec5afc44b3dff1daeb293447638e810
Save this as /etc/init.d/rrdbackup and set +x permission

#!/bin/sh /etc/rc.common

# OpenWrt /etc/init.d/ script to backup and restore the rrd (collectd) database, to preserve data across reboots
# Note: generate backupfile with name of date, then intentinoally do not clean up after copying
#  to simulate wear leveling on cheap USB (simple overwrite might wear out flash)
#  expected backup size of 500kb * 365 days * 5 years = 900MB
#
# howto:
# - adjust BACKUP_DIR below to point to the target directory for the backup (e.g., a USB drive)
# - save this file as /etc/init.d/rrdbackup
# - add execute permissions
# 	chmod +x /etc/init.d/rrdbackup
# - run first backup
#	/etc/init.d/rrdbackup start
# - for periodic backups insert into crontab:
#      0 */6 * * * /etc/init.d/rrdbackup backup
#   adjust interval to own taste (example above backs up every 6 hours)

# queue this init script to start (i.e., restore) right before collectd starts (80)
# and stop (i.e., backup) right after collectd stops (10):
START=79
STOP=11

# add commands "backup" to manually backup database (e.g. from cronjob)
# and "restore" to restore database (undocumented, should not be needed in regular use)
EXTRA_COMMANDS="backup restore"
EXTRA_HELP="    backup  backup current rrd database"

# directories and files
# RRD_DIR needs to be relative to root, i.e. no slash in front (to mitigate tar "leading '/'" warning)
RRD_DIR=tmp/rrd
BACKUP_DIR=/mnt/sda1/backups

backup() {
        BACKUP_FILE="tmp-rrd-$(date +%Y-%m-%d).tar.gz"
        tar -czf /tmp/$BACKUP_FILE -C / $RRD_DIR
        mkdir -p $BACKUP_DIR
        mv -f "/tmp/$BACKUP_FILE" "$BACKUP_DIR/$BACKUP_FILE"
        #Delete in case move failed to avoid filling up RAM (if usb was not available)
        rm -f /tmp/$BACKUP_FILE
}

restore() {
        #Find the newest backup file, this returns the full path
        RESTORE=$(ls -t $BACKUP_DIR/tmp-rrd* | head -1)
        [[ -f "$RESTORE" ]] && tar -xzf "$RESTORE" -C /
}

start() {
        restore
}

stop() {
        backup
}

Run /etc/init.d/rrdbackup backup to confirm it works. You should see a new file in your target backup directory.

Since routers don't get rebooted often, I recommend you add an entry to your crontabs to run on an interval. This example runs a backup every 6 hours.

0 */6 * * * /etc/init.d/rrdbackup backup

It is already decoupled.

Configure the RRDTool on the Output plugins tab to have a longer cache collected data for. This is how long it waits to write to disk with a default of 300 seconds.

1 Like

I am using a Cron job to copy the rrd files from /tmp once per day as a tar.gz archive.

(As daily files have datetime in filename, I need to manually delete accumulating old files every now and then.)

Thank you all, I guess I have what I need.

@dan3 and @hnyman: I most likely will make something out of your suggestions.

@DBAA: I didn't know about that setting. But after playing with it I found that doesn't only prevent the rrdtool from writing its data to the rrd file but it also prevents the graphs UI from showing the accumulated data. If I set "luci_statistics.collectd_rrdtool.CacheTimeout" to 1800 (half an hour), new data on the UI only appears every half an hour. Which indicates the UI doesn't get its data from the rrdtool collection process but reads it from the rrd file.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.