Thanks to @efahl who shared the comment below in the 23.05.2 thread:
I've decided to create this topic to add more visibility to this new feature, since it is important to anyone using luci_app_statistics. Previously we depended on custom scripts to have some sort of automatic collectd database backup (such as discussed in this topic), but now I am happy to see this officially supported as part of this upgrade.
Collectd dabase is saved to directory /etc/luci_statistics during shutdown/reboot and restored during system startup. More details (documentation) about this new feature is available here and here.
Not me! The big goes to @atownlede for doing all the work and pushing it into production.
Anyone using this needs to also read about configuration here for information on keeping the data off flash memory, or at least writing to it as little as possible:
You might be able use a symbolic link from /etc/luci_statistics to somewhere else, but some of the logic about backups/restores currently assumes that is the real name. Probably it wouldn't be hard to add a config option for a different backup location, but it will still need to be on non-volatile storage that can be saved/restored during a sysupgrade.
That would be really good... after all that is what we recommend for the current scripts to achieve the same, store to some other place than precious (as hard to replace on failure) system NOR/NAND...
If such a location is available it should be used for the Storage directory itself.
The default is to put RRD files in RAM (/tmp/rrd) and now (with this new feature) optionally back it up to device flash occasionally like when upgrading. If additional storage is available - like an external USB flash drive - it would make more sense to place the data files there (likely /mnt/sda1/rrd/ or similar) instead in RAM (/tmp/rrd) to begin with and backup is not needed.
I disagree, I might like to:
a) have ephemeral/fast access to the data
b) store it on potentially dog slow storage like USB flash drives occasionally...
that also covers the case that the USB flash wears out and dies (or is accidentally unplugged), would be nice if the router would still boot up...
Though, I guess your position is not any way better/worse than mine, but simply different...
I recently adopted multiple rootfs partitions as a long term x86 sysupgrade strategy. For this use-case, I want to keep the running RRD data in tmpfs, but backed up to a separate data partition, thus allow seamless restores when switching rootfs slots (in typical A/B testing or upgrade scenarios).
rootfs A (active)
rootfs B (standby)
data (to be mounted by rootfs slots)
I made the following patch to support relocating $BACKUP_DIR (with fallback), and if the relocated $BACKUP_DIR can persist a sysupgrade, optionally drop RRD artifacts out of the sysupgrade backup file list.
--- /mnt/sda2/etc/init.d/luci_statistics [original]
+++ /etc/init.d/luci_statistics [modified]
@@ -6,7 +6,9 @@
STOP=11
USE_PROCD=1
-BACKUP_DIR="/etc/luci_statistics"
+### support relocating $BACKUP_DIR (with fallback)
+BACKUP_DIR=$(uci -q get luci_statistics.collectd_rrdtool.backup_dir)
+BACKUP_DIR=${BACKUP_DIR:-/etc/luci_statistics}
BACKUP_FILE="${BACKUP_DIR}/rrdbackup.tgz"
SYSUPGRADE_BACKUP_FILE="${BACKUP_DIR}/rrdbackup.sysupgrade.tgz"
SYSUPGRADE_BACKUP_TWIN_A="${BACKUP_DIR}/sysupgrade.trustme.txt"
@@ -162,11 +164,15 @@
### Copy the backup to use for sysupgrade
copy_backup_for_sysupgrade
}
- ### Edit the backup file list to remove everything else
- sed -i -e /${BACKUP_DIR//\//\\/}/d $filelist
- ### Add only the files we need to ensure proper
- ### restore behavior
- echo ${SYSUPGRADE_BACKUP_FILE} >>$filelist
- echo ${SYSUPGRADE_BACKUP_TWIN_A} >>$filelist
+ ### if the relocated $BACKUP_DIR can persist a sysupgrade,
+ ### we can safely drop RRD artifacts from the backup file list.
+ [ -f "$BACKUP_DIR/.safe_for_sysupgrade" ] || {
+ ### Edit the backup file list to remove everything else
+ sed -i -e /${BACKUP_DIR//\//\\/}/d $filelist
+ ### Add only the files we need to ensure proper
+ ### restore behavior
+ echo ${SYSUPGRADE_BACKUP_FILE} >>$filelist
+ echo ${SYSUPGRADE_BACKUP_TWIN_A} >>$filelist
+ }
}
}
I am looking for feedback on my approach. If it is deemed safe and useful for others with the same use-case, I can then submit a PR.
I've installed luci-app-statistics recently but I can't understand why does it rewrite it's backup constantly? I added separate USB drive for storing luci-app-statistics data, but that app resumes to rewrite it's backup file regardless reboot/poweroff/update of router.
uptime && date && ls -lah /etc/luci_statistics/rrdbackup.tgz
11:42:43 up 1:16, load average: 0.00, 0.04, 0.06
Fri Jun 21 11:42:43 CEST 2024
-rw-r--r-- 1 root root 327.7K Jun 21 11:22 /etc/luci_statistics/rrdbackup.tgz
It's 1h and 16min router uptime but backup was made 20min ago. Can somebody please explain why it was created? I thought it should be created only before reboot/poweroff. Please correct me if I'm wrong.
Looks like I've found where was the issue (my misunderstanding). I changed configuration of luci_app_statistics and it was restarted. So backup was created before service restart.
Sorry to deadpost, but I was thinking of doing this but I wondered if it would result in frequent read/write ops that might be bad for an sdcard. Thoughts?
It can be made to only do writes at sysupgrade, or it can be configured to do interim backups via a cron job (to minimize loss on unscheduled power loss, for example).
Much thought was put into the design to reduce writes to flash, so it mostly depends on how you set it up. If I recall, there's a bunch of discussion of this on the PR which introduced the functionality.
goes to 
