Luci_app_statistics now has a backup feature starting with 23.05.2

Thanks to @efahl who shared the comment below in the 23.05.2 thread:

I've decided to create this topic to add more visibility to this new feature, since it is important to anyone using luci_app_statistics. Previously we depended on custom scripts to have some sort of automatic collectd database backup (such as discussed in this topic), but now I am happy to see this officially supported as part of this upgrade.

Collectd dabase is saved to directory /etc/luci_statistics during shutdown/reboot and restored during system startup. More details (documentation) about this new feature is available here and here.

Kudos to @atownlede @hnyman @efahl for this new feature! :clap:

4 Likes

Not me! The big :+1:t3: goes to @atownlede for doing all the work and pushing it into production.

Anyone using this needs to also read about configuration here for information on keeping the data off flash memory, or at least writing to it as little as possible:

3 Likes

What about an option 'backup_directory'? Instead of a hard-coded backup directory.

You might be able use a symbolic link from /etc/luci_statistics to somewhere else, but some of the logic about backups/restores currently assumes that is the real name. Probably it wouldn't be hard to add a config option for a different backup location, but it will still need to be on non-volatile storage that can be saved/restored during a sysupgrade.

That would be really good... after all that is what we recommend for the current scripts to achieve the same, store to some other place than precious (as hard to replace on failure) system NOR/NAND...

If such a location is available it should be used for the Storage directory itself.

The default is to put RRD files in RAM (/tmp/rrd) and now (with this new feature) optionally back it up to device flash occasionally like when upgrading. If additional storage is available - like an external USB flash drive - it would make more sense to place the data files there (likely /mnt/sda1/rrd/ or similar) instead in RAM (/tmp/rrd) to begin with and backup is not needed.

1 Like

I disagree, I might like to:
a) have ephemeral/fast access to the data
b) store it on potentially dog slow storage like USB flash drives occasionally...

that also covers the case that the USB flash wears out and dies (or is accidentally unplugged), would be nice if the router would still boot up...

Though, I guess your position is not any way better/worse than mine, but simply different...

I recently adopted multiple rootfs partitions as a long term x86 sysupgrade strategy. For this use-case, I want to keep the running RRD data in tmpfs, but backed up to a separate data partition, thus allow seamless restores when switching rootfs slots (in typical A/B testing or upgrade scenarios).

  • rootfs A (active)
  • rootfs B (standby)
  • data (to be mounted by rootfs slots)

I made the following patch to support relocating $BACKUP_DIR (with fallback), and if the relocated $BACKUP_DIR can persist a sysupgrade, optionally drop RRD artifacts out of the sysupgrade backup file list.

--- /mnt/sda2/etc/init.d/luci_statistics [original]
+++ /etc/init.d/luci_statistics [modified]
@@ -6,7 +6,9 @@
 STOP=11
 USE_PROCD=1
 
-BACKUP_DIR="/etc/luci_statistics"
+### support relocating $BACKUP_DIR (with fallback)
+BACKUP_DIR=$(uci -q get luci_statistics.collectd_rrdtool.backup_dir)
+BACKUP_DIR=${BACKUP_DIR:-/etc/luci_statistics}
 BACKUP_FILE="${BACKUP_DIR}/rrdbackup.tgz"
 SYSUPGRADE_BACKUP_FILE="${BACKUP_DIR}/rrdbackup.sysupgrade.tgz"
 SYSUPGRADE_BACKUP_TWIN_A="${BACKUP_DIR}/sysupgrade.trustme.txt"
@@ -162,11 +164,15 @@
 			### Copy the backup to use for sysupgrade
 			copy_backup_for_sysupgrade
 		}
-		### Edit the backup file list to remove everything else
-		sed -i -e /${BACKUP_DIR//\//\\/}/d $filelist
-		### Add only the files we need to ensure proper
-		### restore behavior
-		echo ${SYSUPGRADE_BACKUP_FILE} >>$filelist
-		echo ${SYSUPGRADE_BACKUP_TWIN_A} >>$filelist
+		### if the relocated $BACKUP_DIR can persist a sysupgrade,
+		### we can safely drop RRD artifacts from the backup file list.
+		[ -f "$BACKUP_DIR/.safe_for_sysupgrade" ] || {
+			### Edit the backup file list to remove everything else
+			sed -i -e /${BACKUP_DIR//\//\\/}/d $filelist
+			### Add only the files we need to ensure proper
+			### restore behavior
+			echo ${SYSUPGRADE_BACKUP_FILE} >>$filelist
+			echo ${SYSUPGRADE_BACKUP_TWIN_A} >>$filelist
+		}
 	}
 }

I am looking for feedback on my approach. If it is deemed safe and useful for others with the same use-case, I can then submit a PR.