Log to disk, archived by date

Installing and Using OpenWrt
1

I know the tools are out there but the implementation is beyond me. Can someone help me with the tools/cron commands needed to acheive this, currently my log setup is OpenWRT default.

  1. Logs stored into the RAM buffer are written to disk instead to survive power cycles
  2. Logread command still works (as I have other scripts that rely on this)
  3. Logs are written to 'backup' files each day, or when the log-size exceeds the max log size (so nothing is lost). Filenames are like:
    openwrt.log.ddmmyyyy (first file)
    openwrt.log.ddmmyyyy.1 (subseqenet files if the max log size is exceeded)
  4. Logs are kept for 2 weeks (14 days worth) and automatically deleted after.

I know it's a combination of the OpenWRT log settings, logrotate and cron but any help joining the dots would be helpful so I don't have to learn by trial and error!
Thanks!

3

My two cents:

  • Install "syslog-ng", but:

    • Write output to a file in RAM, called "/var/log/messages_${YEAR}${MONTH}${DAY}.log".
    • Redirect logs from OpenWrt (and other devices in the network) to it.

  • Every day, execute a script that:

    • Moves and compresses the older log files from RAM to disk
    • Deletes the oldest files from disk

This is my "syslog-ng.conf" file:

#############################################################################
# OpenWrt syslog-ng.conf specific file
# which collects all local logs into a single file called /var/log/messages.
# More details about these settings can be found here:
# https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition

@version: 4.4
@include "scl.conf"

options {
	create_dirs(yes);

	chain_hostnames(no); # Enable or disable the chained hostname format.
	keep_hostname(yes);  # Enable or disable hostname rewriting.
	use_fqdn(no);        # Add Fully Qualified Domain Name instead of short hostname.

	log_fifo_size(256); # The number of messages that the output queue can store.
	log_msg_size(1024); # Maximum length of a message in bytes.

	stats_freq(0);  # The period between two STATS messages (sent by syslog-ng, containing statistics about dropped logs) in seconds.
	flush_lines(0); # How many lines are flushed to a destination at a time.
};

source net {
	udp(ip(0.0.0.0) port(514));
};

destination msg {
	file("/var/log/messages_${YEAR}${MONTH}${DAY}.log");
};

log {
	source(net);
	destination(msg);
};

#
# Finally, include any user settings last so that s/he can override or
# supplement all "canned" settings inherited from the distribution.
#
@include "/etc/syslog-ng.d/" # Put any customization files in this directory

And this is the script that rotates the files:

#! /bin/sh

BASE="/var/log"
DEST="/mnt/flash/syslog"
MASK="messages_*.log"

SELF=`basename "$0"`

echo "${SELF} Rotating log files"

find "${BASE}" -name "${MASK}" -mtime +0 | while read FILE; do
	echo "${SELF}  Rotating '${FILE}'"
	gzip "${FILE}" && mv "${FILE}.gz" "${DEST}/"
done

And another two cents: installing "syslog-ng" will overwrite and break your "logger" binary, and you will need to reinstall it again.

2 Likes
4

Or

dateext 
rotate 14

options in logrotate.

5

Yep - I get that bit, where I'm a little fuzzy is how to get logrotate to work nicely with the in-built logger (if not wanting to move to syslog-ng).

Also - something I have always failed to understand with logrotate is how it deals with log-size limits. As it only runs on cron, presumably you need to set the logrotate max log size to something slightly (or significantly depending on log volume) smaller than the OpenWRT logger max-size so that you can be sure logrotate runs before the log overflows?

6

Default Openwrt log is round robin, large log size on disk is usually not a problem.

If you rotate the log on a daily basis, it always captures the last 24h.

If you want to minimize size on disk, add compression to logrotate, but it obviously only applies to the old logs, not the current one.

7

So just set the OpenWRT log location to be something on-disk like /var/log and then set the log max size to somthing huge, then rotate daily with logrotate?

8

I guess, yeah.