Most effective way for log rotation

For Developers
1

The task is super simple - rename a number of files, compress last log file into, let's say, ".0" added to original file name, and clear current log file.
Is there any effective recommended way - in particular regarding compressing the content of existing file? Zlib does not provide simple command to compress file - I need to open it, load contents (most probably in parts because log could be big), save etc; OS gzip is super simple, and if using it I will be dependent on separately installed components.

Your opinion for doing it in pure C code?

2

Since you do mention separately installed components like gzip, I assume you want to achieve this inside your own application without external dependencies? Or what would be the reason to not use existing tools like logrotate + gzip?

3

I can not answer your question, the only requirement is having least dependency, predictability of execution and fault tolerance. I see I can run gzip using popen(). I suspect I can use zlib. What else is out there? How would you do it?

5

Logrotate is probably the simplest method.

6

If using logrotate really isn't an option, there are a few things to consider when moving/clearing log files (looking at logrotate's source / documentation may help here):

  • File locks / race conditions
  • File permissions

I've personally had a bad experience when using zlib manually, so I personally would always use an external gzip application, if you don't know exactly what you're doing.
Apart from that, renaming the current logfile and creating an empty new one is a quite simple task.

7

What is about sharing control - if logrotate currently works on current log file, and my application will fail opening log file for writing?

8

You are mixing up the log handling program and log rotate program.

9

Thank you. Apologies for the question, what is a proper way to invoke gzip and wait for its execution completion? There're system(), exec() family of commands, application will also need to locate gzip utility in reliable way, etc...

10

My application handles the log. And that's why I want it to continue handling it in terms of rotation (renaming/deletion/compression).

11

This is a common use case. The logrotate program has the ability to execute a post-rotate command.

So, if using syslog-ng, the entry would look something like

/var/log/messages {
   missingok
   rotate 7
   daily
   postrotate
         /usr/sbin/syslog-ng-ctl reopen
   endscript
}

A simple way to implement this in your application would be to implement a signal handler for SIGUSR1. In the signal handler, reopen the log file. In the logrotate configuration postrotate command, put a kill -10 <pid>

12

You are wanting to unnecessarily reinvent the wheel. The solution I gave you above is the correct way to do it. The system administrator should be able to configure the log rotation method and policy and if you try to handle log rotation in your application, you're removing this choice and flexibility and adding unneeded complexity to your application.

It's much better to just add a signal handler to reopen the file. This way an external log policy handler can control log rotation policy. It results in simpler code and does not override global administrator control of log file rotation policies.

1 Like
13

And not to mention that this is the way Debian based Linux systems by preinstalled default handles logs. At least in Ubuntu and RaspberryPi.