Introduction

Logging strategy varies between Linux distributions as some (like Arch) make heavy use of systemd's journal but they all usually store some log files in /var/log with automatic rotation configured meaning old log files get renamed and compressed, and eventually destroyed to only keep a certain amount of them.

For instance, Debian systems have a default configuration for the Apache web server log rotation expecting to find them in /var/log/apache2.

The default is to rotate Apache logs every single day and keep a maximum of 14 old log files, compressed using gzip.

Sometimes you might want more of less than that, or have configured log files outside of /var/log/apache2 where it won't be automatically rotated at all and could result in log files of more than 1GB of text, even for small websites.

The program responsible for log rotation is called logrotate. It's actually stateful and keeps a "status" file that can be read here: /var/lib/logrotate/status and is almost always invoked daily by Cron.

Global configuration

The main config file is at /etc/logrotate.conf. On Debian, it's pretty minimalist by itself but has a directive that will read and process any file present in /etc/logrotate.d, which is where all the custom log rotation should be configured.

The usual purpose of the main config file is to set the default options for log rotation. These may vary between distributions, I like to leave them as they are and override what is needed by being very explicit in custom config files.

Custom configuration

On Debian systems, most of the common system logs are rotated using the config at /etc/logrotate.d/rsyslog which tends to look like this:

/var/log/syslog
{
  rotate 7
  daily
  missingok
  notifempty
  delaycompress
  compress
  postrotate
    /usr/lib/rsyslog/rsyslog-rotate
  endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
  rotate 4
  weekly
  missingok
  notifempty
  compress
  delaycompress
  sharedscripts
  postrotate
    /usr/lib/rsyslog/rsyslog-rotate
  endscript
}

We'll see momentarily what these options are about. For the most part, it's somewhat intuitive.

We see we can provide multiple log file locations separated by line feeds or spaces. We can also use (POSIX + globbing) wildcards.

In short we could match two log files and have them share the same rotation config like so:

/path/to/log1.log
/path/to/log2.log
{
  option1 value
  option2 value
}

Or, using a wildcard (could match a lot more files):

/path/to/*.log
{
  option1 value
  option2 value
}

Paths are always absolute and options (sometimes called directives) and their possible values are idented using a tab character.

Rotation frequency and retention count

The first directives in logrotate configurations are usually one of the rotation frequencies or the rotate count.

Looking at the syslog example from above:

rotate 7
daily

Means log rotation is triggered every day and we remove the oldest log files after 7 rotations.

To be clear, 7 rotations means we keep 7 rotated log files additionally to the current (the one that is being written to) log file.

Using rotate 0 is a special case where the old log data is completely removed at each rotation, meaning no actual rotation is taking place but the log is flushed at the rotate frequency.

The available rotation trigger frequencies are the following:

• daily
• weekly [weekday] — Where the optional week day indicates the exact day when it triggers (careful that 0 is sunday) — 7 has the special meaning to rotate every 7 days exactly regardless of what day it is;
• monthly
• yearly
• hourly

A common rule of thumb is to lower the trigger frequency for intense log files and leave it high or increase it for somewhat passive ones.

For instance, a Debian-based mail server could use the daily rotate option instead of its default of weekly if using the default mail.log or mail.info log file but a workstation can easily keep or increase up to monthly with no consequence.

While we're at it, it's possible to configure log centralization or your own archiving system but this is out of scope for this article.

Common configuration options

Let's make ourselves a quick reference guide of the most useful options.

create [mode owner group]

Causes the creation of a new log file after rotation (before a possible postrotate script is ran).

It's often a good idea to add the file mode (permissions in octal), owner and group as in:

create 0640 www-data adm

Where the new log file gets the read-write permissions for the www-data user and read permission for the adm group, anyone else denied all access.

Not providing these options causes logrotate to copy the attributes from the previous log file. Having them in makes sure permissions don't get accidentally modified so that a program can no longer log to the file.

The file creation might fail if a program is keeping a lock on the previous file. This option is often used in combination with a postrotate script that orders the program to re-open the new log file immediately.

The "create" option can be ignored completely if you trust the logging program to create a new log file on its own (most will just do that) but it's always safer to make it explicit.

copytruncate

The normal rotation involves renaming the previous log file (and possibly creating a new one if create is present).

With copytruncate, logrotate first copies the entire log file (instead of renaming) then truncates (set file size to 0, remove the whole content of) the current log file.

As a result, no new log file is created, it's still the old one that's been emptied completely, allowing the logging program to keep an active lock on the log file.

The copytruncate option disables and replaces create if present and is usually seen as the "safer" option since it'll very rarely fail with any kind of program, including the ones you don't know how to notify a new log file was created.

It's a surefire way to do log rotation but still allow the logging program to keep a lock on the log file.

However, it's possible logging data might be lost in the time window between the copy and the truncating. Small log files and/or very fast storage makes this is almost a non-issue but it's very important to keep in mind.

missingok

Don't produce an error if the log file is missing. That option is almost always present for system and custom scripts, you should probably always have it.

compress

Compresses rotated logs with gzip. Very useful to save storage space, not enabled by default on Debian-based distributions.

delaycompress

Doesn't have any effect unless compress is also present.

Leaves the latest rotated log file uncompressed.

This option is seen quite often for safety reasons in combination with create as it allows the logging program to keep a handle on the renamed log file (the inode should be the same as before) and keep logging to it until the program is notified a new log file has been created (usually in a postrotate script).

Regardless of the technical details, remember this option should always be used when create and compress are present unless you know what you're doing.

maxsize [size]

Rotate logs when they grow bigger than the specified size in bytes even if the configured log rotation trigger wasn't reached yet.

As always, the amount of files configured in the rotate directive will be kept.

size [size]

Does the same thing as maxsize but completely replaces any time related-rotation option (such as daily, weekly, etc.) whereas maxsize works in combination with rotation timing options.

Consider it a remplacement for daily, weekly, etc.

minage [days]

Do not rotate at all unless the log file is at least <days> old.

maxage [days]

Remove rotated logs that are older than <days> amount of days. Remember it'll be used in combination with the standard rotate.

shred

Uses the shred program before removing files, which rewrites over the storage multiple times where the file was to make it extremely hard to recover the content.

Can be interesting to enable on sensitive log files, has a performance cost depending on the file size to be shredded.

start [count]

By default, rotated files are renamed using a trailing number that starts from 0.

This option allows specifying something else than 0 as the start for rotated log files numeration.

dateext

Use the current date as the file extension instead of the default numbering (adding .0).

Very rarely seen in practice but could be useful.

notifempty

Do not rotate if the log file is empty at the time where it should be rotated.

Can be useful for very low verbosity log files though there are other options you could use such as size-based rotation or very long timing triggers.

The no* options

Most of the options presented above have an equivalent preceded by "no".

For instance, nocompress is the opposite of compress.

These options can be used to work around the default ones set in /etc/logrotate.conf and to make sure some option you really don't want is enabled without you knowing.

Scripting postrotate, prerotate et al.

The postrotate and prerotate directives allow running commands before and after the rotation process takes place, respectively, for the log files concerned by that current configuration.

Generally, postrotate is where the logging program is notified to reload the new log file.

How this is done depends on the program and ranges from sending it a signal up to completely restarting the whole process.

Here is an exemple for the MySQL log rotation on Debian:

/var/log/mysql/mysql.log
/var/log/mysql/mysql-slow.log
/var/log/mysql/mariadb-slow.log
/var/log/mysql/error.log {
  daily
  rotate 7
  missingok
  create 640 mysql adm
  compress
  sharedscripts
  postrotate
    test -x /usr/bin/mysqladmin || exit 0
    if [ -f `my_print_defaults --mysqld | grep -m 1 -oP "pid-file=\K.+$"` ]; then
      # If this fails, check debian.conf!
      mysqladmin --defaults-file=/etc/mysql/debian.cnf --local flush-error-log \
        flush-engine-log flush-general-log flush-slow-log
    fi
  endscript
}

The main part is that it invokes mysqladmin to reload all the log file.

The syntax isn't unintuitive, logrotate expects the script to set between postrotate and endscript. Same thing for prerotate except that one is rarely used.

To just run a single script or binary in there, you'd write:

postrotate
  /usr/local/bin/myscript
endscript

There is an extra option called sharedscripts that changes the behavior of prerotate and postrotate so that they only run once when the log entry has multiple files configured (using wildcards or providing multiple files fits this situation).

The normal behavior is to expand wildcards and run the scripts individually after every single file.

As seen in the MySQL rotation config above, their postrotate script notifies MySQL to reload all of the log files at once, it should only be ran once at the end, hence why they added sharedscripts. Most complex logrotate configs will have this option present.

Testing logrotate config files

On Debian systems, logrotate is invoked through cron in /etc/cron.daily. That may vary depending on the distribution, and you could completely disable log rotation by removing the relevant cron config.

It's possible to invoke logrotate manually with just:

logrotate /etc/logrotate.conf

Which is exactly what the cron script does.

You could run logrotate against a single config file and not the global /etc/logrotate.conf which includes all of the other files.

It's possible to invoke logrotate in "debug" mode and trigger a simulation to run without any modifications to the filesystem (the state file isn't modified either):

logrotate -d /etc/logrotate.d/something.conf

You can force a rotation even if it shouldn't take place yet (or because the configured size or another threshold hasn't been reached) using the -f option.

If you want more details but actually want the rotation to happen, you could use:

logrotate -vf /etc/logrotate.d/something.conf

Conclusion

We hope comprehensive log rotation will help maximize storage space effectiveness.

It's not uncommon for us to find forgotten log files with sizes above 100GB.

As mentioned previously, we didn't touch on log exportation or centralization strategies but these are the next step you may want to explore starting from here.