The Bookkeeper

Perhaps one of the most underreported yet regular tasks of a system administrator is to parse logfiles. Scrolling through logfiles isn't fun, but they are an indispensable resource for helping troubleshoot an issue. Seasoned admins will tell you that they almost always instinctively pull up the logs whenever they face any issues with their installations.

Linux, for its part, is ever vigilant and makes careful logs of everything that takes place within your system. Most logfiles under Linux reside under the /var/log/ directory. These are all system and service logs, which come in handy when troubleshooting system-wide issues. Different apps also write app-specific logs, which are kept in a configurable location usually under a user's home directory.

You don't necessarily need to use any special tools to read logfiles, since they are stored as plain text. Not only can you search the logfiles, you can also create scripts to traverse them and then perform any action based on the results. While you can usually read logfiles without leaving the command-line interface (CLI) (see the box titled "View Logfiles Using the CLI"), many distributions also ship with a graphical tool to help you with the task (Figure 1).

grep sshd /var/log/auth.log | less

tail /var/log/auth.log

tail -n 20 /var/log/auth.log

Figure 1: Gnome users can use the Logs utility gnome-system-log to search through the various logfiles using several useful filtering options.

Archived Logs

Looking through the /var/log/ directory, you may notice that some logfiles have a number suffix, such as auth.log.1, auth.log.2.gz, etc. These are rotated logfiles. Over a period of time, just about all logfiles grow too large and become difficult to read. When that happens, the original logfile is renamed, and a new logfile is started. Log rotation is the process that renames a current logfile, such as changing auth.log to auth.log.1, and sets up a new logfile, auth.log, for new log entries. Depending on the importance of the underlying tool or service, some logfiles will have older iterations with higher numbers, while others will have lower numbers. The system might also compress the old logfile before creating a new one to save space, which is especially true for logfiles that take up too much disk space.

Rotating logfiles is important for several reasons. First, you probably don't want older logfiles eating up too much of your disk space. Second, you don't want the logfiles to be extremely large, because that'll make it even more cumbersome when you need to analyze the data within them. Log rotation, like any system administration task, is an involved process. You'll have to decide which logfiles are to be rotated, when and how often, and whether the rotated logfiles need to be compressed, along with several other questions. The logrotate utility makes log rotation fairly straightforward by helping you automate the process. It offers enough dexterity, and you can easily configure the tool to rotate logs per your requirements, as I'll demonstrate in this tutorial.

Getting Started with logrotate

Chances are your favorite distribution already ships with logrotate right out of the box, but if it doesn't, you can easily pull it from the official repositories with a simple apt, yum, or dnf command. The logrotate utility, which is governed by the /etc/logrotate.conf file, is responsible for creating new logfiles, as well as renaming and compressing the old logfiles. Also of note is the /etc/logrotate.d directory, which contains configuration files for individual logfiles such as apt, dpkg, etc.

When logrotate runs, it reads its configuration files to decide where to find the logfiles that it needs to rotate, how often the files should be rotated, how many archived logs to keep, and other details. There are primarily two ways to write a logrotate script. You can either tweak its default configuration file /etc/logrotate.conf or create a separate configuration file for each app or service that logs data inside the /etc/logrotate.d directory. By default, the tool uses a combination of both of these methods. The /etc/logrotate.conf file typically looks like Listing 1.

# rotate logfiles weekly
weekly
# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) logfiles after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your logfiles compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d

In Listing 1, we've set several global options. Remember, however, that local definitions, which we'll get to in a bit, override global ones. The weekly option asks the tool to attempt to rotate the logs on a weekly basis. You can change the frequency by using either the daily or the monthly option. The rotate 4 option specifies that the tool should keep the last four rotated logs. When it's time to rotate the fifth one, logrotate will delete the oldest log and rename the others to maintain the sequential file names. The create keyword is used to create a new logfile while renaming the old one. The su root syslog specified at the top of the file determines the logfiles' permissions. Here, I tell logrotate to archive the logs using the specific user root and group syslog to prevent issues with permissions.

Besides these, there are a couple of commented out options as well. If enabled, dateext will use the current date as the old logfiles' extension when rotating files. Then there's the compress option, which asks the utility to compress old rotated logs. By default, it uses gzip, but you can also specify another compression utility. For instance,

compresscmd /bin/bzip2

asks logrotate to use the bzip2 compression utility and together with the option

compressext .bz2

appends the specified extension to the compressed files.

If you need to read a rotated logfile that's been compressed, you'll first have to uncompress it. This is rather inconvenient if you need to use rotated files regularly. In such a case, you can use the delaycompress option that asks the utility to not compress the rotated file until the next rotation cycle. This will ensure that the most recent rotated logfile is in an uncompressed state and can be used without extra effort.

At the end of Listing 1, the include option asks logrotate to pull in the specified file's contents, which in this case is the entire /etc/logrotate.d directory. The directory contains logrotate configuration files for various utilities; this is where you can add a configuration file for a service or tool per your requirements. As an example, take a look at the contents of the Uncomplicated Firewall (UFW) configuration file in Listing 2.

/var/log/ufw.log
{
  rotate 12
  monthly
  missingok
  notifempty
  compress
  delaycompress
  sharedscripts
  postrotate
    invoke-rc.d rsyslog rotate >/dev/null 2>&1 || true
  endscript
}

For the most part, Listing 2 uses the same options as the global configuration file, but it also introduces some new ones. For starters, instead of weekly rotations, I am doing them once every month for UFW. Secondly, I am preserving the logs for a year by keeping them for 12 rotations. As I mentioned earlier, in case both the global configuration file and the local configuration file mention different values for the same option, logrotate will follow the values set in the files under /etc/logrotate.d. This is why it'll stick to the rotation policy (monthly and rotate 12) set in this file instead of the ones set in the global file (weekly and rotate 4). The missingok option asks logrotate to continue working even if there is no logfile. The default option, nomissingok, asks the tool to throw an error if it doesn't find an existing logfile. The notifyempty option asks logrotate not to rotate a logfile if it is empty.

One important option is postrotate, which lists custom commands and scripts that will be run after the logs have been rotated. When used together with the sharedscripts option, logrotate will run only once when all the configuration files that write to the same log directory have been rotated. To force rsyslog to reopen the files it's using to write logs, use the command:

invoke-rc.d rsyslog rotate > /dev/null

This is required, because rsyslog will continue to write to the file that first called it, which has since been rotated. So this command ensures that rsyslog writes to the current logfile. You can also use postrotate to reload services, like the Apache web server:

postrotate  /bin/systemctl restart apache2.service > /dev/nullendscript

Here, I reload the Apache web server, after its logfiles have been rotated, to make sure it writes to the newly created logfiles. There's also the prerotate option, which will execute the mentioned commands/scripts before the log rotation process. When either the prerotate or postrotate option is triggered, it'll continue executing the commands until it runs into the endscript keyword.

I've only mentioned a handful of the most frequently used logrotate options. There are several others that you can read about on the logrotate's manpage [1].

Running logrotate

After you've defined or tweaked a logrotate configuration file, it's best to use the -d debug option to execute a dry run to see what the utility would do when it's actually executed (Figure 2).

Figure 2: The -d debug option spits out various useful information including when and how the logs are rotated.

While you can invoke logrotate manually, it's best to run it through cron. By default, the installation of logrotate creates a cron job file named /etc/cron.daily/logrotate. You can copy the logrotate cron job inside any other cron directory (there's /etc/cron.hourly, /etc/cron.weekly, or /etc/cron.monthly) to change the execution frequency. The /var/lib/logrotate/status file is updated whenever the logrotate cron job is executed. The status file shows the date and time when each of the logfiles was last rotated.

There is no disputing the importance of logs in the life of a system administrator. A utility like logrotate helps you take charge of the management of such a crucial resource. It's an elegant solution that offers lots of configurational flexibility while still being easy to comprehend and deploy.

The Author

Mayank Sharma is a technology writer and you can read his scribblings in various geeky magazines on both sides of the pond.