Critical Support

Even the best sys admins can feel overwhelmed with the logfiles generated by a single production server. The problem is especially severe if you are running multiple applications on one machine. Meet logwatch, a "customizable log analysis system [that] parses through your system's logs and creates a report analyzing areas that you specify" [1]. In this article, I look at this very popular tool, which I suspect many of you already use, and explore some of the lesser known options.

Lifesaver

A server that writes to many logs presents a couple of conspicuous challenges. The first challenge is physical limitation: Do you have enough spare space on your /var/log disk partition? If a server performs several roles, you might have firewall logs, user login logs, and HTTP logs – all in addition to the multitude of mandatory system logs. Before you know it, you might have dozens of applications vying for access to your overburdened /var/log directory.

The second issue is how to navigate the seemingly infinite screeds of recorded data successfully. It's an understatement to say the potentially critical information is surrounded by 10 times its volume of relatively useless noise. Faced with such a volume of data, it is imperative that you be able to zero in on the information you really need.

Timing is especially important when you are presented with a business-critical server issue, such as an attack. You need to know in advance the variety of logging formats employed by different packages, certain idiosyncrasies attributed to quirky applications, and whether you need to enable extra detail with debugging mode. All of these facets are essential if you want to pinpoint that elusive empirical evidence required to identify how an attacker has gained access to one or more of your servers. Empowered with that critical information, you can plug the hole through which the attackers gained access, and you can then use the log evidence to report the culprits to the authorities.

Basics

Straight out of the box, the excellent logwatch can sift through your seemingly endless logfiles and dig out the information you request, dutifully generating a well-considered, nicely formatted report, having sorted the wheat from the chaff.

In this article, I'll be using a Debian-based server for my examples, and I simply run the following command to install the package from the repositories:

# apt-get install logwatch
The following NEW packages will be installed:
libdate-manip-perl libyaml-syck-perl logwatch

As a result, about 12.5MB of new software is installed.

Logwatch gets configuration details several ways:

• from script or command-line arguments,
• /etc/logwatch/conf/ (for your system)
• /usr/share/logwatch/dist.conf/ (for your operating system)
• /usr/share/logwatch/default.conf/ (general defaults)

You can customize logwatch by editing the files in /etc/logwatch/conf/. You might want to create a file with only the modified variables (and include newly added scripts, if needed), or you might copy a config file from /usr/share/logwatch/default.conf to its namesake location in /etc/logwatch/conf and change what is needed in there. The comprehensive Debian HOWTO can be found at:

/usr/share/doc/logwatch/HOWTO-Customize-LogWatch.gz

(see the Documentation section).

Precedence of settings is from top (command line) to bottom (general defaults) in the list. If you see a config option with the format --<option>, you can assume it is used at run time; otherwise, you can assume the options are in the config files.

So powerful is logwatch that you might want to make just one change to the standard installation so that the root user does not receive a lot of email. To make this change, go to /usr/share/logwatch/default.conf/logwatch.conf. This config file presents a number of useful, configurable options that help, along with some other files I explore later, to control logwatch. In the MailTo = root line, you can replace root with a full email address. You might also want to change the Real Name option to something like Logwatch For Mail Server #1. A little farther down logwatch.conf is the mailer = sendmail -t entry, which I discuss in the "Mailer" box.

/usr/sbin/sendmail -> /etc/alternatives/mta

Reporting

If you have a Syslog server that pulls logs from multiple devices (e.g., servers, routers, switches) to a centralized locations, you might be pleasantly surprised to discover that the flexible logwatch can cope with the complexities presented when scanning logfiles from multiple hosts. For example, you can use --multiemail at run time to set up an email address for each host processed.

If the mail server admin doesn't need or want to know about a router's logwatch report, you can split log reporting recipients very simply using this option. If you want the reporting for many different hosts (i.e., networked devices) to be in one email report, then the following change in your config file will split up the entries by hostname:

SplitHosts = yes

In this case, do not limit hosts with HostLimit = no; otherwise, only the server's logs will be processed. Don't forget, of course, to keep email recipients to just you by selecting MultiEmail = no inside the config file (rather than the use --multiemail at run time as mentioned previously). If you want to send a report to multiple people, it's just a case of separating email addresses with a space in the config file.

You can also change where logwatch drops its temporary files, which might be important on servers with a number of applications installed, so that logwatch is therefore dealing with many different log files. You can change where temporary files are dropped by altering the TmpDir = /var/cache/logwatch variable. Also, if you are used to logging to a nonstandard directory (e.g., not /var/log), change the LogDir or add a new one in logwatch.conf. If you want to include archived logfiles (e.g., messages.1.gz or syslog.6.gz) in addition to the logging directory, you use the following option (which is No by default):

Archives = Yes

By default you are set up to check a date range of only yesterday; however, this is configurable to today or all:

Range = All

I'd suggest experimenting until you're happy with the level of output.

Incidentally, if you want, you can also output the report to the console with --output stdout at the command line or Output = stdout in logwatch.conf. This might be useful if you need the salient information quickly and you're in a closed environment where SMTP is not available.