Log Shepherd

Collecting data is all the rage, but gaining meaningful insights from data is an art form. For example, if you want to find correlations between events in various logfiles, the Simple Event Correlator (SEC) is an obvious choice.

SEC handles a large chunk of the work of security information and event management (SIEM) software, which tries to make sense of a vast amount of data in (security) logs. Unlike, say, OSSIM (Open Source Security Information Management), SEC does not do this proactively; instead, it restricts itself to logfiles that come from other sources.

If you are looking forward to an installation marathon, you will be disappointed: The small tool is ultimately just a Perl script. The remaining files are documentation and various startup scripts that integrate SEC into various Linux distributions, FreeBSD, and Solaris. An installation program does not exist, so you need to copy the man page and binary manually to a suitable location. SEC was developed around 2000 by Risto Vaarandi, but the latest version is from January 2014.

The functional principle of SEC is to look for patterns in logfiles and, if found, initiate a predetermined action. The program can handle multiple logfiles simultaneously and perform actions that can depend on each other in many ways. SEC offers a wide range of preconfigured options. If that's not enough, you can still edit Perl code, which will probably cover most applications.

For the simplest case, SEC uses the Single rule type in which a single find of the pattern triggers an action. The syntax looks like this:

type=Single
ptype=RegExp
pattern=Failed password for root
desc=Matched: $0
action=logonly

The rule type in the first line is followed by the pattern type (here, regular expression), the search pattern, the variable definition for the pattern description, and finally the action to perform. To test the configuration, you only need to store it in a file, such as ssh.conf. For test purposes, it is good to know that the SEC processes text from standard input. You need to call the program like this:

sec -conf=ssh.conf -input=-

If you now enter arbitrary strings in a terminal window, nothing happens; but, if you enter Failed password for root, SEC acknowledges this by outputting Matched: Failed password for root . The logonly action tells SEC to the output to the logfile if one is set and to standard output otherwise (Figure 1).

Figure 1: In the simplest case, SEC matches a string and then performs an action.

To call SEC with a logfile, pass in the logfile's name as an argument for the input parameter. If you want to track multiple logfiles simultaneously, you can use multiple input options. SEC offers six debugging levels, which are enabled by the --debug switch. If everything works as desired, and you decide to run SEC in the background, the --detach command-line switch does the trick.

Although the pattern used above is a regular expression, it is only true if the string appears exactly as listed. Typically, you would use well-known wildcards to provide greater tolerance. Thus, the above pattern could look like this:

pattern=Failed password for (\w+)

Now SEC alerts for each failed SSH login, because the \W+ regex stands for one or more "word-characters" (i.e., numbers, letters, hyphens, underscores). The parentheses state a match group that can be used later (e.g., in desc=Matched: $1). In this construction, $0 stands for the complete match and $1 is the first match group; $2 would be the second match group, and so on.

In addition to the logonly action, SEC offers a number of other possibilities. Similar to logonly, write sends the output to a file of your choice, or - sends the output to stdout, the standard output channel. If you want to send email, for example, you can use shellcmd to run an arbitrary external program. spawn is similar, but the output of the called program is returned to SEC for further processing. A list of available actions can be seen in Table 1.

Up to this point, the functionality of SEC is not particularly impressive, especially considering the many similar programs that can monitor logfiles. However, SEC has a lot of other rule types that expand its functionality considerably (Table 2).

The Simple Event Correlator does justice to its name when it comes to the Pair rule, which ignores all further identical matches after the occurrence of an event and waits a certain amount of time for a second event. If it occurs, SEC triggers the configured action.

Additionally, the PairWithWindow rule type works almost like Pair but only executes one of two possible actions if both events occur in the time window. If only one event occurs, and not the other, SEC performs the second action. All other rule types are described in Table 2.

One particularly powerful feature for cross-referencing several logfiles is contexts, which are useful for exchanging data between multiple rules and matches.

Contexts are implemented as Perl hashes, which is why they can store most data types. Most of the rules have a condition called context, which is satisfied if the corresponding context exists. Similarly, you can create a context in the scope of an action:

action=create FOO_CONTEXT

For a time-limited context, you need to specify the time duration in seconds behind the context name.

Listing 1 shows a pair of rules, one of which generates a context, whereas the second checks for its existence.

type=Single
ptype=RegExp
pattern=bar
desc=$0
action=create FOO_CONTEXT
06
type=Single
ptype=RegExp
pattern=foo
context=FOO_CONTEXT
desc=$0
action=write - Context exists

Besides leveraging the use of contexts, you can also work with its contents.

For example, the add command in an action adds an event to a context as follows:

action=add MY_CONTEXT "ipmatch"

An alternative to add is the fill command, which does almost the same but empties the context first. You can output such an event once again with the report command, which transfers the contents of the event to a shell command.

As you can easily imagine, contexts are useful for all kinds of applications; for example, for switching to a different state  – in a style similar to SingleWithSuppress  – and handling future events differently. The spawn action is also useful for correlating multiple logs. When an event occurs, you can open one or several different logs and search them at the same time for interesting patterns.

Conclusions

SEC is a flexible and powerful tool for analyzing logfiles and finding correlations, and it does not have any major installation overhead. In simple cases, it can be used without extensive training.

SEC's vocabulary consists of rules and actions, which makes it suitable for a wide range of applications. If that's not enough, SEC also supports the use of short or long Perl functions, which ultimately can solve any problem.