Where Did They Go?

The filesystem file and directory structure under the directory /etc/audit is shown in Listing 2. Remember, you make changes in the rules.d/ directory, which is then written to the audit.rules file after regenerating rules with augenrules .

Listing 2: /etc/audit Structure

chrisbinnie ~ # cd /etc/audit/
chrisbinnie audit # ls
auditd.conf  audit.rules  rules.d
 
chrisbinnie audit # ls -al
total 20
drwxr-x---  3 root root 4096 Dec 16 12:34 .
drwxr-xr-x 95 root root 4096 Dec 16 12:34 ..
-rw-r-----  1 root root  701 Dec  9  2014 auditd.conf
-rw-r-----  1 root root  373 Dec  9  2014 audit.rules
drwxr-x---  2 root root 4096 Dec 16 12:34 rules.d
 
chrisbinnie audit # ls -al rules.d/
total 12
drwxr-x--- 2 root root 4096 Dec 16 12:34 .
drwxr-x--- 3 root root 4096 Dec 16 12:34 ..
-rw-r----- 1 root root  373 Dec  9  2014 audit.rules

Apologies that I’ve been a little “now hit the Enter key” with the directory listings displayed in Listing 2. It’s purely to emphasize that the changes you make inside the rules/auditd.rules file then appear magically inside the file audit.rules in the directory /etc/auditd after the execution of the augenrules command, which I will look at now.

As mentioned, after you make a change (assuming “immutability” isn’t enforced, which I’ll cover a little later), you follow a procedure each time you update your rules. The file /sbin/augenrules is used to merge (concatenate, really, in a “sorted” order) all the rules files that are present in the rules.d directory. The rationale is that you might have several files in that directory grouped into rules by application. In this way, it’s much easier to administer an organized set of rules when you potentially can have a heap of configured rules. The mighty auditd offers some examples to put inside this directory:

$ ls /usr/share/doc/auditd/examples
auditd.cron  capp.rules.gz  lspp.rules.gz  nispom.rules.gz  stig.rules.gz

I will leave you to read the comments in these files if you want to pick up some tips and sample rules to add to your own monitoring configuration. Containerized environments or not, you can definitely make good use of monitoring (e.g., user creation and network event rules, among a zillion other things).

Back to the matter in hand, however: The command to run after each rule change in rules.d/audit.rules is:

$ augenrules --load

Running --check instead of --load will tell you whether the trimmed-down, concatenated /etc/audit/audit.rules file needs updating because it differs from the rules.d/audit.rules file.

Your Point Is?

To whet your appetite further, I’ll now get to the good stuff and cover a useful Docker-orientated auditing rule. Inside your rules config file (this is the last time I’ll remind you that it’s rules/auditd.rules and not /etc/audit/audit.rules ), you can add a variety of rules along with what they monitor.

With auditd , you have three distinct types of rules, all of which are configured in rules.d/audit.rules . They comprise “control rules,” which do things like set buffer sizes and flush currently running rulesets to keep things ticking over sensibly, avoid duplication, and so on; “watch rules,” which are focused on a specific file’s activity; and “syscall rules,” which catch and log kernel system activity around any matching task, exit, user, and exclude lists (essentially filters).

The syntax for watch rules is broken down as follows,

-w <path-to-file> -p <permissions> -k <keyname>

where <permissions> are:

• r – read access

• w – write access

• x – execute access

• a – change in a file or directory attribute

The syntax for syscall rules is:

-a <action>,<list> -S <syscall> -F <field>=<value> -k <keyname>

The <action> is either always or never , and < list > is a kernel rule-matching filter (i.e., task , exit , user , exclude ); <sys_call> is the name or number of a system call. You can see a list of <field> types and <value> s on the auditctl  man page.

Watching The Watcher

Using the watch rules layout above, I'll set up a command that keeps a close eye on a Docker daemon binary. Thankfully, it’s not as tricky as you might think when you know how.

If you prefer that a rule does not persist after a reboot, you use the auditctl command, which I might do If I were testing in a sandbox (or if my /var/log/audit/audit.log logfile is really noisy and I need a quieter logfile to grep ). Generally, I find building up the rules.d/audit.rules file section by section is fine, as long as I remember to run augenrules afterward to check for errors (e.g., watch rules pointing at non-existent files).

The auditctl approach is as simple as entering:

$ auditctl -w /usr/bin/docker -p rwxa -k docker-daemon

Briefly, this simple rule takes the path of the Docker binary; adds a watch for events relating to read, write, execute, and attribute changes; and writes to the trusty logfile, appending a docker-daemon label (the < keyname > in the command template) at the end of each line to help with searches.

To check which rules are currently applied on your system, enter:

$ auditctl -l

The output will look just like the command minus the auditctl at the start:

chrisbinnie ~ # auditctl -l
-w /usr/bin/docker -p rwxa -k docker-daemon

In the/var/log/audit/audit.log logfile, you can now see that your faithful Docker daemon is being monitored closely. To give yourself something to look for (more on searching effectively in a moment), run the command:

$ docker pull chrisbinnie/super

(If you’re container crazy, check out the article “Troubleshooting Kubernetes and Docker with a SuperContainer” to see why I pulled the super image.)

Now when you run the Docker pull command to grep for the latest command, you see

$ cat /var/log/audit/audit.log | grep -i chris
type=EXECVE msg=audit(1513507468.995:124): \
  argc=4 a0="/bin/sh" a1="/usr/bin/docker" \
  a2="pull" a3="chrisbinnie/super"

which I’ve abbreviated for ease of reading.

That log entry hopefully makes some sense. Although interpreting the above is logical enough, the correct way to search is with the ausearch command, which the considerate auditd makes available as part of its bundled tools.

Cast your mind back to the watch rule’s syntax with a -k < keyname > (label) added to the end of the rule (in my example, it was docker-daemon ). Using the ausearch tool, you can search for that label or keyname to retrieve the logs for ALL of the activity that the Docker binary has generated. As warned, these logs can be noisy, so the output in Listing 3 has been heavily abbreviated. For those familiar with SELinux, you might be able to spot some recognizable information.

Listing 3: ausearch by Label

$ ausearch -k docker-daemon
type=SYSCALL msg=audit(1513507481.041:143): \
    arch=c000003e syscall=59 success=yes exit=0 \
    a0=e5c9f0 a1=e5a110 a2=e5c660 a3=7ffee7cf14b0 \
    items=3 ppid=1140 pid=1541 auid=0 uid=0 gid=0 \
    euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 \
    tty=pts0 ses=1 comm="docker" exe="/usr/bin/bash" \
    subj=unconfined_u:unconfined_r:unconfined_t:\
    s0-s0:c0.c1023 key="docker-daemon"

This apparently simple search tool is actually very sophisticated. You can put it through its paces for a specific PID (process ID) (Listing 4).

Listing 4: ausearch for PID

$ ausearch -p 1431
----
time->Nov 11 11:11:11
type=VIRT_CONTROL msg=audit(1513507481.075:145): \
  pid=1431 uid=0 auid=4294967295 ses=4294967295 \
  subj=system_u:system_r:container_runtime_t:s0 \
  msg='user=root auid=0 exe=? reason=api op=create \
       vm=? vm-pid=? hostname=? \
       exe="/usr/bin/dockerd-current" hostname=? \
       addr=? terminal=? res=success'

The ausearch  command has a heap of options, such as searching on event IDs, user groups, syscall names, SELinux context, and words and strings that exactly match, to name but a few. The man page

$ man ausearch

helps out if you want to explore it further.

I would be remiss not to mention an accompanying tool that sits alongside ausearch very nicely called aureport . The man page describes it as having the ability to summarize auditd system logfiles into reports as follows:

The reports have a column label at the top to help with interpretation of the various fields. Except for the main summary report, all reports have the audit event number. You can subsequently lookup the full event with ausearch -a event number. You may need to specify start & stop times if you get multiple hits. The reports produced by aureport can be used as building blocks for more complicated analysis.

If you think this will be useful, this example checks for login activity:

$ aureport -au --summary
Authentication Summary Report
=============================
total  acct
=============================
366  somedude
21  chrisbinnie
16  root

For a count of executable events with ausearch , the command output (abbreviated) is:

$ ausearch --start today --raw | aureport -x --summary
Executable Summary Report
=================================
total  file
=================================
1223  /usr/bin/dpkg
591  /bin/ls
118  /bin/cp

As the man page stated about using start and stop times, be warned that this is very CPU-heavy and dependent on the time range you select.