Investigating container security with auditd

Container Check

Talkin' Loud and Sayin' Nothing

All this log output isn't very useful if you've no idea what it means. Interpreting the audit.log file is clearly very important, and I won't claim it's always easy, but with some exposure you'll get the hang of it eventually.

Bear in mind that many of the logs are event based and the timestamps, which are written something like

msg=audit(1513507481.075:145)

are how the system clock records each event: a timestamp (the longer number) followed by a colon and then the ID applied to it (the shorter, trailing number). If a number of events happen at once (which you'd expect on modern systems with high-power threaded CPUs), many entries could have the same time and ID. Using this knowledge, you can check when and what happened, in what order.

If you've used Linux for a while, then fields such as pid (process ID) and ppid (parent process ID) should make some sense. A handy entry to look out for is auid , which is the "actual user ID" of the user that generated this activity (even if they became root as UID 0, the original user is logged).

With time, this becomes easier to fathom, so don't despair if it initially induces eye strain; just refine searches with queries per event across short time periods.

Immutability

One important thing to note is the "immutability" feature in auditd . As mentioned, auditd makes its configuration and logfiles accessible only to the root user for very good reasons. Imagine if an attacker got into a system and wanted to affect an application (e.g., sshd) to hide a second SSH port for back door logins. If they knew auditd was dutifully logging all their actions, they would clearly want to disable some, if not all, of the SSH ruleset to hide their actions.

Fear not. Simply append this tiny additional config parameter to the very end of your rules.d/audit.rules file:

-e 2

The result of adding this line to your config is that the only way to alter the auditd rules on a running system after it has been booted up with that setting in place is to rewrite the rules in the config file (i.e., edit, then rerun augenrules, and perform another reboot). As a result, of course, you need to pay much more attention to rebooting systems and treat random reboots as suspicious activity, which you should be doing anyway.

One gotcha is being mindful of the fact that the system is using the immutability flag in its configuration.

You can tell if immutability is enabled by running the --load option with augenrules:

$ augenrules --load
/sbin/augenrules: No change

In this case, the audit system is in immutable mode, so no rule changes are allowed.

If You Strike Me Down Now

A plethora of auditd options are available, and I've only looked at catching one binary so far; however, the sheer number of other rules (watch rules, control rules, and syscall rules) are a little mind-blowing. Ultimately, this is what makes auditd so powerful: its ability to capture anything and everything going on within your systems. The following are samples from the bottom of the auditctl man page (note the -a for the syscall rules)

To watch a file for changes (two ways):

auditctl -w /etc/shadow -p wa
auditctl -a always,exit -F path=/etc/shadow -F perm=wa

To watch a directory recursively for changes (two ways):

auditctl -w /etc/ -p wa
auditctl -a always,exit -F dir=/etc/ -F perm=wa

To see if an admin is accessing other users' files:

auditctl -a always,exit -F dir=/home/ -F uid=0 -C auid!=obj_uid

I'd suggest using your favorite online hunter-gatherer engine for more information and example rules.

If you're interested in threat modeling, then the powerful auditd also provides a tool called autrace, which you can point at specific binaries and glean a whole host (pun intended) of useful logging data. A simple example command is: