« Previous 1 2 3 4 Next »
Investigating container security with auditd
Container Check
Your Point Is?
To whet your appetite further, I'll now get to the good stuff and cover a useful Docker-orientated auditing rule. Inside your rules config file (this is the last time I'll remind you that it's rules/auditd.rules
and not /etc/audit/audit.rules
), you can add a variety of rules along with what they monitor.
With auditd
, you have three distinct types of rules, all of which are configured in rules.d/audit.rules
. They comprise "control rules," which do things like set buffer sizes and flush currently running rulesets to keep things ticking over sensibly, avoid duplication, and so on; "watch rules," which are focused on a specific file's activity; and "syscall rules," which catch and log kernel system activity around any matching task, exit, user, and exclude lists (essentially filters).
The syntax for watch rules is broken down as follows,
-w <path-to-file> -p <permissions> -k <keyname>
where <permissions>
are:
r
– read accessw
– write accessx
– execute accessa
– change in a file or directory attribute
The syntax for syscall rules is:
-a <action>,<list> -S <syscall> -F <field>=<value> -k <keyname>
The <action>
is either always
or never
, and <list>
is a kernel rule-matching filter (i.e., task
, exit
, user
, exclude
); <sys_call>
is the name or number of a system call. You can see a list of <field>
types and <value>
s on the auditctl
man page.
Watching the Watcher
Using the layout above for watch rules, I'll set up a command that keeps a close eye on a Docker daemon binary. Thankfully it's not as tricky as you might think when you know how.
If you prefer that a rule does not persist after a reboot, you use the auditctl
command, which I might do if I were testing in a sandbox (or if my /var/log/audit/audit.log
logfile is really noisy and I need a quieter logfile to grep
). Generally, I find building up the rules.d/audit.rules
file section by section is fine, as long as I remember to run augenrules
afterward to check for errors (e.g., watch rules pointing at non-existent files).
The auditctl
approach is as simple as entering:
$ auditctl -w /usr/bin/docker -p rwxa -k docker-daemon
Briefly, this simple rule takes the path of the Docker binary; adds a watch for events relating to read, write, execute, and attribute changes; and writes to the trusty logfile, appending a docker-daemon
label (the <keyname>
in the command template) at the end of each line to help with searches.
To check which rules are currently applied on your system enter:
$ auditctl -l
The output will look just like the command minus the auditctl
at the start:
chrisbinnie ~ # auditctl -l -w /usr/bin/docker -p rwxa -k docker-daemon
In the /var/log/audit/audit.log
logfile, you can now see that your faithful Docker daemon is being monitored closely. To give yourself something to look for (more on searching effectively in a moment), run the command:
$ docker pull chrisbinnie/super
(If you're container crazy, check out the article "Troubleshooting Kubernetes and Docker with a SuperContainer" [1] to see why I pulled the super
image.)
Now when you run the Docker pull
command to grep
for the latest command, you see
$ cat /var/log/audit/audit.log | grep -i christype=EXECVE msg=audit(1513507468.995:124): argc=4 a0="/bin/sh" a1="/usr/bin/docker" a2="pull" a3="chrisbinnie/super"
which I've abbreviated for ease of reading.
That log entry hopefully makes some sense. Although interpreting the above is logical enough, the correct way to search is with the ausearch
command, which the considerate auditd
makes available as part of its bundled tools.
Cast your mind back to the watch rule's syntax with a -k <keyname>
(label) added to the end of the rule (in my example, it was docker-daemon
). Using the ausearch
tool, you can search for that label or keyname to retrieve the logs for ALL of the activity that the Docker binary has generated. As warned, these logs can be noisy, so the output in Listing 3 has been heavily abbreviated. For those familiar with SELinux, you might be able to spot some recognizable information.
Listing 3
ausearch by Label
$ ausearch -k docker-daemon type=SYSCALL msg=audit(1513507481.041:143): arch=c000003e syscall=59 success=yes exit=0 a0=e5c9f0 a1=e5a110 a2=e5c660 a3=7ffee7cf14b0 items=3 ppid=1140 pid=1541 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="docker" exe="/usr/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="docker-daemon"
This apparently simple search tool is actually very sophisticated. You can put it through its paces for a specific process ID (PID) (Listing 4).
Listing 4
ausearch for PID
$ ausearch -p 1431 ---- time->Nov 11 11:11:11 type=VIRT_CONTROL msg=audit(1513507481.075:145): pid=1431 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_runtime_t:s0 msg='user=root auid=0 exe=? reason=api op=create vm=? vm-pid=? hostname=? exe="/usr/bin/dockerd-current" hostname=? addr=? terminal=? res=success'
The ausearch
command has a heap of options, such as searching on event IDs, user groups, syscall names, SELinux context, and words and strings that exactly match, to name a few. The man page
$ man ausearch
helps out if you want to explore further.
I would be remiss not to mention an accompanying tool that sits alongside ausearch
very nicely called aureport
. The man page describes it as having the ability to summarize auditd
system logfiles into reports as follows:
The reports have a column label at the top to help with interpretation of the various fields. Except for the main summary report, all reports have the audit event number. You can subsequently lookup the full event with ausearch -a event number. You may need to specify start & stop times if you get multiple hits. The reports produced by aureport can be used as building blocks for more complicated analysis.
If you think this will be useful, this example checks for login activity:
$ aureport -au --summary Authentication Summary Report ============================= total acct ============================= 366 somedude 21 chrisbinnie 16 root
For a count of executable events with ausearch
, the command output (abbreviated) is:
$ ausearch --start today --raw | aureport -x --summary Executable Summary Report ================================= total file ================================= 1223 /usr/bin/dpkg 591 /bin/ls 118 /bin/cp
As the man page stated about using start
and stop
times, be warned that this is very CPU-heavy and dependent on the time range you select.
Boot Up the Behind
Another important component to think carefully about is what happens when the newer init system, systemd, starts up a modern day Linux after a reboot. The command
$ systemd-analyze Startup finished in 1.026s (kernel) + 5.925s (userspace) = 6.952s
shows you the slowest services to load when you boot up your system, so you can name and shame those services (and then fine tune them afterward). Just under six seconds of userland applications are slowing down the system. The next command shows the top five worst offenders:
$ systemd-analyze blame | head -5 2.084s fail2ban.service 1.958s cloud-init.service 1.851s cloud-init-local.service 1.597s apache2.service 1.134s postfix.service
As you can see, during the last reboot, the fail2ban
rules took a whopping two seconds. Be aware these values change with each boot depending on a number of events, so don't focus entirely on these metrics, just the slower packages. Remember that systemd can also load its services using parallelism.
What's this got to do with auditing, you might ask? Quite a lot, actually.
It's very important to log services during the boot process with auditd and not just let those services do whatever they like (e.g., think about trojans, keyloggers, and boot-level malware) without generating any logging that can then be referenced later.
Out of the box, auditd needs a little help. I suspect this might not be the default (on Red Hat machines, anyway) because of performance-related concerns. If you have a multitude of badly organized rules, you can incur a slight kernel performance hit as each bit of activity is checked then logged before being executed. You should do more research if you're using a shipload of rules in a production environment.
Assuming you're using the GRUB bootloader, you can append a simple audit=1
to the kernel
or linux
line within your GRUB config, depending on your Linux flavor.
In Debian, I add the line
GRUB_CMDLINE_LINUX="audit=1"
to the /etc/default/grub
file, which then is added to /boot/grub/grub.cfg
after the update command:
$ grub-update
This is slightly different on Red Hat derivatives (which, strictly speaking, create a new config file; potentially, there might also be an update
equivalent that you can use instead).
$ grub2-mkconfig > /boot/grub2/grub.cfg
As you can probably guess, it's now just a case of rebooting the system and looking at the start of the logfile to see which services are now also to be logged to the auditd
logfile /var/log/audit/audit.log
, as well.
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)