Special Thanks: This article was made possible by support from Linux Professional Institute

Thanks to the unremitting, ever-present threat of a multitude of attacks to which a Linux system can be subjected, it’s critical to capture important changes and events made by users and processes on your running systems.

Highlighting such changes could potentially point toward something as innocuous as a simple misconfiguration but, equally, might proactively help stop an impending attack dead in its tracks. Additionally, having trustworthy, detailed logging data is exceptionally useful for post-event forensic analysis, especially when you are trying to discern how an attacker originally managed to compromise your system and get a foothold.

One such package I have been using recently on a large AWS server estate is called auditd . Its man page states: “auditd is the userspace component to the Linux Auditing System.”

One of the pages on the Linux Audit Documentation project GitHub site describes its (very old) original design as being based around the following aims:

The main goals were to provide system call auditing 1) with as low overhead as possible and 2) without duplicating functionality that is already provided by SELinux (and/or other security infrastructures).

For the uninitiated, “system calls,” which are more commonly referred to as “syscalls,”occur when processes ask the kernel for a hand with something. The syscalls man page reports, “The system call is the fundamental interface between an application and the Linux kernel.”

Simply think of every task, such as opening a socket for network communications, mounting a disk volume, or even creating a directory, as needing some form of assistance and therefore validation from a host’s kernel. In many cases, a handling program like the all-pervasive glibc (GNU C Library) will invoke the syscall directly and not the underlying application that is asking for help. If you’re keen to encourage narcolepsy, you can find a list of system calls with some more glibc wrapper detail thrown in for good measure on the syscalls man page mentioned above.

In this article, I’ll help you set up auditd and give you some food for thought in relation to what syscall information you might want to capture when it comes to containers.

Bring Out the DevOps

The advantages of carefully monitoring such key system changes are clear, but you might rightly ask: “Why is this even more important in a containerized environment?”

The reason is surprisingly simple. On a host running many containers (I’ll use the popular Docker as an example), all share the same Linux kernel. Without some jiggery-pokery, your superuser root inside one container or more is effectively the root user on your host, which, from a security perspective, is unwelcome for a myriad of reasons.

As a result, if a vulnerability is used to exploit a container to a high enough degree, then the other containers and potentially the host itself are at risk of being compromised. Such an event, wherein a bad actor (an attacker) breaks into the larger host system by way of a relatively minuscule container, is known as a “container escape.” The implications are varied, but a favorite is gaining access to the host’s filesystem, reading secrets (passwords), and being able to access anything visible to the host itself using its credentials. That might be far beyond the host’s local cluster or VLAN and into other trusting, geographically disparate environments.

Vendors such as Docker have made a number of welcome strides to mitigate such risks, but do not imagine for one moment that the issues around container security are their responsibility. The Docker Security page opens candidly with the following warning:

There are four major areas to consider when reviewing Docker security:  the intrinsic security of the kernel and its support for namespaces and cgroups;  the attack surface of the Docker daemon itself; loopholes in the container configuration profile, either by default, or when customized by users ;  the “hardening” security features of the kernel and how they interact with containers.

For these reasons, among others, tracking issues to a high degree of detail is critical to the uptime of your containerized environments, especially in an enterprise environment. You can lessen the possibility of a compromise in one environment being successful in another environment if the attack vector that was exploited is determined early enough using such information.

In simple terms, a container suddenly opening up a network port that it’s never used before or catching a misbehaving container trying to set the host’s clock to an arbitrary time would almost definitely not be welcome surprises.

Needless to say, if you like squinting at kernel logs or want to monitor users and applications on your systems to a really high level of detail, then the auditd package is for you. It is noisy in its output for good reason: detail.

The installation command on Debian derivatives such as Ubuntu (as per Debian 8 Jessie) is shown in Listing 1.

Listing 1: Installing auditd on Debian

$ apt install auditd
 
chrisbinnie ~ # apt install auditd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libauparse0
Suggested packages:
  audispd-plugins
The following NEW packages will be installed:
  auditd libauparse0
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 253 kB of archives.
After this operation, 833 kB of additional disk space will be used.

Notice that a dependency is listed when installing auditd on Debian derivatives. Somewhat surprisingly, on Red Hat derivatives (CentOS 7, anyway), I need to install the audit package and not the auditd package, as on Debian:

$ yum install audit

Don’t let this trip you up if it’s not already installed for some reason.

Audit This

The all-seeing, omnipotent auditd and supporting packages allow you to report on fine-grained system actions at varying levels of detail. With too many configuration options to cover in this article, bear in mind the following aspects if you want to try it out for yourself.

First, auditing takes place directly from the kernel and is saved, usually to /var/log/audit/audit.log , at least. That file is readable by the root superuser only and for very good reason: You don’t want anyone or anything tampering with this file if you are relying on it to examine your system forensically after either an attempted or successful compromise. For this reason, the auditd package also insists that permissions are set to root:root after rotating logs, … which leads me to the next aspect of auditd to be considered.

Professionally I deal with Docker and Kubernetes security. If you’ve used it before, you might be able to guess how often the Docker run time (the main Docker binary file running as a daemon) makes syscalls to the kernel. If you can’t imagine then let me tell you: It’s a lot!

With a few containers running on a host, your auditd logs will soon be creaking at the seams with pull , push , run , and kill commands if a human is interacting with the Docker API; however, if an orchestrator like Kubernetes or OpenShift is accessing that API, then the logs will be filling up even quicker. As a result, it’s important to forward the logs off-host to a syslog server or something similarly functional (e.g., Splunk in an enterprise environment) for security, in case an attacker manages to elevate to root and somehow affect your logging. Meanwhile, you are preserving resources because you only need to keep a few days of logs on a host.

Inside the main auditd configuration file (/etc/audit/auditd.conf ), you can change the maximum size of each logfile and how many logs you want to keep; it is unquestionably wise to set these to something sane as soon as you possibly can. Following each change, be sure to restart the daemon as root in systemd:

$ systemctl restart auditd

Also, you should make sure that the daemon is set to load after each reboot:

$ systemctl enable auditd

Note that to keep you on your toes, the daemon/service is named auditd and the package is called audit . I am always using the command

$ systemctl list-unit-files

to find systemd service names; if you forget and think you’re losing your marbles because your service isn’t starting or stopping, simply run this command and search for audit .