Pedigree

Containers have changed the way software releases occur in modern DevOps environments. The portability they offer is exceptional, and the consistency (being able to package units of functionality together) between development and production environments is invaluable for operational predictability and, ultimately, uptime.

A serious issue, however, that came to the fore some time after the initial flurry of adoption, when enterprises started deploying workloads with value, was how to secure them effectively. You can find more information about the minefield around the security model from one of my earlier articles [1], although it is admittedly a little dated because of the rate of innovation in this space.

In this article, I walk you through automating the analysis of container images and have a look at how easily a trojan downloaded from a popular registry (e.g., Docker Hub) might be installed into a public container image.

Four Is the Magic Number

Four key container security areas jump to my mind:

1. The usual package concerns. In other words, you have your Common Vulnerabilities and Exploits (CVEs) [2] that, if you're lucky, are monitored by your operating system (OS) and updated so you can use a package manager to install a newer package with a fix for the security hole. Then, you can rebuild your container.
2. Misbehaving containers. Once your container is running (having been built from an image you trust), you want to keep an eye on its behavior. Misbehaving containers usually mean one of two things: Either you've misconfigured something and broken a service, or someone else has broken into your container and is about to start breaking things. I call this "anomalous behavior." Red flags could include network ports being opened unexpectedly, disk volumes being mounted without previously having been present, and processes spawning that weren't there before. Thanks to the declarative nature of Dockerfiles (i.e., they are statements of exactly what you expect to be inside the resulting containers), it can be relatively easy to know which anomalies are present if they ever do appear.
3. Orchestrators (e.g., Kubernetes or OpenShift) that might be configured to allow access to the host OS. Should a container have root user privileges, and should it be able to adjust host-level settings? The answer should almost definitely be no. With orchestrator features such as Security Context Constraints, Pod Security Policies, and network isolation, you can isolate containers.
4. Untuned host OSs. One area that's overlooked far too often these days is tuning the OS on the container host so the OS will play nicely with your containers and orchestrator. When it comes to fine-tuning your OS, you have the choice of a mountain of changes – from filesystems and the network stack to performance tweaks at the kernel level. I'll leave you to investigate these options further.

With that reassuring discussion about how and where to start mitigating some security issues, I'll now look at using the excellent Docker Scan tool [3], which primarily deals with container images.

Checkers

As you might have guessed from its name, you'll look at Docker as the run time. Docker Scan classes itself, according to its GitHub page, as belonging to "Docker analysis & hacking tools." A YouTube video about the tool [4] jumps straight to the point by admitting that it dissects Docker images and abuses an image registry!

The slides, presented at RootedCON 2017, paint a clear picture, telling you not to trust image tags; instead, you should use digests in the FROM line within your Dockerfile and set up lots of tests during build time so as not to fall foul of nasty container image content. You'll find much more advice, so I'd recommend you watch the video and note some of the attack vectors when it comes to container images.

Invasive Surgery

In its documentation, Docker Scan uses one of the most popular Docker images currently in use worldwide: the performant web server, Nginx. In my examples, I'll use that image, too, because it's slick and lightweight.

Before I go much further, look at Figure 1 for a quick reminder of what an image looks like to Docker, which talks to the Docker API to run its commands. It is the relatively old but suitably simple chrisbinnie/super Dockerfile that I used for a project described in a previous article about SuperContainers [5]. The basic elements, which are present in most Dockerfiles, include the OS, the packages involved, and the commands to run.

Figure 1: This example Dockerfile shows how a container image is layered at build time.

To pull down the nginx image enter:

$ docker pull nginx

Next, you can use the history command to see what the Docker image comprises (Figure 2). For example, the command for nginx:latest is:

$ docker history nginx:latest

Figure 2: The innards of the very popular Nginx container image.

If you look at the Docker Hub page for Nginx and follow links to the Dockerfiles of the respective image versions, you should be able to figure out which Dockerfile built the image you're looking at. I'm guessing it is the mainline Debian stretch-slim [6], because it seems to fit if you work from the bottom of the Dockerfile backward.

In Figure 2 you can see the different layers and their sizes that were added by the Dockerfile that helped build this image. I find the history command can be invaluable for troubleshooting.

The --no-trunc command option will show an abbreviated output of the full commands used for each layer. This output can be sifted through to pinpoint more accurately where images suddenly bloat in size and where issues might creep in.