Herding Containers

Deployment Pipelines

Kubernetes is extremely well suited to deployment pipelines. In addition to the containers, Dockerfiles can easily be audited and transported. On the basis of the audited Dockerfiles, you can build containers for testing and production from the files in safety-critical areas without the influence of the developers. A Git repository makes the audit more easily traceable; the cluster operator can then fully automate the entire process. It is basically also possible to let experienced developers roll out images to simplify the infrastructure in DevOps teams.

CoreOS

If you follow the container paradigm, the only role of an operating system is to start a container safely and reliably and to guarantee its life cycle. This makes the few programs of even the leaner Linux distributions redundant. CoreOS [28] takes this idea and reverts to the core of an operating system. The /usr partition weighs in at just 1GB, of which CoreOS is less than 700MB. Kubernetes itself requires more than double the amount of disk space at 1.5GB.

The small size of CoreOS makes it possible to use a second disk as an alternative for an update. The distribution comes with two /usr partitions (USR-A and USR-B). During updates, they take turns and are mounted as read-only. If something goes wrong, you can at any time restore the old system as a fallback. Such immutable operating systems were previously only known from little-used embedded Linux distributions, which are used in the security field. They eliminate the need for package management – and actually also for configuration management – at the same time.

To meet the mission statement of CoreOS, you can enable a Trusted Platform Module [29] for the hardware and use rkt [30] as a container runtime alternative for Docker that only executes signed containers.

New Build

One gap has yet to be closed: Most images (depending on the counting method, two thirds or even five sixths of the images on Docker Hub) contain vulnerabilities, one third of which are considered critical. In secured environments, you should therefore only install self-made images FROM scratch from their own registries [31]. It also helps to renew containers regularly – that is, to use them for a maximum of one week or even rebuild them every day on an isolated build system.

Configurations can be grouped in config_maps [32] or accommodated in the environment variables. Passwords can be stored in secrets [33], which appear to the containers to be RAM disks and are not stored on the node's disk. An automatic build pipeline can handle the required steps; Jenkins can be easily extended with appropriate plugins for this purpose.

If you prefer to avoid this overhead, you can use Clair [34], a docker registry developed by CoreOS. It scans the included containers for known common vulnerabilities and exposures (CVEs) and raises the alarm if it finds an insecure library in a container. As a hosted alternative, there is also Quay.io, which is part of the CoreOS project.

By the way, you should remember to dimension the registry and a repository server for the legacy packages on a big enough scale. The reason being that a mass roll-out of containers after patching a low-level library usually triggers a full roll-out of all images. The lean technology of rolling out only deltas cannot be used in such a case.