Hazardous Goods

Correcting Images

The second step after discovering that a security problem exists is something I have already implicitly worked through in this article. The most important basic rules can be listed briefly: Security updates in containers can always be solved when you roll out a new version of the container instead of tinkering with the running container. For this to work, the architecture of a setup must be designed to be implicitly redundant from the start. Central tools such as databases must be clustered so that the failure of a container does not cause the entire environment to collapse.

However, how exactly you find updated images on the basis of which you then launch new containers is something that I cannot meaningfully describe in this article. The number of potential setups in the field that use and generate images is huge. In an article on containers and security [2], I looked in detail three years ago why it is not a good idea to source containers on the web, roll them out, and only briefly test whether they deliver the desired functionality. To this day, nothing has changed in this regard; what's more, anyone who seriously runs container workloads somehow has to get their containers out of a CI/CD pipeline, which is not a particularly convenient task to construct in such an environment.

Commercial Distributions Help

The manufacturers of the commercial Kubernetes distributions are aware of this problem and help you in various ways. If you use Red Hat's OpenShift or SUSE's Rancher, for example, you can source all the software required for a CI/CD tool chain in a version preconfigured by the manufacturer, which makes the setup far easier, especially because the manufacturers now also deliver their solutions with easy-to-understand, intuitive user interfaces.

The Update

Assume you have identified the MariaDB container from the example as a container with a security problem and now want to update it. You have already sourced and retrieved the new image, so the next step is concrete action. If you do not use a fleet manager like Kubernetes, the whole process is quickly completed: Stop the old pod, start the new pod, repeat until all the pods have been rotated – done.

However, setups with containers that are not governed by a fleet manager are very rare these days, so this example can hardly be considered true to life. Typically, you will instead be dealing with an arbitrary delivery form of Kubernetes, such as OpenShift, Rancher, or the vendor's Kubernetes distribution. At this point, a technique the Kubernetes developers refer to as a "rolling update" comes into play, and it is basically intended for just such scenarios.

Basically, you do not update the individual containers in a pod. Instead, you phase in new pods alongside the existing pods. If so desired, Kubernetes can handle inserting new pods with newer images and phasing out old pods automatically – a rolling update. Under the hood, it uses the Kubernetes controllers for replica sets to do its job. A Kubernetes tutorial [3] describes the technique in detail for each case. The advantage of this approach is that you don't have to worry about removing the obsolete containers. Instead, you can safely assume that the vulnerable software will disappear into a black hole.