Keeping container updates under control

Hazardous Goods

Handling Updates

The strategies and theories described here do help you update your containers in such a way that end users experience no downtime. However, some passionate admins rightly complain that these tips do not answer the question of the technical procedure for updating software in containers in the broadest sense.

On the other hand, some very concrete questions arise: How do you replace a container in a pod on a network of active containers? Where do you get the updated image, and what do you need to watch out for? The remainder of this article therefore summarizes the manual procedure that dominates the container update scene. For easier understanding, you ideally need to think in terms of several process steps.

Something Is Wrong

As with conventional systems, the first step in handling security problems with containers is discovering that something is wrong in the first place. Because the container is built such that it forms a logical unit, the security checks that regularly scan "normal" systems for known security vulnerabilities simply do not take place here.

How admins usually build Docker or Podman containers also plays a role. If you regularly create your own containers, you are likely to do so on the basis on an image (e.g., Alpine Linux, Ubuntu, Debian) (Figure 1). If a library or program with a vulnerability crops up in the images, the next version of the container will probably contain the fix because a repaired package is available on the Debian servers and will then be included in the new package. Formal announcements (e.g., those from the security team announcing that a gap has been plugged) are not yet available for container images – although I will discuss a little helper named Clair in a moment.

Figure 1: Most Docker images used by application developers are based on the base images provided by vendors, such as Canonical's image for Ubuntu.

If you stray from the path of using prebuilt base images and instead build the complete image yourself, things get even more complicated. In such situations, you have to rely on manual work. One method could be to use tools such as apt-listchanges to monitor the changes regularly in the packages that the respective image contains. If the security string appears, it could indicate that an update is on the cards. However, solutions like this are not available off the shelf, and companies that use a CI/CD tool chain to build their own images need to develop this aspect of the build process themselves.

Clair to the Rescue

If you build your own Docker images so as to preserve the original layers of a base image, a small utility named Clair can come to your rescue (Figure 2). Clair analyzes container images on the basis of their metadata and compares the images with a list of published security issues. If it finds vulnerabilities, it raises an alert and prompts you to update your base images (Figure 3).

Figure 2: Automated monitoring of security vulnerabilities in container images can be handled by Clair, but only for the static layers of the respective image. © Red Hat

Figure 3: If Clair detects a problem with an image, it warns you and prompts you to install the new image. However, a rollout of the respective pods is required to follow up. © Red Hat

On the one hand, this process requires a CI/CD pipeline that can locally construct a new image with a new base image and the local changes. On the other hand, tools like Clair do not discover security issues in the locally modified parts. That said, an approach that uses Clair is still far more convenient than staging the handling of the entire gamut of security problems in your own infrastructure. A general tip at this point is to let the community and the manufacturers handle the topic of security, as you would in a standard environment.

Clair, by the way, belongs to Quay, which Red Hat acquired a few years ago. Clair is therefore integrated into OpenShift: For example, the images of active pods can be checked directly from the OpenShift graphical user interface. Again, Clair does not examine the contents of the container – only the layers of the image – for known security problems (Figure 4).

Figure 4: Clair's capabilities can be used in OpenShift both in the graphical user interface and from the command line.

For the sake of completeness, I should also mention that most manufacturers of external software also rely on base images from the major distributors for their Docker containers (Figure 5). For example, the container that MariaDB delivers with the database is based on the Ubuntu Focal image provided by Canonical. For use in Clair, this proves to be a great advantage, because it means that Clair can at least check those parts of the MariaDB image kept from the original Ubuntu image.