Bug Hunt

Docker must have passed the peak of its hype. New versions no longer trigger the same kind of hustle and bustle in the community and no longer provoke a flood of specialist articles, not least because a considerable amount of interest has shifted away from Docker in recent months toward solutions that use Docker as a basis for larger setups – such as Kubernetes.

Docker is now mainstream. Today it is no longer unusual to use the container solution where you might have relied on virtual machines (VMs) in the past. In this respect, Docker is only one of many virtualization solutions. However, the number of users who deal with Docker in the production environment is growing. As a result, an increasing number of developers and admins, who are unfamiliar with Docker, initially have problems with building and starting a Docker container.

In fact, many things can go wrong when building and operating a Docker container. If you approach a Docker problem with the mindset of an experienced kernel-based VM (KVM) admin, you will not get very far: Most things in Docker are fundamentally different from real VMs. Although you can log in to the latter and investigate a problem in the familiar shell environment, Docker offers its own interfaces for reading logfiles and displaying error messages, for example.

In this article, I describe in detail the options for debugging Docker and Docker containers. To start, I introduce various tips to make Docker containers debugging-friendly right from the start.

A Debugging Mindset

Building your own containers makes sense, because prebuilt containers from Docker Hub expose you to the risk of a black box model. On the outside, the container often does what it is supposed to do – what is inside the container and how it has been built, however, remain unclear. Compliance issues already preclude this approach in many companies. If you source your containers on the web, you will also have a hard time debugging.

However, it is not difficult to build Docker containers by starting with the manufacturer's distribution image, as discussed in a previous article on Docker and GitHub [1]. In the end, it takes little more than an error-free Dockerfile containing the central instructions. If you follow this path and build your own containers from the start, you can design them so that they are easy to debug from the beginning, if you follow several basic rules.

One Container per Application

First and foremost – and even experienced Docker administrators often disregard this logical recommendation: Put each application into its own Docker container instead of combining several applications into one large container.

The idea of using only one container for several applications probably still reflects the long-standing thinking of VM admins: KVMs produce a certain amount of overhead that multiplies when many small programs are spread over many individual VMs.

However, for containers, this factor is only of minor importance, because a container needs very little space on disk, and container images from vendors are so small that they hardly matter.

As compensation, accepting the disk space overhead offers considerable benefits in terms of debugging. For one thing, investigating a container in which only one application is running does not immediately affect your other applications. Moreover, a single container with one application is easier to build and replace than a conglomerate of several components.

Last but not least, Docker is by definition (because of the way Dockerfiles are constructed) predestined for one application per container. Because Docker containers usually do not have init system like systemd, the admin usually calls the program to be started directly in the Dockerfile, which receives virtual process ID (PID) 1 within the container; the container considers the program to be its own init process, so to speak.

Combining several services and programs into the container against your better judgement almost inevitably leads to problems. You then have to work with shell scripts in the background, which might achieve the desired effect but will definitely make debugging far more difficult.

Proper Container Startup

Launching Docker containers is a complex matter and of great importance with regard to debugging. Whether and how you can debug a container depends on how you call the desired application. Startup is a topic that regularly leads to despair when creating Docker containers, because in the worst case, the container crashes immediately after the docker run command.

When starting a Docker container, two entries in the Dockerfile play a role: ENTRYPOINT and CMD. Many administrators don't realize the difference between these two at first. Within the container, the CMD is passed as a parameter directly to the program or script defined in ENTRYPOINT. If the container itself does not specify an ENTRYPOINT, it is usually /bin/sh -c. If the CMD is /bin/ping and you do not define a separate ENTRYPOINT, the container calls the /bin/sh -c /bin/ping command at startup (Figure 1).

Figure 1: Overwriting the ENTRYPOINT – as shown in GitLab's Docker containers – determines the init process within the container.

The two values can therefore also be used in combination: For example, it is possible and customary to define a program as the ENTRYPOINT and to supply it with parameters with CMD. The entry in ENTRYPOINT is then the process with virtual PID 1 within the container.

As already mentioned, ENTRYPOINT should ideally not be a shell script that starts multiple processes. If this cannot be avoided for some reason, all commands in that shell script, except for the last one, must either terminate cleanly or retire to the background with the use of &. However, the last command must neither expire with a return value of   nor disappear into the background because, then, the entire Docker container terminates.