Bug Hunt

Temporarily Stopping a Container

The following tip does not refer so much to the container's inner workings: Docker supports the pause and unpause commands. These commands let you pause running containers without stopping them completely or even deleting them. In everyday life, this can be useful if you need to debug one container and stop another to prevent it from sending further data to the container under investigation. It would take considerably more effort to stop the other container completely and then restart it. However, the combination of pause and unpause ensures that the application in the container simply continues to run at the end of the process as if nothing happened in the meantime.

Debugging Inside the Container

Even if a container launches successfully, things can still go wrong. If you have little experience with Docker, you may be confused. Unlike a real VM, you can't just SSH in to containers to see what's going on. But don't panic, Docker offers a solution, which is probably it's most powerful debugging tool: docker exec.

This command can best be compared with chroot at the command line. After calling docker exec <Name Command> at the command line, Docker calls the whole command within the container. At the system level, then, Docker calls the specified command within the namespaces defined for that container, such as the network and process namespaces that already exist for the container.

If you append the -it parameters docker exec also works interactively:

Docker exec -it /bin/bash

This command calls a Docker shell inside the ping container, which you can use just like a normal shell. If you are looking for a way to "log in" to the running Docker container, you will find it in docker exec (Figure 3).

Figure 3: docker exec catapults you into a shell inside the container, where you can look around.

Inside the Glass House

Accordingly, you have the possibility to call all binaries available in the container from a shell started in this way. If something doesn't work, the usual suspects like ss or ip will help – provided they are included in the Docker image on which the container was built. If you build the containers yourself, you should pay attention to your basic configuration, because the Linux distributors' basic images do not include all the tools you are used to in your daily work.

Other Linux tools like ls or ps also work in the Docker container. However, you should not forget that you really only have one view inside the container: the view you are forced into by the various namespaces and security policies. If you want to debug components outside the container, you first need to leave it.

Also be aware that filesystem changes only affect the overlay image of the running Docker container. If you delete this container and restart it from the base image, any modifications are gone. If you encounter changes you want to make, you will ideally also want to rebuild the container's base image.

In this context, it is important that a running container's main process cannot simply be restarted from within the container. After changing a configuration file, it can only be reloaded with SIGHUP if the tool supports the signal. A SIGKILL with the plan to restart the process afterward would immediately stop the running shell, because the service is virtual PID 1 in the container. An init crash on a Linux system leads to a kernel panic. With Docker, the affected container just terminates immediately.