The Target

The final step is identifying a problematic container to troubleshoot and then firing up a SuperContainer on the same host as that misbehaving container. Without further ado, here's the clever part.

The target container is shown below in Figure  1 using the docker ps command. Note that the container is an Nginx container, which I have explicitly named nginx. In Kubernetes, the name would be a pod name such as nginx-kv152x. You should first find out which minion (or node) your pod is running on using a command such as:

$ kubectl get nodes

In Figure 1, you can see the target container has the name nginx, on the right-hand side, and is exposing the HTTP port (TCP port 80) from the host via TCP port 80 on the container under PORTS.

Figure 1: I'd better take a closer look at the problematic web server container, named nginx.

The run-time parameters, which make the SuperContainer so useful, are based on launching the SuperContainer within the same process namespace and network namespace as the target container.

To achieve this kernel-trickery, you need to fire up the SuperContainer with the following command line switches and also provide the CAP_SYS_ADMIN capability. If you're not familiar with the CAP_SYS_ADMIN capability, it is a powerful mechanism for opening up system permissions. However, it is not quite as daunting as running a container in privileged mode, which also circumvents a kernel's cgroup resource limitations. Look online for more information on kernel capabilities [3].

The capabilities man page warns about the power of SYS_ADMIN access by saying "Don't choose CAP_SYS_ADMIN if you can possibly avoid it!" A long overloaded list shows the complications associated with running a process at this capability level.

Back to the all-important command line, I run the SuperContainer as follows:

$ docker run --rm -it --pid=container:nginx   --net=container:nginx --cap-add sys_admin   chrisbinnie/supercontainer

Take a closer look at this clever command line. I am running the container with --rm, so it is destroyed afterwards (so attackers can't take advantage of it), and I'd also recommend deleting the image after using it on a host. I use the --pid switch to enter the same process namespace as the container name nginx. In exactly the same way, I use --net to access its network stack. Then I finally add the tricksy SYS_ADMIN.

If you struggle to get access to your applications from inside the target container, you can add other capabilities and, as a last resort, use --privileged mode. For example for running the strace package as a syscall debugging tool (syscalls are made whenever a process needs to request a service from the kernel), you should change your custom Dockerfile as follows:

CMD ["strace", "-p", "1"]

You would also need to add the CAP_SYS_PTRACE capability.

If you're interested in more detail about CAP_SYS_PTRACE, the kernel man page lists the following system resource access:

• Use ptrace(2) to trace arbitrary processes.
• Apply get_robust_list(2) to arbitrary processes.
• Use process_vm_writev(2) to transfer data to or from the memory of arbitrary processes.
• Use kcmp(2) to inspect processes.

Super Power

Having executed the run command you have spawned an interactive terminal offering access to your SuperContainer.

Figure 2 shows the SuperContainer running with an arbitrary hashed name (c7b61b1a8b82).

Figure 2: I have entered the SuperContainer using run as the superuser root.

Super Process Tables

Be warned that what follows might take a moment to get used to; it is clever, simple, and yet a little surprising. If you have ever used a chroot in the past, it isn't quite as discombobulating.

A reminder that the plan is to access another container from within the SuperContainer by using the same kernel namespaces as the target.

Figure 3 shows a standard command to check the process table. But… look! The Nginx web server is definitely not running inside the SuperContainer, so I know for certain that I'm accessing the target container, named nginx.

Figure 3: Running the ps -ef command inside the SuperContainer shows the process table of another container.

I can identify which processes are running within the container, under which user, and with which PID and parameters, as shown in Figure 3.