« Previous 1 2 3 Next »
Troubleshooting Kubernetes and Docker with a SuperContainer
Super Powers
The Target
The final step is identifying a problematic container to troubleshoot and then firing up a SuperContainer on the same host as that misbehaving container. Without further ado, here's the clever part.
The target container is shown below in Figure 1 using the docker ps
command. Note that the container is an Nginx container, which I have explicitly named nginx
. In Kubernetes, the name would be a pod name such as nginx-kv152x
. You should first find out which minion (or node) your pod is running on using a command such as:
$ kubectl get nodes
In Figure 1, you can see the target container has the name nginx
, on the right-hand side, and is exposing the HTTP port (TCP port 80) from the host via TCP port 80 on the container under PORTS
.
The run-time parameters, which make the SuperContainer so useful, are based on launching the SuperContainer within the same process namespace and network namespace as the target container.
To achieve this kernel-trickery, you need to fire up the SuperContainer with the following command line switches and also provide the CAP_SYS_ADMIN
capability. If you're not familiar with the CAP_SYS_ADMIN
capability, it is a powerful mechanism for opening up system permissions. However, it is not quite as daunting as running a container in privileged mode, which also circumvents a kernel's cgroup resource limitations. Look online for more information on kernel capabilities [3].
The capabilities man page warns about the power of SYS_ADMIN
access by saying "Don't choose CAP_SYS_ADMIN if you can possibly avoid it!" A long overloaded
list shows the complications associated with running a process at this capability level.
Back to the all-important command line, I run the SuperContainer as follows:
$ docker run --rm -it --pid=container:nginx --net=container:nginx --cap-add sys_admin chrisbinnie/supercontainer
Take a closer look at this clever command line. I am running the container with --rm
, so it is destroyed afterwards (so attackers can't take advantage of it), and I'd also recommend deleting the image after using it on a host. I use the --pid
switch to enter the same process namespace as the container name nginx
. In exactly the same way, I use --net
to access its network stack. Then I finally add the tricksy SYS_ADMIN
.
If you struggle to get access to your applications from inside the target container, you can add other capabilities and, as a last resort, use --privileged
mode. For example for running the strace
package as a syscall
debugging tool (syscall
s are made whenever a process needs to request a service from the kernel), you should change your custom Dockerfile as follows:
CMD ["strace", "-p", "1"]
You would also need to add the CAP_SYS_PTRACE
capability.
If you're interested in more detail about CAP_SYS_PTRACE
, the kernel man page lists the following system resource access:
- Use ptrace(2) to trace arbitrary processes.
- Apply get_robust_list(2) to arbitrary processes.
- Use process_vm_writev(2) to transfer data to or from the memory of arbitrary processes.
- Use kcmp(2) to inspect processes.
Super Power
Having executed the run command you have spawned an interactive terminal offering access to your SuperContainer.
Figure 2 shows the SuperContainer running with an arbitrary hashed name (c7b61b1a8b82).
Super Process Tables
Be warned that what follows might take a moment to get used to; it is clever, simple, and yet a little surprising. If you have ever used a chroot
in the past, it isn't quite as discombobulating.
A reminder that the plan is to access another container from within the SuperContainer by using the same kernel namespaces as the target.
Figure 3 shows a standard command to check the process table. But… look! The Nginx web server is definitely not running inside the SuperContainer, so I know for certain that I'm accessing the target container, named nginx
.
I can identify which processes are running within the container, under which user, and with which PID and parameters, as shown in Figure 3.
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)