Inspection

Containerized environments are complex and comprise several layers, especially if Kubernetes is involved as a fleet orchestrator. Container security is a particular challenge because today's cloud stack combines so many components from so many different sources, in a more or less meaningful way, that it is not easy to keep track of and identify security updates for the various sources, finding the ones that you need for your own environment, and installing them in good time. As if that weren't enough trouble, more or less the same thing applies to compliance. Most glaring security issues are not caused by bugs, but by trivial misconfigurations that nobody notices in the review. If all the internal control processes fail, your own container landscape is left as open as the proverbial barn door in a worst case scenario.

To ensure that containerland does not turn into a horror movie, companies need to bear a few things in mind when they look to operate a large number of containers. After all, Kubernetes and others of the same ilk do not maintain themselves, and container-based approaches are no less complex than their traditional predecessors; you have to deal with even more loose ends than in conventional setups. The runtime environment for containers, Kubernetes itself, a number of on-top solutions such as the Istio service mesh, various package managers such as Helm, and the various sources from which container images can be obtained today are just a few examples.

This is where Kubescape [1] enters the scene. Its developers make some bold promises, claiming that it is the first tool that can completely automate the process of checking the entire container stack of an environment for security and compliance problems according to accepted rules (e.g., from the US National Institute of Standards and Technology (NIST), the not-for-profit MITRE organization, or the joint US National Security Agency (NSA)-Cybersecurity and Infrastructure Security Agency (CISA)). Kubescape not only looks at the basic services that belong to Kubernetes itself but also checks YAML files with resource definitions, code in directories (e.g., GitHub or GitLab), the artifacts that arise from continuous integration and continuous delivery (CI/CD) tools such as Jenkins or Argo, and the deployment itself and associated components.

Complex Kubernetes Setups

According to the Kubescape developers, all you have to do is install, launch, and be happy. Kubescape really does an amazing job, and once you have familiarized yourself with the program, you will be thrilled with the versatile feature set it offers. What's more, Kubescape is free software, an official Cloud Native Computing Foundation (CNCF) sandbox project freely available online, which is reason enough for Kubernetes administrators to take a closer look at the tool and try it out.

Before you do, however, you need to look at a bit of theory – after all, Kubescape is also complex under the hood. To make sure you don't end up with yet another tool whose functionality you can only guess at in a vague way, you need to take a closer look at the capabilities of Kubescape, which, in turn, means taking a closer look at the structure of a standard Kubernetes setup. In this way you can identify the components that play a role in terms of security and compliance and see where Kubescape enters the scene when you want to monitor these components.

The components of Kubernetes (K8s) are defined and obvious, as are the helpers that K8s needs to operate containers. They include the Kubernetes API, the scheduler, the K8s controller cluster, and the agents (aka kubelets) on the target systems. Moreover, K8s is practically useless without a runtime environment for containers on a compute node, which means you need either the Docker Community Edition or Podman, which also offers the compatible CRI-O runtime environment. Together, these components are all it takes to manage containers across the boundaries of individual compute systems.

In the vast majority of today's environments, though, these basic components are not enough because they only offer very basic functionality for complex software-defined networking (SDN) and software-defined storage. Without these components, container fleets in particular cannot be operated meaningfully. After all, if you are building a scalable platform, you will also need a scalable network and scalable storage.

More components are quickly added, such as Calico container and network security, or Rook, which bundles the Ceph object storage solution into K8s and makes it manageable. Both Rook and Calico make extensive use of custom resource definitions, which, strictly speaking, must be handled as a separate factor in the security context; even a hardened Kubernetes is useless if it becomes vulnerable because of the extensions you added or just installed.

The prebuilt Rook solution is by no means the only project that manipulates Kubernetes from the outside. Tools such as Istio are also likely candidates, extending the Kubernetes API with their own settings and services (Figure 1). To make matters worse, all of these tools and extensions come with their own software in their own containers, and you need to at least monitor the containers to keep your environment secure.

Figure 1: Kubernetes comprises several layers and can be extended externally with complex service definitions, such as from Istio, as shown here. Ensuring security and compliance is a highly complex task.

No mention has yet been made of tools that you add to your setup in other ways. For example, the Helm package manager promises to facilitate software installation tasks but itself consists of software that is potentially susceptible to security problems. Remember that, in a Kubernetes cluster, all these challenges are just the icing on top of those that come with your normal systems.

Containers do not run in a vacuum – they require standard Linux systems with a runtime environment, which can also be affected by security issues or incorrect settings. The trend in container environments is to degrade the host system to a container playback box, but you also have a Linux kernel and a basic set of userland software, which can cause worries in terms of security.

Kubescape Scans Everything

Kubescape enters the scene with the promise of making your life easier in this complex situation by applying defined best practices and screening your entire environment in a completely automated way. The operating principle is simple: Download the tool. For a change, Kubescape does not have to run as a Kubernetes component to perform its task. It is fully external and therefore avoids the criticism levied at many other Kubernetes tools that it only sees things "from the inside" and cannot identify problems outside of the Kubernetes universe.

Kubescape scans everything – resources in Kubernetes, the container images you use, and the host system configurations with any software installed there. At the end of a run, it displays an overview of the security and compliance problems found and sorts its findings according to how threatening they are. Clear and urgent security problems are highlighted in red and are found at the top of the list of results. Results where Kubescape is not entirely certain, but which – if true – would also be problematic, are highlighted in yellow in the middle of the list. Less important entries follow at the end.

Kubescape also applies a point system, awarding points for each problem it finds. If an installation exceeds a defined point limit, Kubescape sounds the alarm and prompts you to take action. In the default configuration, critical vulnerabilities will always mean that the number of points exceeds the alert threshold. If this principle reminds you of Chef's InSpec, well spotted. You would not be mistaken to describe Kubescape as a technology twin of InSpec that is specialized for use in Kubernetes environments.

Installing Kubescape

After all this theory, you still need to know how to put Kubescape to practical use, and it's not as complicated as you might expect, given the range of functions I just described. For a smooth start, your working environment needs to fulfill only a few conditions. The first is almost self-explanatory: You must have access to the entire K8s instance from the system on which you want to run Kubescape. If not already in place, you are advised to set up something like a cluster workstation that can access the entire Kubernetes cluster and the host systems.

In many places, IT and security departments tend to hide setups behind as many ridiculous firewall constructs as possible; and in some cases, this even means having firewalls between the individual systems in an environment. If this is true in some of your cases, you might need to use more than one host to perform scans. Of course, the host for Kubescape operation can be virtual. How you install Kubescape essentially depends on your personal preferences. The easiest way is to download the program directly from GitHub onto any Linux system:

curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash

The developers expressly point out that Kubescape is a security solution. For compliance reasons alone, you will want to download and investigate the shell script referenced by this command before running it. As an alternative, prebuilt packages are available for a number of Linux distributions. If you want to run Kubescape on a recent Ubuntu system, for example, you will find complete packages in a Launchpad PPA directory:

sudo add-apt-repository ppa:kubescape/kubescape
sudo apt update
sudo apt install kubescape

Make sure this directory is enabled on the system, and install the package required for Kubescape. This approach ensures that updates in Launchpad automatically find their way into your installation, which is not the case with the shell script variant described first.

The approach for RHEL is somewhere in between; the developers do not offer a usable repository, but at least you can use the prebuilt RPM packages. The packages are available online [2] and can be installed with dnf. Again, you are responsible for handling the updates yourself.