Isolation Ward

The debate surrounding systemd, originally launched with the simple goal of replacing the ancient SysVinit scripts in most Linux distributions with a contemporary solution, has caused even venerable projects like Debian GNU/Linux to split into a pro-systemd faction (Debian) and an anti-systemd faction (Devuan).

However you look at it, though, success has proved systemd originator Lennart Poettering right. No major distribution today would seriously consider replacing systemd with another solution. The init system's relevance is dwindling in any case in the age of containerized applications. If MariaDB is just a container you need to launch, then the init system hardly needs to perform any magic.

If you follow Red Hat, SUSE, and its offspring, clearly containers is where the journey is headed (see the "Container Advantages" box). A container-first principle now applies to all enterprise distributions, with the exception of Debian. Systemd has a few aces up its sleeve that most admins don't even know about – not least because of the sometimes almost hysterical controversies surrounding the product.

In the container context, these functions include Nspawnd and Portabled. When deployed correctly, they draw on features from the container world to make conventional applications more secure. If you use Nspawnd wisely, you could even save yourself the trouble of needing Docker or Podman. In this article, I provide an introduction to these two functions and explain how you can use the solutions to supplement your own setups.

Unknown Container Runtime

When asked about runtime environments for containers, most admins intuitively think of one of two candidates: Docker or Podman. Docker returned containers on Linux to the land of the living and provided a decent business model. That containers are considered commercially attractive at all today is largely thanks to Docker's persistent work. Podman, on the other hand, is known by most admins as the anti-Docker solution created by Red Hat that exists because the Docker developers once tangled with the crimson chapeau and, as expected, got the wrong end of the stick.

Because Podman is meant to work as a one-to-one replacement for Docker. However, it adopts much of its architectural assumptions, and they're tough, because the Docker notion of containers is complex (Figure 1) and can overwhelm you with feature bloat. Containers should be simple. All container implementations on the market ultimately rely on a relatively small set of security features that the Linux kernel itself has offered for a few years.

Figure 1: Docker comprises a multitude of services and components. If you only need simple protection, you could quickly feel overwhelmed by the features (credit: Docker docs [1]).

No container implementation can do without namespaces, which logically separate individual parts of the system (Figure 2). A network namespace, for example, lets you create virtual network cards without giving them direct access to the physical NICs of the host. Instead, this access must be established by a bridge or some other means. Namespaces do not only exist for network stacks; they also apply to individual points in the filesystem, to process IDs, and to the assignment of user IDs on a Linux system. They always work along the same principle: As soon as a certain process starts in a namespace, the namespace acts like a jail from which it is impossible to break out.

Figure 2: The kernel namespaces feature has many uses in the context of containers, allowing areas in a virtual system to be isolated from the main system (credit: Ivan Zahariev [2]).

Control groups (cgroups) are added on top in many container environments. Again, they are deeply embedded in the Linux kernel. In very simplified terms, cgroups control access by individual processes to the system's resources. They complement namespaces nicely because they help you enforce an even tighter set of rules for applications and processes than would be possible with namespaces alone.

More than Runtimes

If Nspawnd is a runtime environment for containers, yet at least two well-functioning environments already exist in the form of Docker and Podman, why, some might ask, does Poettering have his fingers in the pie again? The answer to this question is stunningly simple: systemd-nspawnd targets admins who really only want to use basic kernel features to isolate individual processes.

The problem with Podman and Docker, after all, is that you never just get the program in question. Instead, they come with a huge pile of assumptions and prerequisites about how to run a container well and sensibly. You might not even want to deal with things like volumes, software-defined networking, and other stuff if all you want to do is put an Apache process in a virtual jail. Also, you might not want to install dozens of megabytes of additional software for Docker or Podman, thereby raising the maintenance overhead, although this step is not strictly necessary from a functional point of view. Anyone who can see themselves in this scenario – simple containers that use built-in tools without too much tinsel – is a candidate for the target group that Nspawnd has in mind, even if the scope of Nspawnd has naturally expanded in recent years.

The daemon has been part of systemd since 2015, so it's an old acquaintance. The "N" in the name – you probably guessed it after following the article up to this point – stands for "namespaces." Reduced to the essential facts, Nspawnd is a tool that sets up the namespaces required for isolated operation of applications and then starts the applications. Some developers jokingly refer to it as "Chroot on steroids," which works well as a metaphor. In the context of concrete technology, however, the comparison is misleading.

Containers, Pronto!

Nspawnd is now included in most distributions, so a container can be created on a normal Linux system in next to no time. Creating a usable template takes longest; in Docker or Podman parlance, this would be referred to as an image. Nspawnd only requires a working filesystem on a Linux distribution. You can put this in place in different ways.

The following example assumes Debian GNU/Linux 11 alias "Bullseye" as the distribution used in the container. In the first step, you build an empty folder on a Debian system after installing the debootstrap package (Figure 3), which contains a basic Bullseye system:

# debootstrap --arch amd64 bullseye /mnt/containers/bullseye-1 https://debian.inf.tu-dresden.de/debian/

To log in to the container as root, pts/0 must be in /etc/securetty:

echo "pts/0" >> /mnt/containers/bullseye-1/etc/securetty

If you now want to start a running container from the directory you just created, type:

systemd-nspawn -D /mnt/containers/bullseye-1

Figure 3: A container suitable for running in Nspawnd or Portabled is quickly built with typical Debian tools like debootstrap.

You can now run passwd to change the password for root in the container or add new users. All other commands that you will be familiar with from a normal Debian system are available to you. The recommendation is to store central files such as the package sources in the template and to update the package sources in the template immediately by running apt update. You need to delete the /etc/hostname file in the template so that the container uses the name assigned by Nspawnd later.

Finally, D-Bus needs to be installed in the container because the machinectl userland tool (Figure 4), which you use to control the containers from the host, cannot communicate with the particular container otherwise. Once the template is created, copy it to a location with a suitable name (e.g., /var/lib/machines/webserver-1). The next step is to start the container with:

systemd-nspawn -M webserver-1 -b -D webserver-1

Figure 4: The machinectl command manages containers in Nspawnd in the shell. If so desired, the process can be automated with a unit file.

After that, you can install the userland software that you want to run in the container (e.g., an Apache web server).