Active Separation

Whether a virtualization environment comprises only a few hosts or a complex cloud landscape, the majority of admins today who plan to use Linux as the basis of their hypervisors favor KVM. All common Linux distributions already have the necessary software packages, which often facilitates the decision to use KVM, as well as the virtualization project itself.

The architects of such a setup all too rarely pay attention to the security of their design. The widespread distribution of container technologies illustrates the low degree of protection under which many users are willing to work; however, you cannot assume that Linux hypervisors such as KVM are automatically secure simply because it's Linux and because virtualization inherently isolates its guests in a superior way compared with Docker containers, for example. In this article, I describe hardening strategies and look at common configuration errors.

The most far-reaching danger for a virtualization server is a malicious virtual machine (VM) managing to break out and gain access to the host and other guests. The danger becomes tangible with errors in the virtualization components. Antidote number one is rapid deployment of all security patches, but the admin can also use other means to reduce the likelihood of a guest breaking out.

To begin, you need to be very familiar with the main software components of your KVM ecosystem and their interactions. KVM [1], which stands for kernel-based virtual machine , is at the heart of the Linux virtualization universe. The Linux kernel's KVM component only provides the basic functions for running VMs, so the application that uses these functions does not have to deal with all the details of the hardware.

KVM and Qemu

Any Linux application can use the KVM functions provided by the kernel, but in practice, only one is relevant: The Quick Emulator (Qemu) [2] uses KVM to run a guest system in a largely native way on the physical hardware [3]. However, Qemu performs security-critical operations of the guest under its own steam – in particular, access to the (virtual) hardware, which involves the computer interrupting native execution and KVM taking control. KVM then typically calls on Qemu to implement hardware access; that is, it only enters the game if the guest needs not only more CPU and main memory, but also hard disks, the network, or the graphics adapter.

Not even the team of Qemu and KVM can successfully cover the entire field of modern virtualization. This all changes with a third core component: On the one hand, libvirt [4] lets you run multiple Qemu processes and, thus, several guests on a host. On the other hand, libvirt provides an interface for starting and stopping the guests or grabbing snapshots of VMs.

The virsh and virt-manager tools rely on the libvirt interface to help admin staff manage the whole enchilada, but even programmers can use the libvirt services, in particular for complex cloud solutions (e.g., OpenStack). Figure 1 outlines the interaction between KVM, Qemu, and libvirt.

Figure 1: An architectural overview of the KVM ecosystem with libvirt/Qemu.

Hardening Systems

The lion's share of the code in the overall solution is not contributed by the KVM kernel component, but by Qemu in the form of various platform emulators. Therefore, you should restrict the Qemu process to user space as far as possible. In practice, this can be done quite simply and effectively with the use of SELinux or AppArmor.

Although both access control systems have the reputation of being difficult to handle – indeed, manual (!) maintenance of control lists does not deliver a good return on the effort invested – distributions that offer one of the two as a standard package also include a working configuration for all packages, including libvirt.

The task of hardening Qemu essentially consists of choosing a Linux distro that supports SELinux or AppArmor out of the box. In this case, libvirt automatically generates the rules for appropriate protections, limiting the privileges of Qemu processes in an appropriate manner. In particular, this limits access to third-party disk images that are stored locally.

Libvirt also uses other hardening features such as cgroups to secure the Qemu process effectively, provided the operating system supports it, which is another reason to look closely at the Linux distro you use. Debian is not recommended, for example, because it does not support SELinux and AppArmor. However, Ubuntu, Red Hat Enterprise Linux (RHEL) 7, and CentOS 7 are recommended with AppArmor or SELinux on board.

Kernel Samepage Merging

RHEL and its clone fail elsewhere, however: When you install Qemu, Kernel Samepage Merging (KSM) [5] is activated automatically. This well-meant technique is intended to deduplicate memory pages with identical content and promises efficiency on a virtualization host, because guests with the same operating systems have many memory pages with identical content. Thanks to KSM, the host only needs to keep them in RAM once.

Apart from the fact that KSM often saves substantially less memory in practice than expected, it comes with significant security risks. The problem is that two guests who have access to the same physical memory page can determine the behavior of the other guest on that memory page. On first access, the page is sent to the CPU cache, which delivers the data in a measurably faster way as of the second read. With KSM, a second guest can leverage this fact to determine whether the first guest has read data identical to its own.

If you think that this is only significant under laboratory conditions, think again: Irazoqui et al. [6] have shown how a guest in a real cloud could discover the keys of an AES implementation of OpenSSL.

Any admin of virtualization environments in production use should thus ensure that KSM is disabled. In RHEL 7, CentOS 7, and others, you need to do this manually. Listing 1 shows how to see whether KSM is active. If the output from your system differs, it is highly probable that KSM is active. In this case, Listing 2 shows how KSM can be switched off on a CentOS 7 system.

# cat /sys/kernel/mm/ksm/run
0
# cat /sys/kernel/mm/ksm/pages_shared
0
# pgrep ksmtuned
 **
#

# systemctl stop ksmtuned
Stopping ksmtuned:                  [   OK   ]
# systemctl stop ksm
Stopping ksm:                       [   OK   ]
# systemctl disable ksm
# systemctl disable ksmtuned
# echo 2 > /sys/kernel/mm/ksm/run