Lighting the Fuse

For many admins, virtualization is "done and dusted": something you can just leave to run on its own. It is all the more surprising when someone comes along with a genuine innovation – or at least new features. Amazon set off fireworks last November, implementing its Firecracker [1] virtualizer under an open source license. Thus far, the tool had mainly been the driving force behind the AWS Lambda service [2], where it enables basic virtualization of small VMs. However, Firecracker can do far more and looks set to conquer territory, even some that is currently occupied by containers.

Technical Underpinnings

Amazon doesn't often publish an in-house solution used in its AWS environment under an open source license. In the case of Firecracker, the approach makes sense, because Firecracker is closely connected to another open source technology: KVM.

For some time now, Amazon has been offering its Lambda service (Figure 1) as part of AWS, with the aim of providing users with simple virtual machines (VMs) on lean virtual hardware that are significantly cheaper than classic AWS instances.

Figure 1: Amazon's Lambda service was the inspiration for Firecracker: The service runs defined programs on demand to process data.

However, Amazon obviously didn't want to rely on containers for this product, explaining that the security and isolation functions of the common container conversions in Linux were not very impressive. KVM, on the other hand, was far too heavy as an off-the-shelf virtualizer. For example, if KVM generates 20 percent overhead per VM, that's a huge sum, given the quantities AWS sells. The solution was obvious from Amazon's point of view: The perfect combination would be to use the functions of KVM without the overhead of the Qemu emulator.

Again, much of the overhead in KVM is not caused by KVM itself, but by the way Qemu works. The KVM team have gradually adapted Qemu to KVM and optimized it for use with KVM; however, various topics, such as resource usage, are so deeply rooted in Qemu's design that it is impossible to eliminate the associated bottlenecks without a rewrite. When Amazon sent Firecracker into the ring, they introduced a potent competitor for Qemu, not a replacement for KVM.

Firecracker is definitely not a 100% Amazon native. The software is also based on previous work (e.g., from Google's Chromium team). Even the approach is not completely new, because Kata Containers [3], which are currently experiencing a kind of mini-hype themselves, follow a similar pattern. Without a doubt, however, Amazon is the company that is already using such a solution in production operations on a huge number of systems. The logical conclusion, then, is that Amazon must know what it is doing.

Technically Firecracker comprises several components. First is the core component, a hypervisor in its own right, which builds directly on the KVM implementation of the Linux kernel. It thus grabs all the advantages offered by KVM: paravirtualization for individual devices in the virtual machines or direct access to the CPU to take advantage of its virtualization capabilities. Anyone who has worked with virtio-net or the paravirtualized Virtio disk driver in KVM and Qemu knows the advantages of this approach.

Under the hood, the hypervisor in Firecracker is based on crosvm, which has its origins in the development of the Chrome browser. Like crosvm, Firecracker is therefore also written in Rust.

Jailer Provides Security

The hypervisor is supported by a component known as the Jailer, which comprehensively tackles security issues. On the one hand, the Jailer component starts the Firecracker application, which then creates minimal VMs on the hypervisor base.

At the same time, Jailer puts handcuffs on the various stack components and configures groups and kernel namespaces to determine right from the start what the VM can access. Jailer also restricts the permissions of a Firecracker VM with seccomp filters, and because that's not enough, it also dynamically removes access rights from VMs or grants them where needed.

That's a good helping of up-to-date security, but here it becomes obvious that Amazon has at least partially reinvented the wheel. After all, a powerful framework for managing virtual machines is already available for Linux in the form of libvirt, and this very probably could have been extended to include comparable functions, if required.

One genuine innovation is the RESTful API of the Firecracker package. As you would expect, an application developed today can hardly dare leave the company without a RESTful API with a control function. Far removed from the considerations of hip development paradigms, the Firecracker API at Amazon also has some very practical uses: Users can launch Firecracker VMs directly.

Libvirt offers a similar remote control function, but not via the RESTful API, and because the cloud is the measure of all things these days, it seems important that technology can be controlled with typical cloud tools. An API based on REST principle guarantees just that.

Not a Drop-In Replacement

If you are under the impression that Firecracker is only a leaner replacement for Qemu and libvirt, you would be wrong. The differences between the solutions are considerable, and they quickly become apparent. For example, Firecracker does not let you launch any old distribution at the drop of a hat. Instead, you need a special kernel package and a root filesystem, for both of which Amazon is currently the only provider.

The package is Alpine Linux, which is not one of the more common distributions. As Firecracker becomes more widespread, Firecracker packages from other vendors will not be long in coming. According to Amazon, this is explicitly one of the reasons why the solution is now being released as open source software. Together with the community, Amazon is not only looking to push forward with Firecracker development, but also to make it more attractive and put it on the foundation of a large developer community.

Firecracker can only pass a few virtual devices into the VM, such as a virtual network card, the virtual block device driver, a serial console for intercepting output, and a keyboard that has only one key and is used for emergency purposes.

Firecracker's reduced approach pays off: Amazon calculates that the Alpine kernel and root filesystem can give you an executable VM within 125ms, and this is a fully fledged Linux system. Qemu can't keep up with this – even the Linux distributions that usually start as a complete system in a combination of Qemu and KVM need a multiple of this time, and the BIOS that Qemu emulates needs more time to load. Operation "Lean VM" was obviously successful for Amazon.