Cloudless

A strange trend has surfaced in which everyone suddenly wants to use OpenStack, even if it is only to manage a few virtual machines (VMs). OpenStack projects that are launched without any other motivation typically disappear again very quickly, especially when the company realizes the overhead it is taking on with OpenStack.

Without a doubt, if you only want to manage a few VMs, you are significantly better off with a typical virtualization manager than with a tool designed to support the operation of a public cloud platform. Although classic VM managers are wallflowers compared with the popular cloud solutions, they still exist and are very successful. Red Hat Enterprise Virtualization (RHEV) enjoys a popularity similar to SUSE Linux Enterprise Server (SLES) 12, to which you can add extensions for high availability (HA) and which supports alternative storage solutions.

Another solution has been around for years: Proxmox Virtual Environment (VE) by Vienna-based Proxmox Server Solutions GmbH. Recently, Proxmox VE reached version 5.0. In this article, I look at what Proxmox can do, what applications it serves, and what you might pay for support.

KVM and LXC

Proxmox VE sees itself as a genuine virtualization manager and not as a cloud in disguise. At the heart of the product, Proxmox combines two virtualization technologies from which you can choose: KVM, which is now the virtualization standard for Linux, and LXC, for the operation of lightweight containers. Proxmox also gives you the choice of paravirtualizing the whole computer or relying on containers in which to run individual applications (Figure 1).

Figure 1: Proxmox VE supports KVM virtualization and LXC container virtualization.

On the system level, a Proxmox-managed server does not initially differ from a system on which you manually roll out KVM or LXC instances. What sets Proxmox apart is an intermediate layer displayed to the user that supports the management of VMs and any attached storage from the top down and offers standardized interfaces with which you can determine your desired configuration from the bottom up.

This intermediate layer in Proxmox is very powerful: On request, it sets up redundancy for virtual systems or persistent memory, creates new VMs on demand, and always keeps you up to date with the current load. To achieve this goal, the developers have combined different tools.

The Proxmox API

The most important of these tools is the API, which goes by the slightly clumsy name pvedaemon in the Proxmox universe. It forms the central switching point in the entire setup. If you want to run an operation within the platform (e.g., start a VM or create a new volume on connected storage), this command ultimately ends up with one of the running API instances.

At this point, Proxmox is not much different from a cloud computing solution. The concept of the central API also is available from all sides; however, unlike OpenStack, in which an instance of the API runs on every host, in Proxmox VE, each host has its own API.

Nevertheless, you do not have to address a specific API when running commands. Because Proxmox has the ability to send the target host as part of a command, the pveproxy helper service, which runs on every host, is all you need. No matter to which proxy instance in the cluster the user sends a command, the command always initially ends up in the proxy of the respective host, and the host forwards it to the target, if it is not the target host itself.

The PVE stats daemon, pvestatsd, supports the Proxmox tools as they work. It regularly lists the running and available resources for each host and shares this information with all other nodes in the cluster. Each node has a complete image of the resource situation at any point in time, which makes it significantly easier to, for example, launch additional VMs by making it far easier to discover the host on which the new resource needs to be created.

High Availability

A significant difference between classic virtualization and the "anything can break at anytime" mantra that was propagated for clouds, is the way the setup responds to failures. In the cloud, either the software author or the administrator who operates the software is responsible. Together, they need to design and roll out the software so that the failure of an individual server, or any other infrastructure component important for operations, does not result in service failures.

Classical setups try to field failures by introducing measures at the infrastructure level. To do this, for example, the VM has to reside on redundant storage so that its data remains available, even if the previous computing nodes fail. Additionally, a cluster resource manager (CRM) has to ensure that the VM is restarted automatically on another node if the previous home node has crashed.

Because Proxmox sees itself as a classical virtualization solution, the developers have taken precisely this approach: The pve-ha-lrm and pve-cluster services establish a complete HA cluster that fields crashes of individual nodes. The resource manager pve-ha-lrm runs the commands it receives from pve-ha-crm on the local system as part of pve-cluster and the cluster-wide resources manager: If a node crashes, the CRM service initially decides on which hosts to relaunch the VMs previously running on the crashed node; then, it sends the corresponding commands to the local resource managers (LRMs).

Although Proxmox relies on Corosync to enable communication between the cluster nodes, it does not use Pacemaker as the CRM, which is the norm in this combination. Instead, the developers have written their own cluster manager, ha-manager, which they have been using since Proxmox VE v4 (Figure 2).

Figure 2: High availability plays a big role in classical virtualization. Proxmox includes a complete cluster manager.

The pve-cluster service includes even more: The team has also developed a database-based filesystem known as pmxcfs, which takes care of keeping configuration files in sync across the hosts of a setup. That was not always so easy in the cluster history of Linux. Solutions such as csync2 have regularly proved very inflexible in the past.

Classical automation tools don't help, either. Some files kept in sync across hosts change during operation. Regular replacement with a static template also does not work. From today's perspective, it is questionable whether you really need to tackle this problem with your own, partially Posix-compatible, FUSE-based filesystem. Today, both Consul and its etcd competitor offer comparable functionality without using their own filesystems. However, it is possible that the two services simply didn't exist when Proxmox started the development of pmxcfs.