Clever and Smart

Out of the ashes of OpenSolaris, the SmartOS [1] cloud platform is rising to position itself as a competitor to Amazon EC2. The Unix-like operating system is developed by Joyent, a virtualization and cloud computing company.

After Sun published the source code of Solaris under the name OpenSolaris in 2005 and then Oracle discontinued open development of the software soon after its takeover in 2010, a small OpenSolaris community quickly developed, including companies such as Nexenta, who used the free system as the basis of their products.

Nexenta, a core of Solaris engineers, Joyent, and other players founded and contributed to the Illumos project based on a fork of the free Solaris kernel. The Illumos kernel then became the basis for a variety of free Solaris offshoots, such as OpenIndiana, Illumian, EON, OmniOS, and Joyent's SmartOS.

Joyent has put much effort into the development of SmartOS to create a highly flexible basis for its own cloud. To ensure this flexibility, Joyent, or more specifically Max Bruning, ported the Linux KVM hypervisor to the OpenSolaris kernel. This project took about six months before Joyent had KVM running on SmartOS and thus able to support any guest systems that work with KVM.

KVM for Solaris

Before the KVM port, SmartOS already had Zones, which originated with Solaris, that offered operating system virtualization with containers and very little ballast.

However, Zones normally restrict you to the host operating system, even through branded Zones with Linux already existed in Solaris. In any case, full virtualization with KVM offers more options for running various systems as guests, including Microsoft.

Another advantage of SmartOS is that it provides complete support for the DTrace tracing framework that can trace kernel and userspace programs at run time with very little overhead (Figure 1).

Figure 1: A flame graph created with DTrace visualizes the call stack for MySQL.

Although Linux has similar tools, such as Kprobes and, more recently, a DTrace port, both of these products are still in their infancy compared with Solaris DTrace.

Joyent has written extensive programs based on DTrace that allow customers to analyze their own cloud in detail. Users no longer have to deal with DTrace scripts; rather, they are presented with various graphs that give them an impression of capacity utilization and bottlenecks.

Even ZFS, the SmartOS filesystem, has a special place in the SmartOS virtualization concept. The system benefits from the general capabilities of the Solaris filesystem, such as the possibility to implement RAID systems and extend them arbitrarily. Additionally, virtual machines leverage ZFS features, in that filesystems on VMs are simply snapshots of the VM template, thus saving space.

Using SmartOS

If you want to experience the benefits of SmartOS for yourself, you do not have to be a Joyent customer, because SmartOS is available for free. Among the several download options are various images for CDs, USB sticks, and virtual machines, as well as files for a PXE boot. SmartOS is not installed on a hard disk, because an up-to-date system is most likely to be guaranteed if there are no old files in the filesystem. The operating system is thus always booted from media, whereas the virtual machines are stored permanently in a ZFS storage pool. In the simplest case, you boot SmartOS from a USB stick or CD; in production environments, it is advisable to go for PXE via a server.

Booting SmartOS on a physical computer makes the most sense, because it assumes the existence of the (Intel only) processor features VMX (VT) and EPT (Extended Page Tables), which are not necessarily available on virtualized systems. The install is also possible in VMware Fusion, which passes on those two features to guest systems. VirtualBox also lets you install SmartOS, but the performance is not the best. With nested KVM, Linux at least provides VMX in virtual machines. However, nested EPT support was only introduced with Linux kernel 3.12, so that SmartOS at least installed without error messages, although it would hang if you tried to launch a KVM machine.

Although you don't have to worry about installation, you do need to set up a few things after booting from CD or USB: the root password, network settings, and storage options. SmartOS requires at least an entire hard disk as storage. With multiple disks in the ZFS pool, you can also implement various RAID setups. Even at this point, Linux users need to adapt, because the devices use typical Solaris-style names. For example, in device c0t0d0, c0 stands for the first controller, t0 for the first bus, and d0 for the first disk.

After logging in to the Solaris environment of a new SmartOS system, you need to familiarize yourself with a few new tools. Although the well-known Linux tool top does not exist, SmartOS has a program called prstat. A handy cheat sheet for Linux users listing the appropriate commands in SmartOS is available online [2].

The configuration files that SmartOS stores on the ZFS filesystem are located in /usbkey/config. The global zone in which you find yourself after logging in is not persistent; instead, it is on a RAM disk, which means, in particular, that you cannot create users who persist after a reboot; this additionally applies to any files in the /etc, /root, and /usr directories. However, /opt (and /var) are on ZFS and can thus be used to save files.

If you want, you can even install the pkgin package manager there and upload software to your heart's content. To download and unpack the tarball for the package manager, you only need to enter:

cd /
curl -k http://pkgsrc.joyent.com/packages/SmartOS/bootstrap/bootstrap-2013Q3-x86_64.tar.gz | gzcat | tar-xf -

To rebuild the package database and update it, enter:

pkg_admin rebuild
pkgin -y up

You can now use the pkgin program to search for packages and install and uninstall them. All packages are stored in the /opt/local directory and survive a reboot.

In the /opt directory, you can also set up services permanently. The Service Management Facility (SMF), which manages services in both Solaris and SmartOS, also checks the /opt/custom/smf directory at boot time; you can store the configuration files for your own services there.

Getting Images

Two simple commands let you manage virtual machines: imgadm manages the virtual machine images that the SmartOS administrator installs with vmadm. By default, imgadm is configured so that it makes use of the image server below https://images.joyent.com , as typing imgadm sources reveals. imgadm list displays the local directory of VM templates; until you have imported at least one template, there is nothing to see.

New servers can also be added using this command. To discover which images Joyent offers, type imgadm avail; typing imgadm update updates the list. To import an image to your local image store, use imgadm import <UUID>. Basically, you will see two types of images: smartos for Solaris Zones ("Joyent brand" in SmartOS-speak) and KVM images. The Joyent repository currently offers linux and bsd.

As you can see in Figure 2, SmartOS uses confusing hashes to identify images and virtual machines that can be cumbersome for manual administration, but at least you can copy and paste.

Figure 2: The templates available for virtual machines can be imported to the local store.

After importing a VM template, you can proceed to create a virtual machine. For this purpose, you need both a template and a specification of the virtual machine; this is composed in JSON format and essentially determines the virtual hardware of the VM (i.e., the disks, network cards, and so on).

An example specification is shown in Listing 1. The brand entry specifies the type of virtualization. In the example, it is kvm; the brand for Zones is joyent. An alias is convenient because you will otherwise find it difficult to locate the machine later. In terms of disk hardware models, the IDE and SCSI standards provide an alternative to virtio, but virtio is still the best solution for Linux.

01 {
02   "brand": "kvm",
03   "alias": "ubuntu1",
04   "vcpus": 1,
05   "autoboot": false,
06   "ram": 2048,
07   "resolvers": ["192.168.111.254"],
08   "disks": [
09     {
10       "image_uuid": "1fc068b0-13b0-11e2-9f4e-2f3f6a96d9bc",
11       "boot": true,
12       "model": "virtio"
13     }
14   ],
15   "nics": [
16     {
17       "nic_tag": "admin",
18       "model": "virtio",
19       "ip": "dhcp",
20       "primary": 1
21     }
22   ]
23 }

You can use this option for Windows virtual machines, but you need the appropriate driver, especially for the installation. The same goes for the network card, where emulations of popular Ethernet hardware exist as an alternative to virtio.

Make sure in the disks section that you have an image_uuid entry that matches that of the imported templates. Armed with this configuration, calling

vmadm create -f vmspec.json

generates a new virtual machine. If all goes well, its (new) UUID is output. The command vmadm list displays a list with the now active machine (Figure 3). An overview of all options available in the JSON configuration files can be found online [3].

Figure 3: Displaying the installed virtual machines with vmadm list.

Some configuration variables can be customized at run time with vmadm. For example,

vmadm update alias=name

changes the alias of the virtual machine. Alternatively, the command also parses JSON sections and uses them to update the configuration. Thus, you can integrate your own metadata into the configuration. Listing 2 shows the corresponding JSON data and how to load it into the configuration.

01 # cat meta.json
02 {
03     "set_customer_metadata":
04          {"opsys": "ubuntu"}
05 }
06 # vmadm update ed9879e8-764a-4c15-a298-6b51e815b68e < meta.json
07 Successfully updated ed9879e8-764a-4c15-a298-6b51e815b68e

You can log into the virtual machine, for example, by calling:

vmadm console <VM-UUID>

However, the guest must then be configured so that a getty (get teletype) process is listening for a login on the first serial port.

Alternatively, SmartOS offers Remote Desktop Access via VNC for each VM; the port is automatically assigned. To find out which port the current machine uses, the vmadm command supports the info option (Figure 4), which also returns the VNC configuration.

Figure 4: With a call to vmadm info you can find the VNC port for a virtual machine.