Well Kept

Ceph and its core component RADOS are considered a prime example of an open source object store implementation that has earned a reputation as a reliable piece of software that need not fear competition with proprietary and commercial offerings. Some observers tend to forget the quite extensive history of the environment. In the eyes of many, Ceph is still quite young, even though it has been around for more than 15 years and has been marketed commercially for more than 10 years. The software is by no means a newcomer; quite the opposite, it is well established.

Ceph has not been quiet. To this day, developers continue to surprise users with an explosion of features in a relatively short time, which is reason enough to take a closer look at what has been happening at Ceph in recent weeks and months – both in terms of the people working on Ceph and of the product itself. What new options are available to administrators today, compared with slightly older Ceph versions, such as Nautilus, and how can administrators benefit from those innovations and improvements?

Step by Step Takeover

Somewhat uncharacteristically, I start with the organizational aspects behind the scenes at Ceph, because at least as much has changed there recently as on the technical side. The reason is obvious: After Red Hat swallowed the erstwhile Ceph provider Inktank several years ago, the company initially continued to run as a company within a company. This approach was not unusual for Red Hat. Several times in the past, Red Hat has not integrated companies into its own organizational structure immediately after acquisition, and for good reason.

After the acquisition of Inktank, Red Hat would have had virtually no teams into which Inktank could have been integrated. Some structures in the company were streamlined over the years; for example, parts of the sales department formally moved from Inktank to Red Hat, as did parts of the marketing department.

IBM has now taken over Red Hat, and as much as IBM always emphasizes that it wants to "preserve the DNA of purchased companies," Big Blue is not well known for keeping this promise. The inevitable happened, and after a grace period, IBM announced extensive changes to both the corporate structure and the portfolio. Initially, all the storage sales moved from the Red Hat side to the established IBM storage teams.

IBM justified this step with arguments that are quite conclusive at first glance. Currently, they claim, different sales teams have to travel to customers with IBM and Red Hat products, but some of them offer the same products or variations thereof. Now, customers will be able to purchase the vendor's entire storage portfolio directly from IBM, even if the product originally came from Red Hat. Red Hat was practically dropped as a factor in sales communication and was not to everyone's liking. IBM affirmed that the contact persons for existing customers would remain the same, both in terms of sales and technology, but this would not be the first time such promises fell victim to "reorganization" at a later stage.

Less Ceph, More IBM

IBM is also tweaking the Red Hat product portfolio. In the future, Ceph will be increasingly marketed as software bundled with storage hardware from Big Blue. Ceph will only be available as a standalone product with commercial support in combination with Red Hat's OpenStack platform. This arrangement is likely to be a bitter blow for some companies, because by no means does everyone who operates Ceph also rely on the OpenStack private cloud environment.

Red Hat previously expressly pointed out that Ceph in the form of Red Hat Storage is also available without OpenStack, and presumably the number of active users of this solution is greater than that of the cloud combination. No problem, IBM assures you, existing contracts will not be changed. However, administrators do not have any real certainty that IBM will not slam the door on the previous standalone Ceph installations, which, understandably has led to a sinking feeling in the stomach of some admins, although there does not seem to be any reason to panic.

Inktank might still be the sole Ceph part of Red Hat upstream and, hence, as part of IBM. However, companies that offer consultancy for Ceph also exist outside of Red Hat headquarters. These consultants include small, external service providers, as well as large companies like Canonical. At least a few of those same companies will in all likelihood also ensure that Ceph with commercial support can continue to be used without OpenStack. The approach that smaller consulting firms take with regard to Ceph may be better received by many Ceph users anyway. Although the typical all-in contracts are still widespread in the US, in many other countries arrangements can be more individual, which, in some cases, leads to independent service providers offering commercial support for setups that the Red Hat people at IBM just wave off.

Speaking of waving off, a Ceph personnel story made the rounds in the community a while ago and caused an uproar, at least briefly. None other than Ceph inventor Sage Weil announced his intention to withdraw completely from Ceph development in the foreseeable future and leave the helm to others. During the US presidential campaign, he said, he had taken part in various voter mobilization groups, gaining interest in election and civil rights issues, and Weil has been active in precisely these areas, more or less turning his back on IT. Fortunately, the Ceph world did not stop spinning after the departure of its warlock. Because the rights to Ceph now belong to a foundation under US law, Ceph can be considered independent enough to withstand even a severe change in the available personnel.

Incidentally, some suspect the spin-off of the Storage team from Red Hat to be the start of IBM's Red Hat carve up. This wouldn't be the first time that IBM bought a competitor, gutted it, and then sold off the unprofitable remains. For the moment, at least, no real evidence supports this thesis. Since Red Hat has belonged to IBM, it has been growing as a division more strongly than before, achieving double-digit annual growth rates. Quite possibly, other parts of Red Hat will be merged into IBM sooner or later. However, this should not be seen as a departure from the technologies that IBM has incorporated with Red Hat – which includes Ceph.

New Delivery Format, New Setup Tool

The news that Ceph has a new setup tool will probably not provoke more than a yawn of boredom from old hands in the Ceph business. After all, anyone who has worked with the software for a few years has seen many tools come and go (e.g., mkcephfs, ceph-install, ceph-deploy), not even mentioning the tools supplied by distributors. Ceph can also be rolled out by Puppet, Chef, and Ansible. However, the developers promise this new deployment tool, cephadm [1], is here to stay (Figure 1).

Figure 1: Admins create Ceph daemons and object storage daemons (OSDs) with cephadm teamed with ceph-mgr, which replaces Ceph deployment tools.

However, cephadm is only half the truth. Strictly speaking, it really only contains the functionality necessary to get the basic services running on a host. When cephadm has done its job, the management component ceph-mgmt takes over. The developers have been quite generous. The cephadm and ceph-mgmt team is de facto a separate automation and orchestration framework that virtually replaces Ansible and others, at least for the Ceph part of a setup.

However, the introduction of cephadm is accompanied by another change that is by no means to the liking of every admin and that has already caused extensive discussions on the Ceph mailing list. Red Hat is mercilessly following through with its "containers first" strategy for Ceph, too. Red Hat Storage, for example, is already completely containerized. The installation no longer rolls out packages in RPM or DEB format on the system, but installs Podman as a runtime environment for containers and then starts the required Ceph services in containerized form (Figure 2). From the administrator's point of view, this method introduces many a weird effect.

Figure 2: If the Red Hat people have their way, Ceph will run in containerized form, which is practical for the manufacturer but means a change of habit for admins.

You can no longer call ceph -w at the command line because the ceph binary simply does not exist on the system. To do so, you first need to run cephadm shell to launch a local container with the shell, in which the ceph command will then run. Admittedly, the problem is relatively easy to get around. If you install ceph-common and make sure that /etc/ceph contains the appropriate ceph.conf, including the key of CephX user client.admin, ceph will again run from the command line. However, having Ceph services run in containers also has other implications. If you want to see the system messages of these services, you have to use podman logs to access their containers, because the messages are no longer available in text files.