Put to the Test

Fun with ceph-deploy

The quick-start documentation [8] helps you get going with Ceph, whereas the ceph-deploy tool takes over installation and configuration. Because it uses SSH connections for the managed systems, you should look through its Preflight section [9] first. The following are the essential steps:

• Create a ceph-deploy user on all systems.
• Enter this user in the sudoers list so that the user can execute all commands without entering a password.
• Enter an SSH key in this user's ~/.ssh/authorized_keys file so that the admin system can use it to log on.
• Configure ~/.ssh/config so that SSH uses this user and the right key file for the ssh host command.

You need to perform these steps on all four computers (i.e., on the storage servers, the client, and the admin server). You can supplement the EPEL repository [7] again and install Ceph 10.2.3 under CentOS on the storage servers. For Debian and Ubuntu systems (client and admin server), the Preflight section of the documentation provides instructions [9] that explain where on the Ceph website the repository for Ceph 10.2.2 is, which was used in the test.

The ceph-deploy tool collects data such as keys in the current directory. The documentation therefore recommends creating a separate directory for each Ceph cluster. The command

ceph-deploy new mon-node

starts the installation. Among other things, it generates the ceph.conf file in the current directory. For the test setup to work with two storage nodes (Ceph uses three by default so it has a majority), the configuration file needs the following line in the [global] section:

osd pool default size = 2

The installation of the software itself follows, which ceph-deploy also performs:

ceph-deploy install <admin-machine> <Fileserver_1> <Fileserver_2> <Client>

It expects a list of hosts as arguments on which ceph-deploy needs to install the software. The tool distributes the packages to the client and to the storage nodes, which are called OSDs (Object Storage Devices) in the Ceph world, as well as to the admin computer on which both the admin and monitor nodes run, with the latter having control.

Because ceph-deploy is also conveniently familiar with the various Linux distributions, it takes care of the installation details itself. To activate the first monitor node then, I use the command:

ceph-deploy mon create-initial

The command also collects the keys of the systems involved so that the internal Ceph communication works.

At this point, I generate the volume in two steps:

ceph-deploy osd prepare <Fileserver_1>:</directory> <Fileserver_2>:</directory>
ceph-deploy osd activate <Fileserver_1>:</directory> <Fileserver_2>:</directory>

These commands create the volumes on storage servers <Fileserver_1> and <Fileserver_2>. As with Gluster, the directories already exist and contain the filesystem where Ceph saves the data; then, I just need to activate the volume as an admin:

ceph-deploy admin <admin-machine> <Fileserver_1> <Fileserver_2> <Client>

The command distributes the keys to all systems. The argument is again the list of computers that need a key.

The still-missing metadata server is created with

ceph-deploy mds create mdsnode

The ceph health command checks at the end whether the cluster is working.

The Client

Three options are available for the Ceph client. In addition to the kernel space implementations of CephFS and Ceph RADOS, CephFS is available in a userspace version. The test fields Ceph RADOS and CephFS with a kernel space implementation.

To create and activate a new RADOS Block Device (RBD), I use:

rbd create lmtest --size 8192
rbd map lmtest --name client.admin

The device is then available under the path /dev/rbd/rbd/lmtest and can be equipped with a filesystem that the client can mount like a local hard disk.

However, the map command throws up an error message on the Ubuntu computer. To create the RADOS device, I need to deliver some arguments to disable features. The

rbd feature disable lmtest exclusive-lock object-map fast-diff deep-flatten

command produces the desired result after a few tests.

After that, a Ceph filesystem (CephFS) needs to be prepared. The first step is to create a pool for data and one for metadata:

ceph osd pool create datapool 1
ceph osd pool create metapool 2

The two numbers at the end refer to the placement group index. The cluster operator executes this command on the admin node running on the admin computer, and then generates the filesystem, which uses these two pools. The statement reads:

ceph fs new lmtest metapool datapool

The client now mounts the filesystem but then needs to authenticate it. The ceph-deploy command generates the password and distributes it to the client. There it ends up in the /etc/ceph/ceph.client.admin.keyring file.

The user name in the test is simply admin ; the client passes this onto the filesystem using the command:

mount -t ceph mdsnode:6789:/ /mnt/cephfs/ -o name=admin,secret=<keyring-password>

The filesystem is then ready for the test.

LizardFS

LizardFS is similar to Ceph in terms of setup. The data ends up on chunk servers (which should also include the hard disks), and a master server coordinates everything. The difference is the web interface mentioned at the start of this article (Figure 1).

To install version 3.10.4 under CentOS, you need to add the http://packages.lizardfs.com/yum/el7/lizardfs.repo repository to the /etc/zum.repos.d directory. On the master server (the admin computer), you can then install the lizardfs-master package on the client lizardfs-client and on the chunk servers with the hard disks lizardfs-chunkserver.

The configuration can be found under /etc/mfs on all systems. On the master computer, you can convert the mfsgoals.cfg.dist, mfstopology.cfg.dist, mfsexports.cfg.dist, and mfsmaster.cfg.dist files into versions without the .dist suffix. The contents of all the files are then suitable for a test setup; only the mfsmaster.cfg files require the entries in Listing 1.

PERSONALITY = master
ADMIN_PASSWORD = admin123
WORKING_USER = mfs
WORKING_GROUP = mfs
SYSLOG_IDENT = mfsmaster
EXPORTS_FILENAME = /etc/mfs/mfsexports.cfg
TOPOLOGY_FILENAME = /etc/mfs/mfstopology.cfg
CUSTOM_GOALS_FILENAME = /etc/mfs/mfsgoals.cfg
DATA_PATH = /var/lib/mfs

The configuration files are waiting in the /etc/mfs directory on the chunk servers. Only the MASTER_HOST entry, which must refer to the master, belongs in the mfschunkserver.cfg file. You can enter the directories where Lizard is to store data on the chunk server in the mfshdd.cfg file. Because the service is running under the mfs user ID, the operator needs to make sure this user has write rights to the directories used.

Finally, you can start the master and chunk server services on the respective computers:

systemctl start lizardfs-master
systemctl start lizardfs-chunkserver

Everything is therefore ready, so the client can mount and access the filesystem:

mfsmount -o mfsmaster=lizard-master <Mountpoint>

To use the web interface mentioned at the beginning, install the lizardfs-cgiserv package which, as a server, then accepts connections to port 9425.