« Previous 1 2 3 Next »
SDS configuration and performance
Put to the Test
Fun with ceph-deploy
The quick-start documentation [8] helps you get going with Ceph, whereas the ceph-deploy
tool takes over installation and configuration. Because it uses SSH connections for the managed systems, you should look through its Preflight section [9] first. The following are the essential steps:
- Create a ceph-deploy user on all systems.
- Enter this user in the sudoers list so that the user can execute all commands without entering a password.
- Enter an SSH key in this user's
~/.ssh/authorized_keys
file so that the admin system can use it to log on. - Configure
~/.ssh/config
so that SSH uses this user and the right key file for thessh host
command.
You need to perform these steps on all four computers (i.e., on the storage servers, the client, and the admin server). You can supplement the EPEL repository [7] again and install Ceph 10.2.3 under CentOS on the storage servers. For Debian and Ubuntu systems (client and admin server), the Preflight section of the documentation provides instructions [9] that explain where on the Ceph website the repository for Ceph 10.2.2 is, which was used in the test.
The ceph-deploy
tool collects data such as keys in the current directory. The documentation therefore recommends creating a separate directory for each Ceph cluster. The command
ceph-deploy new mon-node
starts the installation. Among other things, it generates the ceph.conf
file in the current directory. For the test setup to work with two storage nodes (Ceph uses three by default so it has a majority), the configuration file needs the following line in the [global]
section:
osd pool default size = 2
The installation of the software itself follows, which ceph-deploy
also performs:
ceph-deploy install <admin-machine> <Fileserver_1> <Fileserver_2> <Client>
It expects a list of hosts as arguments on which ceph-deploy
needs to install the software. The tool distributes the packages to the client and to the storage nodes, which are called OSDs (Object Storage Devices) in the Ceph world, as well as to the admin computer on which both the admin and monitor nodes run, with the latter having control.
Because ceph-deploy
is also conveniently familiar with the various Linux distributions, it takes care of the installation details itself. To activate the first monitor node then, I use the command:
ceph-deploy mon create-initial
The command also collects the keys of the systems involved so that the internal Ceph communication works.
At this point, I generate the volume in two steps:
ceph-deploy osd prepare <Fileserver_1>:</directory> <Fileserver_2>:</directory> ceph-deploy osd activate <Fileserver_1>:</directory> <Fileserver_2>:</directory>
These commands create the volumes on storage servers <Fileserver_1>
and <Fileserver_2>
. As with Gluster, the directories already exist and contain the filesystem where Ceph saves the data; then, I just need to activate the volume as an admin:
ceph-deploy admin <admin-machine> <Fileserver_1> <Fileserver_2> <Client>
The command distributes the keys to all systems. The argument is again the list of computers that need a key.
The still-missing metadata server is created with
ceph-deploy mds create mdsnode
The ceph health
command checks at the end whether the cluster is working.
The Client
Three options are available for the Ceph client. In addition to the kernel space implementations of CephFS and Ceph RADOS, CephFS is available in a userspace version. The test fields Ceph RADOS and CephFS with a kernel space implementation.
To create and activate a new RADOS Block Device (RBD), I use:
rbd create lmtest --size 8192 rbd map lmtest --name client.admin
The device is then available under the path /dev/rbd/rbd/lmtest
and can be equipped with a filesystem that the client can mount like a local hard disk.
However, the map
command throws up an error message on the Ubuntu computer. To create the RADOS device, I need to deliver some arguments to disable features. The
rbd feature disable lmtest exclusive-lock object-map fast-diff deep-flatten
command produces the desired result after a few tests.
After that, a Ceph filesystem (CephFS) needs to be prepared. The first step is to create a pool for data and one for metadata:
ceph osd pool create datapool 1 ceph osd pool create metapool 2
The two numbers at the end refer to the placement group index. The cluster operator executes this command on the admin node running on the admin computer, and then generates the filesystem, which uses these two pools. The statement reads:
ceph fs new lmtest metapool datapool
The client now mounts the filesystem but then needs to authenticate it. The ceph-deploy
command generates the password and distributes it to the client. There it ends up in the /etc/ceph/ceph.client.admin.keyring
file.
The user name in the test is simply admin ; the client passes this onto the filesystem using the command:
mount -t ceph mdsnode:6789:/ /mnt/cephfs/ -o name=admin,secret=<keyring-password>
The filesystem is then ready for the test.
LizardFS
LizardFS is similar to Ceph in terms of setup. The data ends up on chunk servers (which should also include the hard disks), and a master server coordinates everything. The difference is the web interface mentioned at the start of this article (Figure 1).
To install version 3.10.4 under CentOS, you need to add the http://packages.lizardfs.com/yum/el7/lizardfs.repo
repository to the /etc/zum.repos.d
directory. On the master server (the admin computer), you can then install the lizardfs-master
package on the client lizardfs-client
and on the chunk servers with the hard disks lizardfs-chunkserver
.
The configuration can be found under /etc/mfs
on all systems. On the master computer, you can convert the mfsgoals.cfg.dist
, mfstopology.cfg.dist
, mfsexports.cfg.dist
, and mfsmaster.cfg.dist
files into versions without the .dist
suffix. The contents of all the files are then suitable for a test setup; only the mfsmaster.cfg
files require the entries in Listing 1.
Listing 1
mfsmaster.cfg
PERSONALITY = master ADMIN_PASSWORD = admin123 WORKING_USER = mfs WORKING_GROUP = mfs SYSLOG_IDENT = mfsmaster EXPORTS_FILENAME = /etc/mfs/mfsexports.cfg TOPOLOGY_FILENAME = /etc/mfs/mfstopology.cfg CUSTOM_GOALS_FILENAME = /etc/mfs/mfsgoals.cfg DATA_PATH = /var/lib/mfs
The configuration files are waiting in the /etc/mfs
directory on the chunk servers. Only the MASTER_HOST
entry, which must refer to the master, belongs in the mfschunkserver.cfg
file. You can enter the directories where Lizard is to store data on the chunk server in the mfshdd.cfg
file. Because the service is running under the mfs
user ID, the operator needs to make sure this user has write rights to the directories used.
Finally, you can start the master and chunk server services on the respective computers:
systemctl start lizardfs-master systemctl start lizardfs-chunkserver
Everything is therefore ready, so the client can mount and access the filesystem:
mfsmount -o mfsmaster=lizard-master <Mountpoint>
To use the web interface mentioned at the beginning, install the lizardfs-cgiserv
package which, as a server, then accepts connections to port 9425.
« Previous 1 2 3 Next »
Buy this article as PDF
(incl. VAT)