Automation Tool

If you have to run standard tasks in an environment with a large number of systems (e.g., a compute cluster, a server farm, or a cloud environment) you might want a tool to help you save time and avoid duplication of labor. Logging in on each server and typing your commands hundreds of times manually is too slow, too error prone, and too inefficient. Many admins would rather have a tool that lets them run standard tasks on all clients in parallel, without typos and in a reproducible manner.

Tools such as Puppet, Chef, SaltStack, and Ansible provide this functionality through an agent running on the client, and a special description language lets the user define tasks or the target state. Rex takes a different approach. The Rex configuration management tool uses SSH as the transport medium and Perl as the command language, which means any computer can act as a Rex client without the need for additional software.

The fact that Rex doesn't rely on a client agent program also means the user won't run into conflicts between newer and older Rex versions. And you won't have to learn a new, specialized command language: As long as you know some Perl, you'll be ready to get started.

Installing Rex

Only the command center – the Rex host, which is sometimes called Rex Control Master – needs a few software modules. By the way, the Rex developers call the program (R)?ex , which is totally unpronounceable, so I'll settle for plain old Rex for the rest of the article. You can install these software modules either via a package manager in Linux or FreeBSD or via Git. If you're using a package manager, you'll want to include the Rex [1] repository to leverage automated updates by the distribution. Listing 1 shows how to install via Git. The advantage of using Git is that the Git sources have the freshest version. If you are accustomed to working with Perl, a third way to install Rex is through Perl's CPAN archive [2].

01 git clone https://github.com/krimdomu/Rex.git
02 cd Rex
03 perl Makefile.PL
04 make
05 make test
06 make install

First Steps

Commands are transmitted to the remote Rex client through a so-called Rex file. Listing 2 shows a fairly simple example. The first three lines define the name of the user who executes the commands, the password, and the authentication method.

01 user "root";
02 password "secret";
03 pass_auth;
04
05 group server => "hercules", "sugar";
06
07 desc "Get the uptime of all servers";
08
09 task "uptime", group => "server", sub {
10  my $output = run "uptime";
11  say $output;
12 }

You might notice that Listing 2 holds the root password in unencrypted form. Rex also allows authentication using SSH keys. For encrypted authentication, you need to create an RSA or DSA key pair without a password on the Rex host and then copy the public key over to the client (Listing 3). After that, a test login on the client without password should work.

01 root@hercules:~/.ssh# ssh-keygen -t rsa
02 Generating public/private rsa key pair.
03 Enter file in which to save the key (/root/.ssh/id_rsa):
04 Enter passphrase (empty for no passphrase):
05 Enter same passphrase again:
06 Your identification has been saved in /root/.ssh/id_rsa.
07 Your public key has been saved in /root/.ssh/id_rsa.pub.
08 The key fingerprint is:
09 9b:e4:e2:27:92:04:4a:9b:ee:82:cc:9f:4d:4b:4d:c1 root@hercules
10 The key's randomart image is:
11 +--[ RSA 2048]----+
12 |                 |
13 |    .            |
14 |    E            |
15 |    ..   .       |
16 |    ..o. .S      |
17 |    .o . oo o    |
18 |    = . +..+     |
19 |    o+ B.o..     |
20 |    o..o +.o     |
21 +-----------------+
22 root@hercules:~/.ssh# ssh-copy-id root@sugar
23 root@sugar's password:

Check ~/.ssh/authorized_keys to make sure you haven't added extra keys that you weren't expecting.

Then, change the first lines of the Rex file into the following:

user "root";
private_key "/root/.ssh/id_rsa";
public_key "/root/.ssh/id_rsa.pub";

The CPAN Net::OpenSSH module also supports the possibility of Kerberos as an authentication method.

Installing Packages

Listing 2 shows how you can use the Rex file to query the client for information such as the uptime. You can use the same technique to obtain other client values, such as the amount of free memory (free or vmstat), the fill factor of the hard disks (df), the network utilization (netstat), and the I/O performance (iostat). You can also easily filter and format the output with Perl.

If you want to archive the results, a tool like the System Activity Reporter sar or the Performance Co-Pilot PCP [3] would be more appropriate, because they are designed for long-term data handling. Sar comes with most Linux distributions (package sysstat), but it is not installed by default. So, you have to install it first, and Rex can even help with installing other tools.

Rex comes with several commands written in Perl and one of these commands is install, which you can use to install software package. Append the lines from Listing 4 to the Rex file. After that, type:

rex install_sysstat

01 use Rex::Commands;
02 use Rex::Commands::Pkg;
03
04 desc "Install sar (sysstat)";
05 task "install_sysstat", group => "server", sub {
06  install package => "sysstat";
07 };

The install command automatically takes care of the translation into real commands for each platform on which it runs. Thus, install works well with a heterogeneous group of servers, as long as each group member belongs to the supported systems (CentOS 5/6, Debian 5/6, Fedora, Gentoo, Mageia, openSUSE, RHEL 5/6, Scientific Linux, Ubuntu version 10 or greater, Solaris 10/11, FreeBSD, NetBSD, OpenBSD). Rex knows how to install packages on these platforms and will use the appropriate command (rpm, apt, pkg, emerge, urpmi,opkg, yum, pkgadd or zypper). However, all systems in a group have to use the same package name.