Automation software such as Ansible is popular with admins, but occasionally other colleagues need to step in to perform jobs on servers, and not everyone is a big fan of the command line. Rundeck [1] is the perfect software tool for these tasks because it lets admins configure tasks that others can then perform, such as build processes or even simple things like rebooting a server. Rundeck's highly granular access controls let you configure the appropriate rights for users and groups, as well as configure and map complex workflows with dependencies (Figure 1).

Figure 1: A Rundeck server can be controlled using the web GUI or the API. In addition to local resources, external resources can be integrated.

Thanks to a series of plugins, Rundeck can also be used in conjunction with popular tools like Puppet, SaltStack, and Ansible. Rundeck integration for Foreman lets you export managed hosts from Foreman for Rundeck. Even cloud environments are not unknown territory, because Rundeck has plugins for Amazon Web Services (AWS). If you need high availability, multinode clusters, or centralized logging, you can also take a look at Rundeck Pro. Additionally, the Rundeck developers at SimplifyOps also offer support and advice for Rundeck.

Java Required

Rundeck is a Java program; thus, a Java Runtime Environment (JRE) or a Java Development Kit (JDK) must exist on the computer. I installed the free OpenJDK run-time environment on an Ubuntu 16.04 server:

$ sudo apt-get install openjdk-8-jre

Rundeck can be downloaded from the homepage as a JAR file or a Debian or RPM package before installing:

$ wget http://dl.bintray.com/rundeck/rundeck-deb/rundeck-2.6.7-1-GA.deb
$ sudo dpkg -i rundeck-2.6.7-1-GA.deb

Installing a package is practical because it integrates a Rundeck daemon into the system. The corresponding script (SysVinit) is in /etc/init.d; a config file for the Upstart init system resides in /etc/init/rundeckd.conf. However, Ubuntu 16.04 uses systemd as its init system, for which no startup file is supplied. Listing 1 shows a unit file for systemd that starts the Rundeck service. Systemd lets you set environment variables but does not support script execution, which you need to handle in the scope of the start command (ExecStart).

[Unit]
Description=Rundeck service
[Service]
Type=simple
ExecStart=/bin/sh -c '. /etc/rundeck/profile ; $JAVA_HOME/bin/java $RDECK_JVM -cp $BOOTSTRAP_CP com.dtolabs.rundeck.RunServer /var/lib/rundeck $RDECK_HTTP_PORT > /var/log/rundeck/service.log 2>&1'
[Install]
WantedBy=multi-user.target

Once the Rundeck server is running, you can log in with the URL http://rundeck-server: 4440 . To make changes, you need to run a text editor on the Rundeck server and edit the /etc/rundeck/realm.properties file. Amazingly, the software even allows a plain text password; alternatively, an MD5 hash is possible. The exact format is documented in the comments with the file. As an alternative to local user management, Rundeck supports a connection to LDAP or Active Directory via the Java Authentication and Authorization Service (JAAS).

The managed nodes are typically configured through text files in XML, JSON, or YAML formats. Alternatively, other providers are possible; again, Rundeck is flexible and open in this respect. If you manage AWS instances, you can use the Rundeck EC2 Nodes plugin, which queries active computers directly on Amazon. The Rundeck wiki on GitHub [2] offers some tips, such as how to scan a network with Nmap or use a script that produces a list of nodes.

In addition to storing a configuration in the filesystem, SQL databases are supported. Rundeck's flexibility shines here, too, because it uses the JDBC interface. MySQL, PostgreSQL, and other databases should thus work without trouble.

Projects and Jobs

Rundeck organizes work in projects and jobs. When you create a new project, the program asks for some basic parameters such as the Default Node Executor, which will probably be Secure Shell SSH in most cases. Here is also where you will find the SSH key storage path, which you can set with variables so that it can be used for all nodes involved in the project. Rundeck offers context variables for this, which are automatically assigned for each job execution (e.g., with the project name, job name, or username). You can assign the SSH key storage path as follows:

/keys/projects/webserver/nodes/${node.name}/${node.username}.pem

As an alternative to management in the web interface, you can create a project on the Rundeck server:

$ rd-project -p myproject --action create

Either way, after creating the project, a new directory named /var/run/projects/myproject appears on your hard drive; it contains the project.properties file with the project settings in the etc subdirectory.

Now, you need to save the keys for authenticating against the managed computers in Rundeck. You will find the corresponding menu on the configuration page of your project; use the gear icon at top right to access it. The project menu then appears on the left; among other things, it contains the Key Storage entry. In the main panel, navigate to the keystore so that the hierarchy matches the path stored earlier as the SSH key storage path. With the settings created previously, you would save each private key under the root.pem name.

By default, Rundeck stores the keys unencrypted in the filesystem (under /var/lib/rundeck/var/storage/) or in the database, but a plugin lets you encrypt the keystore (Figure 2). If the SSH key has a passphrase, Rundeck asks for it when running the job. SSH passwords are possible as an alternative to using private/public key authentication.

Figure 2: Keys and passwords can be encrypted for storage with the supplied plugin (bottom).

Configuring Jobs

You create the nodes running Rundeck jobs in a file on the server. To create a template, click on Nodes in the top menu of the web interface and search for existing nodes. The file /var/rundeck/projects/webserver/etc/resources.xml (which you can create manually, if necessary) must be available on the hard drive. The automatic create process makes localhost (i.e., the Rundeck server) part of the node list. A few of the possible node attributes are shown here:

<node name="localhost" description="Rundeck server node" tags="" hostname="localhost" osArch="amd64" osFamily="unix" osName="Linux" osVersion="3.13.0-83-generic" username="rundeck"/>

The name and hostname attributes, at least, are necessary, and the hostname can also include an IP address. Another important attribute is the username, which determines the account that Rundeck relies on to log in to the remote host via SSH. The ssh-key-storage-path attribute lets you configure the previously described key storage path for each node.

To create a new job, go to the Jobs menu and select Create a new Job . Alternatively, you can upload a job definition through the web interface in XML, JSON, or YAML format. In addition to a name and a description, you need to add at least one "step" in the Workflow, which can be a single command, a script in the web interface, or a script on the server. Other jobs can be added here, and you can determine what should happen if the executed step fails: cancel the whole job or proceed to the next step. As the execution mode, you can select between Dispatch to Nodes and Execute locally . In the first case, you need to specify a node filter to define the node on which the step runs. This web interface has many more options (e.g., deciding whether the job can be executed automatically at any given time).

Once you are done creating jobs, you can run them against the computers and choose between two log levels: Normal and Debug . After the run, Rundeck returns the success rate of the configured jobs it logs over the course of time. Reports and logfiles are archived but can, of course, also be deleted. Job definitions can be exported in the aforementioned file formats and imported back again. Rundeck can send alerts for processed jobs via email or HipChat. You can use webhooks to adapt messaging to suit your own needs and, for example, integrate messages into other web-based systems.