Building Helpers

The open source community offers two projects for routine administrative tasks that complement each other perfectly: Puppet and Foreman. Puppet [1] is an open source configuration management toolset and framework written in Ruby for the management and configuration of servers, whereas Foreman [2] is a lifecycle management tool.

Puppet

Puppet gives admins an approach to installing and configuring additional software that is relatively independent of the underlying operating system. The supported operating systems are the most popular Linux distributions, Unix, and now also Windows.

A typical Puppet environment comprises a master (server) and one or more agents (clients). Communication between the master and agent is secured with self-signed SSL certificates. The master stores a manifest that describes the role and the target state of the client. A target state would be, for example, "web server with PHP and local users with SSH keys." The manifest can be written in a domain-specific language (DSL).

Additionally, Facter [3] runs on the client. Facter collects information about the system and forwards it to the Puppet master when Puppet runs. On the basis of collected facts, the master can generate a dynamic catalog from the pre-defined manifest. In turn, the catalog is executed on the agent, which issues the correct commands based on the Facter information. Facter collects many facts about the operating system, hardware, software, network, and so on. If you need more information, you can define custom facts. For example, if you want to install Apache2 as a web server, an excerpt from the manifest looks like this:

package { "apache2" :
     ensure => "installed",
 }

For Debian, Puppet would select the Apt provider for the package and then run apt-get install apache2. On Red Hat, the Yum package manager would replace Apt.

However, running yum install apache2 would lead to an error message, because the package on RHEL is not called apache but httpd. To design this manifest for multiple operating systems, more facts are necessary. Facts can be used in the manifest as variables for decisions (Listing 1).

01  **case $operatingsystem {
02  ** ** ** **/(Debian|Ubuntu)/: {
03  ** ** ** ** ** **$apache_package_name = 'apache2'
04  ** ** ** **}
05  ** ** ** **/(RedHat|CentOS|Fedora)/: {
06  ** ** ** ** ** **$apache_package_name = 'httpd'
07  ** ** ** **}
08  ** **}
09
10 package { $webserver_package_name :
11         ensure => "installed",
12 }

The Puppet DSL offers inheritance, classes, modules, arrays, variables, parameters, templates, and scopes very much like a programming language. In other words, you have plenty of opportunities to make your manifests flexible and modular. If you take this one step further, you can ensure that the configuration for Apache is rolled out including any virtual hosts and that the web server is restarted at boot time and after changes to the Apache configuration.

Both the apache::service and apache::config classes require the apache::install class to run. The apache::install class only installs the httpd or apache2 package. A logical, contiguous set of classes is referred to as a module (Listing 2).

01 class apache {
02         include apache::params, apache::service, apache::install
03 }
04
05 class apache::params {
06  ** **case $operatingsystem {
07  ** ** ** **/(Debian|Ubuntu)/: {
08  ** ** ** ** ** **$apache_package_name = 'apache2'
09  ** ** ** ** ** **$apache_service_name = 'apache2'
10  ** ** ** ** ** **$apache_config_name = '/etc/apache2/apache.conf'
11  ** ** ** **}
12  ** ** ** **/(RedHat|CentOS|Fedora)/: {
13  ** ** ** ** ** **$apache_package_name = 'httpd'
14  ** ** ** ** ** **$apache_service_name = 'httpd'
15  ** ** ** ** ** **$apache_config_name = '/etc/httpd/httpd.conf'
16  ** ** ** **}
17  ** **}
18 }
19
20 class apache::config {
21  ** **file { $apache::params::apache_config_name:
22  ** ** ** **notify => Class[apache::service'],
23     source => "puppet:///modules/apache/${apache::params::apache_config_name}",
24  ** **}
25 }
26  **
27 class apache::service {
28  ** **service { $apache::params::apache_service_name :
29  ** ** ** **ensure => 'running',
30     enable => 'true',
31  ** **}
32 }
33
34 package { $webserver_package_name :
35          ensure => "installed",
36 }

After adapting your manifest to suit your needs, you can easily set new or additional servers, or the same server after reinstalling, to the defined target state. Puppet also has a large community, and you can use existing modules from Puppet Forge or GitHub.

In summary, Puppet is a tool that allows you to configure a server in minute detail. But Puppet does not handle the actual operating system installation; this is where Foreman enters the game.

Foreman

The Foreman lifecycle management tool covers the complete lifecycles of physical and virtual servers, from creating a host and installing the operating system, through Puppet-based management, to deletion.

Installation of Foreman can be handled by an installer directly from the sources of the Git repository, or you can use the pre-built packages. Foreman interacts closely with Puppet, but also with other pre-existing infrastructure elements (Figure 1); it is highly dynamic and can be used with more or fewer features, depending on the requirements.

Figure 1: The interaction of the components in an environment that uses Foreman.

For some of its features, Foreman relies on a smart proxy. A smart proxy is a server that, for example, handles DNS record entries or provides the PXE installation. Any number of smart proxies can be installed on different servers or the same server as Foreman itself.

As with Foreman, you can draw on pre-built packages for the supported distributions or install directly from source. Communication between Foreman and a smart proxy is handled by a RESTful API. If you have, for example, a multiple subnet network without DHCP relay, you will need to use multiple smart proxies to handle automatic installation on each network segment. The smart proxy features include:

• DHCP: Handles automatic assignment of IP addresses on defined subnets or IP address ranges.
• DNS: Provides name resolution for forward and reverse host records. Only ISC-compatible products, such as BIND and Microsoft DNS servers, are supported. The nsupdate tool handles the update to a dynamic zone; it is authenticated on the name server using a pre-defined and configured key.
• TFTP and PXE: Silent installations require a configured DHCP server and can install an operating system on a PXE-bootable server. The installation is handled by the TFTP server for the available image and is then seeded via Kickstart (Fedora, RHEL) or Preseed (Debian, Ubuntu) with the predefined installation responses. Besides Kickstart and Preseed, other techniques can be used, such as JumpStart.
• PuppetCA: Thanks to this feature, Foreman can sign the Puppet SSL certificates between master and agent. After installing the operating system, Puppet can be run without manual intervention to roll out the configuration from the Puppet manifest.

Features without Smart Proxy include:

• ENC: Puppet provides an external node classifier. It runs a script that reads and outputs the class assignment from another data source. Using the web interface in Foreman, you can create a host and assign Puppet classes to it.
• CMDB: Foreman can also be used as a configuration management database, thus removing the need for separate inventory maintenance. The inventory typically takes the form of the Facter information. Missing or additional information can be easily mapped with custom facts as mentioned previously.
• Compute Resources: Foreman can provision physical and also virtual machines in private or public clouds. Instead of a bare-metal server, it then directly deploys, for example, an AWS EC2 instance in the Amazon cloud. VMware or OpenStack are also supported.
• Reporting: The report from a Puppet agent is usually sent to its puppet master, and the report can be sent to Foreman for viewing and statistics.
• Auditing: The Audit feature lets you view changes by colleagues or clients.
• LDAP: Foreman user authentication can address an existing LDAP or Active Directory and thus use a central authentication authority.

Saving Time

Installation and integration into the existing infrastructure relies on having an overview of all the components and often takes some time. However, once you have installed all the components and features, Foreman and Puppet can save time.

If you need a new Debian server application, for example, an administrator can log in to Foreman using Active Directory credentials and complete the form to create a new host. This involves choosing a hostname, the operating system, and the Puppet modules. The next free IP address is suggested on the basis of the domain assignment. The Puppet modules could be ssh, apache2, php, or haproxy.

Once you have confirmed the form, an entry that installs the server with the assigned MAC address and Debian Linux is generated on the PXE/TFTP server. The DHCP server has already set up a static lease for the server and will assign the IP address when the server requests it. The server boots and is installed by Preseed. Then, the preparations for the upcoming first Puppet run are completed on the puppet master.

You could extend the script here to add the server to your monitoring system. After a reboot, Foreman knows that the server has been installed and changes its PXE boot record so that the server will boot from disk rather than from the installation image.

After this, the puppetd daemon runs and immediately triggers a Puppet run. The Puppet agent asks its puppet master for its manifest and executes the manifest. The report on the successful installation of the puppet classes is then returned to Foreman, so the administrator can track the status of the server through the web interface.

Any errors would be seen there immediately. The Puppet modules have told the server to install Apache2 with PHP; all the vhosts were also set up, and their information was communicated by the haproxy module to the load balancer and configured at the same time. The server can thus enter production without any delays.

The administrator can then try an SSH login to the server to make sure everything is working. The SSH login uses the server's SSH key and hostname because Foreman created the DNS entries and Puppet installed the administrator's SSH key. All told, the process should not take the system administrator more than 5 to 10 minutes, including installation.

The example described in this article clearly shows the value of this type of solution: The time benefits as well as the completeness of the setup are impressive. With a manual setup, mistakes will happen, and even simple ones can cause problems sooner or later.

Foreman is under active development and has just been updated to version 1.1. This release implements some long-awaited features, such as parameterized classes, support for Puppet 3, locations and organizations, and other improvements.