Server Configuration Management with puppet

The configuration management craze is all over the start-up world, but many, many computers in this world are not managed in any way apart from by very careful sysadmins. I have nothing against them – I am glad that they’re careful. But I’m not careful, so I need to cheat: I need config management.

Config management sets up servers. It installs stuff, configures it, and makes sure it is working as you requested.

The important word is: you . It sets it up as you requested.

I think it is this that scares people away. Admins have learned how to configure a server by installing things manually, and automating the configuration steps can lead to uncertainty. Puppet takes a little time to set up, but I found that once I understood what the hell it was doing, I felt much more able to trust it – even on legacy servers.

So what I’ve written here is the really simple, fast and impatient guide to getting stuff with puppet. Once you’ve worked through this, you will be able to install and configure most of the major packages out there. It’s really very simple.

The Really Fast Intro to puppet

A lot of admins are using puppet for increasingly complex systems. The technology has picked up pace in the past year, but for those of us who have either used it a little or not at all, puppet requires a heap of learning.

In this whirlwind tour, I want to convince you that you should start using puppet to manage everything, right now. This is a tour of the terminology and a high-level description of what puppet does. Instead of treating this as a fresh installation that is going to run a really complex server, I’ll take you from installing puppet locally and using it to control just a few things. Puppet is a really strong tool, and can do great things, but until you trust it, you’ll treat it with hesitation.

Configuring a server is largely about writing text to files, and this is one of the many things puppet can do for you. Puppet takes a description of what should be there, compares it with what is there, and changes it to match. For a simple example, configuring an Apache site might require the existence of the following file:

/etc/apache2/sites-enabled/my-site.conf

Or, you might want to make sure a module is configured as you desire:

/etc/apache2/conf.d/some-module.conf

In both cases, all that’s required is that that file exists and the content is as you want it. Would that be enough to run your site? Probably not, because for this command to work, you’ll need to install Apache plus maybe a couple of modules. You also need the document root to exist.

A more complete list of what you need to have to get your site working is:

• apache
• some modules
• a couple of config files
• a file in the document root

Puppet gives you building blocks to set all this up.

Install puppet

Start by installing the latest version of puppet. Not all versions have the same options, and you don’t want to spend your time translating between the options in different versions, so get the latest if you can. For the sake of not cluttering the Internet with information that will soon be out of date, I’m only covering the latest version at this time of writing, which is version 2.7.x.

Using Ubuntu, run

do-release-update

(*only* if you’re happy upgrading your server) and then:

apt-get update

Then install puppet with:

apt-get install puppet

Make sure you have the latest version installed:

$ puppet --version
2.7.1

This tour is focused on the puppet command line tool, because I want to make sure you can see the link between what’s in the puppet manifest files and what puppet does to the server. Understanding this relationship makes working with puppet on the small and large scale make sense, and it makes puppet considerably less terrifying.

Create the puppet Manifest File

We’ll start with a very simple puppet manifest file. Puppet files are named .pp . The syntax is very simple, looking a bit like JSON, but it is worth noting that a lot of puppet modules make use of ruby (including erb files) and other scripting tools to add more cleverness. We’re not there yet though …

The puppet cli tool runs manifest files, which are suffixed with .pp . When you want to apply a puppet manifest file to your server, apply it so that, if you create an empty file and run the following command:

puppet apply myfile.pp

puppet will successfully do absolutely nothing. Put the following in the file to spit out a message:

notice("Let's build a server!")

And run it:

$ puppet apply myfile.pp 
notice: Scope(Class[main]): Let's build a server!
notice: Finished catalog run in 0.01 seconds

Now create a simple and largely useless file in the same script. Drop the file into a home directory. Change the manifest file to:

    file { "/home/dan/hello.md":
      ensure => "present",
      mode    => '0664'
    }

And run it:

    $ puppet apply myfile.pp 
    notice: /Stage[main]//File[/home/dan/hello.md]/ensure: created
    notice: Finished catalog run in 0.02 seconds

Now have a look and you’ll see that /home/dan/hello.md exists. Using this technique alone, you can litter the server with all the files you need but keep the best settings in one place: in your puppet project.

This technique is hugely useful for deploying to new servers and tracking changes. Say, for example, you run some experiments and find that changing the memory limits on Apache, along with the max children improves performance. Instead of remembering this, you would put it in your puppet files and ensure that your config files are the ones that are in use.

Update the Message of the Day

Nothing impressive so far, so create a project with the following structure:

    ./files/
    ./manifests/
    ./manifests/init.pp

With the project structure created, I can move a simple file declaration into the init.pp file:

    $ cat ./manifests/init.pp:
    package { "update-motd": 
      ensure => installed
    }
    file { "/etc/update-motd.d/10-mymotd":
      ensure => "present",
      source  => "puppet:///modules/mypuppets/misc/10-mymotd",
      owner   => 'root',
      group   => 'root',
      mode    => '0755',
      require => Package['update-motd']
    }

Next I create:

    ./files/misc/10-mymotd

Containing:

    #!/bin/bash
    echo "Welcome to my puppet-managed server!"

In the preceding code, package asks puppet to install the update-motd package. file asks it to ensure that the file is present and has the permissions as described, but also that this is done after the package update-motd is installed.

The contents of 10-mymotd are placed in a separate file and referenced using puppet’s syntax as:

puppet:///modules/mypuppets/misc/10-mymotd

With only this and a little googling, you should be able to rinse through dozens of services, packages and configuration files, simply copying from your best configuration into puppet.

MySQL is just a matter of the mysql package plus your my.cnf file. Setting up /etc/logrotate.d/our-logs with puppet means you don’t have to remember it. Creating /etc/profile.d/binpath.sh will mean that a customization no one can quite remember (adding s3cmd to the PATH) doesn’t have to be remembered. Most stuff on servers is just files, which is what makes puppet so useful.

Now for the following line:

require => Package['update-motd']

This line means that the corresponding file declaration won’t be run until the package update-motd is installed. Puppet doesn’t run top-to-bottom but compiles a list of dependencies and runs the setup in dependency order. This approach lets you chain together dependencies and make sure things are installed in the right order. It also means that running puppet apply twice isn’t the same as running puppet apply on a vanilla system twice, which can be painful for debugging. I don’t have room for the long answer here, but the short answer is: you should use something a virtualization tool like EC2, Vagrant, or VMware to test your puppet configuration on vanilla systems before you trust them.

Install and Configure Something Useful

So far I’ve created a useless file and then created a mostly harmless file and installed a package. Now that I have a small puppet project, I’ll install something useful and fairly typical.

Start a new project with the following lines in the init.pp file:

    package { "apache":
      ensure => present,
      name => $apache,
      require  => Exec['apt-get update']
    }
        exec { 'apt-get update':
    }

Next change the default configuration of apache:

    file { '/etc/apache2/sites-enabled/000-default':
        ensure  => present,
        source  => 'puppet:///modules/myproject/apache/000-default',
        owner   => 'root',
        group   => 'root',
        mode    => '0664',
        require => Package['apache'],
        notify  => Service['apache']
    }

And so we have to create the 000-default file:

# Nothing here anymore.
# Ubuntu normally has a default on /var/www but we don’t want it.

Now I’m going to create a document root and put an HTML page in it. This could be your app or just a holding page, but it gives you a starting point.

    file { '/var/www/index.html':
        ensure  => present,
        source  => 'puppet:///modules/myproject/web/index.html',
        owner   => 'root',
        group   => ‘root’,
        mode    => '0664',
        require => Package['apache'],
        notify  => Service['apache']
    }

Populate web/index.html with a little HTML – anything you like. Now try something on your own: populate the file /etc/apache2/sites-enabled/mysite with:

    <VirtualHost *:80>
      ServerName    exampledomain.localhost
      DocumentRoot /var/www/vhosts/
    </VirtualHost>

Now you’ve got a web server config in your puppet project. Run:

    puppet apply manifests/init.pp

And watch puppet do the hard work.

What Just Happened?

Once you get into the cycle of writing puppet code, applying it, and testing it, the puppet system becomes much easier to work with. It’s really just like write a code-refresh browser, but on the server.

A puppet module, like the one you just made describes resources: users, files, packages, and services. You go from a vanilla installation to a full configuration by layering resources on top of resources, which means they have to run setup in the right order. These are the dependencies, or meta parameters as puppet calls them: before , require , subscribe , and notify .

Puppet only makes changes when changes are needed. If the file exists and looks correct, puppet won’t make the change, but if it looks wrong, puppet will update the file when you run puppet apply .

Puppet configurations are distributed through modules. You’ll find a heap of modules at Puppet Labs Forge. With the latest versions of puppet, you can install these modules so that you don’t have to code complex implementations of MySQL from hand. For example, Puppet Labs has master/slave configuration setups.

Learning Puppet

Coming from a DB/web app world, I found that learning puppet was fairly annoying because each test cycle requires configuring a machine, so it is important to set up a good test environment. I personally like using Vagrant because I can launch a VM and test the puppet module by passing it to the machine.

If you don’t want to or can’t test locally, you can use any cloud hosting provider where you’re able to pass it boot-up-time commands such as the user-data on EC2. After a couple of hours getting your scripts together, you can bootstrap instances with a single script:

    apt-get install -y puppet
    wget http://your-domain-with-your-stuff.com/your-puppet-module...
    puppet apply path/to/your/init.pp

That’s much nicer than being careful on servers.