Config Management with Bash

07/12/2011 01:33 pm

The guys at Netflix, who built the ChaosMonkey app I talked about last time, say that, in the move to cloud, you have to be prepared to unlearn everything you know about hosting.

Cloud hosting differs in one important way. You can trace many cloud architectures, applications, and best practices back to this difference. Every time you plan an architecture, it comes back to the same thing.

Cloud instances are transient.

One of the tools to work around instances disappearing is configuration management. Anyone who has run more than a handful of servers will have used config management, whether home-spun or through a tool like Chef or Puppet.

So configuration management isn't really a cloud-specific topic, but it does make things a hell of a lot easier for you.

Configuration management manages the O/S and software stack running on top of each instance. Large data-centers have used this kind of thing for years to update hundreds of servers at once because it saves time. For anyone working with just a few servers, it doesn't make sense to spend time and money on something that you can do in a few minutes manually.

But cloud instances are transient. They will probably die, and when they do, who's going to configure them? Not you, if you have any sense. The config management kicks in, installs all of the required packages, and uses your preferred configuration.

You can use tools such as Chef and Puppet, but to start with, let's do the poor man's configuration using a bash script.

The following script - ok pseudo script - installs Apache, grabs your code from S3, and starts the server up.

apt-get -y install apache2
cd /tmp
wget -O your-code.tgz &nbsp;<a href="http://my-bucket.s3.amazonaws.com/your-project.tgz">http://my-bucket.s3.amazonaws.com/your-project.tgz</a>
mkdir tmp-extract && cd tmp-extract
tar xzf your-project.tgz
mv * /var/www/vhosts/
service apache2 restart

Once this is done, the instance is up and running with your app.

To inject this into an EC2 instance when it boots up, thereby creating an cloud instance that actually does something, you can make use Ubuntu's CloudInit script.

If you're on AWS, you can punch in the user data on boot up (Figure 1).

CloudInit makes the instance do stuff when it boots up, which is a really good way to get around bundling everything into images all the time.

To use this, just shove the script from above in the “runcmd:" section - e.g.

runcmd:
- apt-get -y install apache2
- cd /tmp
- wget -O your-code.tgz http://my-bucket.s3.amazonaws.com/your-project.tgz
- mkdir tmp-extract && cd tmp-extract
- tar xzf your-project.tgz
- mv * /var/www/vhosts/
- service apache2 restart

Sticking with the poor man's config management system, you can still make things a little nicer by pushing the script to S3 and then downloading it from there:

runcmd:
- wget -O boot-up-script.sh http://my-bucket.s3.amazonaws.com/boot-up-script.sh
- chmod +x boot-up-script.sh && ./boot-up-script.sh

This way you can version the script in git, try it out on a vanilla instance (and isn't cloud all about trying stuff out cheaply). When you're happy, push it up to S3 and all your new instances will start up with the new version.

Soon I'll cover chef and puppet for config management, but these tools rely on your ability to get the app running 100% from a script. There can be no human help in the cloud.

Do you have a home-spun config system like this? Many do. Or are you using a different, off-the-shelf system? I'd like to hear how you've solved this problem.

Until next time...