Cloud Orchestration with Chef

Configure a Chef Client

To orchestrate the cookbooks recipes that you create, first you need to install software on your designated Chef workstation. This software is called the Chef Development Kit. I’m using an Ubuntu 16.04 system. You don’t have to – you can use almost any platform you wish. To install the requisite software, simply go to the following URL:

https://downloads.chef.io/chefdk

You’ll then be able to download the code, compare the hashes for the sake of security, and then get things going. My particular Ubuntu system requires that you meet the following dependencies:

• Ruby (included in the Chef Development Kit)
  Note: For Windows systems, you’ll need to make sure that that PowerShell is installed and enabled.

• Set the proper execution policy: To do so, issue the following command at your command prompt: Set-ExecutionPolicy RemoteSigned
• Install git. To obtain his code, go to: https://git-scm.com/downloads
• Start the Chef server Web user interface by issuing the following command: chef generate app. This code will download the starter kit, and make the workstation ready to start creating and managine recipes and cookbooks.

The next step is to create an organization name:

$ chef-server-ctl org-create chef "Stanger Network Consulting." 
  -f /tmp/chef.key

After this, you then need to create a user. In my case, I created a user named James Stanger:

$ chef-server-ctl user-create jstanger James Stanger jstanger@company.com
  Cr8*p@sswrd*here -f /tmp/jstanger.key

Now that you’ve created a certificate and a user, make sure that the PEM files are in the right place, which is the ~/.chef/ folder:

p /tmp/stanger-validator.pem ~/chef-repo/.chef \ 
   cp /tmp/stanger.pem ~/.chef/.chef.pem

Then, create the knife.rb file, which informs the system that you have the user name and certificate information to orchestrate files:

current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name                'stanger_workstation'
client_key               "#~/.pem/stangernet.pem"
validation_client_name   'stangernet-validator'
validation_key           "#~/.pem//stangernet-validator.pem"
chef_server_url          'https://api.chef.io/organizations/stangernet'
cache_type               'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path            ["#{current_dir}/../cookbooks"]

Then, create the cookbook repository:

$ mkdir ~/chef/-repo/

Finally, you need to download the SSL certificates from the Chef server:

knife ssl fetch

If you wish, you can then verify the installation by going to the chef-repo/ directory, and issuing the following command:

knife client list

You will then see a list of workstations that have access to your Chef architecture.

Configuring a Chef Server

You have the options to use a server hosted by Chef, or a stand-alone server. Two stand-alone versions are available:

1. Standalone – where all Chef software is on one system.
2. High availability, which federates front-end and back-end systems.

I’ve installed a stand-alone system for my particular production environment. But, for the purposes of this article, I’ve decided to install the workstation and server separately on my Ubuntu system. You can use Windows or Linux-based systems. I prefer Linux, so I sourced a 64-bit system with 10 GB of RAM and 20 GB of free space. To fulfill Chef’s prerequisites, I made sure I had:

1. Ports 80 and 443 both open.
2. A fixed-IP system with a host name that is fully resolvable.
3. The server configured as a functional Network Time Protocol (NTP) client.
4. Cron functioning as the default job scheduler.
5. A properly-functioning mail transfer agent. In my case, I used postfix. As long as yours is running and able to send e-mail to root and any other defined user, you’ll be fine.
6. An instance of PostgreSQL, with a standard local and group account.

Once you have these prerequisites complete, you then download the server software package from Chef by pointing your Web browser to the following URL:https://downloads.chef.io/chef-server. For the sake of this article, I downloaded the package into the /chefsource/ directory.

Become a root user, then install the package. Because I use an Ubuntu system, I use the following command:

$ sudo dpkg -i /chefsource/chef-server-core-6.1.deb

Note: CentOS or Red Hat users would, of course, use the rpm utility.

Once the software is downloaded, then start all of the services using the command:

$ chef-server-ctl reconfigure

Chef will take several minutes to start. Use the ps command with the aux switches to see if chef-server is running for sure. Once Chef is running, you’ll need to create an administrator account:

$ chef-server-ctl user-create jstanger James Stanger jstanger@company.com
  'this*is*the*password' --filename /bin/chef/private/.jstanger.pem

The above command creates an RSA key pair. The jstanger.pem file is the private key. Make sure to place this inn a safe location by specifying a proper location for Chef.

The next step is to create an organization file:

$ chef-server-ctl org-create stangernet 'Stanger Network Consulting' 
  --association_user jstanger --filename /bin/chef/private/jstanger.pem 

Then, you have a choice: You can either download the remaining Chef server packages as part of the installation process, or download them and install them later. I decided to install them as part of the process. So, on the server, you issue the following commands, in order:

$ chef-server-ctl install chef-manage
$ chef-server-ctl reconfigure
$ chef-manage-ctl reconfigure

Once you’ve installed the server, use your Web browser to go to the server’s URL, which would be the following, if you’re browsing it from the server itself:

https://localhost/cookbooks

You’ll see the Chef Server default page, as shown in Figure 3.

Figure 3: The Chef Server default page

Connecting to a Cloud Service

The next step is to connect your Chef environment to a cloud service, such as AWS, Azure, or Oracle. In this case, I’ll choose AWS, as I live in the Seattle area, and you would pretty much expect me to pick a provider from the Northwest.

The good news, is that Chef makes it easy to work with Amazon. Chef has images available for people who have properly licensed AWS. The images are available in the AWS Marketplace. The Amazon images, known as an Amazon Machine Image (AMI), are also pre-installed with Chef. To launch the AMI, you take the following steps:

1. First, get into your AWS marketplace. I’m assuming you already have your AWS login information, and know how to use the AWS interface.
2. Find the Chef Automate page and take the steps to navigate to the image.
3. Create a template, called a “role,” for your AWS instance, doing the following:
   a. Choose the Amazon EC2 service type.
   b. Attach the following policy: AWSMarketplaceFullAccess
4. Then, click on the following button next to the policy: Launch with EC2 Console
5. AWS will then ask you for the instance type. This can be:
   a. Amazon Virtual Private Cloud (VPC)
   b. SSH key pair
   c. IAM Role and assign a public address
6. You’ll want to choose option C. The key is to allocate enough space. In many cases, you’ll want to choose only about 40GB. If you plan on having many nodes, then a significantly larger size is a good idea: I’d use 80GB, if at all possible.
7. Then, configure the security group to include at least the following ports:
   a. 80
   b. 22
   c. 443
   d. 8989
8. Then, assign the AMI role for your company.

You can then launch the Amazon Machine Image (AMI) and begin deploying your Web server, database, and load balancing server.

Creating a Cookbook with Recipes Using knife

To begin creating a cookbook, go to the designated cookbook repository directory. we need to go to our ~/chef-repo directory on our workstation:

$ mkdir ~/chef/-repo/

Once you’re in this directory, you can then use Chef’s knife utility to create a cookbook. The knife utility is Chef’s main configuration tool when it comes to creating any cookbook. You also use it to connect to the Chef server.

Here is the syntax for using knife to create a cookbook:

knife cookbook create cookbook_name

I’ll be deploying an instance of Apache Server, so I’ll use a fairly logical name for the cookbook:

knife cookbook create apache2_server
** Creating cookbook apache2_server
** Creating README for cookbook: apache2_server
** Creating CHANGELOG for cookbook: apache2_server
** Creating metadata for cookbook: apache2_server

The above code creates a file and directory structure inside of the apache2_server cookbook. You can view all of the files by issuing the following commands from within knife:

cd cookbooks/nginx
ls

To implement a cookbook for your MySQL instance and your load balancing server instance, you would follow much the same procedure.