Tasty Recipes

Chef has been around since 2009 and is one of the most popular systems for configuration management. Originally the company was called Opscode, but it later changed its name to Chef. Since the end of 2020, Chef has belonged to US software company Progress, which added a DevOps solution to its broad portfolio of development tools. Many well-known companies rely on Chef [1] to manage their infrastructure or even parts of their cloud. Facebook, Alaska Airlines, SAP, and Carfax are prominent examples.

Chef is written mostly in Ruby and is based on a client-server architecture with various optional components [2]. It can also be used in a standalone setup for simple use cases. Chef is surrounded by a robust toolchain for software quality, testing, and software life-cycle management, and this toolset is constantly being developed by an active community and Chef itself. The portfolio now includes 4,000 freely usable cookbooks for the configuration of software and several hundred plugins and additional tools. Unlike other tools, Chef focuses solely on the management of operating systems and applications, but not on the provisioning of virtual machines (VMs) or cloud infrastructure.

Basic Terms

To begin, we need to clarify some basic concepts on which the following explanations are based. Because the term Chef originates from cuisine, the names of many tools have been borrowed from the kitchen world with a wink of the eye; even newcomers will get this right away. Cookbooks contain recipes, and a supermarket provides the community additional cookbooks free of charge.

In addition to the cookbooks maintained by Chef itself, more than 150 cookbooks maintained by sous chefs supplement the ecosystem. This group of volunteers continually updates and expands all of the projects they oversee and also holds weekly public meetings in Chef Community Slack. Furthermore, roles use recipes from cookbooks and connect them with each other. These roles can be used by a node, where a node can be a VM, a server, or generally any system on which software can be installed or configured.

A tool named Knife uploads cookbooks, lists resources, and uses the supermarket. To make sure you don't lose track of a node, the small Ohai tool lets you read out the current configuration of a system.

If needed, custom attributes supplement the information collected by Ohai. These attributes are used much like variables to influence execution. The connection between roles, attributes, run lists, and cookbooks is established by policy files, which contain all the information needed for all environments and can be versioned in Git.

Recipes use an easy-to-understand domain-specific language (DSL) named Chef DSL. If you can't use it, you have the option of falling back on Ruby as a programming language, although this is not usually necessary.

Components

A typical Chef environment contains different components depending on the desired scope [3]. The minimum setup that will let you proceed is the Chef Infra Client, which plays the role of the active component. If you run it locally on a node, it needs a run list, which contains at least one recipe.

Centralized management of run lists is handled by the Chef Infra Server, which registers and links nodes and has other advantages, such as centralized management as a source of truth for cookbooks, run lists, environments, roles, attributes, and policy files. In the past, the server also acted as a reporting tool, providing a quick overview of the completed Chef Infra client runs.

Having already discontinued the ever-expanding web interface for Chef Infra Server, Chef decided to launch Chef Automate, which initially served as a plain vanilla reporting tool for infrastructure changes (Figure 1). However, it has evolved so much over the past few years and been enhanced with so many features that it is expected to replace the Chef Infra Server web interface. Additions include a REST API, single sign-on, and integration as a source for the ServiceNow configuration management database (CMDB).

Figure 1: Chef Automate started life as a reporting tool but is now well on its way to replacing Chef Infra Server.

Over time, Automate has evolved to the point that you can even use it as a centralized compliance monitoring platform with more than 400 industry-standard benchmarks included, and it lets you communicate directly with and automatically query the APIs of Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) or with the target environments and systems over SSH and Windows Remote Management (WinRM). Chef InSpec takes care of continuous verification.

In addition to the compliance profiles included in Automate, the manufacturer also offers matching cookbooks in the form of Chef Compliance, which automatically solves identified problems. These remediation packages are officially certified by the Center for Internet Security (CIS) and provide enhanced IT security without additional development overhead.

You can support the internal Chef Community or use public Cookbooks through the Chef Supermarket, which provides an overview of existing cookbooks – no point in reinventing the wheel. Developing new cookbooks and policy files and operating a Chef environment does not require the complex installation of additional components such as Ruby or other tools. You just need to install Chef Workstation. After initialization with a short command, all the required components and packages are ready and can be used immediately. Windows even has a specially created Docker image.

Using Chef

Chef Infra works with an agent installed on the node that periodically queries the Chef Infra Server with the pull principle to see if more recent cookbooks, run lists, or attributes are available. If not, the cookbook cached locally during the previous run is restarted to check the desired state and restore it if necessary. If the Chef Infra Server has newer information, only the updated components are downloaded.

The client identifies itself to the server with a certificate over the HTTPS protocol; you do not have to punch holes in the firewall for each individual system, as is the wont with push-based systems. This approach ensures very secure connections and economical use of resources on the Chef Infra Server, which can already manage around 100,000 nodes as a standalone installation.

The previously mentioned cookbooks, like many other tools, follow a standardized structure that varies depending on the complexity of the task (Figures 2 and 3). At least two parts are optional: the metadata.rb file, which contains information such as the name and version of the cookbook, and one or more recipes in a recipes subdirectory that specify the desired state:

# metadata.rb
name 'webserver'
description 'Installs an Apache2 webserver'
version '1.0.0'

Figure 2: The structure of a minimal Chef cookbook.

Figure 3: The structure of a more complex Chef cookbook.

To make sure creating a new cookbook does not translate to unnecessary work, use the

chef generate cookbook <Name>

command. Because each organization has different requirements, the template for the command can be customized (e.g., with a centrally provisioned Gem package).

A cookbook always needs to represent a clearly defined task (e.g., "Install an Apache 2 web server"). The cookbook's subtasks are the individual recipes – for example, installing a piece of software, configuring it, upgrading it, or uninstalling it. Inside the recipes, Chef does away with DIY scripts and uses Chef DSL to abstract the system resources (Listing 1).

# recipes/default.rb
package 'apache2' do  # Resource type+package name
  version '2.4.43'    # Package version
  action :install     # Desired action
end

Besides the package resource, many recipes also contain the remote_file (file download), file (definition of files and, if necessary, their content), and template (for file templates with variables and logic) components. Chef automatically selects the correct procedure depending on the identified platform. When this article went to press, Chef Infra Client supported a total of 167 different resources across 12 platforms and all major architectures.

However, a cookbook must be able to respond not only to the platform, but also to different use cases. To that end are the automatic data from Ohai on the one hand and arbitrarily definable attributes that make the behavior dynamic on the other. Therefore, you can specify the version of a package, but a different value can be assumed if needed. The value here,

# attributes/apache2.rb default['apache2_version'] = '2.0.1'

can be used in recipes or templates by typing:

node['apache2_version']

Sensitive data like passwords or certificates can be handled with the integration of solutions such as Chef Vault, AWS Secrets Manager, or HashiCorp Vault.

The highly dynamic nature of cookbooks, because of different platforms and the use of attributes and secrets, places high demands on quality. Fortunately, Chef offers a complete ecosystem for testing. Static code and style analysis is handled by Cookstyle, unit tests by ChefSpec, and integration tests by InSpec. The Test Kitchen tool runs tests in parallel across multiple platforms, which makes test-driven development a breeze.

InSpec in particular plays a key role: Its tests can be executed completely independently of Chef (Listing 2).

# test/integration/default/apache2.rb
describe package('apache2') do
  it { should be_installed }
  its('version') { should eq '2.4.43' }
end
describe port(80) do
  it { should be_listening }
end

Once a cookbook has been authored and tested extensively, it can optionally be published in your own Chef Supermarket or shared publicly with the community. In Chef, the task that a node has to perform is defined by the policy files (Listing 3). They contain the required dependencies, the order of the recipes to be executed, the environments (e.g., test), and the attributes.

# Policyfile.rb
name 'webserver'
cookbook 'apache2', '~> 1.0'
run_list [
  'apache2::install'
]
default['apache2_version'] = '2.4.43'
# Webserver runs a newer version on test machines
default['test'] = {
  'apache2_version': '1.0.1'
}

The commands

chef install
chef push test webserver

resolve all external dependencies and handle the upload to the Chef Infra Server. As soon as the admin assigns a node to this policy by typing

knife node policy set server01 test webserver

the system is complete, and the Chef client can be executed.