The principles of Dev(Sec)Ops are based on test-driven development, which states that you need to consider how you will test before you write a single line of code. Monitoring and alerting, as well as security compliance and detection tests, should be in place from the beginning. The ability to enact and clean up a VM-based environment quickly is important for unblocking on-demand development and testing (and is a vital part of continuous integration and continuous delivery or deployment, CI/CD).

Many admins like to model their infrastructure in advance from a personal laptop or lightweight test system, but multiple VMs can be quite demanding for a single system. For instance, if you have used the combination of Vagrant and VirtualBox for deploying and managing virtual machines, you probably realize the virtualization provided requires special hardware acceleration and significant computing resources not found in lightweight systems. In my own experience, only four or five VMs are enough to throttle even a nice four-core, 16GB MacBook Pro and make it scream like a Steam Engine.

Containers are a bit easier on resources because the devices and kernel are shared, and only the userspace is sandboxed. One solution that could stretch your test system a bit farther is Footloose, a tool that provides "containers that look like VMs" [1]. Footloose [2], which is built around Docker, runs systemd as PID 1 and uses ssh daemon to log into the container machines (containers that look like virtual machines). The advantages are few dependencies, except the Docker daemon, and no need for virtualization support from hardware. Footloose, which runs on both GNU/Linux and macOS, can help you model your virtual environment without overloading your hardware. You can even run dockerd inside a Footloose container machine.

Footloose

Running Footloose is very easy, with no installation step: Just grab the static binary and move it to any location configured in your system path (e.g., /usr/local/bin). To download the latest version of Footloose, move it to the system path, and make it executable (requires sudo privileges on GNU/Linux), enter:

curl -sSLk "https://github.com/$(curl -sSLk https://github.com/weaveworks/footloose/releases | grep download | grep linux-x86_64 | awk -F '"' '{print $2}'|head -1)" -o /usr/local/bin/footloose && chmod +x /usr/local/bin/footloose

If you don't have sudo privileges on your machine, use the following command to download Footloose to the current directory and use it from there:

curl -sSLk "https://github.com/$(curl -sSLk https://github.com/weaveworks/footloose/releases | grep download | grep linux-x86_64 | awk -F '"' '{print $2}'|head -1)" -o ./footloose && chmod +x ./footloose

Going forward, you could either prepand every Footloose command with ./ or add the current directory in your system path. You can find the instructions for this step many places online. Next, verify whether the downloaded binary works by executing the footloose command (Figure 1).

Figure 1: The footloose command on an Ubuntu 18.04 LTS laptop.

The creation of container machines is driven by a configuration file that declares various machine properties required by Footloose. Execute the command

footloose config create

to create a starting configuration in footloose.yaml in your current directory (Figure 2).

Figure 2: Basic configuration properties required by Footloose.

The privateKey parameter is the SSH private key created by Footloose to let you ssh into the created container machine(s). The count parameter denotes the number of desired container machines to be created. The names of the created container machines are a combination of the cluster name, the machines name, and the count index starting from 0. The image parameter is the docker image, with the systemd and the SSH daemons to instantiate the container machine. The portMappings parameter is about exposing your container ports on the localhost for access. You could further configure the default values of the properties as desired through various flags passed to the

footloose config create

command. Those flags are dumped through -h or --help suffix to the command (Figure 3).

Figure 3: The help screen for footloose config create.

The Footloose project currently provides the prebuilt container machine images [3] as quay.io/footloose/ , for CentOS7, Fedora 29, Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, Amazon Linux 2, Debian 10, and Clear Linux, but you are not limited to the prebuilt images. You could use any prebuilt dockerfile image to derive a customized image.

For example, I used the dockerfile in Listing 1 to create a customized local Rocky Linux 8 image to instantiate my container machines when moving from the old prebuilt CentOS7.

FROM rockylinux/rockylinux:8
LABEL "com.richnusgeeks.vendor"="richnusgeeks"
LABEL "com.richnusgeeks.category"="base"
LABEL version="latest"
LABEL description="docker image for RockyLinux to replace CentOS"
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN dnf install -y openssh-server libnsl && \
    dnf install -y 'dnf-command(config-manager)' && \
    dnf clean all
STOPSIGNAL SIGRTMIN+3
CMD ["/bin/bash"]

The Footloose configuration shown in Listing 2 uses the locally created rocklinux8 image with

docker build . -t rockylinux8

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 3
  spec:
    image: rockylinux8
    name: kafka%d
    networks:
    - cldinabox-demo
    portMappings:
    - containerPort: 22
    - containerPort: 2181
    - containerPort: 8080
    - containerPort: 9092
    - containerPort: 38080
    - containerPort: 52812
    - containerPort: 58080
    privileged: true
    volumes:
    - type: volume
      destination: /var/lib/docker

which brings up three container machines with hostnames kafka0 through kafka2, in which dockerd is later installed and configured (which requires the privileged flag). The configuration also exposes a number of ports for multiple services running in each container machine. You could also mix and match various kinds of container machines under the machines section in the same configuration.

To begin, create the required network, bring up the container machines, and dump their information with the commands (Figure 4):

docker network create cldinabox-demo
footloose create
footloose show

Figure 4: Rocky Linux machines.

You can clearly see the localhost port corresponding to each exposed container machine port. The command

footloose ssh root@<HOSTNAME>

gets you into any of the created container machines where you can run any normal GNU/Linux command (Figure 5). Finally, you can clean up the created cluster with

Figure 5: Accessing a container machine and running a command.

footloose delete

(Figure 6). You should feel comfortable now creating and using your container machines and becoming productive right away.

Figure 6: Shutting down the container machines.

Advanced Tooling with Footloose

I've created a solution over Footloose that uses wrapper scripts, Docker Compose manifests, and Ansible roles to spin up – on-demand – the fully configured container machines running various kinds of industry-standard cloud computing clusters (e.g., Cassandra, Consul, Elasticsearch, Kafka, Spark, etc.) [4]. This solution doesn't need or require you to know anything about Footloose or its configuration or download and install anything because everything runs in the various Docker containers. The top-level wrapper script just takes a simple footloose.cfg configuration (Listing 3).

# Name Count Image Networks Ports([hostPort:]containerPort)
kafka 3 rockylinux8 cldinabox-demo 22,2181,8080,9092,38080,52812,58080
spark 3 quay.io/footloose/centos7 cldinabox-demo 22,7077,8080,8081,38080,52812,58080
monitoror 1 rockylinux8 cldinabox-demo 22,52812,38080,58080,58880:58880
vigil 1 quay.io/footloose/centos7 cldinabox-demo 22,52812,58080,58888:58888
consul 3 quay.io/footloose/centos7 cldinabox-demo 22,52812,58080,58500
consulesm 1 quay.io/footloose/centos7 cldinabox-demo 22,52812,58080
hashiui 1 quay.io/footloose/centos7 cldinabox-demo 22,52812,53000,58080

The user provided config file just defines the name, count, image, and portMappings for your desired container machines; the rest of the magic is performed by this solution.

To get started, download the sources and go to the scripts directory:

git clone https://github.com/richnusgeeks/devops.git
pushd CloudInABox/ContainerMachines/scripts

You could use cd instead of pushd, but I prefer pushd for its intelligence. The wrapper script execution should show you a help screen (Figure 7).

Figure 7: Wrapper script help screen.

Now I can try to create a stack of 13 container machines to run clusters of Consul, Kafka, and Spark on the three nodes and ConsulESM, Hashi-UI, Monitoror, and Vigil (Listing 3) on a single node.

In a few tens of seconds,

./create_container_machines_stack.sh create

brings up your container machines. (It might take more time to create the necessary Docker images the first time only.) You'll also see Ansible trying to connect to every container machine host, verify that Python is usable, and return pong on success. Entering

./create_container_machines_stack.sh show

(this is the already familiar Footloose show command) dumps the info about the created container machines (Figure 8).

Figure 8: Info dump of 13 container machines.

A set of common components like monit (host monitoring/watchdog/autohealing), goss (host validations), and docker/docker-compose (running and managing containers) are working for each container machine. To initiate Ansible provisioning of these common components, use:

./create_container_machines_stack.sh test docker

The Monit dashboard (guest and guest login credentials) runs on container port 52812, the Goss HTTP endpoint container port is 58080, and the cAdvisor dashboard container port is 38080. You could find the respective local port to access these using

./create_container_machines_stack.sh show

Figures 9 and 10 show these web user interfaces on my laptop.

Figure 9: Monit dashboard webui page.

Figure 10: cAdvisor webui page.

Running the command

./create_container_machines_stack.sh config

creates a footloose.yaml file, which could be used directly with footloose on the command line. Executing

./create_container_machines_stack.sh delete

cleans up everything that was created. If you have some understanding of docker-compose and ansible, then you could easily add more functionality to this solution beyond the server configuration automation it is providing currently. This tooling will enable anyone to bring up and clean up the completely configured container machine stack quickly with a few commands.

Conclusion

Cloud-age techies need to be able to enact and clean up a working environment on a laptop or a VM. Footloose, along with the other wrapper utilities covered in this article, is the easiest way to accomplish this with cross-platform container machines. In fact, over the past few years I have been using and delivering "In a Box" local infrastructure stacks with the use of these tools to develop and verify various pieces of Dev(Sec)Ops automation and to remove impediments in the development and testing environments on demand.