Container Validation with dgoss

In many cases you only have access to the physical or virtual servers at the application level. Containers have largely replaced running native applications. Packing the applications in container images and running them containerized makes everything immutable to upgrade and rollback on the fly, with fine-grained control over server resources. This scheme also greatly improves the packing density of applications to optimize the available server resources.

Containers present different kinds of challenges, though, because their true colors only show at runtime. Container images are always expected to be as complete as possible but have the minimum possible footprint to launch and keep containerized applications running dynamically. Acceptance testing becomes very important with containers because software version control and configuration drift are a problem. I myself have witnessed folks creating containers with images without any standards and then suffering major outages because of an unstable life cycle. The good news is that you can use your knowledge of goss to create Acceptance-as-Code gates of good quality for container workloads.

The dgoss shell wrapper over goss adds some extra functionality [5]. One benefit is that you can use dgoss to create an Acceptance-as-Code gate in your container image's build and release pipelines. The usual way to run dgoss is to download it and then launch it with your goss test configuration, but I'll use dgoss through a Docker driver image to run it containerized without downloading anything. To begin, I'll create a dgoss driver image with all the necessary artifacts in the working directory where the Dockerfile (Listing 7) and goss.yaml file (Listing 8) are located:

docker build -f Dockerfile_DGossDriver . -t dgoss0320driver

FROM alpine:3.12
LABEL "com.richnusgeeks.vendor"="richnusgeeks"
LABEL version="latest"
LABEL description="dgoss test driver docker image"
ENV GOSS_VERSION 0.3.20
SHELL ["/bin/ash", "-o", "pipefail", "-c"]
WORKDIR /tmp
RUN apk add --no-cache --virtual=goss-deps ca-certificates curl && apk add --no-cache bash tini && curl -sSLk -o /tmp/docker.tgz "https://download.docker.com/linux/static/stable/x86_64/$(curl -sSkL https://download.docker.com/linux/static/stable/x86_64/|grep '^ *<a'|grep docker|grep -v rootless|awk -F '\"' '{print $2}'|sort -nr|head -1)" && tar zxf docker.tgz && mv docker/docker /usr/local/bin && rm -rf docker docker.tgz && curl -sSLk "https://github.com/aelsabbahy/goss/releases/download/v${GOSS_VERSION}/goss-linux-amd64" -o /usr/local/bin/goss && curl -sSLk "https://github.com/aelsabbahy/goss/releases/download/v${GOSS_VERSION}/dgoss" -o /usr/local/bin/dgoss && chmod +x /usr/local/bin/*goss && mkdir -p /etc/goss && apk del goss-deps
WORKDIR /etc/goss/
ENTRYPOINT ["tini", "--"]
CMD ["dgoss", "-h"]

port:
  tcp:22:
    listening: true
    ip:
    - 0.0.0.0
  tcp6:22:
    listening: true
    ip:
    - '::'
user:
  sshd:
    exists: true
    uid: 101
    gid: 65534
    groups:
    - nogroup
    home: /run/sshd
    shell: /usr/sbin/nologin
process:
  sshd:
    running: true

Now you can see dgoss in action by running the command:

docker run -it --rm -v ${PWD}/:/etc/goss:ro -v /var/run/docker.sock: /var/run/docker.sock:ro -e GOSS_SLEEP=2 -e GOSS_FILES_STRATEGY=cp -e GOSS_OPTS='--color -format tap 'dgoss0320driver dgoss run dgoss0320driver sleep infinity

The failures flashed in the run (Figure 4) are from a quick testing criterion of sshd running in containers, which is not true in the case of this image. The image is not a continuously running service, and that's why I supplied a sleep command, so that the launched service container is there during dgoss operations, such as copying goss and goss.yaml into it and launching a goss test run in the newly launched service container. The dgoss behavior is configurable by a number of environmental variables.

Figure 4: Output from a dgoss test run for the driver container.

A more standard example of a popular Docker image is the high-performance Apache Spark analytics engine. To begin, pull the official Spark Docker image:

docker pull bitnami/spark:latest

Next, prepare the spark.yaml goss test configuration file in your current working directory (Listing 9).

port:
  tcp:7077:
    listening: true
    ip:
    - 0.0.0.0
  tcp6:7077:
    listening: true
    ip:
    - ::
  tcp:8080:
    listening: true
    ip:
    - 0.0.0.0
  tcp6:8080:
    listening: true
    ip:
    - ::
process:
  java:
    running: true

Now, run the command

docker run -it --rm -v ${PWD}/:/etc/goss:ro -v /var/run/docker.sock: /var/run/docker.sock:ro -e GOSS_FILE=spark.yaml -e GOSS_FILES_STRATEGY=cp -e GOSS_OPTS='--color -format tap 'dgoss0320driver dgoss run -e SPARK_MODE=master bitnami/spark

to launch a test run against the official Spark image. It looks a bit surprising that, although the Spark Java service is running, the master and user interface port tests are failing (Figure 5).

Figure 5: A dgoss test run for the Spark container.

I'll try once again by executing the command:

docker run -it --rm -v ${PWD}/:/etc/goss:ro -v /var/run/docker.sock:/var/run/docker.sock:ro -e GOSS_SLEEP=2 -e GOSS_FILE=spark.yaml -e GOSS_FILES_STRATEGY=cp -e GOSS_OPTS='--color -format tap' dgoss0320driver dgoss run -e SPARK_MODE=master bitnami/spark

Now things look better. If you have a Dockerized service that has startup delays to run and respond (e.g., Java, Ruby, etc.), then GOSS_SLEEP helps to delay before launching a goss run in the spawned service container. You should see all the IPv4 port tests passing now (Figure 6).

Figure 6: The dgoss test run with GOSS_SLEEP for the Spark container.

Another important dgoss feature is useful when you have a set of precondition tests. If you create the goss_wait.yaml file (Listing 10) in your working directory and execute the docker command shown in the previous section, it should wait for the tests in the goss_wait.yaml file to be passed. In this case, I ensured that no SSH functionality was part of the Spark image (Figure 7).

port:
  tcp:22:
    listening: false
    ip:
    - 0.0.0.0
user:
  sshd:
    exists: false
process:
  sshd:
    running: false

Figure 7: A dgoss test run with goss_wait.yaml for a Spark container.

Try to change a test in the goss_wait.yaml to true, and you should see dgoss abort the test run because of the test failure. The dgoss run after setting the sshd process running state to true in goss_wait.yaml is shown in Figure 8.

Figure 8: A dgoss test run with a failing goss_wait.yaml.

Container Stack Validation with dcgoss

Docker Compose is a popular tool that lets you define and run multicontainer applications. Another wrapper created over dgoss tests the containers of the entire Compose stack: Put the code in Listing 11 in your current working directory and run the command

docker build -f Dockerfile_DCGossDriver . -t dcgoss0320driver

to build the driver image.

FROM alpine:3.16
LABEL "com.richnusgeeks.vendor"="richnusgeeks"
LABEL version="latest"
LABEL description="dcgoss test driver docker image"
ENV DCPS_VERSION 2.12.2
ENV GOSS_VERSION 0.3.20
SHELL ["/bin/ash", "-o", "pipefail", "-c"]
WORKDIR /tmp
RUN apk add --no-cache --virtual=goss-deps ca-certificates curl && apk add --no-cache bash tini && curl -sSLk -o /tmp/docker.tgz "https://download.docker.com/linux/static/stable/x86_64/$(curl -sSkL https://download.docker.com/linux/static/stable/x86_64/|grep '^ *<a'|grep docker|grep -v rootless|awk -F '\"' '{print $2}'|sort -nr|head -1)" && tar zxf docker.tgz && mv docker/docker /usr/local/bin && rm -rf docker docker.tgz && curl -sSLk "https://github.com/docker/compose/releases/download/v${DCPS_VERSION}/docker-compose-linux-x86_64" -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose && curl -sSLk "https://github.com/aelsabbahy/goss/releases/download/v${GOSS_VERSION}/goss-linux-amd64" -o /usr/local/bin/goss && curl -sSLk "https://raw.githubusercontent.com/aelsabbahy/goss/master/extras/dcgoss/dcgoss" -o /usr/local/bin/dcgoss && chmod +x /usr/local/bin/*goss && mkdir -p /etc/goss && apk del goss-deps
WORKDIR /etc/goss/
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["dcgoss", "-h"]

Next, create the docker-compose.yml file shown in Listing 12 and the kafka.yaml file shown in Listing 13 to bring up the ZooKeeper plus Kafka container stack over which the goss tests are run.

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

port:
  tcp:2181:
    listening: true
    ip:
    - 0.0.0.0
  tcp:9092:
    listening: true
    ip:
    - 0.0.0.0
process:
  java:
    running: true

Now you can fire dcgoss to acceptance-test each service container,

docker run -it --rm -v ${PWD}/:/etc/goss:ro -v /var/run/docker.sock: /var/run/docker.sock:ro -e GOSS_FILE=kafka.yaml -e GOSS_SLEEP=5 -e GOSS_FILES_STRATEGY=cp -e GOSS_OPTS='--color -format tap' dcgoss0320driver dcgoss run zookeeper

and run all goss tests in the ZooKeeper service container. You should see that all the ZooKeeper tests pass (Figure 9).

Figure 9: The dcgoss test run for the ZooKeeper service.

To run an acceptance test for the Kafka container, use the command:

docker run -it --rm -v ${PWD}/:/etc/goss:ro -v /var/run/docker.sock:/var/run/docker.sock:ro -e GOSS_FILE=kafka.yaml -e GOSS_SLEEP=5 -e GOSS_FILES_STRATEGY=cp -e GOSS_OPTS='--color-format tap 'dcgoss0320driver dcgoss run kafka

You should see all the Kafka tests passing now (Figure 10).

Figure 10: The dcgoss test run for the Kafka service.

I have combined the ZooKeeper and Kafka goss tests in a common file for quick testing, showing ok , not ok , and skip test results, but you could separate the tests into their respective gossfiles and mention the corresponding file with the GOSS_FILE environment variable. Finally, to clean up the running service containers, use the command

docker ps -q|xargs -I % docker rm -f %

to say goodbye to dcgoss.

Last but not the least, Kubernetes has become a de facto platform to run and manage containerized services at scale, so the goss project also provides another wrapper in the form of kgoss to acceptance-test containers running in Kubernetes pods. You need to run kgoss on a machine with kubectl already installed and configured to talk to your local or remote Kubernetes cluster. The kgoss documentation [6] clearly describes the installation and usage.

Conclusion

Server misconfigurations can create a lot of havoc, resulting in big losses. Goss is an elegant modern solution to bake acceptance testing in your servers. The validation tool is highly dynamic, supporting templating and serving reports on an HTTP endpoint. The icing on the cake is its added value when integrated with modern cloud tools like Ansible, Molecule, Kitchen, and Packer. The goss wrappers created for services running in containers are very useful for quality gating in modern build-and-release pipelines.