Continuous Progress

Zuul 3 brings a new approach to the DevOps challenge. (For more on these important DevOps concepts, see the "CI and CD" box.) Whereas other CI/CD tools like Jenkins, Bamboo, or TeamCity focus on constant building and testing, Zuul's main design goal is to be a gating mechanism. In other words, as soon as a proposed change is verified and approved by a human, it can undergo final tests and be deployed automatically to the production environment. Such a system requires a lot of computing power, but Zuul has been proven reliable and highly scalable by its designers: the OpenStack community.

CI and CD

CI and CD are terms that pop up frequently in discussions of modern software development (Figure 1). CI stands for continuous integration , a practice that focuses on making releases easier to prepare. CD can mean either continuous delivery or continuous deployment , and although these practices have a lot in common, they also have significant differences [1].

Figure 1: Simply put, continuous integration is part of both continuous delivery and continuous deployment. Continuous deployment is like continuous delivery, except that releases happen automatically.

In the continuous integration software engineering practice, developers integrate code into a shared repository several times a day to obtain rapid feedback on code feasibility. CI enables automated build and testing so that teams can work together rapidly on a single project [2] [3].

In the continuous delivery software engineering practice, teams develop, build, test, and release software in short cycles. Continuous delivery depends on automation at every stage so that cycles can be both quick and reliable [4].

In continuous deployment, qualified changes in software code or architecture are deployed to production as soon as they are ready and without human intervention. Developers can focus on building software, and they see their work go live minutes after they've finished working on it [5].

The general concept behind the Zuul series (Zuul 3 at the time of writing) is to provide developers a system for automatically building, testing, and finally merging new changes to a project. Zuul is extensible and supports a number of development platforms, like GitHub and Gerrit Code Review [6].

Zuul originated as OpenStack CI testing. For years, OpenStack, the Infrastructure-as-a-Service (IaaS) cloud [7], got all the attention. Over time, people began to realize that as impressive as OpenStack was, the CI system behind it also deserved attention, because it enabled contributors and users across many different organizations to work together and quickly develop dozens of independent projects [8] [9]. Finally, it was decided to spin off Zuul 3 from OpenStack and run it as a standalone project under the OpenStack Foundation.

Components

Zuul 3 is organized in the form of services (Figure 2). For small projects, all of the components can be running on a single host. For larger projects, it is better to distribute the entire solution onto several hosts.

Figure 2: Zuul 3 components dependency map.

All Zuul processes read the /etc/zuul/zuul.conf file (an alternative location may be supplied on the command line), which uses an INI file syntax. Each component may have its own configuration file, although you might find it simpler to use the same file for all components.

Zuul Scheduler

The heart of the system is Zuul Scheduler, which is responsible for listening to and receiving events from a Git hosting service like Gerrit or GitHub. During the initial inspection of an event (e.g., to determine whether an event is associated with any of the projects to be observed), Scheduler forwards the event to be handled by one of the Executors through the Gearman server. In the end, after the flow has been completed, Scheduler reports the results to Gerrit or GitHub.

Although Scheduler is one of the few components that cannot be replicated, it is designed to perform only lightweight tasks and should hardly ever be overwhelmed by load. The OpenStack Foundation states that a small host, even with 1GB of memory, should be enough for a standard composition. Nevertheless, according to available metrics, OpenStack's Scheduler at most requires 8GB. Regardless of the details, almost any machine today can provide enough resources for Zuul Scheduler to work without throttling.

Gearman

Gearman is a separate project, but Zuul has its own implementation inside Zuul Scheduler. Gearman is responsible for distributing tasks between services, which communicate through the Gearman protocol. Scheduler puts a task in Gearman's queue, and the next free Executor takes care of it. Similarly an Executor, after the change is processed, submits the result to the queue, so Scheduler can report the result to Gerrit. Although Gearman can be used as a separate service, the built-in service is sufficient for handling up to 10,000 tasks at once.

If the built-in server is used, other Zuul hosts will need to be able to connect to Scheduler on Gearman TCP port 4730. It is also strongly recommended to use SSL certificates with Gearman, because secrets are transferred from Scheduler to Executors over this link.

Zuul is stateless; hence, all ongoing tasks are kept in Gearman operating memory and are lost on service restart. If you want to restore a queue after a maintenance reboot, for example, you first need to make a snapshot with the built-in zuul-check.py script and load it back to Gearman after restart.