Continuous Progress

Workflow

The workflow in Zuul is similar to that in standard CI/CD tools like GitLab CI/CD: The developer pushes the change for a review, which triggers a CI/CD tool. The tool then proceeds with steps configured as required for judging whether a change is valid or not. If the flow succeeds, you receive approval; otherwise, you get a denial. Now it is up to the developer (and probably other human reviewers) to decide when this change is merged.

In Zuul, however, this is only the first part of the process, called "verification." When a change is approved by verification and accepted by reviewers, you can proceed to the second part known as "gating." The goal of gating is to perform an exhaustive number of tests that confirms the change is production ready before applying it. (See the "Gating" box.)

Gating

A gating flow example (Figures 3-10).

Figure 3: Four changes are to be applied on the master branch and therefore deployed on production. Assume the changes are independent and non-conflicting.

Figure 4: The first two changes are verified and approved by reviewers. Gating comprises three parallel tests, as indicated by the stacked blue lines. When the changes are ready, Zuul applies them in serial order in its Merger service and starts gating, as follows: master + change 1; master + change 1 + change 2.

Figure 5: Gating of changes 1 and 1+2 is ongoing, but two more arrive: changes 1+2+3 and 1+2+3+4. If resources are sufficient, they will take place in parallel.

Figure 6: Flow change 1 is going well, but change 1+2 fails in one test!

Figure 7: Zuul aborts the two last flows, because they are based on change 2. Unless a Depends-On footer is present, this action is the default and logical action, because there is no use testing something that failed at some point. Of course, flow 1 continues and completes.

Figure 8: The successful gating of change 1 is reported, and the change is merged into the master. Developers need as much feedback as possible, so the gating of change 1+2 continues. In the meantime, changes 3 and 4 are rebased onto the last successful gating – change 1 – and begin from scratch.

Figure 9: Gating continues smoothly: Change 2 issues and successes are reported; the change 3 flow is completed and merged to the master (on top of change 1, of course).

Figure 10: Finally, all changes have gone through gating, with only change 2 denied, which the developer can reclaim, fix, and push for review once again.

Web UI

Previous Zuul iterations had Jenkins as the graphical user interface, but OpenStack parted with Jenkins and enforced the implementation of a new service responsible for displaying the current tasks.

Figure 11 shows the main view. Running tasks appear in the columns (except the last one, in this case). Note that the checks in column 1 are independent, but the gate pipeline in column 2 combines its tasks.

Figure 11: The main dashboard of the Zuul 3 Web component.

The last column displays less commonly used pipelines that come into play when Zuul acts on different events, including merging changes to the master.

If you take a closer look at a chosen task by clicking on it, it expands to show the jobs and the status of each (Figure 12).

Figure 12: A closer look at the job statuses in one of the pipeline tasks.

In the menubar at the top of the screen, the Projects menu takes you to a list projects of that Zuul monitors, displaying the hosting platform (GitHhub or Gerrit) and how Zuul trusts its content. Usually there will be only one trusted repo.

The Jobs menu takes you to a list of all jobs known to Zuul, which might be helpful if you are adding a new test to your pipeline: Perhaps someone has written something similar that you could reuse.

The Labels and Nodes menu items are interconnected. The first lists all possible labels you can use to define where jobs ought to be executed, and Nodes lists all nodes known to Zuul, with their labels and other details.

Finally, the Builds item lists all recently completed job,s and Buildsets shows all recently finished pipelines, along with their target (which commit), date, and result.

Summary

The Zuul system, originally developed to manage the massive multiple-contributor OpenStack software platform, has been updated to handle any complex collaborative software project. Zuul 3 is currently in use by multiple international companies (e.g., BMW, Software Factory, and, of course, OpenStack Foundation). In all cases, Zuul has proved to be a reliable and highly scalable solution [10].

With Zuul, "Humans don't merge code, machines merge code" [11]. Moreover, it works with complex code sets: Zuul can coordinate integration and testing across multiple, at first glance independent, repositories. Moreover, Zuul has a growing community. Thanks to a concept similar to Ansible roles [12], community members can reuse existing Zuul roles (e.g., pushing to the Docker registry) or write and share their own playbooks.

To sum up, Zuul 3 [13] is a free-to-use, highly scalable, and flexible solution that fulfills all three modern software development paradigms: continuous integration, continuous delivery, and continuous deployment. Projects that use GitHub or Gerrit repository hosting services can reasonably consider introducing Zuul into the development process.