Zuul 3, a modern solution for CI/CD

Continuous Progress

Workflow

The workflow in Zuul is similar to that in standard CI/CD tools like GitLab CI/CD: The developer pushes the change for a review, which triggers a CI/CD tool. The tool then proceeds with steps configured as required for judging whether a change is valid or not. If the flow succeeds, you receive approval; otherwise, you get a denial. Now it is up to the developer (and probably other human reviewers) to decide when this change is merged.

In Zuul, however, this is only the first part of the process, called "verification." When a change is approved by verification and accepted by reviewers, you can proceed to the second part known as "gating." The goal of gating is to perform an exhaustive number of tests that confirms the change is production ready before applying it. (See the "Gating" box.)

Gating

A gating flow example (Figures 3-10).

Figure 3: Four changes are to be applied on the master branch and therefore deployed on production. Assume the changes are independent and non-conflicting.
Figure 4: The first two changes are verified and approved by reviewers. Gating comprises three parallel tests, as indicated by the stacked blue lines. When the changes are ready, Zuul applies them in serial order in its Merger service and starts gating, as follows: master + change 1; master + change 1 + change 2.
Figure 5: Gating of changes 1 and 1+2 is ongoing, but two more arrive: changes 1+2+3 and 1+2+3+4. If resources are sufficient, they will take place in parallel.
Figure 6: Flow change 1 is going well, but change 1+2 fails in one test!
Figure 7: Zuul aborts the two last flows, because they are based on change 2. Unless a Depends-On footer is present, this action is the default and logical action, because there is no use testing something that failed at some point. Of course, flow 1 continues and completes.
Figure 8: The successful gating of change 1 is reported, and the change is merged into the master. Developers need as much feedback as possible, so the gating of change 1+2 continues. In the meantime, changes 3 and 4 are rebased onto the last successful gating – change 1 – and begin from scratch.
Figure 9: Gating continues smoothly: Change 2 issues and successes are reported; the change 3 flow is completed and merged to the master (on top of change 1, of course).
Figure 10: Finally, all changes have gone through gating, with only change 2 denied, which the developer can reclaim, fix, and push for review once again.

Web UI

Previous Zuul iterations had Jenkins as the graphical user interface, but OpenStack parted with Jenkins and enforced the implementation of a new service responsible for displaying the current tasks.

Figure 11 shows the main view. Running tasks appear in the columns (except the last one, in this case). Note that the checks in column 1 are independent, but the gate pipeline in column 2 combines its tasks.

Figure 11: The main dashboard of the Zuul 3 Web component.

The last column displays less commonly used pipelines that come into play when Zuul acts on different events, including merging changes to the master.

If you take a closer look at a chosen task by clicking on it, it expands to show the jobs and the status of each (Figure 12).

Figure 12: A closer look at the job statuses in one of the pipeline tasks.

In the menubar at the top of the screen, the Projects menu takes you to a list projects of that Zuul monitors, displaying the hosting platform (GitHhub or Gerrit) and how Zuul trusts its content. Usually there will be only one trusted repo.

The Jobs menu takes you to a list of all jobs known to Zuul, which might be helpful if you are adding a new test to your pipeline: Perhaps someone has written something similar that you could reuse.

The Labels and Nodes menu items are interconnected. The first lists all possible labels you can use to define where jobs ought to be executed, and Nodes lists all nodes known to Zuul, with their labels and other details.

Finally, the Builds item lists all recently completed job,s and Buildsets shows all recently finished pipelines, along with their target (which commit), date, and result.

Summary

The Zuul system, originally developed to manage the massive multiple-contributor OpenStack software platform, has been updated to handle any complex collaborative software project. Zuul 3 is currently in use by multiple international companies (e.g., BMW, Software Factory, and, of course, OpenStack Foundation). In all cases, Zuul has proved to be a reliable and highly scalable solution [10].

With Zuul, "Humans don't merge code, machines merge code" [11]. Moreover, it works with complex code sets: Zuul can coordinate integration and testing across multiple, at first glance independent, repositories. Moreover, Zuul has a growing community. Thanks to a concept similar to Ansible roles [12], community members can reuse existing Zuul roles (e.g., pushing to the Docker registry) or write and share their own playbooks.

To sum up, Zuul 3 [13] is a free-to-use, highly scalable, and flexible solution that fulfills all three modern software development paradigms: continuous integration, continuous delivery, and continuous deployment. Projects that use GitHub or Gerrit repository hosting services can reasonably consider introducing Zuul into the development process.

Infos

  1. Duvall, Paul M., Steve Matyas, and Andrew Glover. Continuous Integration: Improving Software Quality and Reducing Risk . Addison-Wesley Professional, 2007
  2. "Continuous Integration" by Martin Fowler: https://martinfowler.com/articles/continuousIntegration.html
  3. Meyer, M. Continuous integration and its tools. IEEE Software . 2014;31(3):14-16
  4. Chen, L. Continuous delivery: Huge benefits, but challenges too. IEEE Software . 2015;32(2):50-54
  5. Humble, Jez, and David Farley. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation . Addison-Wesley Professional, 2010
  6. Gerrit Code Review: https://gerrit-review.googlesource.com/Documentation/intro-quick.html
  7. Sefraoui, Omar, Mohammed Aissaoui, and Mohsine Eleuldj. OpenStack: Toward an open-source solution for cloud computing. International Journal of Computer Applications 2012;55(3):38-42
  8. "OpenStack Spins Off Zuul, a Gated CI Pipeline for Multiple Repositories" by Joab Jackson. The New Stack, May 2018: https://thenewstack.io/openstack-spins-off-zuul-a-gated-ci-pipeline-for-multiple-repositories/
  9. "Zuul: Proven Open-Source Continuous Integration/Continuous Delivery" by Steven J. Vaughan-Nichols. ZDNet, May 2018: https://www.zdnet.com/article/zuul-proven-open-source-continuous-integrationcontinuous-delivery/
  10. "Zuul Case Study: The OpenStack Foundation" by Nicole Martinelli. Superuser, September 2018: https://superuser.openstack.org/articles/zuul-case-study-the-openstack-foundation/
  11. "Zuul v3 for Gating" by Monty Taylor. OpenDev 2018: https://youtu.be/6177329H4Tg
  12. Mohaan, Madhurranjan, and Ramesh Raithatha. Learning Ansible . Packt Publishing Ltd., 2014: https://subscription.packtpub.com/book/networking_and_servers/9781783550630
  13. Zuul files: https://opendev.org/zuul

The Author

Tomasz Szandala is a PhD student at the Wroclaw University of Science and Technology and a Site Reliability Engineer at NewVoiceMedia in Wroclaw, Poland. He enjoys embracing and improving new open source tools and playing the "not-so-dead" game Heroes of the Storm.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus