Continuous Progress

Project

The project config element (Listing 3) spans pipelines and jobs and defines which jobs should be run for which pipeline. In this example, I expect to run, on verification, a set of three jobs. The first, clang, is a static C++ code analysis. The other two are compilation variants according to the processor architecture: one for i686 and the other for x86_64. In this case, the playbooks used in the job compilation are generic, so they expect variables to be passed. The second compilation call also requires the first clang to be completed successfully.

01 - project:
02   verification:
03     jobs:
04       - clang
05       - compilation:
06         vars:
07           hardware: i686
08           technology: legacy
09       - compilation:
10         vars:
11           hardware: x86_64
12           technology: legacy
13         require:
14           - clang

Ansible

To run the job – where the job hits the metal, as it were – the Zuul launcher dynamically constructs an Ansible playbook. This playbook is a concatenation of all playbooks the job needs to run. By using Ansible to run the job, all the flexibility of the orchestration tool is now available to the launcher. For example, a custom console streamer library allows you to live stream the console output for the job over a plain TCP connection on the web dashboard. In the future, Ansible will allow for better coordination when running multinode test jobs; after all, this is what orchestration tools such as Ansible are made for! Although the Ansible run can be fairly heavyweight, especially is you're talking about launching thousands of jobs an hour, the system scales horizontally, with more launchers able to consume more work easily.

At the start of each job, an Executor prepares an environment in which to run Ansible that contains all of the Git repositories specified by the job with all the dependent changes merged into their appropriate branches. The branch corresponding to the proposed change will be checked out in all projects (if it exists). Any roles specified by the job will be present, as well (with dependent changes merged, if appropriate), and added to the Ansible role path. The Executor also prepares an Ansible inventory file with all of the nodes requested by the job.

Trusted/Untrusted

The configuration described above is divided into two sections: trusted and untrusted configurations. The first, trusted, is defined in Zuul's central repository. This repository (or a few, if needed) has stricter access rules, so only a limited number of people can modify it. The trusted repository is also the location where you define the base job, which can later be inherited.

The untrusted repositories are usually those that are being verified by Zuul. The main difference is that Zuul takes YAML only from master branches in trusted repositories for verification and from the review or pull request that is being verified. In other words, if a developer changes a job definition from their repository, Zuul will include those changes in verification. But all "trusted" YAML has to be merged to the master to be considered in the flow.

Playbooks called from the untrusted repositories also are not permitted to load additional Ansible modules or access files outside of the restricted environment prepared for them by the Executor. Additionally, some standard Ansible modules are replaced with versions that prohibit some actions, including attempts to access files outside of the restricted execution context.

When Zuul starts, it examines all of the Git repositories specified by the system administrator in the tenant configuration and searches for files in the root of each repository. Zuul looks first for a file named zuul.yaml or a directory named zuul.d and, if not found, .zuul.yaml or .zuul.d (with a leading dot). In the case of an untrusted project, the configuration from every branch is included; however, in the case of a config project, only the master branch is examined.