Change of Course

If you believe what Kohsuke Kawaguchi, inventor of the open source automation server Jenkins [1], is saying, open source continuous integration software is stuck in a "local optimum." Although traditional users like Jenkins, it can't attract new users. Meanwhile, new competition is growing on Jenkins' stomping ground [2].

The continuous integration/continuous delivery (CI/CD) landscape has changed completely in recent years. CI/CD was a new concept when Jenkins first started, but it is a matter of course today, and often a central component of many services. Although Jenkins has written its own success story over the past 10 years, some problems have grown with it and have not been solved.

Users today are looking for more plugins, more workloads, and more availability from Jenkins, pushing the software to its limits. According to Kawaguchi, the software is too complex. The operation of a larger Jenkins instance involves too much overhead, and sometimes daily restarts are necessary. Problems include pipeline execution, processes going haywire, and memory requirements.

Upgrade Pain

Many admins hesitate to update Jenkins and its plugins. Even simply adjusting job settings can sometimes cause undesirable side effects, which draws a picture of test software that itself is not well tested. Moreover, the Jenkins project, instead of moving forward, has developers struggling with compatibility issues.

According to Kawaguchi, Jenkins traditionally follows the Lego principle: Admins pick the parts they need and assemble their solution. This approach is no longer appropriate; more plugins are not a solution. On the contrary, Jenkins has to be far easier to use and ready for use immediately, which would improve support for users and keep the developer community going.

Kawaguchi offers a solution. His company, CloudBees, is looking to tackle the problems in collaboration with the Jenkins project. To achieve this goal, the project needs to pursue two key objectives: launch a cloud-native Jenkins and increase the speed of development for Jenkins 2.

Cloud Native Jenkins

A generalizable CI/CD engine running on Kubernetes is the idea behind Cloud Native Jenkins (CNJ). Thanks to a fundamentally different architecture and a new expansion mechanism, in the future, CNJ will scale arbitrarily on the basis of Kubernetes. The Cloud Native Special Interest Group (SIG) is responsible for implementation.

According to the plans, CNJ will enable a serverless or Functions-as-a-Service (FaaS) build execution, which already partially exists thanks to Jenkinsfile Runner [3] (Jenkins pipeline execution packaged as a command-line tool), and offer certain functions as microservices. Additionally, the services will use custom resource definitions (CRDs) to talk to Kubernetes.

Developers could extend the storage back end beyond local file systems to cloud-based storage solutions. Here, too, scaling is an essential point. Jenkins Configuration as Code (JCasC) [4] is set to play a central role.

Jenkins doesn't have to be reinvented completely for the cloud. The Jenkins X project (Figure 1) [5], for example, is already preparing the software for use in the cloud – specifically, on Kubernetes. The developers could customize Jenkins X to become Cloud Native Jenkins, and CNJ could then act as an engine for Jenkins X in the development process for other Cloud Native Apps.

Figure 1: Jenkins X is one of the already existing future-oriented projects in the Jenkins universe.

Jenkins Evergreen [6], the automatically updating rolling distribution system, is an attempt to get away from the Lego principle, or to prepare Jenkins so that users can get it up and running quickly. Finally, the project has to focus more on security for CNJ (Secure by Design).

Minimum Viable Product

Kawaguchi proposes first to develop a minimum viable product (MVP, i.e., a slimmed down prototype) that will consist of a FaaS-like build engine that Jenkins X could use. In this way, the MVP would bring together existing projects such as Pipeline (plugins that support implementing and integrating CD pipelines into Jenkins), Evergreen, Jenkinsfile Runner, and JCasC.

Specifically, a webhook receiver can receive webhooks from GitHub and trigger the build engine, which in turn would launch a pipeline execution with the help of the Jenkinsfile Runner, where Jenkinsfile Runner exists as a FaaS function. JCasC files take care of configuration and plugin management. In the form of Evergreen, however, Jenkins is looking to achieve a manageable number of combinable plugin variations and is also set to step in as a test environment for CNJ itself.

What the MVP lacks is a graphical user interface. From the MVP, the Jenkins project will further enhance the software, adding new services, revising the extension system, and making up for lost features.