Featured

2021 OpenInfra Annual Report: Zuul

The OpenInfra Foundation annual report for 2021 is now live, including sections on open infrastructure growth, all OpenInfra projects, Foundation updates, SIG and WG updates, and more.

January 31, 2022
image

Zuul is an open source CI/CD platform designed for test-driven open source projects and software development organizations who need to gate against multiple projects and systems before landing a single patch. Since 2012, Zuul has been proven at scale as a critical part of the OpenStack development process. In 2018, Zuul version 3.0 was released with the intention of making Zuul easier to use outside of the OpenStack project. Since then we’ve seen the number of users and use cases grow tremendously. You will find Zuul behind cloud, e-commerce, and automotive software ensuring that every commit passes tests and is ready to be merged.

2021 was a big year for Zuul, in large part due to the efforts to remove all single points of failure from its component services. The goal is to allow you to run a highly available Zuul cluster that requires no downtime to perform maintenance. We have called this effort “Zuul v5” naming it after the release version that will include this functionality.

The first step towards version 5.0.0 was version 4.0.0 released in February 2021. Version 4.0.0 makes two assertions about the databases that Zuul communicates to. First that the SQL database is always present, and second that the ZooKeeper database communication is always encrypted. These two changes serve as the foundation for the incremental updates necessary to reach version 5.0.0. Importantly, Zuul components rely on these two databases (which themselves can be operated in an HA manner) to store necessary information allowing every Zuul component to run with redundant processes. The encrypted ZooKeeper connectivity is important because Zuul secrets and other sensitive information are stored in the ZooKeeper database which is necessary to run multiple Zuul Scheduler instances.

Seventeen additional version 4.x releases have been made in 2021 ending with 4.11.0. Many of these releases included incremental improvements on the road to version 5.0.0. If you are running 4.11.0 configuration data, secrets, merge requests, build requests, pipeline queues and more are now stored in the ZooKeeper database. This means that adventurous users can now run a fully HA Zuul deployment with no single points of failure on Zuul 4.11.0. However, we have not released version 5.0.0 yet as there are still old components specific to the previous design that need to be removed as well as documentation updates.

This step-wise development approach to the 5.0.0 release has worked well. The branched rewrite for version 3.0.0 was necessary at the time, but was very disruptive for users and made upgrades much harder, a result we were looking to avoid this time around. One major upside to the new approach is that deployments like OpenDev’s have been able to more effectively run pre release Zuul updates and provide early feedback. Through Zuul’s cooperation with OpenDev we have been able to do this and ensure that each release you consume is ready for use after non trivial real world trials. These same trials ensure that the incremental changes towards version 5.0.0 will all be ready when you upgrade to version 5.0.0 too.

These eighteen releases have brought more than just background operations specific updates; there are new features and bug fixes too. Version 4.6.0 brought a number of important security updates to Zuul that users should be aware of and plan upgrades for if they haven’t already. Administrative tasks can be performed through Zuul’s REST API using JWT tokens. When version 5.0.0 is released this will be required for administrative tasks as Gearman will be removed. Windows test node support has been greatly improved and is now quite useable. The Zuul Kubernetes operator has migrated to the KOPF operator platform. The addition of liveness/readiness probes and Prometheus support make running Zuul in Kubernetes easier.

A number of improvements have been made to Nodepool as well. The Azure driver has been rewritten using the new generic driver framework, making it more efficient and adding new features. A metastatic driver was added which allows Nodepool and Zuul to treat dynamic test instances as static nodes, timing them out after periods of inactivity.

To make all of this happen more than 80 contributors submitted changes to 14 project repositories. More than 1000 changes in total were merged to Zuul and its related repositories in 2021. The Zuul project would like to thank all of its contributors for a productive 2021.

The Zuul community saw some interesting changes and events during 2021 as well. Notably, the Zuul synchronous communications platform transitioned from IRC to Matrix. This brings a more modern web experience to chat for Zuul and should be familiar to more people. Matrix is a fully open source platform that stores channel history on the server side which means new participants in Zuul chat can review the entire history of the Matrix room in their client without needing to maintain a persistent connection. Community members also presented at events like Ansiblefest (requires registration), OpenInfra Live, and OpenInfra Live Keynotes whose recordings you can use to learn more about Zuul. Another great starting point is Zuul’s new “Get Started” page which includes information about running your own Zuul, finding support, or purchasing a managed Zuul service.

Looking ahead to 2022 we expect version 5.0.0 to be available early in the year. Deployments with additional redundant components can expect better uptime and less painful upgrades. The Zuul documentation is being reorganized to be easier to navigate and include complete information on topics within a single page. IBM cloud support for Nodepool is also in the works. Once version 5.0.0 is finalized, I’m sure we will find a number of exciting new features to focus on as well.

Find us:

Read the full 2021 OpenInfra Annual Report here!

Tags: , ,

Superuser
Latest posts by Superuser (see all)