Zuul case study: Tungsten Fabric

Zuul drives continuous integration, delivery and deployment systems with a focus on project gating and interrelated projects. In a series of interviews, Superuser asks users about why they chose it and how they’re using it.

Here Superuser talks to Jarek Łukow, CI engineer at Codilime, about Tungsten Fabric. Łukow currently operates the OpenStack-backed CI/CB system for Tungsten Fabric (formerly OpenContrail). In one of its primary deployment scenarios, Tungsten Fabric serves as a networking provider for OpenStack. “This means that we face challenges similar to the OpenStack project in our continuous integration system,” Łukow says. As a result, Tungsten Fabric uses Zuul to gate changes made in over 70 source-code repositories.

The days of annual, monthly or even weekly releases are long gone. How is CI/CD defining new ways to develop and manage software in your work?

One of Tungsten Fabric’s strengths is a wide variety of integrations and deployment scenarios it provides. Automated testing ensures that new changes and features keep the CI system working across the required range of supported systems, platforms and versions. It also helps us respond faster to changes in software we integrate with. Fortunately, the CI system has been in place from the very beginning and Zuul has always been a large part of its history. At the beginning of this year we migrated from Zuul 2 to 3, making our core continuous integration and delivery pipeline powered entirely by Zuul and its companion services.

What features drew you to Zuul?

I would say that the most important features, in terms of positive impact on our operations, are dynamically spawned disposable slaves and cross-project change dependencies.
The Nodepool service duo (builder and launcher) provides automated image provisioning and CI cluster scalability. This ensures the jobs run in well-defined and reproducible environments, effectively relieving us from the need of maintaining a static pool of shared slaves. The cross-project dependency mechanism is important because our project consists of over 70 repositories and 30 of them are required to build the core of the system. The feature is very handy for the developers and gives them more confidence when testing changes that span multiple repos.

How are you currently using Zuul?

Zuul tests every change going into Tungsten Fabric’s Gerrit review system. It manages the process of compilation, packaging and containerization of our software. The VM-based test environments are well suited for us as some of the tested components need exclusive access to an underlying operating system. We also re-use the build pipelines (or workflows, not to confuse with the Zuul nomenclature) to perform periodic builds and releases.

What other benefits have you seen so far?

The Git-centric Zuul approach of doing things has helped us to get to the “version control everything” model, where all the things, from the CI jobs, through the third-party dependencies to the infrastructure configuration, are version-controlled, reviewed by team members and verified by the CI system.
Also, using Ansible as the main job DSL plays out really well in our organization, as many of people working on our Zuul jobs are already familiar with it and can jump in without the need for learning any new syntax.

What challenges have you overcome?

Understanding trusted and untrusted job contexts is a very important part of learning Zuul and can help you get the most out of it. We started with all of the job configuration in a single config-project and decided to split into a separate untrusted project afterwards. We recommend designing your Zuul jobs so that only the really essential minimal parts are kept in trusted projects and the rest in untrusted ones, so they can be easily developed and tested. Additionally, as some parts of our infrastructure run inside private network, we had to do some additional plumbing to use a worker cloud from behind a firewall.

What can you tell us about future plans?

The next things on our roadmap are: running nested Hyper-V and ESXi testbeds directly under Nodepool/Zuul v3 and making first trials of the third-party CI approach for performance testing. We expect Zuul to shine in these fairly complex test scenarios.

What are you hoping the Zuul community continues to focus on or delivers?

Zuul has proven to be a reliable and well-designed tool for some of the most complex software projects. Now it may require some more integrations beyond the core ones to become a good fit for a wider group of potential Zuul adopters.
New connection types and Nodepool drivers can be beneficial for current users, like us, to discover new ways of running and optimizing pipelines and will also attract new users that may have specific platform requirements.

Anything else you’d like to add?

I want to acknowledge the quality of the Zuul software suite. We’ve been running its bleeding-edge pre-release version for a long time and did not notice any kinds of instability. The documentation is rock solid, pretty much every feature is tested right away by running in production for one of the largest CI environments – the OpenStack project. Also the community is impressively responsive via IRC and mailing lists.

For more details, check out this session on lessons learned from the Tungsten implementation from the recent Berlin Summit and this blog post over at Codilime.