Zuul case study: BMW

Zuul drives continuous integration, delivery and deployment systems with a focus on project gating and interrelated projects. In a series of interviews, Superuser asks users about why they chose it and how they’re using it.

Here Superuser talks to Tobias Henkel, software engineer at BMW, about benefits, challenges and hopes for future development. Henkel will be sharing more details about the case study in a breakout session at the OpenStack Summit Berlin.

The days of annual, monthly or even weekly releases are long gone. How is CI/CD defining new ways to develop and manage software at BMW?

Software has been an integral part of cars for several decades and has become one of the key enablers for many modern safety and comfort features. The amount of software required to implement all these features, as well as the complexity inducted by the many configuration options of current cars, is constantly rising.

The SW architecture in vehicles has evolved from more-or-less independent electronic control units (ECUs) to a set of highly connected functions spread over many ECUs. Without the right strategies, it’s not possible to manage all the required software projects that must converge on a strict schedule to deliver the BMW experience to customers.

The wide adoption of CI/CD in our internal and external development teams is one of the essential tools to deliver and integrate all software components on time with the required quality, despite the complexity of current and future software projects. Today, most BMW software projects rely on CI/CD for automating use cases of their daily work.

Are there specific features that drew you to Zuul?

After using CI/CD systems for many years for an ever-increasing amount of projects, the limitations of the existing CI solutions were starting to impact our software development efforts. With the increasing size and complexity of today’s software projects such as autonomous driving, the scaling capabilities of our CI/CD solution have become a crucial prerequisite of future development.

While scalability is an absolute must-have for our developers, testers and integrators, there are other important requirements for CI/CD:

1. Support for a centrally hosted instance for many projects

2. Support for complex cross-project CI configurations

3. Compatibility with our existing infrastructure

4. An active open-source community

The Zuul solution, especially after release of version 3.0, fully supports all our requirements to provide a centrally hosted solution that can be shared by many internal software projects. This dramatically reduces operations overhead and frees up valuable developer time to continuously improve all aspects of our CI system setup.

Zuul integrates seamlessly with our in-house OpenStack cloud and our repository systems Gerrit and GitHub. It also has an active community and provides the flexibility that our projects need.

How are you currently using Zuul at BMW?

For several years we’ve been operating several Zuul V2 instances for big software projects that need high CI/CD performance. This, of course, came at the cost of operating many instances with similar configurations.

In recent months, we’ve been preparing a centrally hosted CI/CD instance based on Zuul V3 and many projects using previous CI/CD solutions are already in the progress of migrating to the new Zuul V3 instance.

Hosting many projects on one central platform has many advantages for operation overhead and resource sharing in the cloud, but hosting many projects on one CI/CD instance directly translates to high stability and availability.

To maximize the availability of our central CI/CD service, we’re running Zuul, Nodepool and Zookeeper services in an OpenShift cluster, hosted on OpenStack. In addition to improved availability, we’ve seen several development and operation benefits for our internal CI/CD development team.

What other benefits have you seen so far?

The wide adoption of CI/CD in our software projects is the foundation to deliver high-quality software in time by automating every integral part of the development cycle from simple commit checks to full release processes.

With the introduction of Zuul-based CI/CD instances, projects were able to use the cloud resources for their automation in a very dynamic way. As a result, projects can do much more excessive testing in less time, which directly results in higher quality software, while being faster in development and integration.

With the introduction of Zuul V3 we also see a lot of benefits from the operators perspective by providing a centrally hosted CI/CD instance, as opposed to many small ones that have to be managed individually.

What challenges have you overcome?

A centrally hosted CI/CD platform for many projects faces the challenge to support many different use cases or restrictions, that projects inherently have. A common restriction for our projects is the need for non-Linux build nodes, because some required applications or tool chains are only supported for some operating systems like explicit versions of Microsoft Windows.

From the operator’s perspective, we don’t want special solutions for updating the node images, we want Nodepool to automatically do that for us, just like it does for Linux-based images. This required and still requires some extra effort.

Another interesting challenge is the management of the Zuul and Nodepool configuration, or, to be more precise, the responsibilities for managing the configuration. On one hand, we want to provide the projects as much configuration flexibility as possible, but on the other there are still centralized configuration files that we need to manage centrally. One example is the registration of static nodes at the CI system. We’re still working out how to manage these centralized configuration files effectively.

What can you tell us about future plans?

We’re currently migrating many projects to the centrally provided CI/CD instance based on Zuul V3. This instance will be the go-to solution for many existing and new software projects of BMW. We anticipate a continuous growth of project count and sizes, as well as a massive increase of our user base, which includes internal and external project members.

Given the strategic importance of Zuul and Nodepool for our development infrastructure, our main focus will be stability, availability, as well as scalability. While Zuul is already well prepared for most stability and scalability needs, there are still availability improvements required.

The main issue to solve is the removal of all single points of failure by making all services of the Zuul CI system highly available (HA). The CI/CD service should stay fully operational at all times, even if there are issues in single (virtual) machines or even a whole OpenStack availability zone.

What are you hoping the Zuul community continues to focus on or delivers?

While Zuul V3 provides a solution for most of our software projects out of the box, we still see room for improvement.

Our users of the Zuul CI/CD system would appreciate improvements to the Zuul-Web component, e.g. to provide more information on current and past jobs, cancelling a running job or configuring the status page layout per tenant.

However, the highest priority from our perspective is the removal of any single point of failures to support a configuration with high availability.