Zuul case study: GoDaddy

Internet domain registrar and web hosting company GoDaddy has more than 17 million customers and over 7,000 employees worldwide. A former Superuser Awards finalist, they’re longtime proponents of OpenStack and active members of the OSF.

Here Superuser talks to Clint Byrum, a senior cloud software engineer at GoDaddy, about how they’re currently using Zuul, the importance of community and the perils of neophobia.

If you’re at the Vancouver Summit, catch him moderating an collaborative session on CI/CD and all the sessions on Zuul here.

The days of annual, monthly or even weekly releases are long gone. How is CI/CD defining new ways to develop and manage software with open infrastructure?

We have a lot of free/open software and a lot of physical infrastructure to manage underneath it. Every day, hundreds of GoDaddy developers and tens of thousands of GoDaddy customers rely on our systems to stay usable while we add features, fix bugs and generally deal with the problem of change. Without a strong investment in CI/CD, we’d just be pulling manual levers, sitting on already-complete work and generally never make any progress.

Are there specific Zuul features that drew you to it?

The community that built Zuul was probably the first feature. We have a very large OpenStack installation at GoDaddy and a few of us have been using Zuul since its very early days as OpenStack developers. This experience has always been pleasant and the folks in charge of it are extremely responsive and welcoming.

Beyond that, cross-repo gating allows us to keep our concerns separate and build CI/CD pipelines out of a combination of upstream free/open software, such as OpenStack, in concert with our more custom GoDaddy-specific integrations. Having built-in multi-node testing is a massively important feature for us. We are trying to test deployment that must orchestrate several pieces of infrastructure, such as databases, message queues, and hypervisor software like libvirt/KVM. It just doesn’t work to do this all on one VM, and containers lack the ability to accurately simulate our production bare metal environments. It doesn’t hurt that Zuul makes use of Ansible and our specific team is very well versed in Ansible, making it easy to get up to speed quickly writing test cases in Zuul, vs. Jenkins, which requires people to learn Groovy or the pipeline DSL to write complicated test cases.

Finally, the fact that it can scale your test bandwidth as big as your available OpenStack is appealing to us, since we have quite a bit of OpenStack capacity available to us.

How are you using Zuul at GoDaddy now?

Zuul is still in somewhat of a pilot for us, primarily in use for testing our OpenStack deployment automation and also for managing changes to some of our Kubernetes deployment automation. We also use it to test and deploy itself. Hopefully being self-aware won’t turn it into SkyNet.

As for nuts and bolts, we have a small-ish team of about eight developers using it every day, with a few other devs from other teams contributing as well. The Zuul control plane runs on a single 16GB VM, and we currently run between 40 and 75 8GB VMs for tests across two regions of our internal OpenStack cloud. We use the GitHub driver to interface with our GitHub Enterprise installation, which allows teams to partially adopt Zuul while still using the code review workflow they’re used to.

What benefits have you seen so far?

We’ve replaced a lot of slow, brittle Jenkins jobs with highly parallelized, more realistic Zuul test jobs. We’ve also been able to accelerate development on some new projects by setting up rapid proof-of-concepts in PR’s to GitHub. Some of these go nowhere, and some of them eventually become the deployment automation that goes into production.

In addition, since all of Zuul’s job and project configuration is done in YAML files stored in Git repositories, we’re able to treat our testing infrastructure as code and reap all the benefits of that without having to take notes and repeat manual steps.

What challenges have you overcome?

Zuul v3.0 was only just released with a stable interface. We were early adopters and thus had to deal with tracking the extremely aggressive pace of development in Zuul. There’s also substantial inertia around Jenkins and it’s not always 100 percent obvious why Zuul is a better fit in some cases when Jenkins is just so popular. However, once shown the power of it, we’ve had no complaints from members of the team about Zuul, so this was largely just a case of neophobia.

What can you tell us about your future plans?

Now that 3.0 is released, it’s a much easier conversation to have with internal dev teams about potentially adopting it. I plan to personally evangelize Zuul inside GoDaddy and outside GoDaddy in the open-source communities where I’m active.

We also want to help steer the future of Zuul and make sure we can use it in conjunction with Kubernetes more smoothly. We have a lot of institutional knowledge of Kubernetes, but Zuul just doesn’t have much to offer when all you need is a container.

What are you hoping the Zuul community continues to focus on or deliver?

In addition to container integration, I’d really like to see a focus more on the CD part. The CI part is incredibly strong in Zuul, but there are some real challenges to deploying things at scale with Zuul. I’d also like us to see about integrations with some other tools, like Spinnaker, in this area, where Zuul might be handy for the dev-> deploy hand-off.

Anything else you’d like to add?

So many great people worked very hard on Zuul, it’s really an honor to be able to speak about what they’ve done. I just want to say thank you to the OpenStack community and the Zuul community for making our lives a little better each day.

Superuser wants to hear what you’re up to with open infrastructure, get in touch: editorATopenstack.org

Cover photo: GoDaddy Tempe office, courtesy Go Daddy.

Tags: CI/CD, GoDaddy, Zuul