Take these OpenStack Infrastructure tools and run!

When you’ve got thousands of proposed patchsets, comments and test environments flooding in every day, you need the right tools to handle them. OpenStack created these free software tools to handle its burgeoning scale – here’s what they can do for you.

In a packed room of about 150 people, Thierry Carrez, director of engineering at the OpenStack Foundation, showcased these tools recently at the Free and Open Source Software Developers’ European Meeting, aka FOSDEM.

While some of these tools such as Puppet, Jenkins, and Gerrit are well known, the Foundation has also created its own free software tools. Many of these are potentially useful for any software project development infrastructure.

The numbers behind OpenStack tell why these tools are necessary. Every day, OpenStack handles 1,000 proposed patchsets, 7,500 posted comments and votes on Gerrit, 16,000 test environments spawned and 250 changes merged.

These tools were devised by the Infrastructure Team, an internal team that provides the code reviews system, testing automation, continuous integration and other tools used by OpenStack project development. They’re fully automated, deployed by Puppet and all the manifests are published, too. Configuration files and all open source tools make it possible for people to clone all OpenStack infrastructure. So have at it!

Here are some of the tools you can use:

Gating

Zuul is a pipeline-oriented project gating system.
Its main characteristics include:

Merges changes only if tests pass
Receives events about proposed changes, trigger tests and reports back
Handles cross-project dependencies
Uses parallel pipelines using speculative gating instead of testing changes in a series. All tests are ordered in a queue and expect the tests to pass
Supports purely serial gating, if needed

Zuul is still a strict gating system, optimized gating for speed, although it uses a lot of resources. At best, it’s as fast as parallel gating and worst case scenario it’s as slow as serial gating, Carrez says. The pipelines in Zuul are visualized in http://status.openstack.org/zuul. OpenStack test clusters run on public clouds hosted by Rackspace and Hewlett-Packard. Zuul doesn’t need OpenStack to run tests, since it has a lot of configurable backends. It also watches for changes in Gerrit, using triggers to integrate with it.

Automation

Jenkins needed a job builder that translates YAML into XML jobs suitable for Jenkins. (Because who likes to write XML?) Enter a handy tool called Jenkins-Job-Builder that features a lot of useful templates and macros to make maintenance of Jenkins jobs much easier.

d9unowvozjc6jzhjd7ej

Code Review

Jeepyb makes managing Gerrit easier. OpenStack uses it to integrate Gerrit with Launchpad and GitHub, for example. When people submit pull requests in Github, Jeepyb closes them automatically. Jeepyb also periodically expires changesets with negative votes and no activity for a certain time.

Facilitating tools

Git-review is a tool that helps submit changes to Gerrit; it sets the Gerrit remote automatically if missing. It can also lend a hand with with reviews, downloading changesets for review and comparing different patchsets. Gertty is a console-based interface to the Gerrit Code Review system, it has a workflow similar to an email/newsgroup and lets you work offline.

Workflow coordination

StoryBoard helps OpenStack solve the problem of cross-project/cross-team coordination by making sure that all people and all systems are in sync, notified, etc. Bugs need to have tasks attached, and same with blueprint/features requests. Launchpad is limited, so the OpenStack foundation decided to build their own tool to connect these dots.
In StoryBoard, stories describe the problem to solve and these stories have tasks effecting projects. The API server represents the stories, and the static Javascript webclient interrogates the server.

This is just a sample of what’s available. The rest of the projects managed by OpenStack Infrastructure Team are available at git.openstack.org/cgit/openstack-infra or at ci.openstack.org for documentation.

If you’ve used any of these tools, drop us a line. We’d love to talk to you about your projects.

Cover Photo by Dr. Mark Kubert // CC BY NC; At the OpenStack booth during FOSDEM: Ubuntu developer and Debian contributor Adrien Cunin, Erwan Gallen of Red Hat, OpenStack Ambassador Marton Kiss, Stefano Maffulli of the OpenStack Foundation and Sergii Golovatiuk.