OpenStack at Bloomberg

Bloomberg has been running a number of production quality OpenStack clusters supporting various workloads, including web, machine learning, software development, databases and more since 2014.

To meet our needs for customization and integration with internal systems, we run our own Ubuntu-based OpenStack distribution called chef-bcpc. Our next-generation of chef-bcpc will be based on a modern OpenStack release such as Pike and will use OpenStack Neutron with pure L3 networks, via networking-calico, BGP and routed racks and will roll out in early 2018. In the meanwhile, however, we operate a number of stable clusters running an older version of our stack.

The curious case of the disappearing docs

Starting in late 2016, we updated our stable version to OpenStack Liberty. Developing, testing and then rolling out this release to all AZs of all production-SLA clusters took us almost six months. If we updated twice a year, we would get very little else done. Other production OpenStack deployments are similarly behind the curve (see below). We were very pleased with the Liberty upgrade. However, as our users started using more and more OpenStack features, we noticed that it was hard to answer some highly detailed API questions, since the OpenStack Liberty documentation had been removed from docs.openstack.org.

I asked our point of contact at Canonical, our OpenStack vendor, to look into this, since — from our point of view — OpenStack Liberty is still supported. His first contact with the dev community on this issue elicited the suggestion to “just build from the EOL tag.” Liberty itself is EOL (end of life), so the Liberty docs were removed and the last tag that allows us to find the Liberty versions is actually tagged liberty-eol. Our Canonical consultant did just that, but soon discovered that actually building the documentation from a tag is non-trivial.

Fast forward past the details of scripting and patching: after a few days of work, we had a build of the last available Liberty docs, but it turns out that a local build of the documentation is very much inferior to the what was previously on the website. Navigation dropdowns were empty and many links were broken in the offline build, since the Liberty docs were written assuming they would refer to other parts of the live OpenStack Liberty doc set on docs.openstack.org.

I requested our rep take this up with the OpenStack docs community. After some further discussion, it turned out that the end-of-lifing of older releases was part technical and part policy.

The policy part relates to the OpenStack release cadence: With two releases a year, older releases are quickly superseded by several newer ones, at which point they are deemed dead, support ceases and they are removed. (There’s more to say about this – see below.)

The technical part of removing the docs relates to matters of security, branching and available developer effort. The JavaScript/CSS parts of the docs were being maintained in-tree, with the code rendered directly into each page, so for necessary functional updates (e.g., for security), it required a new docs release. The devs did not want to do that – a reasonable objection – but it leads to an unreasonable result: documentation for versions of OpenStack that are still supported by vendors disappear.

After yet more discussion with the docs team, it was agreed to change the OpenStack documentation system to remove the maintenance problem. The code-like parts of the documentation (JS/CSS) are being removed from the OpenStack release branches and they will be maintained and hosted separately. This, in turn, allows old releases that work with the latest documentation tooling to be maintained indefinitely. Code pushes don’t require new doc releases. This led to an agreement that, as of OpenStack Mitaka, the docs for each OpenStack release will stay available on docs.openstack.org indefinitely. This is a great result for the wider OpenStack Operators community which currently runs a wide range of older releases – many even older than OpenStack Liberty. For example, in this post about “Archiving old documentation on docs.openstack.org,” we see that “According to the last User Survey, 9 percent of production OpenStack installations were still running Icehouse, 17 percent were running Juno and 40 percent were running Kilo.”

Proposing further changes: Upstream long-term support releases

Documentation retention is only a small part of the OpenStack release cadence and short-term/long-term support policy.

I presented the news about the documentation agreement to the OpenStack Operator community at the OpenStack Ops Meetup meetup in Mexico City in August. While continued access to the docs was seen as a plus by many of those present, the debate rapidly expanded to pull in the issue of the overall OpenStack release cadence and the possibility of OpenStack LTS (long-term support) releases. Many customers of the various OpenStack distro vendors (Canonical, RedHat, SUSE, etc.) do indeed love the two releases per year rapid release cadence that OpenStack has followed since 2010. However, there is another part of the operator community who, like us, are unable to keep up and would be HIGHLY interested in less frequent releases with longer support. Many operators reported “we just can’t keep up” (see for example this recent article about NTT; Shintaro Mizuno from that team stated at the Mexico meetup that staying current is practically impossible).

So, something Bloomberg what would like to see is the aforementioned OpenStack LTS releases, for example longer support for every other OpenStack release (a “tick/tock” system), or perhaps something like the Canonical/Ubuntu LTS system.

Happily, “Upstream LTS releases” is now a topic at the Forum in the upcoming OpenStack Summit in Sydney (as per draft Forum schedule on 2017-10-12). Several of the distro vendors will attend and I encourage all parts of the community to support this session in Sydney and make your voice heard.

For Bloomberg, if LTS releases were introduced, we’d try hard to keep up with those, as opposed to the twice-yearly releases that are quite simply impossible for us to deploy.

About the author

Chris Morgan is the cloud services team leader at Bloomberg, focused on providing OpenStack-based private cloud services within the firm. He regularly chairs the OpenStack Ops Meetups team and was co-organizer of the 2016 Ops Mid-Cycle Meet-up in New York City.

Cover Photo of the Bloomberg Tower // CC BY NC

Tags: Bloomberg