Experts Discuss Tradeoffs, Frequency, and more around Upgrades of Large Scale OpenStack Deployments

OpenInfra Live is a new, weekly hour-long interactive show streaming to the OpenInfra YouTube channel every Thursday at 14:00 UTC (9:00 AM CT). Episodes feature more OpenInfra release updates, user stories, community meetings, and more open infrastructure stories.

In this continuation of the May 20th OpenInfra Live episode, a panel of large scale OpenStack infrastructure operators from Blizzard Entertainment, OVHcloud, Workday, Vexxhost and CERN, join us again to further discuss upgrades.

Enjoyed this week’s episode and want to hear more about OpenInfra Live? Let us know what other topics or conversations you want to hear from the OpenInfra community this year, and help us to program OpenInfra Live! If you are running OpenStack at scale or helping your customers overcome the challenges discussed in this episode, join the OpenInfra Foundation to help guide OpenStack software development and to support the global community.

What are the tradeoffs of falling behind?

In Episode 6 of OpenInfra Live, the motivations behind upgrading in large scale OpenStack were discussed, mentioned Thierry Carrez, Vice President of Engineering at the OpenInfra Foundation. But this time, Carrez expounds on the question to ask “What are the tradeoffs of falling behind?”

Mohammed Naser, CEO at VEXXHOST, lists community-maintained bug fixes as the number one tradeoff to staying up-to-date with the latest release. “The closer you are to the supported versions, the easier it is to get your bugs fixed,” said Naser.

Joshua Slater, Senior Cloud Systems Engineer at Blizzard Entertainment, lists stability maintenance as Blizzard Entertainment’s top priority. This means that they stay with a reliable release until they are able to say “Okay! Like that’s the killer feature for us to finally go ahead and upgrade,” said Slater. He did, however, mention the tradeoff of “the farther we get behind, the harder it might be to actually perform the upgrade.”

Imtiaz Chowdhury, Senior Principal Software Engineer at Workday, lists security and staying close to the latest releases as Workday’s top priorities.

Arnaud Morin, DevOps at OVH, notes that “it’s a mixture of all the things [mentioned]” for OVH. “If you stay on an old release…you will end up in situations with security issues you can end up with…And if you want to upgrade you cannot because your OpenStack old version will not work with the new OpenStack version.” He also notes that when you want to introduce new features “you can still cherry-pick and bring them back [to the old OpenStack version], but you will have conflict and it’s more complex.”

Belmiro Moreira, Cloud Architect at CERN, tells us that CERN is always working to run the latest releases. By running the latest release, CERN, like VEXXHOST, gets active support from the community. “when we find the issue, and we expose that issue to the community…it’s much easier to explain it in the latest release, then we go back like five releases,” said Moreira.

Is it ok to skip releases?

At Blizzard Entertainment, skipping releases is something they commonly do. However, “there are, of course, releases that you cannot skip…One example was when Nova Cells was introduced, we absolutely had to upgrade to that version. And if we wanted to go to a version after that, upgrade again after that, but we commonly [skip upgrades], because we’re not able to keep up every six months for sure,” said Slater.

Similar to Blizzard Entertainment, OVH “skips a lot of releases, but plays every migration script on the database in order to make sure that everything will be,” said Morin.

How frequently do you upgrade to the latest version of OpenStack?

Workday, similarly to OVH, has “done big jumps…it is difficult to roll out or do an upgrade on enterprise setting,” said Chowdury. He goes on to mention that “sometimes the network provider, like an SDN provider, can add additional constraints of how often and which release, you can upgrade to.”

Finding an effective upgrading process is key to being able to see these upgrades. VEXXHOST has taken a few steps to get upgrades to be as painless as possible. By “using something like Kubernetes as the control plane for all of our cloud, we can easily roll out a change that just goes out and Kubernetes just goes out and rolls it out across the entire fleet one by one. Because it’s also image-based, like our hosts systems are pretty much as pristine as they can be,” said Naser.

Moriera agreed with Naser, that tooling does make a difference, however, he then mentions that it is “also OpenStack itself.” There have been huge improvements to the upgrade process over the last five years, however, “I still think that the developers still need to be more aware of the challenges of operators, we continue to see, for example, like between releases, configuration changes,” said Moriera.

Check out the full episode on YouTube to hear more questions from the audience including:

Do you use fast-forward upgrades when doing big upgrade jumps?

What are the best practices for database upgrades?

How does your deployment system of choice affect upgrades?

Have you’d deployed OpenStack with Airship?

How do you handle Operating System upgrades?

Do you use containers?

Where does the concern for removing images and the security side of it come from?

Have you seen production use cases where OpenStack is managed by Kolla Ansible?

Next Episode on #OpenInfraLive

Victoria Martinez de la Cruz, an OpenStack contributor who joined the community through the Outreachy internship program, will be joined by Kendall Nelson, OpenInfra Upstream Developer Advocate, and contributors who are just getting started. The group will discuss internships, mentoring, and how you can get started contributing upstream to open source projects like OpenStack, Airship, Kata, StarlingX and Zuul.

Tune in on Thursday, June 17th at 1400 UTC (9:00 AM CT) to watch this #OpenInfraLive episode: How to Start Contributing Upstream to OpenInfra.

You can watch this episode live on YouTube, LinkedIn and Facebook. The recording of OpenInfra Live will be posted on OpenStack WeChat after each live stream!