An introduction to OpenStack fast forward upgrades

OpenStack momentum continues to grow as an important component of hybrid cloud, particularly among enterprise and telecomunications.

One area that Red Hat customers ask about is the rapid release cycle of OpenStack. And while this speed can be advantageous in getting key features to market faster, it can also be quite challenging to follow for customers looking for stability.

With the release of Red Hat OpenStack Platform 10 in December 2016, we introduced a solution to this challenge – we call it the Long Life release. This type of release includes support for a single OpenStack release for a minimum of three years plus an option to extend another two full years. We offer this via an ELS (Extended Life Support) allowing our customers to remain on a supported, production-grade OpenStack code base for far longer than the usual six-month upstream release cycle. Then, when it’s time to upgrade, they can upgrade in-place and without additional hardware to the next Long Life release. We aim to designate a Long Life release every third release, starting with Red Hat OpenStack Platform 10 (Newton).

Now, with the upcoming release of Red Hat OpenStack Platform 13 (Queens), we’re introducing our second Long Life release. This means we can, finally and with great excitement, introduce the world to our latest new feature: the fast forward upgrade.

Fast forward upgrades take customers easily between Long Life releases. It’s a first for our OpenStack distribution, using Red Hat OpenStack Platform director (known upstream as TripleO) and aims to change the game for OpenStack customers. With this feature, you can choose to stay on the “fast train” and get all the features from upstream every six months, or remain on a designated release for longer, greatly easing the upgrade treadmill some customers are feeling.

It’s a pretty clever procedure since it factorizes the process of three consecutive upgrades and therefore greatly reduces the number of steps needed to perform it. In particular, it reduces the number of reboots needed, which in the case of very large deployments makes a huge difference. This capability, combined with the extended support for security vulnerabilities and key backports from future releases, has made Red Hat OpenSack Platform 10 very popular with our customers.

openstack_graphic_large

Red Hat OpenStack Life Cycle dates

Under the covers of a fast forward upgrade

The fast forward upgrade starts like any other upgrade, with a Red Hat OpenStack Platform 10 minor update. A minor update may contain everything from security patches to functional enhancements to even backports from newer releases. There is nothing new about the update procedure for Red Hat OpenStack Platform 10, what changes is the packages that will be included such as kernel changes and other OpenStack specific changes. We’re placing all changes requiring a reboot in this minor update. The update procedure allows for a sequential update of the undercloud, control plane nodes and the compute nodes. It may also include an instance evacuation procedure so that there is no impact to running workloads even if they reside on a node scheduled for reboot after the update. The resulting Red Hat OpenStack Platform 10 cloud will have the necessary operating system components to operate in Red Hat OpenStack Platform 13 without further node reboots.

The next step is the sequential upgrade of the undercloud from Red Hat OpenStack Platform 10, to 11, to 12, to 13. There is no stopping during these steps and in case of a needed rollback of this portion you must return to Red Hat OpenStack Platform 10.

During the fast forward upgrade there are opportunities to perform backups. These should be performed since there’s no automated rewind included but rather a restore from these backups.

A lot of things have changed between Red Hat OpenStack Platform 10 and 13. The most notable is the introduction of OpenStack services in containers. But don’t worry! The fast forward upgrade procedure takes the cloud from a non-containerized deployment to a resulting cloud with OpenStack services running in containers, while abstracting and reducing the complexity of upgrading through the middle releases of Red Hat OpenStack Platform 11 and 12.

In the final steps of the fast forward upgrade procedure, the overcloud will move from Red Hat OpenStack Platform 10 to 13 using a procedure that syncs databases, generates templates for 13, and installs 13’s services in containers. While some of the content for these steps may be part of a release which is no longer supported, Red Hat will provide full support for the required code to perform the upgrade.

What’s next …

In order for this procedure to be supported, it needs to be validated with the released code, and carefully tested in many situations. For this reason, it is scheduled to be ready for testing from Red Hat OpenStack Platform 13 general availability (GA); however, we will warn users launching the procedure that it should not be used in production environments. We encourage you to try the procedure on test environments during this period, and report any issues you find via the normal support procedure. This will greatly help us ensure that we are covering all cases. During this time support cases relating to fast forward upgrades will not be eligible for high priority response times.

Once we have thoroughly field tested the procedure, fixed bugs, and are confident that it is ready, we will remove this warning and make an announcement on this same blog. After this happens, it will be OK to proceed with fast forward upgrades in production environments. You can follow this progress of validation and testing by following this blog and staying in touch with your local Red Hat account and support teams.

Stay tuned for future fast forward upgrade blog posts where we dig deeper into the details of this procedure and share experiences and use cases that we’ve tested and validated.