OpenStack Ussuri is Python3-Only: Upgrade Impact

A brief history of Python2 -> Python3:

Python version 2.0 was officially released in 2000, OpenStack was founded in 2010 and has since used Python 2.0 as its base language. The Python Foundation realized that in order to prevent users from having to perform tasks in a backward or difficult way, big improvements needed to be made to the software.

We released Python 2.0 in 2000. We realized a few years later that we needed to make big changes to improve Python. So in 2006, we started Python 3.0. Many people did not upgrade, and we did not want to hurt them. So, for many years, we have kept improving and publishing both Python 2 and Python 3.

In 2015, The Python Foundation made a very clear announcement on multiple platforms to migrate to Python 3 and discontinue Python 2. This initial plan was later extended to 2020. 

We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

OpenStack Starting Support of Python 3:

With the announcement of the sunset of Python 2, it became very clear that OpenStack also could not support Python 2 for much longer. Because it would have been impossible to fix any security bugs on Python 2, it was better for OpenStack to drop its support completely and instead concentrate on Python 3.

OpenStack’s support of  Python 3 started in 2013, and many developers contributed towards the enormous task of transitioning the software. After so much hard work from the community, the Stein cycle (September 2018) was the time when running OpenStack under Python3 as default work became a community goal. The community goal is a way to achieve common changes in OpenStack. OpenStack runs under Python3 as default was a great effort and includes a lot of hard work by many developers. Doug Hellmann was one of the key developers and showed coordination and leadership with other developers and projects to finish this goal.

OpenStack Train (Oct 2019): Python3 by default:

In the OpenStack Train release (October 2019), OpenStack was tested on Python 3 by default. This meant that  you could upgrade your Cloud to Python 3 environment with full confidence. OpenStack Train was released with well tested Python 3 support, but still also supported Python 2.7. At the same time, we kept testing the latest Python 3 version, and the OpenStack Technical Committee (TC) started defining the testing runtime for each cycle. OpenStack is targeting Python 3.8 in the next development cycle beginning soon.

OpenStack Ussuri (May 2020): Python3-Only: Dropped the support of Python2:

With the Ussuri cycle, OpenStack dropped all support of Python 2. All the projects have completed updating their CI jobs to work under Python 3. This achievement allows the software to be able to remove all Python 2 testing as well as the configuration that goes along with it..

Very first thing in the Ussuri cycle, we started planning for the drop of Python 2.7 support. Dropping Python 2.7 was not an easy task when many projects depend on each other and also integrate CI/CD. For example, if Nova drops Python 2.7 support and becomes Python 3 only, it can break Cinder and many other projects’s CI/CD. We prepared a schedule and divided the work into three phases, dropping support from services first, then library or testing tools.

    Phase-1: Start of Ussuri -> Ussuri-1 milestone: OpenStack Services to start

             dropping the py2.7 support.

    Phase-2: milestone-1 -> milestone-2:  Common libraries and testing tooling

    Phase-3: at milestone-2: Final audit.

Even still, a few things got broken in initial work. So, we made DevStack as Python 3 by default which really helped move things forward. In phase 2, when I started making Tempest and other testing tools as python3-only, a lot of stable branch testing started breaking. That was obvious because Tempest and many other testing tools are branchless, meaning the master version is being used for testing both the current and  older releases of OpenStack. So all Python 2.7 testing jobs were using the Tempest master version. Finally, capping and fixing Tempest installed in py3 venv made all stable branches and master testing green.

Just a couple of weeks before the Ussuri release, we completed this work and made OpenStack as python3-only, with an updated wiki page. Two projects, Swift and Storlets, are going to keep supporting Python 2.7 for another one or two cycles.

What “OpenStack is Python3-Only” means for Users/Upgrades:

If your existing Cloud is on Python 3 env, then you do not need to worry at all. If it is on Python 2.7 and you are upgrading to Ussuri,then you need to check that your env has the Python 3.6 or higher version available. From the Ussuri release onwards, OpenStack will be working on Python 3.6 or higher only. For example, if you want to install Nova Ussuri version, then it will give an error if Python 3.6 or higher is not available. It is done via metadata (“python-requires = >=3.6”) in setup configuration file. Below is the screenshot of how the setup config file looks in the Ussuri release onwards:

python-requires = >=3.6

classifier =

      Environment :: OpenStack

      Intended Audience :: Information Technology

      Intended Audience :: System Administrators

      License :: OSI Approved :: Apache Software License

      Operating System :: POSIX :: Linux

      Programming Language :: Python

      Programming Language :: Python :: 3

      Programming Language :: Python :: 3.6

      Programming Language :: Python :: 3.7

      Programming Language :: Python :: 3 :: Only

      Programming Language :: Python :: Implementation :: CPython

If you are using a distribution that does not have Python 3.6 or higher available, then you need to upgrade your distro first. There is no workaround or any compatible way to keep running OpenStack on Python 2.7. We have sunset the Python 2.7 support from Ussuri onwards, and the only way is to also upgrade your python version. There are a few questions on the python upgrade which are covered in the FAQ section below.

FAQ:

Q1: Is Python 2 to Python 3 upgrade being tested in Upstream CI/CD?

Answer: Not directly, but it is being tested indirectly.We did not set up the grenade testing (upstream upgrade testing) for py2 setup to py3 setup. However, previous OpenStack releases like Stein and Train were tested on both the python versions. This means that the OpenStack code was not working or well-tested on the previous version before it was python3 only. This makes sure that upgrading the py2->py3 for OpenStack has been tested indirectly. If you are upgrading OpenStack from Stein or Train to Ussuri, then there should not be any issues.

Q2: How are the backport changes from Ussuri onwards to old stable branches going to be python2.7 compatible?

Answer: We still run the Python 2.7 jobs until Stable Train testing so that any backport from Ussuri or higher (which are tested on Python 3 only) will be backported on Train or older stable branches with testing on Python 2.7 also. If anything breaks on Python 2.7, it will be fixed before backporting. That way we will keep Python 2.7 support for all stable branches greater than Ussuri.

Q3: Will testing frameworks like Tempest which are branchless (using the master version for older release testing) keep working for Python 2.7 as well?

Answer: No. We have released the last compatible version for Python 2.7 for Tempest and other branchless deliverables. Branchless means that the tools master version is being used to test the current or older OpenStack releases. For example, Tempest 23.0.0 can be used as a Python 2.7 supported version and Tempest 24.0.0 or master is Python 3 only. But there is a way to keep testing the older Python 2.7 release also (until you upgrade your cloud and want Tempest master to test your cloud). You can run Tempest on a Python 3 node or virtual env and keep using the master version for testing Python 2.7 cloud. Tempest does not need to be installed on the same system as other OpenStack services, as long as the APIs are accessible from the separate testing node, or the virtual env Tempest is functioning.

For any other questions, feel free to ping on the #openstack-dev IRC channel.