eBay in Production: Migration from Nova-Network to Neutron

OpenStack Networking (Neutron) is the focus of development for a full software-defined networking stack. However, many organizations still use nova-networking in production, even though development on the networking code that is part of Compute will be deprecated in a future release.

More and more, organizations are at a point of decision with their cloud and would like to migrate to Neutron. But what’s the best way to perform the migration from nova-network to Neutron?

Li Chengyuan, Cloud Engineer at eBay, and Han Zhou, Cloud Architect at eBay, shared how they made the upgrade at the Paris Summit in November.

At eBay, several data centers were using Folsom nova-networking in production with thousands of VMs in service. They successfully upgraded to OpenStack Havana, adopted Neutron as a Network service instead of nova-networking, and replaced Linux bridge by Open vSwitch.

Their work covered both control plane and data plane migration. The control plane includes neutron net/subnet/port management and SDN controller integration, while the data plane includes tap device migration from Linux bridge to openvswitch bridge, DHCP service, security group, and Libvirt/KVM configuration update.

Here’s a look at what they did:

Folsom Nova-network

This figure below shows the original Folsom network environment.

fcwxolrsvc8xiwoc0olc

To the right are OpenStack controller nodes, which are quite normal. To the left side of the Nova compute nodes are the Folsom Nodes. eBay configures them as nova-network nodes. So you can see that there is a Nova-network process are running in each compute node. Also, there are several dnsmasq process in the compute nodes. They provide the DHCP service for all the local VMs, and all the VMs are running in the bridge mode and in Folsom the Linux bridge is used.

In eBay’s configuration there are two interfaces for each VM, each of which connect to one Linux bridge. The figure below shows the target environment in Havana. All the OpenStack components will be migrated to Havana, with a Neutron server is running, along with SDN controllers.

dnbkky5knighym5l4cr1

eBay uses VMware NSX as their SDN controller, and for this nova-compute side they could see that there’s no nova-network running, no DNSmasq, and no Linux bridge. Instead, they installed Open vSwitch, and other existing VMs are still using the bridge mode. This bridge interface migrates completely to the Open vSwitch bridge interface, making it their primary target.

Control Pane Migration

The process by which the eBay team completed control plane migration is multifaceted. First, they kept all Folsom OpenStack controller nodes and set up Havana OpenStack control nodes, including the Keystone, Nova, Glance, Neutron, RabbitMQ, and MySQL. Then, they created another set of nodes for the NSX controller and create a transport zone.

y8wrjr2dkw1rm8tebcmi

Here are the steps (in brief) they outlined in the presentation:

Set up Havana OpenStack nodes
Set up NSX Nodes
Transport zone creation in NSX controller
Direct all Hypervisor registration to NSX controller
Migrate Keystone, Nova, Glance DB
Create Neutron net/subnet according to the information in Folsom nova.networks and nova.fixed_ips tables
Enable fake driver in nova-compute, and register to controller
Create port in Neutron and attach port to VM
Update security group

Data Plane Migration

Data plane migration had to be done for each hypervisor node. This meant moving the live traffic for other VMs running on this hypervisor node from the Linux bridge to the Open vSwitch.

In the session, Chengyuan asked those in the audience using Open vSwitch to raise their hands — more than 2/3 of the audience indicated that they were. Chengyuan’s reason for moving to Open vSwitch is to be able to use the SDN controller to control all the flows, making the process of migrating flows as seamless as possible with as little downtime as possible.

Their steps for data plane migration:

Install Open vSwitch
Remove VM interfaces from Linux bridge
Rename to tapxxxxx (Havana convention)
Remove physical interface from Linux bridge
Remove linux bridge
Start OVS service
Create integration bridge: br-int
Create external bridge: br0, set external-id
Add eth0 to br0
Add host IP to br0, add host route
Add VM interfaces to br-int
Set SDN controller

m3bz6csmnwv7gt7g42jf

To Summarize

The duo from eBay recapped all the steps in one figure.

orubyoidjn36cdldl5ax

First, they shut down the Folsom network and Folsom node compute.
Then they started up the Nova-compute, with fake driver first.
Next, they executed port creation for the control plane, and started the Open vSwitch.
For the data plan, they removed the interface, destroying the bridge interface, and created the Open vSwitch bridge interface.
They finalized those br-int, br0 connection and got the traffic working, and update their files.

Currently, work is being done so that nova-network will be deprecated in a future release — and upgrades without instance downtime may be available as soon as the Kilo release. Read the OpenStack Operations guide for more info about the planned deprecation.

Check out the full video of the presentation at the Paris Summit