Deploying OpenStack on AWS

Cloud on a cloud!
Yes, you read that correctly! This post will show you how to install OpenStack on top of AWS EC2.

Installing OpenStack in a nested hypervisor environment is not a big deal when we use the QEMU emulator for launching virtual machines inside the virtual machine. However, unlike usual nested hypervisor setup, installing OpenStack on AWS EC2 instances, we will have to bypass a few restrictions for the OpenStack setup to work properly. This blog post outlines those limitations and their solutions to run OpenStack on top of AWS EC2 Virtual Machine.

Limitations

An AWS environment will allow the packets to flow in their network only when the MAC address is known/registered in the AWS network environment. Also, the MAC address and the IP address are tightly mapped, as the AWS environment will not allow the packet to flow if the MAC address registered for the given IP address is different.

You may be wondering if the above restrictions will impact the OpenStack setup on AWS EC2.
The answer is: yes, it will!

While configuring Neutron networking, we will be creating a virtual bridge (br-ex) for the Provider Network where all the VMs traffic will reach the Internet via external bridge followed by the actual physical NIC (eth1).

In this case, we will configure the external interface (NIC) with a special type of configuration.

The provider interface uses a special configuration without an IP address assigned to it. We will configure the second interface as the provider interface.

Replace INTERFACE_NAME with the actual interface name. For example, eth1 or ens224.

Edit the /etc/network/interfaces file to contain the following:

# The provider network interface
auto INTERFACE_NAME
iface INTERFACE_NAME inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down

Ref: – http://docs.openstack.org/mitaka/install-guide-ubuntu/environment-networking-controller.html

With this interface configuration, the restriction in AWS will hit the OpenStack networking. In mainstream OpenStack setup, the provider interface mentioned above can be configured with special NIC configuration that will have no IP for that interface.

Moreover, the VM packets reaching the Internet via this specially configured NIC will have the IP of OpenStack tenant router’s gateway IP address as the source IP address in each packet.

Like I mentioned earlier, AWS will only allow the packet to flow when the MAC address is known/registered in their environment. Also, the IP address must match the MAC address.

In our case, the packet from the aforementioned OpenStack tenant router will have the IP address of router’s Gateway in every single packet, but the packet source MAC address will be the same MAC address of router’s interface.

Note: – You could see these details using the “ip netns show” followed by “ip netns exec qr-<router_ID> ifconfig” command in the OpenStack controller’s terminal.

Since the MAC address in unknown/not registered in the AWS environment, the packets will be dropped when it reaches the AWS switch. To allow the VM packets to reach the Internet via AWS switch, we need to do some tricks on our OpenStack setup.

Making use of what we have

Possible hacks include:

Registering the router’s MAC address and its IP address with the AWS environment. However, this hack is actually not feasible. AWS doesn’t have the ability to register any random MAC address and IP address inside the VPC. Moreover, allowing this type of functionality will be a severe security threat to the environment. So, the trick we are going to use is:
Making use of what we have. Since we have used a special type of interface configuration for the provider NIC, you could note that the IP address assigned to the provider NIC (eth1) is left unused. We could use this available/unused IP address for the OpenStack router’s gateway. The following command will do the trick:

neutron router-gateway-set router provider --fixed-ip ip_address=<Registered_IP_address*>

IP address and MAC address mismatch

After configuring the router gateway with the AWS registered IP address, each packet from router’s gateway will have the AWS registered IP address as the source IP address but with an Open vSwitch-generated MAC address.

Like I mentioned in the AWS restriction session above, the IP address must match the MAC address registered or else all of the packets with mismatched MAC and IP address will be dropped by the AWS switch.

To make the registered MAC address match with the IP address, we need to change the MAC address of router’s interface. The following steps will handle it for you:

Install the macchanger tool.
Note down the actual / Original MAC address of provider NIC (eth1)
Change the MAC address of Provider NIC (eth1)
Change the MAC address of Router’s Gateway Interface to the original MAC address of eth1.
Now, try to ping 8.8.8.8 from router namespace

If you get a successful ping response, then we are done with Cloud on Cloud setup.

Key points to remember

Change MAC address:

In my case, I had worked in an Ubuntu 14.04 LTS server which has no issue in changing the MAC address using the macchanger tool. However, when I tried it in an Ubuntu 16.04 LTS one, I got an error saying “No permission to modify the MAC address .“ I suspect the error was due to the cloud-init tool not allowing the MAC address configuration.

So before installing OpenStack setup, try changing MAC address of the NIC.

Floating IP disabled

Associating a floating IP to any OpenStack’s VM will send the packet through the router’s gateway with the source IP address as floating IP’s address. This will make the packets hit the AWS switch with non-registered IP and MAC addresses, resulting in the switch dropping the packets.

So, we could not use the floating IP functionality in this setup. However, we could still access the VM publicly by using the following NAT process.

NAT to access OpenStack VM

Like I mentioned above, we could access the OpenStack VM publicly using the registered IP address that we have assigned for router’s gateway.

Use the below NAT command to access the Openstack VM using the AWS EC2 instance’s elastic IP:

$ip netns exec qrouter-f85bxxxx-61b2-xxxx-xxxx-xxxxba0xxxx iptables -t nat -A PREROUTING -p tcp -d 172.16.20.101 --dport 522 -j DNAT --to-destination 192.168.20.5:22

Using this command, all packets reaching 172.16.20.101 with port number 522 will be forwarded to 192.168.20.5:22.

Note: here, “172.16.20.101” is the registered IP address of the AWS EC2 instance and “192.168.20.5” is the local IP address of the OpenStack VM. Also worth noting is that “172.16.20.101” is already an NAT with AWS Elastic IP, which means all traffic that comes to the elastic IP (public IP) will be forwarded to this VPC local IP (172.16.20.101).

It means you could SSH the OpenStack VM globally by using the elastic IP address and the respective port number.

Elastic IP address

For this type of customized OpenStack installation, we required at least two NICs for the AWS EC2 instance:

One for accessing the VM terminal for the installation and for accessing the dashboard. In short, it acts as a Management network/ VM Tunnel network / API network.
Another for an external network with unique type interface configuration and mapped with provider network bridge (br-ex with eth1).

AWS will not allow any packets to travel out of the VPC unless the elastic IP is attached with that IP address.

To overcome this problem, we must attach the elastic IP to this NIC so that the OpenStack VM’s packets will reach the OpenStack router’s gateway. From the gateway, the packets will be embedded with the registered MAC address and matching IP address and will be sent to the AWS switch (VPC environment) via br-ex and eth1 (special type interface configuration) on its way to AWS’s actual VPC gateway. From there, the packets will reach the internet.

Other cloud platforms

In my analysis, most cloud providers (like DreamHost and Auro-cloud) have the same limitations for OpenStack networking. So we could use the tricks/hacks mentioned above in any of those cloud providers to run an OpenStack cloud on top of it.
Note: – Since we are using an QEMU emulator without KVM for the nested hypervisor environment, the VM performance will be slow.

If you want to try OpenStack on AWS, you can register for CloudEnabler’s Cloud Lab-as-a-Service, which provides a consistent and on-demand lab environment for OpenStack in AWS.

Selvaraj is a cloud engineer at CloudEnablers.