OpenStack security, piece by piece

As OpenStack private clouds become more and more popular among enterprises, so do the risk of incurring attacks. In part 1 of tighten the security of your OpenStack clouds we defined common threats for an OpenStack cloud and discussed general recommendations for threat mitigation tools and techniques.

In addition, you need to ensure to treat each OpenStack component separately and incorporate security best practices according to the component roles and importance within the complex OpenStack environment. In this post we will discuss the different vulnerabilities of all components and list the actions that need to be taken.

Looking at the latest OpenStack security vulnerabilities you will find many related to each of the OpenStack components, such as, Nova, Glance and Neutron. For example, a few Swift versions before Kilo and Liberty contain a hole which allows remote attackers to launch a Denial of Service (DoS). Another example is a Nova version, before the Kilo release, which allows attackers to obtain sensitive information by reading log files.

OpenStack CLI Security

Use of the OpenStack CLI tools requires a username and a password. Many OpenStack guides recommend using the OS_USERNAME and OS_PASSWORD environment variables that are defined in the OpenStack RC file. However, it goes against best security practices to store the credentials in a plain, unencrypted file.

For a more secure option, OpenStack CLI tools can request a username and password for each request, or they can use a provisioned authentication token. Also a dedicated node or VM, located in separated internal DMZ to use OpenStack CLI tools can be used in this additional node for such purposes only. Ensure to disable unnecessary services; allow access by SSH only and from a trusted network; disable the Bash history and store all of the logs in an isolated, remote, secure, highly available storage repository.

Nova security

Nova is probably the most complex OpenStack service, it communicates with many, many other services and has a number of configuration options; therefore Nova is the number one service to ensure security hardening.

The owner of Nova configuration files should be root, the group of configuration files should be nova. Also the permissions should be 640 (read/write for owner, read for group).
Disable PCI passthrough for your hypervisor to restrict direct access from a VM to hostís hardware, such as DMA.
Use sVirt or SELinux/AppArmor to put your hypervisor in a separate security context.
Some of the hypervisors have memory optimization techniques to de-duplicate memory pages used by VMs on the host. For example, Xen has Transparent Page Sharing (TPS) and KVM has Kernel Samepage Merging (KSM). To mitigate cloud tenant threats use a strict separation of tenants per compute; therefore, disable hypervisor memory optimization techniques for such computes.

Store hypervisor logs in a secure remote storage.
Monitor the integrity of your hypervisor executable files. For example, you can use debsums for Debian based operating systems, rpm -V for Red Hat.
Use Address Space Layout Randomization (ASLR) and Position Independent Executable (PIE) for hypervisor executables; qemu-kvm supports this.
Use TLS for VNC or SPICE sessions.
Ensure that Nova securely communicates (via TLS) with the other OpenStack services, such as Keystone or Glance. This is also a general recommendation for all of the OpenStack services.

Glance security

Glance stores images, which are used to launch new VMs. To avoid attacks to the images integrity it is important to keep this service secure.

Do not use pre-built images or Docker containers from untrusted sources, because they may contains security breaches or malicious software.
Use Glance image signing (available in Mitaka).

Neutron security

Neutron provides network connectivity and IP addresses for VMs in the cloud. Neutronís architecture is based on plugins, so it is important to understand which plugins are required and which plugins are used for third party solutions; then disable any unnecessary plugins.

Use isolated management network for OpenStack services
Use L2 isolation with VLAN segmentation or GRE tunnels
Enable security groups in Neutron and disable security groups in Nova (all Nova security group API calls should be forwarded to Neutron)
Secure Neutron API endpoints with TLS
Use iptables along with ebtables rules to prevent MAC spoofing and ARP spoofing attacks.
Use network quotas to mitigate DoS attacks

Message Queue (RabbitMQ) security

Message queue facilitates communication for OpenStack services and RabbitMQ is the most popular solution for the OpenStack cloud. OpenStack does not support message signing, so the message queue should provide secure transport for OpenStack services.

Delete the RabbitMQ guest user
Use a separate RabbitMQ virtual host for each OpenStack service; use unique †credentials and appropriate permissions for each RabbitMQ virtual host
Secure RabbitMQ API with TLS
Store RabbitMQ logs in a secure remote storage

Keystone security

Keystone provides identity services for other OpenStack services and it should be properly protected from spoofing and other attacks.

Keystone does not provide methods to enforce policies on password strength, password expiration time or failed authentication attempts as recommended by NIST. However, Keystone can use an external authentication system that supports all these requirements.
Multi-factor authentication should be enabled through an external authentication system, such as Apache HTTP server.
By default, the token expiration time is 1 hour. The recommended expiration value should be set to a lower value. However, the interval should be long enough for OpenStack services to complete their requests, otherwise the operation will fail if the token expires before the services complete a request. Note that certain operations are especially lengthy, for example, when Nova transfers a disk image onto the host.
Use Fernet tokens, which are are designed specifically for REST APIís as they are more secure than standard tokens and they require fewer resources.
Use Keystone domains for more granular access control for tenants. A domain owner can create additional users, groups, and roles to be used within the domain.

Cinder security

Cinder provides high level API to manage block-level storage devices and is actively used by Nova. It should be protected from denial of service, information disclosure, tampering and other threats.

The owner of Cinder configuration files should be root, the group of configuration files should be cinder. Also the permissions should be 640 (read/write for owner, read for group).
Set the maximum request size. By default it is not defined and an attacker can send a large request to crush Cinder (DoS attack).
To secure delete Cinder volumes, use volume wiping.

Swift security

Swift provides durable object storage for OpenStack, primarily to store Glance images, so it should be protected as well.

The owner of Swift configuration files should be root, the group of configuration files should be swift. Also the permissions should be 640 (read/write for owner, read for group).
Use a dedicated network †for storage nodes.
Use a firewall to protect public interfaces on proxy nodes

In conclusion

In addition to the suggestions above, it is important that you stay tuned to OpenStack security vulnerabilities and strive to maintain an updated environment. Security hardening of your OpenStack environment must be addressed on many levels, starting from the physical (data center equipment and infrastructure), through the application level (user workloads) and organization level (formal agreements with cloud users to address cloud privacy, security, and reliability). There are many issues and actions to consider for hardening OpenStack security and we hope that post gave you the information and tools to secure your OpenStack cloud.

This post first appeared on Stratoscale’s blog.

Superuser is always interested in how-tos and other contributions, please get in touch: [email protected]

Cover Photo/ // CC BY NC

Author
Recent Posts

Superuser

Superuser Magazine is the Open Infrastructure Foundation's official online publication. It covers open infrastructure ecosystem news, case studies, event recaps, product updates and announcements, project releases, and more.