Why AT&T is implementing OpenStack and how they are getting it done.


_This post first appeared on the AT&T Innovation Space Blog._

As readers of this blog know, OpenStack is a critical component of AT&T’s plan to virtualize our network and move it into what we call the AT&T Integrated Cloud (AIC). Three months ago, AT&T won the OpenStack Superuser Award. It was a great honor. But we’ve got ambitious goals for our network transformation, so there’s no time to rest on our laurels.

One of my team members, Greg Stiegler, is speaking today at the OpenStack Days Silicon Valley Conference. He’ll discuss what we’ve been doing and learning over the last three months. For those who can’t attend, I want to give an overview. We know we’re being aggressive with our transformation. It’s hard work. But the payoff is real.

For other service providers or large-scale enterprises unsure about following in our footsteps, we
want to clearly explain the benefits of going this direction.

The benefits fit into four main categories. Some of these benefits might be more important depending on your industry. But the combination makes a compelling case for private or hybrid cloud, no matter the industry vertical.


Reduced cost – Moving from dedicated black box or virtualized platforms to true cloud orchestration creates significant infrastructure savings. And through DevOps and automation, AIC also reduces operational costs. Within AIC, we’re able to reduce costs by offering shared as-a-service components to all development teams.

Improved security – Software-defined data centers enhance cloud security. They use overlay networks to logically separate tenants. They also can isolate tenants to discrete physical devices. This minimizes the impact of security incidents. And security groups segregate workloads from each other.

You can also contain threats by quarantining affected virtual machines. And when a workload becomes affected, no one needs to manually intervene. An automated policy will spin up additional capacity.

And the private cloud allows for company-specific and role-based access controls. Security policies are critical to businesses in any industry. You can customize security policies for the individual needs of each business. This lets you protect your company’s intellectual property and sensitive data.

Addressing your business’ unique needs – Using a private cloud also lets you tailor your product or service to your customer’s needs. For instance, AT&T has developed performance profiles (aka Instance Types) that provide varying levels of performance relative to cost. AT&T needs various flavors for different throughput profiles based upon network functions. Most industry verticals will also need separate infrastructure profiles, i.e. Flavor Series that accommodate their different business needs.

New product and service launches – A private cloud lets businesses bring new products and services to market faster than ever before. That’s a huge competitive advantage.

Integrating a private cloud with your existing capabilities can result in new products and services. Our Network on Demand solution is a perfect example of a service enabled by moving from static, black box vendor products to software that we control and can quickly change.

This let us extend service management capabilities directly to the customer and let them adjust their services in near real time. Additionally, a private cloud provides greater business agility as cloud enables micro-services, automated deployments and faster product release cycles.
So how did we realize these benefits? And how can any industry replicate this success?

To make AIC successful, we needed an OpenStack community engagement strategy. We had to break down traditional commercial sourcing walls and redefine our cloud platform’s core function with OpenStack. This included:

Innovating and integrating with existing solutions – We used some of our existing tools and capabilities to customize OpenStack for specific needs.

In some cases, AT&T programmers had to build additional capabilities into OpenStack that didn’t yet exist. For example, OpenStack upgrades were too slow and cumbersome. So we built what we call “Diesel” – based on OpenStack’s Octane project – to deliver these upgrades seamlessly with backwards compatibility.

Establishing a community team to eliminate technical debt – AIC releases were moving away from OpenStack master code due to multiple customizations that were not worked back into upstream development. So we set up a team dedicated to pushing those changes back into OpenStack and making sure we stayed aligned with the OpenStack developer community.

This team eliminated rework, reduced complexity and enabled greater speed. They simply developed new features directly in OpenStack. Now, rather than modifying an OpenStack version, we’re working directly with the community to deliver functions that come with the next OpenStack release version.

Collaborating with companies with similar objectives – AT&T is collaborating with other large service providers and operators through an OpenStack user group.

Three key goals are developing and prioritizing a roadmap that addresses large scale private cloud features and operational needs, increasing upstream contributions, and proposing a single unified roadmap leader in the OpenStack foundation.

This unified roadmap leader can then work within the OpenStack technical community to ensure necessary planning, collaboration and execution across the multiple OpenStack projects. The approach ensures acceleration of both the satisfaction of any private cloud operator needs and OpenStack’s success.

Developing a strategy to meet availability and operational needs – We’ve deployed more than 74 AIC zones, with a goal of 105 by the end of 2016. This provides AT&T with a highly distributed cloud that allows the zones to operate independently.

A failure in one does not affect the others.

AT&T developed an OpenStack Resource Manager (ORM), which works like a travel reservation website – it’s a single place for tenants and operation teams to discover and manage all the zones. We also established a high-speed inter-zone network strategy to support resiliency needs with the ability to move any workload to any zone.

Cultural Transformation – We’ve found that our cultural transformation and large-scale technology initiatives go hand in hand. Any company looking to transition to a private cloud or other technology-driven transformation is essentially also committing to a cultural transformation.
Our cloud initiative and its aggressive goals required end-to-end coordination efforts across multiple organizations. This allowed us to achieve true DevOps at scale much faster. If you want your company to succeed at cloud transformation, you have to train your employees to think cloud native.

We believe this template can help any company develop their transformation strategy for private cloud based upon open source community based software. We’ve seen great success with this approach. I’ll keep you updated as we continue our journey.

Learn more about AT&T’s OpenStack journey in Saxena’s Austin Summit keynote.

Superuser is always interested in community content, email: [email protected].

Cover Photo // CC BY NC