)
One of the virtues of open source software like OpenStack is that anyone can download it, play with it and learn about it without the hassle of dealing with vendor salespeople or the need to justify the effort internally.
But what happens when it comes time to do more than just kick the tires? How do you get from downloaded to deployed? How can you overcome the organizational hurdles to adopting new and disruptive technology? Where do you start? What do you do next?
Of course, there is much to be learned from the experiences of those who have already deployed OpenStack. To better understand the patterns of OpenStack adoption, I spoke to several teams that have successfully introduced the software to their companies.
MercadoLibre: Tapping into a need and running like hell
If the need is strong enough, sometimes getting agile cloud infrastructure adopted can be almost as easy as ‘build it and they will come.’ In many ways, that is the experience that Alejandro Comisario, Maximiliano Venesio, and Leandro Reox had at their company, MercadoLibre, the biggest e-commerce company in Latin America and the 8th largest worldwide.
In 2011, with the company’s development organization starting down the path of decomposing their then-monolithic platform into one composed of loosely coupled services connected via APIs, the infrastructure team saw a dramatic uptick in the number of infrastructure requests their small team was required to fulfill.
“The shift happened so quickly,” said Comisario, technical leader for cloud services at MercadoLibre. “We literally realized overnight that we wouldn’t be able to keep up without help.”
Comisario, Venesio, and Reox, MercadoLibre’s entire infrastructure team at the time, began looking for technology that would allow them to eliminate the manual steps associated with delivering infrastructure to their developers.
The team set some lofty goals for their effort, which they used to articulate what they were doing to the broader company: They wanted to be able to reduce the time required to deliver a developer a production-ready virtual machine from 2 hours to 10 seconds, and they wanted to be able to do so without human intervention.
When they found OpenStack, they quickly realized that it could be exactly what they were looking for. MercadoLibre’s fast-moving culture allowed the trio to move quickly to put an OpenStack environment in place, in spite of the project’s relative immaturity.
“It turned out that the OpenStack way of doing things—researching, diving into the code, and testing how things work and scale—really fit with the way we do things at MercadoLibre,” said Leandro. “We were able to jump right in and define a set of tests for our OpenStack environment, and start testing.”
Their initial testing, on OpenStack’s second release, identified several issues that would prevent them from going to production, however the transition from the Bexar release to the Cactus release happened just at the right time. Further testing on Cactus gave them the confidence they needed to put their cloud into production.
Once in production, and as their developers realized they could provision infrastructure as fast as they could consume it, adoption took off.
“The whole company was craving something like this, and what it makes possible for us,” said Venesio, senior infrastructure engineer at MercadoLibre.
Still, the team was careful to manage developers’ expectations. They needed to make sure that their developers knew that existing apps wouldn’t be able to run directly on the new private cloud without modification.
“We had to make sure our developers were prepared to write stateless applications for the cloud,” said Comisario. “It was a huge cultural mind shift for them. In some cases we had to hurt developers’ feelings by insisting that not even one instance would persist their data. So they needed to adjust their thinking.”
The team was vigilant in educating their development team and challenging them to adopt the latest best practices for building cloud-ready applications. They sent emails, held brownbag lunches and formal training sessions, and ensured that the cloud environment was properly documented. As a result of their efforts, MercadoLibre’s developers are now as comfortable developing for the private cloud as they were for the company’s traditional virtualized environment.
The automation they were able to achieve with their private cloud has paid off in allowing MercadoLibre to dramatically scale its infrastructure footprint. What started as a highly committed infrastructure team of three people supporting 250 developers, 100 servers, and 1000 VMs has grown into a team of 10 people supporting over 500 developers, 2000 servers, and 12,000 VMs.
Workday: Building the business case for OpenStack
For the team at software-as-a-service company Workday, the decision to adopt OpenStack was less of an operational imperative and more a strategic one.
Workday’s path to adopting private cloud began in earnest in 2013 when company executives agreed to invest in a broad Software Defined Data Center (SDDC) initiative. The hope for the initiative was to deliver greater automation, innovation, and data center efficiencies.
Workday established its vision for its private cloud between the company’s infrastructure, engineering and operations teams, and an agreement to start an exploratory initiative was made. This lead to the company’s hiring of Carmine Rimi as its director of cloud engineering to lead the charge.
Rimi’s first task at Workday was to expand the initial business case for the larger private cloud initiative at the company.
The cornerstone of his business case was the increased agility that would be offered by an SDDC. This increased agility would help the company reach its aspirations of continuously deployed software with zero downtime. The API for the SDDC would allow Workday’s application and platform development teams to innovate in ways that weren’t possible before.
Hardware efficiency also factored into his business case. Workday has aggressive goals to increase the utilization rates of its existing data center hardware and resources.
“We found that we already had middleware technology that could take advantage of a private cloud. That middleware was used to deploy dev/test environments to public clouds. With a private cloud we could extend the middleware to create a hybrid cloud solution. By employing a hybrid cloud strategy, Workday could migrate workloads between public and private clouds, which would give Workday the ability to maximize its hardware utilization rates, delivering savings back to the business,” said Rimi.
Finally, Rimi’s cloud strategy posited that starting with simple workloads that are stateless and scale horizontally would allow Workday to deploy its private cloud with reduced risk, and allow the cloud operations capabilities to mature naturally.
“Getting our feet wet and learning how to operate the new cloud, with a small workload footprint, is akin to traditional R&D efforts that allow for experimentation in safe environments,” offered Rimi.
With a solid business case in place, Rimi evaluated several prominent private cloud offerings, including OpenStack, against a thorough set of evaluation criteria that included each offering’s openness, ease-of-use, flexibility, robustness, resilience, supportability, community, and momentum. Based on the results of his assessment, Rimi and his team selected OpenStack and set out to build a production-ready private cloud.
Having successfully deployed its first viable OpenStack cloud, Workday continues to drive toward greater adoption of the new SDDC environment. To accomplish this, Rimi has taken a multi-pronged approach centered on:
- Narrowing the team’s focus to cloud-ready workloads, particularly those stateless applications in its portfolio
- Defining criteria and a process for what should move over
- Setting goals for engineering for the migration of those applications
- Communicating and educating the Workday stakeholder teams using scrum meetings, demos, videos, and training sessions on OpenStack
“Our cloud supports many workloads, some in production and many more in the pipeline towards production,” said Rimi. “We eventually want to move everything over, and I expect that at some point we’ll reach a tipping point where we’ll see a flash flood of migration activity. We’re putting the pieces in place each day to be able to handle that level of activity when it comes.”
Best Buy: Coloring Outside the Lines
With $43 billion in annual revenues and 140,000 employees, electronics retailer Best Buy is the largest of the companies profiled here by a healthy margin. And so, while the processes navigated by members of bestbuy.com’s infrastructure team in delivering an OpenStack-based private cloud might not be surprising, the agility with which they navigated these processes is impressive.
To deliver their first OpenStack cloud at Best Buy, Steve Eastham, director of web engineering, and Joel Crabb, chief architect, had to rely on their creativity to overcome the many barriers that got in their way.
Best Buy’s OpenStack initiative grew out of an early 2011 effort to understand the various business processes at play in delivering the company’s bestbuy.com e-commerce site. That effort revealed significant inefficiencies in its quality assurance processes. QA was a significant expense in each major release of the site, an event that occurred two to four times each year. A large fraction of this cost was attributed to manual environment setup, reconciling discrepancies, and resolving resource availability problems.
To address these issues, bestbuy.com kicked off a “QA on-demand” effort led by Eastham and Crabb, focused on identifying and rooting out bottlenecks in the bestbuy.com QA process. Among that project’s primary recommendations were automating QA delivery and empowering teams with self-service.
While Eastham and Crabb were able to use the prospect of very significant dollars spent on QA to justify an investment in private cloud, they quickly ran into a problem: While the approval was there, the cash was not. There was no capital budget available to buy hardware for the project.
With necessity as the mother of invention, the team took a novel approach to financing their cloud: they traded two developer headcount with another group for the budget to go buy the hardware.
With the resources that they raised through horse-trading, they set out to buy the hardware they would need. For this they worked with HP, their hardware vendor at the time, to get a good deal. Through shrewd negotiation and ruthlessly cutting hardware requirements, they were able to reduce hardware costs by nearly half.
In the same vein, Eastham and Crabb worked out a deal with the company’s network team to take advantage of capacity on existing core switches, saving the typical expense associated with new networking gear.
“We were skating on pretty thin ice on this one,” said Eastham. “These were not common practices at Best Buy at the time, and they’re still not. We flew under the radar a bit. We should have gotten resistance, but didn’t.”
Navigating financial constraints was only the first of many hurdles. At the time, there was no way for Eastham and Crabb to find OpenStack experts to staff their project. So, they ended up building the team from the ground up by pairing traditional Java developers and system administrators.
“We just threw them in a room and said ‘figure out how to do this’,” said Crabb. “One of the Java guys was like, ‘This is insane, you can’t do this. I don’t know what you’re talking about.’ We had to meld the different styles of two types of teams to get what we wanted—a software-driven, unit-testable, pipelined development process.”
The team’s hustle early on allowed them to rack up some impressive wins. They were able to quickly replace a legacy development environment, reduce the number of QA environments, and in the process transform the way their teams worked and the speed at which they could deliver.
Their success put them in a good position to ask for additional resources for their private cloud initiative. This time around, though, they had the support of C-level company executives.
Eastham and Crabb received the funding they needed to add additional support staff and five new racks of gear. The first cloud in this second wave of projects is an OpenStack environment running Hadoop for analytics. It is already in production.
Conclusion
In the stories of MercadoLibre, Workday and Best Buy there are many principles that can guide you down the path towards successfully introducing OpenStack: being attuned to the needs of developers, the business, and other potential user communities; working within the established processes of your company; collaborating with other organizations; and being willing to color outside the lines when necessary. These are all valuable soft-skills to have in your OpenStack deployment tool belt.
There’s no perfect path to adopting OpenStack within your company—the path to take depends on many factors specific to you, your organization and your situation. While this fact can be disconcerting for OpenStack fans wondering how to get their first project kicked off, it really is good news. It means the sky’s the limit for how far you can go with OpenStack. What you’re able to accomplish is limited only by your creativity and resourcefulness.
- How to introduce OpenStack in your organization - November 14, 2014
