OpenStack enables Workday services to be tested at scale without increasing operational cost nor capital expenditure.


It’s time for the community to determine the winner of the Superuser Award to be presented at the OpenStack Austin Summit. Based on the nominations received, the Superuser Editorial Advisory Board conducted the first round of judging and narrowed the pool to four finalists.

Now, it’s your turn.

The team from Workday is one of the four finalists. Review the nomination criteria below, check out the other nominees and vote before the deadline, Friday April 8 at 11:59 p.m. Pacific Time Zone.

How has OpenStack transformed your business?

Workday has moved from static virtualized environments to a fully elastic and scalable platform based on OpenStack. Our team has created an on-boarding process driven by OpenStack APIs. New application services are enabled in our pre-production environment in minutes; our legacy on-boarding process used to take days. This is 90 percent improvement in response time.

Network isolation in our legacy platform was enforced by changes in our networking infrastructure. With OpenStack, our underlay networking infrastructure does not require changes at all. All is driven by our Neutron APIs and centralized changes in Neutron-managed overlay networks. WPC deploys security policies across multiple data centers from Jenkins jobs. Our network engineers are not any longer distracted with manual changes or unlimited request tickets. OpenStack has provided a huge impact in the scalability tests of some application services. Workday services are now tested at scale without increasing operational cost nor capital expenditure. This has changed not just the reliability of some services, but the culture around testing and increasing the scope of those services. Patches in our applications don’t need to be fully coordinated across multiple environments.

OpenStack has provided a new mechanism to deliver features and updates based on images as a service. Glance hosts hundreds of images that are dynamically uploaded and deleted from our clusters. With a Gluster-based image replication process, we guarantee all users will have exactly the same service version under the same configuration. OpenStack has helped to consolidate all existing application deployment systems in just one.

How has the organization participated in or contributed to the OpenStack community?

Workday Cloud Engineering (CE) team has been active in the OpenStack community since March 2014. Our entire engineering team is registered as active OpenStack developers and code reviewers. Please see this report in Stackalytics.

Workday has presented technical sessions in the following OpenStack Summits: Paris, Vancouver and Tokyo. A couple of sessions have been already accepted for Austin. Workday has moderated diverse sessions in all the Operators Meet-ups from San Antonio in July 2014 to the latest one in Manchester, UK in February 2016. Workday has moderated sessions in networking, operations practices and osops code [12]. Workday team has contributed code upstream in the following projects: neutron, rally, glance, keystone, cookbooks and many others. Workday sponsored a core developer for neutron from June 2014 to Nov 2015. Workday has published in social media networks such as Twitter and LinkedIn capacity metrics of its private cloud. Workday sponsored a member of the User Committee appointed by the OpenStack board of directors. Workday has improved the deployment model based on Chef and sponsored a core developer for the openstack-chef-repo. Workday is the main sponsor for the newly created East Bay OpenStack meet-up. [10] Our team members have presented in the following meet-ups: SF OpenStack and East Coast OpenStack Meetups. Workday has also provided support for new community outside USA. In March we will be presenting at the OpenStack Mexico Meet-up. OpenStack at Workday does not use any vendor specific distribution. All the work has been done by Workday’s team.

What is the scale of your OpenStack deployment?

Workday deployed OpenStack in a multiple data centers across geographical zones: Portland, Atlanta and Dublin, Ireland. By the end of our second quarter, we will have deployed OpenStack clusters in Amsterdam and Ashburn. Including the development environments, by the end of the fiscal year we will have over 650 servers running OpenStack services and more than 50,000 cores as total capacity with a CPU over-allocation ratio of 1:2 and 1:1 for memory. By the next FY, Workday will double down the total capacity. The workload instances creation ratio is over 1000 instances per day including both our production and development clouds. OpenStack on OpenStack deployments with one single command in data centers and not just in development environments. Over 100 monitoring metrics reporting health, state and performance of our clusters. More than 900 virtual networks deployed with over 30 security policies per virtual ports give us over 2500 security rules per cluster. Workday CI/CD for OpenStack automatically deploys multi-node clusters running: Tempest, Rally and a customized validation scripts. Our development cloud has deployed over thousands of these clusters during the last six months.

How is the team innovating with OpenStack?

Workday’s deployment has fulfilled very rigorous security requirements. The most relevant ones are:

  • All APIs utilize SSL.
  • Third party libraries (django, libvirt, etc.) should have all security patches recommended by Workday’s security and compliance team.
  • Certificate based authentication in all end-points.
  • Error and UI logs are not world readable and all based on customized encryption mechanisms.
  • Configuration files containing passwords are not world readable and all based on customized encryption mechanisms.
  • OSSEC enabled in all cluster.
  • OS and 3rd Party RPM Packages signed and verified.
  • RabbitMQ uses TLS
  • MySQL uses TLS On the Operations readability side, our OpenStack deployment fulfills these requirements among others.
  • Centralized logging. All logs are sent to solas system for elastic analysis.
  • Centralized monitoring with alerts based on thresholds.
  • Scalability of the cluster based on regions.
  • Hot patch deployment with no downtime three time per week in non-customer environments and every week for customer clusters fully automated with Chef as a Configuration Management system.
  • Run-books and training for the network operations center (NOC)
  • DB backup and recovery plan.
  • Disaster recovery (DR) plan. Workday has innovated on the way it has implemented its OpenStack Deployment & CI/CD: [Containers-based deployment]( in portable development environments like laptops.
  • OpenStack on OpenStack continuous integration deployments.

Who are the team members?

Workday, Inc. Team: Cloud engineering & infrastructure team

  • Philip Reynolds
  • Vaira Arunachalam
  • Paul McDonnell
  • Derek Organ
  • Rose Kim
  • Steven Tan
  • Imtiaz Chowdhury
  • Sachin Shukla
  • Sirisha Areti
  • Kyle Jorgensen
  • Michal Stolarczyk
  • Howard Abrams
  • Jeff Fischer
  • Megan Baker
  • Adrian Smith
  • Kasey Alusi
  • Guido Patanella
  • Johanni Thunstrom
  • Mick McCarthy
  • Sergio de Carvalho
  • Sitakanta Hotta
  • Ken Dove
  • Mark Gloshen
  • Samuel Cassiba
  • Edgar Magana

Find out more about the Austin Superuser Awards finalists here and cast your vote. Voting is limited to one ballot per person, and closes on April 8 at 11:59 p.m. Pacific Time Zone.

Cover image courtesy Workday, Inc.