)
It’s time for the community to help determine the winner of the Superuser Award to be presented at the OpenStack Boston Summit. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.
Now, it’s your turn.
Paddy Power Betfair is among the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline, Tuesday, April 4 at 11:59 p.m. Pacific Time Zone.
Please specify the team and organization for nomination.
The team is the delivery enablement team from Paddy Power Betfair. The delivery enablement team is made from up from four geographical teams which reside in the United Kingdom, Romania, Ireland and Portugal. It consists of a core team that looks after the core OpenStack platform, consisting of six people with two team members from the London, Cluj, Dublin and Porto offices. The core team is supplemented by an application on-boarding team with six people in each location, which helps development teams use the self-service automation to migrate their applications onto the OpenStack private cloud. Everything we do is automated using OpenStack, so we don’t have a need for huge teams to support the platform.
How has OpenStack transformed your business?
We have built a true DevOps and continuous delivery model for developers using OpenStack as the middleware. We’ve also migrated 25 percent of our production applications onto OpenStack in a year, accounting for over 100 applications. We went from 500 deployments a week to over 1,000 deployments a day using the OpenStack APIs to increase time to market. We have been able to automate the scale out of 600 compute nodes across two datacenters, with the end state 1,300 compute nodes which will make up 100,000 cores and 2.08 petabytes of storage, all running on OpenStack. We have provisioned over 2,000 virtual machines (VMs) a day, deployed apps across two active datacenters to give applications resilience and reduced the mean time to recover from failure. Lastly, OpenStack has helped us automate the deployment of code, virtual/physical machines, networking, compute and storage for our delivery teams.
How have you participated in or contributed to the OpenStack community?
We have contributed modules to the OpenStack Shade library as we use it to create Ansible-OpenStack modules. We have worked with Red Hat on blueprints to make leaf spine architecture available through the Triple O installer. We have written our OpenStack reference stack whitepaper to make it easier for companies looking to go on a similar journey to us. The business has given us investment to build a test lab which will allow us to open up our OpenStack platform to Paddy Power Betfair developers that want to contribute to OpenStack by putting pull requests into our lab environment, which we will then push upstream. We also frequently present at OpenStack days and DevOps conferences.
What open source technologies does Paddy Power Betfair use in its IT environment?
We use OpenStack, KVM, Centos, VirtualBox, Ironic Standalone, Ansible, Jenkins, Thoughtworks, Go, Chef, Packer, Yum, Bower, Docker, Kubernetes, Pacemaker, HAProxy, Nginx, Kafka, Zookeeper, Storm and Cougar.
What is the scale of your OpenStack deployment?
Paddy Power Betfair supports about 1.7 million active users, 135 million daily transactions, 3.7 billion daily API calls, 2.5 terabytes of daily log data output and about 120,000-per-second monitoring points. We run four OpenStack clouds, two in each of our data centers (one for tooling and one for test/production applications). We also run over 6,000 VMs across those four OpenStack clouds and provision over 2,000 VMs a day. We currently run 600 compute nodes (KVM hypervisors) and one petabyte of storage. End state is 1,300 compute nodes (KVM hypervisors) and 2.08 petabyptes of storage across two data centers. We also have a development community of over 500 developers who use the platform for continuous delivery. OpenStack is currently powering over 100 micro-service customer facing applications, which is increasing each day, along with the end state 350 micro-service applications.
What kind of operational challenges have you overcome during your experience with OpenStack?
We have implemented an OpenStack upgrade process where we will spin up a new version of OpenStack in parallel to the existing one, switch a single API endpoint and automatically migrate applications onto the new OpenStack when they next deploy, so upgrades are seamless to our developers. Our network can serve application requests using our leaf spine architecture in under a millisecond. As we hit our APIs very hard with about 1,000 deployments a day, we have had to increase cascading timeouts across our services since we also have around 100 API calls a second to Nova, Neutron and RabbitMQ. In this case, we run three high availability physical controllers per OpenStack cloud and Swift object storage for Glance. Here is a link to our published blogs on our challenges with OpenStack.
How is Paddy Power Betfair innovating with OpenStack?
We are deploying MySQL and Informix databases immutably onto OpenStack, doing rolling updates using Cinder. We are also deploying our Betting Exchange using Cinder to support high performance workloads, which allows us to stream betting information to customers. All of our workloads are designed to be immutable, so a new VM is created every deployment. We also use A/B immutable networking, so applications are deployed onto a brand new network each time we deploy (ACL cleanup is automatic). We use Nuage Networks as our software-defined networking provider to use them as our overlay network to run containers. We have also automated the underlay network that powers OpenStack using Ironic Standalone to build out our leaf spine networking and network kit.
How many Certified OpenStack Administrators (COAs) are on your team?
There are four COAs on the Paddy Power Betfair team.
Find out more about the Boston Superuser Awards nominees here, and cast your vote by Tuesday, April 4 at 11:59 p.m. Pacific Daylight Time. Voting is limited to one ballot per person.
- Exploring the Open Infrastructure Blueprint: Huawei Dual Engine - September 25, 2024
- Open Infrastructure Blueprint: Atmosphere Deep Dive - September 18, 2024
- Datacomm’s Success Story: Launching A New Data Center Seamlessly With FishOS - September 12, 2024
