User Stories

Where are they now? Superuser Awards winner: CERN

Over 90% of the infrastructure in the CERN computer center is managed and provisioned by OpenStack.

May 6, 2020
image

If you’ve been around the OSF community for any amount of time, chances are you’ve heard the name CERN.

Famous for their Large Hadron Collider, Higgs boson, and antimatter studies, the Geneva-based laboratory has spent decades researching physics and the universe. So what does that have to do with OpenStack? All of that research produces massive amounts of data, thus requiring a substantial amount of infrastructure.

Keep reading to find out how CERN’s OpenStack environment has evolved since they won the first Superuser Award at the OpenStack Summit six years ago.

What has changed in your OpenStack environment since you won the Superuser Awards?

At the OpenStack Summit Paris in 2014, CERN received the first Superuser Award from Guillaume Aubuchon, CTO of Digitalfilm Tree.

Presentation of first Superuser Award at Paris OpenStack Summit.

At the time, CERN’s cloud had been in production for a year with 65,000 cores running Havana providing VMs, images and identity. After six years and 13 upgrades, the CERN cloud now covers 11 OpenStack projects adding containers, bare metal, block, share, workflows, networking and file system storage.

What is the current size of CERN’s OpenStack environment?

Snapshot of CERN’s infrastructure dashboard.

Currently, the CERN cloud is around 300,000 cores across 80 cells with big recent growth in OpenStack Magnum to manage Kubernetes clusters, OpenStack Ironic servers for all the computer center hardware, and Fileshares with CephFS.

What version of OpenStack is CERN running?

We are in the process of upgrading from Stein to Train with most components already running Train. We use the RDO distribution.

What open source technologies does your team integrate with OpenStack?

The list is very long! The aim for the CERN cloud environment was to build a toolchain based on a set of open source projects which could also be used by other labs collaborating with CERN. A few examples are:

Cloud and Containers

Configuration

  • Puppet and Foreman for configuration management
  • Terraform for automated provisioning (including external clouds)

Monitoring

Storage

Identity

Workflows

  • Gitlab for version control, continuous integration
  • Koji for builds
  • Rundeck for automation

What workloads are you running on OpenStack?

Over 90% of the infrastructure in the CERN computer center is managed and provisioned by OpenStack. This includes the physics processing and storage, databases along with the infrastructure for the laboratory administration. The remaining hardware in the computer center is now being enrolled into Ironic to ensure strong resource management, accounting and lifecycle tracking.

How big is your OpenStack team?

The production support team in the CERN IT Department is around seven engineers with further students and fellows contributing to various project enhancements.

How is your team currently contributing back to the OpenStack project? Is your team contributing to any other projects supported by the OpenStack Foundation (Airship, Kata Containers, StarlingX, Zuul)?

CERN has made over 1,000 commits to OpenStack since the implementation started in 2011. The largest three OpenStack projects CERN have contributed to are Magnum, Nova and Keystone. CERN’s experiences have been presented at more than 30 talks at OpenStack summits as well as regional events such as the open Infrastructure days which have provided an opportunity to share the experiences of running OpenStack at scale and our current focus areas. This included an OpenStack day at CERN in 2019 covering experiences of OpenStack usage in science and hosting the Ironic mid-cycle meetup in 2020.

The CERN blog is available at https://techblog.web.cern.ch/techblog/ and local developments are shared at https://github.com/cernops.

CERN has also contributed to governance and project management including an elected OpenStack individual board member, two members of the User Committee and PTL/core roles in Magnum, Keystone and Ironic.

What kind of challenges has your team overcome using OpenStack?

Given the demands of the Large Hadron Collider and the CERN experiments, provisioning more computing capacity without increasing the number of engineers was a challenge to overcome. Working with other members of the open source community in areas such as Container Orchestration-as-a-Service, Nova Cells, Identity Federation and Spot Market functionality has allowed these new features to be developed, reviewed by community and further enhanced. OpenStack Special Interest Groups such as the Scientific SIG and Large Scale SIG have provided a useful framework for debate, information sharing and common contribution.

A single framework for tracking, authentication and accounting for bare metal, virtual machines, storage and containers has been a major benefit for the CERN IT department. Allowing users to have self-service resources in a few minutes while ensuring that these are clearly allocated (and expired if appropriate) allows the CERN cloud users to focus on the goals of the laboratory rather than how to get the infrastructure they need.

Stay tuned for more updates from previous Superuser Award winners!

 

Cover Image: CERN

Tags: , , , ,

Ashlee Ferguson
Latest posts by Ashlee Ferguson (see all)