)
OpenStack is used by numerous universities and research facilities around the world as their cloud platform of choice. The Scientific Working Group is dedicated to representing and advancing the use-cases and needs of research and High-Performance Computing (HPC) atop OpenStack. It’s also a great forum for cross-institutional collaboration.
Interview by Gene Kuo via IRC with several members of the scientific OpenStack community – see list below.
Why should researchers care about OpenStack’s Scientific Working Group?
It’s a rare opportunity for researchers to connect with the ops running their research platform. In recent years, the idea of “ResOps” has been introduced: These are people dedicated to outreach into research faculties and to help scientists transition onto the cloud as effectively as possible. Researchers trying to get to understand what OpenStack can bring to their research institute or University will find in the members of the Scientific Working Group like-minded people who can guide them in their journey. Researchers should also be aware that the Scientific Working Group is a body that can advocate for changes in OpenStack.
Why OpenStack for Research?
The traditional HPC model is limited in what it can provide. Researchers often need access to computing capabilities in-between their laptop/workstation and a HPC batch system – both in terms of size and performance, but also with respect to network and data locality. Novel solutions based on Open Source platforms (e.g. Apache Mesos, Google’s Kubernetes, and OpenStack) allow the deployment of specialized solutions on commodity as well as specialized hardware. Research can tightly integrate their research data and computing infrastructure so experiments are reproducible. Two other major factors that make it a good solution for research are its mature community and cost compared to alternative solutions
Being an open source platform, OpenStack gives researchers the flexibility to understand and interpret their solutions better (and faster!). Since they can modify their experimental data iteratively, OpenStack becomes handy for proof-of-concept that can lead to great discoveries!
What are the key differences between scientific OpenStack Clouds and other general OpenStack Clouds?
Scientific OpenStack clouds are usually ahead of major public or private clouds in that they integrate specialized hardware, networking, compute methods and accelerators. As such, the Scientific Working Group also works on support and documentation for the usage of specialized hardware and accelerators (e.g. GPGPUs), HPC capabilities such as large-memory, high-speed interconnects, parallel file systems and bare-metal cloud provisioning. Quantum and edge computing are the latest hot topics for new compute modalities!
In contrast, generic OpenStack clouds –and enterprise virtualization generally– rely on standard CPUs with commodity networking. Scientific clouds have a remit to reduce network latency, optimize ease of compute hardware access/use and achieve new HPC capabilities.
How can researchers speeds things up with OpenStack?
With OpenStack, compute, storage and network are easily managed so that researchers are able to get their workloads up and running faster than using traditional HPC. For example, Heat templates and Ansible can be used to manage VM configurations, while multi-tenant and SDN facilitate private experiments to run.
What kinds of challenges do researchers face when using OpenStack clouds in their organization?
The biggest challenge is that researchers are not system administrators. The complexity of OpenStack deployments need to be simplified for users, allowing them to focus on using the infrastructure for their research. The Scientific Working Group collects use cases and good-practice so that OpenStack as a community will continue to improve for all researchers…
What features are missing in OpenStack to provide better infrastructure for scientific research?
The following come to mind: pre-emptible instances, hypervisor efficiency and placement within the physical network. Through its work, one of the goals that the Scientific WG wants to achieve is combining both the benefits of clouds and the performance of bare metal.
Get involved!
Here are some quick ways to get involved with the group:
- Contact the co-chairs of the Scientific WG: https://wiki.openstack.org/wiki/Scientific_working_group
- Join the Mailing List and do a bit of lurking to find out how and what this community does. Hint: are there other types of OpenStack users or working groups you want to hear from as well?
- Attend the OpenStack Summit – the next one is in Sydney, Australia followed by Vancouver, Canada and Europe in 2018.
We see more and more OpenStack clouds being integrated into research institutes and universities as their cloud platform for running research computing. The agility of OpenStack allows researchers to speed up their research and reduce the cost of managing infrastructure. If you are interested in OpenStack running scientific workloads, don’t miss the chance to meet with the people of the Scientific Working Group.
Contributors to this article, listed by IRC nickname where available:
b1airo – Blair Bethwaite, Monash University, Melbourneb1airo –
rbudden – Robert Budden, Pittsburgh Supercomputer Center
trandles – Tim Randles, Los Alamos National Laboratory
martial – Martial Michel, National Institute of Standards and Technology
jmlowe – Mike Lowe, Indiana University
oneswig – Stig Telfer, StackHPC Ltd
DFFlanders (Twitter/IRC) – David Flanders
Sonia Ramza, OpenStack Foundation
Cover Photo // CC BY NC
- Demystifying Confidential Containers with a Live Kata Containers Demo - July 13, 2023
- OpenInfra Summit Vancouver Recap: 50 things You Need to Know - June 16, 2023
- Congratulations to the 2023 Superuser Awards Winner: Bloomberg - June 13, 2023
Stig Telfer
Superuser
