)
We’re spotlighting users and operators who are on the front lines deploying OpenStack in their organizations to drive business success. These users are taking risks, contributing back to the community, and working to secure the success of their organization in today’s software-defined economy.
Today, we sit down with Jeremy Stanley, a long-time computer hobbyist and technology generalist who has worked as a Unix and GNU/Linux sysadmin for more than two decades. Jeremy currently serves on OpenStack’s Project Infrastructure and Vulnerability Management teams. He lives on a small island in the Atlantic and spends his spare time writing free software, hacking on open hardware projects and embedded platforms, restoring old video game systems and reading articles on math theory and cosmology. You can learn more about him on his OpenStack profile.
Describe how are you using OpenStack. What kinds of applications or workloads are you currently running on OpenStack?
The OpenStack Project Infrastructure consumes OpenStack services generously provided by public service providers, and in turn uses them to support the OpenStack community and associated software ecosystem. This effort includes most of the online systems with which our community interacts such as our code review interface, revision control servers, mailing lists, voice conferencing, paste publishing, etherpad collaboration, blog aggregator and a variety of IRC bots for meeting management, conversation logging, bug linking, announcements, channel permission control and code review notifications. These many services, facilitated by the Infrastructure Team, are configured and managed by the OpenStack contributor community at large.
Perhaps most exciting among the services is the OpenStack continuous integration and testing suite, where OpenStack virtual machines are used to test proposed changes for the ongoing development of OpenStack and related software. This creates a virtuous cycle of improvement since it actively draws the project community into a place where issues spotted through use of these services can be more readily identified, solutions crafted and ultimately implemented not just by the providers donating services, but by everyone running the same software.
What have been the biggest benefits to your organization as a result of using OpenStack?
A major benefit to us is, I think, OpenStack’s broad interoperability. We can and do distribute our application load in parallel throughout environments at multiple public service providers, so if one has maintenance outages or unanticipated trouble we can still continue operating unimpeded. We’re now looking at even supplementing this with our own dedicated "private" OpenStack environments managed and run by the community just like the rest of our infrastructure. This designed-in interoperability also allows for easier prototyping and testing of new features when we can just turn up our own development instances of OpenStack API services locally and be assured that they will accurately mimic the equivalent interactions in existing production environments.
That’s not to say that we don’t have challenges and that there aren’t still aggravating bugs and interoperability gaps to be filled, but this brings up what I think of as our chief benefit from using OpenStack: it’s open-source free software, designed and developed in the open. When we encounter a problem with our system implementations, the extremely vibrant community of users, operators and developers is on hand to help us figure out how to fix it for ourselves and for everyone else as well. If there are upcoming OpenStack features we want to make use of in our various project infrastructure services, we involve ourselves in the design of those features by commenting on draft specifications and reviewing proposed changes.
Through early involvement and input in the process, we’re able to help shape the software in ways which provide greater support for our specific needs.
Can you tell us about a community contribution that you’re particularly proud of?
The OpenStack project infrastructure provides a means for the entire
OpenStack community to collaboratively manage the services they
consume. In this way, we both empower and leverage countless
contributors so that our Infrastructure Team is able to remain
comparatively small while still supporting the systems backing one
of the largest collective software development efforts of all time.
We focus on facilitating cooperative involvement by documenting,
educating and mentoring, but also by soliciting input and keeping in
continuous communication with the rest of the community.
How do you make sure that your team is involved in improving the platform and projects, as well as adding to the knowledge and experience of the community at large?
Diligence is key. When a bug is encountered through our use of
OpenStack services, we make sure to research it as thoroughly as
possible. This includes checking to see if it’s already been
reported and either following up to it in the upstream bug tracker
or opening a new bug if needed, discussing it with developers
focused on the relevant project in IRC or mailing lists, and
engaging in creation or review of any appropriate patches. Not only
do we attempt to follow this pattern for OpenStack software, but
extend the same courtesy to all free software we incorporate into
our infrastructure. After all, we’re all in it together.
What are the current priorities for the InfraTeam?
Our group is more of an autonomous collective, so I can really only
speak to my own priorities (though I know many of theirs are
similar): improving the OpenStack community infrastructure and
keeping it up and running; increasing the body of documentation
surrounding infrastructure systems and process/procedure; helping
grow the size of the Infrastructure Team so we can continue to scale
to meet new demands; making sure OpenStack software is well-tested
and secure; and upholding the open and free principles on which the
OpenStack Project is built, evangelizing it to the larger free
software world and brandishing it as an example of the success of
that model within the computing industry as a whole.
What was a favorite or stand-out experience at one of the many OpenStack user group or operator meetups?
I’ve presented at and otherwise attended many local user group
meetings as well as the operator sessions which we’ve more recently
been running concurrent with the semi-annual OpenStack Design
Summits. It’s tough to single out any one experience above all the
rest, so perhaps it’s best to just say that I’ve been impressed in
the extreme by the passion, dedication and loyalty of the users,
operators and other valued contributors who support our projects by
bringing their suggestions and concerns to these sorts of
gatherings, and who continually collaborate on ways to improve it
for all of us. My deepest thanks go out to you all!
How has your team’s OpenStack deployment grown and developed?
We currently consume OpenStack API services at commercial public
service providers–we didn’t deploy them of course–to obtain and
manage the resources we need to run our infrastructure. The
OpenStack features on which we rely for this have increased over
time as OpenStack development has progressed and as those providers
begin to offer them. However, deployment mechanisms like TripleO and
the continual refinement of the configuration management community
which has evolved around OpenStack is now making it possible for us
to start planning the addition of our own multi-region OpenStack
deployment, both to augment the generous donations from our current
providers and to also allow us the flexibility to run some services
or configurations we couldn’t otherwise.
Rewind to a couple years ago and we were only interfacing with Nova
to manage virtual compute resources. Our ever increasing volume of
test data and release artifacts pushed us to start attaching Cinder
volumes and eventually relocating much of it into Swift containers.
Growth in compute resource needs also drove us to start partitioning
them into multiple Neutron networks for better performance, and a
desire to offload database redundancy concerns brought us to start
relocating more and more databases into Trove. Adding custom worker
images through Glance has given us greater control over consistency
between providers and promises the ability to start providing
downloadable copies of the same for other developers to reuse. I’m
hoping in the near future we could also consider moving DNS for
openstack.org and related domains into a Designate service.
What were the resources at your disposal and scope of the project at the time, and what is it now?
I’m a comparative latecomer to the effort, and only first started
supporting OpenStack’s community infrastructure as a root sysadmin
around the time of the Folsom release. In the two years prior to
my arrival the existing team had already implemented amazingly
innovative testing and validation workflows for developer
contributions along with the fundamental underpinnings of the
services on which we’ve since come to rely. Even so, it consisted of
a mere handful of servers and relatively rudimentary (by our current
standards) configuration management and task worker pooling
supported by three root sysadmins/core reviewers working at the same
employer and living in a single timezone.
In the years since, we’ve doubled the size of our team with
diversity spanning the globe through generous sponsorship of
full-time positions at several different member companies and at the
OpenStack Foundation as well. We’ve brought dozens of useful new
community services online in that time, increasing our footprint to
roughly a thousand systems at peak activity (90% of which are
dedicated to testing OpenStack itself) generating terabytes of test
results each month to further enable quality assurance initiatives
within OpenStack. I can’t wait to see where the coming years take
us!
If you run an OpenStack cloud and would like to be a contributor to Pass the Mic, email us with the subject line: Pass the Mic.
Also Read:
- Check Out What’s Under the Hood At TryStack
- Takeaways from OpenStack’s Mid-Cycle Ops Meetup: A little more conversation, a little more action
- Five Years In: Charting the OpenStack Galaxy
Cover Photo by Torkild Retvedt // CC BY NC
- Exploring the Open Infrastructure Blueprint: Huawei Dual Engine - September 25, 2024
- Open Infrastructure Blueprint: Atmosphere Deep Dive - September 18, 2024
- Datacomm’s Success Story: Launching A New Data Center Seamlessly With FishOS - September 12, 2024
Jeremy Stanley
Kristin Barrientos
Anne Bertucio
