)
James Penick, currently architect director at Yahoo, knows a thing or two about upgrades. Starting out as a self-described “ticket monkey,” he has placed boots have been on the ground in nearly all aspects of large-scale production engineering. He and his team of 20 currently watch over about four million cores.
That’s why Superuser thought it’d be a good idea to get his thoughts on upgrades, one of the topics that’s always top-of-mind for OpenStack users, before the Vancouver Summit. For Summit-goers, here are featured sessions on upgrades and details on both of Penick’s talks.
In Vancouver, one of your talks is with Verizon’s Amrith Kumar about building and operating an enormous OpenStack cloud – can you give us details on the deployment and the size of your team?
The team is relatively small, less than 20 engineers at the moment, including dev ops and PE. In terms of scale we’re currently managing hundreds of thousands of bare metal instances and under 100,000 VM instances. In sum, we’re managing four million cores or so.
What open-source technologies are you using now?
Where to start? The open source culture has produced a rich pantheon of tools and platforms that have become such an integral part of our daily lives that I take them for granted. But off the top of my head: MySQL, RabbitMQ, Apache, KVM, Libvirt, Linux, Chef, Screwdriver, Athenz and of course OpenStack. Frankly there’s too many to name.
There’s no ideal/universal approach to guarantee a smooth upgrade — but what are a few key practices?
Read the release notes! Build a test cluster to test the upgrade process. Make sure to automate the fix to every problem you encounter. Whenever possible, file bugs or even send patches upstream.
In your Sydney talk, you compared moving to OpenStack “like a tug boat pushing at the front of an aircraft carrier to turn it around.” What’s your metaphor for upgrading from Juno to Ocata?
Upgrading from Juno to Ocata seems like climbing Everest, when in reality it’s more like walking up stairs. It’s strenuous, but there’s structure and support to help you get by.
What’s the most common misconception around OpenStack upgrades?
Everything is heresy until it’s common sense. The best way to dispel the FUD around OpenStack upgrades is to get the word out in talks, posts, and presentations, until people roll their eyes at how obvious of a truth it is. As deployers share their success they will change the perception of OpenStack upgrades.
That’s what happened with running OpenStack at scale; there was a time when running a 200-node cluster was considered impossible. Now multi-thousand node clusters are obvious.
Superuser wants to hear your open infrastructure story, get in touch at editorATopenstack.org.
- OpenStack Homebrew Club: Meet the sausage cloud - July 31, 2019
- Building a virtuous circle with open infrastructure: Inclusive, global, adaptable - July 30, 2019
- Using Istio’s Mixer for network request caching: What’s next - July 22, 2019
