News

What you need to know about OpenStack Cinder

Why Cinder’s project technical lead wants you to break things, the key to continuous integration and the influence of an epic 80s boy band.

May 12, 2015
image

Mike Perez is the project technical lead (PTL) of the block storage project Cinder. At the OpenStack Vancouver Summit and the Design Summit you can catch him on several occasions where he’ll be fielding your questions and feedback about Cinder or follow him on Twitter.

What are the most common questions you get about Cinder? And about contributing to the project?

I commonly talk about what exactly Cinder is.
Cinder, like most OpenStack projects, is an abstract layer that provides an interface to implementations.
For Cinder, these implementations are block storage solutions. Typically SAN
implementations, which can be open source or enterprise, that export volumes
over fabrics like Internet Small Computer System Interface (ISCSI) or Fibre Channel. These volumes are persistent storage that can be formatted with a filesystem. Compare this persistent storage idea with ephemeral storage that you get with Nova alone, where the data usually goes away after terminating an instance. With these volumes you can attach them to virtual machine to have data written, and then later attached to another
virtual machine. Another neat idea is writing an image of some Linux
distribution, to later be booted as the root partition of some virtual machine.

New contributors are intimidated by the number of moving parts with Cinder.
I usually tell people to start with understanding the basics of just how
a volume creation gets routed from a user’s request, all the way to the volume
being in a provisioned state.

I’m not embarrassed to admit it, but when I first started, I just stuck Python debugger breakpoints all over the place until I understood things. In general though, don’t be afraid to break things and learn from it. Don’t be afraid to post a patch that’s not going to be approved the first time around, or pass the Tempest integration tests. Work with the Cinder team on what you’re trying to accomplish. We’re a nice, laid-back group, and are very welcoming to new contributors.

What got you initially involved in Cinder?

I started in the OpenStack project back in 2010 during the Bexar release. Back then, there was no Cinder. Just like Neutron, Cinder was forked out of the
giant project Nova, which before then it was known as Nova Volume. During a
summit, discussions started happening to form this project. Of course the most
important question of all, what do we call it? The names I came up with didn’t
get accepted, and for good reasons…Jenga and Lego. Vish Ishaya of course had the best name, The New Kids on the Block. Get it? Because it’s BLOCK storage.

I wasn’t immediately working on the project, but John Griffith, the first and long-time PTL of Cinder made me feel welcome in the project before I made any contributions. My first contribution was creating the version 2 API of Cinder. This was done to begin breaking some legacy Nova pieces off that were left around for compatibility with moving from Nova Volume
to Cinder. By the way, bravo to John for making that transition. It’s not easy, so my hat’s off to the folks making this same exact thing happening on the networking side with Nova Network to Neutron!

For drivers to be included in Kilo, they needed continuous integration (CI) done well by a certain deadline. A few very visible ones didn’t hit that deadline, and you had to say, "Sorry, it’s not going in." How does it feel to say "no" to contributors? What lessons did you learn?

Forget about the contributors who are driver maintainers to some large
companies that have big money riding on this change for a second and think
about operators who are really having a hard time with this change.

These are people who are depending on their vendor to come through with making sure
their solution is stable and working in the OpenStack release. In the past, the
Cinder team had to go on word of the vendor that their solution was
working in Cinder. Later bugs would be filed against certain vendors, and it
was just amazing that some had ever expressed that things working. Operators would be
the ones discovering things didn’t work. It was a mess and frustrating.

I knew from the beginning that this wasn’t going to be easy. The most important
thing with a change this drastic is communication, and really going out of your
way for people.

I made the decision in the Kilo release as my first term as
PTL that it was going to be my priority to make CI a real thing
across all third-party vendor solutions in Cinder.

We were growing to over 72 drivers after accepting a handful of new drivers in the first milestone of
Kilo, and I wanted everyone to feel confident in their solutions being
integrated in the Kilo release. We started the talks — actually a year ago — that
we would start requiring vendors to have a third-party CI, very similar to what
we see today with the OpenStack Zuul system, but much simpler. For every patch that came in from Cinder, a vendor’s third-party CI would need to bring up
a virtual machine running Devstack with that proposed patch, configured to talk
to their real storage solution, and verify that the current volume related
Tempest integration tests pass.

Again to promote good communication and going out of your way for people, I began by sending out emails on the OpenStack dev mailing list
announcing the requirement and deadline. I sent emails directly to every driver
maintainer. If I didn’t know who the maintainer was, I searched the Git history
logs for who was obviously from the company, and contacted them as well.
I would announce this deadline every week at the Cinder meeting.

The OpenStack infrastructure team has also been contributing heavily in helping projects like
Cinder, by making a third-party CI meeting twice a week. This meeting was
specifically created to help people with their CI systems. For the Cinder
project, we also had liaisons available in different time zones to help people
with their CIs. Lastly, I made phone calls to companies who had driver
maintainers but who never got back to me. This was very painful, especially with
large companies. You leave voice mail messages on random extensions to people
who never get back to you. I went out of my way, I didn’t have to, but
I really wanted everyone to be successful in this transition.

The first few results of vendors starting this was absolutely amazing. Some
discovered issues in their drivers immediately. Others discovered their driver
was broken in the upcoming release. The system was really starting to work when
some bigger changes in Cinder core were going to land, and it was obvious that
the changes were going to break integration for other vendors.

This is huge, compared to when the vendor would know something is broken after an angry
operator discovers it once the OpenStack release has already happened. Lastly,
it makes reviewing driver-related code go much quicker, because you can see
right there in the CI testing results whether the new proposed code breaks the
driver, or not.

Amazing. I can actually say with confidence that the drivers in Cinder’s Kilo
release today have been tested. Thanks to all vendors who made this a priority
to your users for a better experience in OpenStack.

You were recently re-elected PTL. What’s your feeling now on this tweet from a few weeks ago: "I don’t wish the #OpenStack project technical lead role on anyone. We have a ways to go in people understanding everyone involved is human."

It’s tough being in a leadership role. While this is probably true anywhere you
go, I feel like open-source projects in general are just filled with angry
people. It’s really unfortunate, because we benefit by having a diverse
community to feed ideas off of to come up with some neat things.

Personally, I feel like some people who I interact with on daily basis are sometimes
angry and discouraging. Some of these people have a huge impact on my
happiness, and feeling like I’m doing something right.

Sometimes I make mistake, or sometimes I have to make decisions that are out of
my control. Being the front person of a team you take a lot of heat, and
sometimes it’s just piles of angry private messages on IRC. I hope people in
this community would remember we’re all human, we make mistakes. Just tell me
I can do better in the future, and don’t be angry.

I’ve mentioned it in my first self nomination for PTL in Kilo, and now Liberty,
making the Cinder project a welcoming team is another one of my priorities.

We can talk and talk about how we’re going to change the world and blah, blah, blah —
but it’s not going to happen unless people want to be part of your team and don’t
get discouraged. Cinder is different, and I’m OK with being part of an open-
source project that doesn’t have some angry jerk leader, where everyone is
actually providing a welcoming environment for contributors.

Perez is giving three talks at the OpenStack Vancouver Summit. Here’s his speaking schedule.

Cover Photo by Amboo Who // CC BY NC

Nicole Martinelli
Latest posts by Nicole Martinelli (see all)