How your feedback sparked what’s next for Cinder

During the OpenStack Project Teams Gathering in Denver, the Cinder team gathered to discuss our development for the Stein release.  Part of the time in Denver was used to look at the responses that users shared about Cinder when responding to the OpenStack user survey.  During the review we noted most of the responses fell into a number of categories:

• Backup/disaster recovery requests
• Driver capabilities reporting
• Multi-attach functionality
• Improved support for actions in in-use volumes
• Standalone Cinder support
• High availability support
• Ease-of-use/configuration issues

With these categories in mind, the team reconvened at the OpenStack Summit in Berlin to discuss this feedback in a Forum session.  We had hoped to engage users more directly during this Forum session but the user/operator turnout for the session was not what we had hoped for.  Instead, the team decided to discuss the feedback again in preparation for this very blog article.

In this post, I’ll provide information on how we’re moving forward given the feedback provided in the user survey. I’ll also share newer functions that are now available that may address some of the concerns. Finally, I’ll share areas of feedback that we need more information to address and provide pointers as to how users/operators can engage the Cinder team to help us understand the requests.

Backup and disaster recovery functionality

A large number of the feedback came around backup functionality.  Backups are an area where the Cinder team has been working for a number of releases to improve support.  For instance, in Rocky the ability to utilize multiple processors on the node running the backups service has been added.  This will allow users to greatly improve the performance of backup functions. We have also added the ability to backup and restore encrypted volumes.

The team is aware that there are gaps in disaster recovery functionality, especially around the handling recovery after a failure when replication is enabled.  In Rocky we added the ability to use the cinder-manage command to fail control from the primary host to the secondary host after a major failure. This makes it so that users no longer need to manually edit the database to failover the host after failure and then, also, makes it easier to failback.

In the Stein release we are working on adding a generic backup driver.  This will make it possible to do backups without a dedicated backups service.  Any volume driver available to Cinder can then be used as a backup target. This will enable users to backup volumes from one storage backend to a secondary storage backend in their environment to help protect against catastrophic failures on one storage backend in the environment.

Another theme in the requests for backup functionality was with regards to adding the ability to schedule backups. While the Cinder team appreciates need for this functionality it is important for users to understand that such support falls outside Cinder’s role.  Cinder and its APIs are focused upon management of block storage resources, not automation of management of those resources. Users who wish to automate storage backup can either automate such processes using Cinder’s API or can use other projects from OpenStack’s ecosystem, like Mistral, to automate such tasks.

Driver capabilities reporting

There were multiple requests for the ability to get more details about what functionality a particular driver/storage backend could support. This is an area where Cinder’s support has been weak for quite some time.  There were champions for improving this functionality back in the Kilo release. The team, however, wasn’t able to agree on a way to implement the functionality and the initiative, more or less, died off. As we have had an increasing number of storage backends and an increasing number of possible capabilities this functionality needs to be re-addressed.

There are members of the Cinder development team who are going to start looking into possible improvements in this functionality.  The goal is to start by getting what capabilities reporting is available on the command-line working properly and then expand it to support a better user experience. Ultimately we would hope to have the ability for admins to see from the Horizon Dashboard what capabilities each storage backends supports so the information then can be used to more easily create volume types.

Multi-attach functionality

Anyone familiar with Cinder knows that this has been a topic of discussion for quite some time.  We finally got base support for this functionality into Cinder and Nova during the Queens release. Since then, driver vendors have been working on implementing multi-attach support for their drivers. Cinder is now up to 13 drivers supporting multi-attach of volumes. Enablement of multi-attach in more drivers will require support from those third party vendors.

The Cinder team is thinking that a number of the requests for multi-attach stem were from the fact that Ceph does not yet support this functionality. This gap in support is being addressed in the Stein release.

There was also a number of requests for read-only multi-attach support. The Cinder team is aware that there are gaps in the ability to properly mount volumes in read-only mode. The team is planning to investigate this functionality during the Stein and Train releases to improve this support.

NOTE:  It’s still important to be aware that the support provided by Cinder is not a ‘magic-bullet’ that makes it possible to attach a Cinder volume to multiple instances in all situations.  The storage and filesystem behind the volume must also support multiple attachments. If a user attempts multi-attachment in read/write mode using a filesystem that doesn’t support multiple writers, filesystem corruption will result.

Actions on in-use volumes

There has been on-going work to improve the operations that can be completed on attached volumes. Recently the ability to extend attached volumes has been implemented. Again, this is a feature that requires support from the vendor’s driver. So this may, or may not, be available in your environment.

A recent area of focus has been on how to handle re-imaging attached volumes so that a base operating system could be recreated for an instance that is booted from volume without having to detach the volume. Efforts are in place to hopefully add this functionality during the Stein or Train releases.

Standalone Cinder support

A number of efforts are currently underway to improve Standalone Cinder Support. Cinder can either be run with or without Keystone in place and there is support for attaching volumes without Nova in place.  Further work is now being done to make the Cinder’s volume drivers available through a new wrapper called ‘cinderlib’. Cinderlib will make it possible for users to interact with Cinder’s volume drivers without all of Cinder’s functionality in place. It’s hoped that cinderlib will provide a way for services like the Container Storage Interface to leverage Cinder’s existing storage drivers.

Improved high-availability support

Incremental movement towards full HA, Active/Active support has been going on within Cinder for the last few releases.  It has been possible to run the API and Scheduler services Active/Active for quite some time. Running the volume service, however, in Active/Active mode has proven more challenging. Depending on whether a driver has any locking in its code, some drivers may be able to run HA A/A but it was not without risk.

In the Stein release, Cinder is planning to adopt the distributed locking techniques used by the Placement Service to ensure that we can safely have multiple volume instances running without potentially ending up with inaccurate quotas or deadlocks attempting to access the database.  The team has also written documentation describing the potential dangers of running the volume service in active/active mode. This documentation should serve as a guide for the storage vendors to test and improve their volume drivers to support HA A/A configurations.

Ease-of-use/configuration improvements

The user feedback came with a number of general requests for improvements in ease of use and configuration. The team is currently doing a number of things to address these concerns.  For instance we are aware that we do not have parity between the functions that are implemented in OpenStack Cinder Client and OpenStack Client/Horizon. To help address this we are planning to put processes in place as we move to using Storyboard as our bug tracker to tag those changes that need to also have changes propagated to OSC and Horizon.  We also have been working to improve our documentation. This entails filling in functions that were never properly documented or that need to have the documentation updated due to changes since the functionality was implemented.

Based on user feedback we have also in recent releases added support for more dynamically configuring logging levels within Cinder, making it possible to make log output more or less verbose without having to restart services.

Much of the other feedback around ease-of-use improvements was quite general. We would like to better understand such requests. Hopefully you can help.

Cinder needs your help

The team has done the best we can to interpret the brief feedback that users can give in the user feedback survey.  Many of the comments, however were very short with limited detail. This is why we need your help to fill in the details.

Some of the areas that had a lot of general feedback that could use additional detail in the request are:  ease of use/configuration issues, requests for improved support for actions on attached volumes, improved usability of snapshots and the addition of more backup functionality outside of the ability to automate creation of backups.

If you have additional details that you would like to share there are a number of ways that you can help.  First, you can send an e-mail to the [email protected] mailing list.  Just tag your post with [cinder] and [user request].  We can then discuss the requests over the mailing list.  We also be happy to have you join us during our Weekly Cinder Team Meeting.

The team meets weekly on Wednesdays at 16:00 UTC on the #openstack-meeting IRC channel on freenode.  If you have a topic, please see this page for instructions to add your topic to the meeting: https://wiki.openstack.org/wiki/CinderMeetings  Finally, the team is always available for discussion in the #openstack-cinder IRC channel on freenode.

About the author

Cover photo // CC BY NC

Tags: block storage, Cinder, OpenStack community