Kilo Update: Swift

Welcome to the PTL overview series, where we will highlight each of the projects and the upcoming features that will be in the next OpenStack release: Kilo. These updates are posted on the OpenStack Foundation YouTube channel, and each PTL is available for questions on IRC.

John Dickinson is the PTL for Swift, the object component of OpenStack. In his PTL update webinar last week, he explained his vision for Swift, the current conversations driving the project, and the upcoming features planned for the Kilo release.

Swift from 30,000 feet

Swift is OpenStack’s object storage engine. It was designed to scale at a massive level, and is optimized for durability, availability, and currency across entire data sets. It’s ideal for unstructured data that can grow without bounds, like content generated by mobile apps, documents, backups, images, videos, etc. In fact, it’s already being used for these real-world applications today. But that’s just the beginning.

Looking ahead, John hopes that Swift becomes something that people use everyday, without even realizing it.

"I want to see people using Swift when they’re helping their kids with their homework, looking up stuff online. I want to see people using Swift when they’re filing expense reports, checking their bank accounts, when they’re watching videos."

From a community standpoint

Swift’s most prolific contributors are familiar names that include RedHat, SwiftStack, Rackspace, Intel and HP.

The project is tangibly maturing. John has seen an uptick in Swift being used and supported by third party ecosystem apps. When you ask someone what they are using as an object storage API, there are two options: S3 and Swift. Major applications like commvault, Veeam, and SME are supporting Swift, and there are more, and larger, clusters of Swift in the overall install base.

John takes all of these proof points to mean that the Swift API is becoming a standard, a first class citizen.

How do clients interact with Swift?

The Swift community has been talking about making things better for applications using Swift. Often the first interaction people have with Swift is in the command line interface or the SDK. Looking ahead to the long-term viability of OpenStack and Swift, the community is considering a different approach.

First of all, the command line interface is not going away. It will be maintained, features will be added, bug fixes will continue to occur, and a layer of polish will be added. The command line is and will continue to be useful for many users and operators.

hxkgierzl1u0szd0yhrc

In the larger OpenStack community, there has been an effort to create a single OpenStack SDK to place all of the projects into a single library where people can consume and talk to all of the projects in a unified place. The OpenStack SDK is a relatively new concept, but the Swift community plans to get involved in the beginning stages. Instead of directing efforts for future releases, new directions, and rewrites in the python swiftclient SDK, the team will focus their efforts on the new work being done on the OpenStack SDK.

John believes that this will make Swift more efficient, and allow the community to leverage the knowledge they’ve already gained in developing the Swift SDK.

John emphasized that the python swiftclient SDK is not going away anytime soon. The team will continue to improve it, fix bugs, and add polish in the immediate future.

GNOME Outreach Program

The GNOME outreach program is an internship designed to get underrepresented groups involved in open source. OpenStack has been a part of the program in the past and currently they have 6 interns working on various projects. One of the six GNOME interns working on OpenStack is on the Swift team and focusing on improving operational tools.

The GNOME intern has already made valuable contributions to Swift and the community is excited to be a part of the program.

New Process – Specs

Specs are a collaborative design process for some of the larger features within Swift. The goal of specs is to increase and improve communication. While specs are used in varying ways in several other OpenStack projects, the community as a whole is still trying to test and determine which approach is most effective.

Specs are in experiment mode, and this is Swift’s second attempt to implement them effectively. This is how they are going about it in the current release cycle:

When a spec is proposed, the team aims to iterate and land at those specs quickly. This is as opposed to having a design doc up front that has every use case and contingency laid out comprehensively. Throughout the collaboration, the team will keep the specs updated as they iterate on conversations. Ideally, this process should make communication across different companies, time zones, and remote offices more effective.

Conversations in the Community

Erasure codes

Storage policies, which allow you to configure how data is stored inside your clusters at a granular level, were recently introduced to Swift. They allow you to build a configuration of how your data is stored across a set of hardware on top of storage policies. Today, we use replication to do that, but the Swift community is building a feature to allow you to use erasure codes alongside of replication.

The community is actively working on developing something that is demo-able, though this is a big subject and the bulk of the work to date has been spent on solving fundamental design issues.

One of the hard problems is giving users the ability to overwrite objects that have been stored as erasure coded data. It’s more complex than it seems at face value and the team is going through great lengths to get it right before actually doing the coding.

Encryption

Encryption is something that John has been hearing about a lot lately, especially from enterprise users. John described the problem with encryption as it relates to storage this way:

“If a hard drive walks out of my cluster, either maliciously or inadvertently, I need to make sure that nobody can read the data stored on it.”

There are many different ways to approach this, and the community is actively discussing and developing specs for a solution that isn’t too complicated, but also isn’t limiting to customers and users who need this kind of encryption.

Composite Tokens

Composite tokens are a new functionality to work with auth systems. The idea is that you have some data in Swift that may be put there by a service. You don’t necessarily want the end users to be able to delete that data without some kind of admin process or permissions. You also don’t want to the system itself to make the decision on behalf of the user.

So, the Swift team is proposing that there are certain levels of permissions that the operation requires two auth tokens — one from the service and one from the end user.

This is currently being worked on in Keystone, and the Swift community is now talking about it. There’s no code for it in Swift yet, but it is an ongoing conversation.

o2qipwu60aldhdhf4mxu

Fast POST

POST is an HTTP verb that is inside of Swift to modify metadata for objects. In most cases, it’s implemented as a copy, and that works well for most current use cases. But it’s an expensive way to do it, and the Swift team wants to improve its efficiency.

Fast POST is an improvement that allows you to skip the server side copy and speed up the process.

Efficiency Improvements

Finally, efficiency is something the Swift community is always working on. It’s designed for massive scale and massive concurrency across entire data sets. John emphasized that Swift should not do too much. The features that it does have should be fast and efficient. Some examples of what the community is considering in terms of efficiency improvements include:

making large containers more efficient
improving efficiency in global replication
overall bug fixing and making it easier for user/operators to validate their clusters simply