OpenStack Swift – Atlanta summit retro

This post originally appeared on the eNovance blog. Christian is a Swift core developer working at eNovance with a focus on storage, testing and automation. You should follow him on Twitter.

There was a lot of interest in Swift during the Juno OpenStack Summit in Atlanta and sessions and workshops were very well attended. More than 30 sessions were directly or indirectly related to Swift, its use cases and discussion on the past, current and future development of Swift. Thanks to all the awesome people who are developing and using Swift every day!

Let’s have a look at some of the current hot topics within the Swift community.

Storage Policies

One talk and design session were only focusing on the upcoming Storage Policy feature. This is a major change in Swift, and quoting the Swift PTL John Dickinson “The Biggest Thing in OpenStack Swift Since it was Open-sourced”.

Storage Policies make it possible to use different classes of hardware and ways to store data. For example, an operator could add some nodes using very fast SSDs and assign a special storage policy to them. Another policy might use a lower replication factor for users that don’t require three replicas for their data. Users are then able to select a storage policy that fits their needs.

So far development took place in a separate branch and the next step is to merge this feature into the master branch. For this a patch chain will be created and the Swift core developers are joining forces to review them in a short time frame. Finally, after storage policies landed the community will continue working on erasure coding, a feature that is able to lower storage costs while preserving the high availability of Swift.

Object server diskfile API and pluggable backends (PBE)

The Object Server DiskFile API was introduced last year and allows the usage of different backends to store objects. By default local disks are used on the object storage nodes, but there is also a reference implementation for in-memory storage.
There are already several external storage backends that are using the diskfile API. One of the major benefits for this is to add the native Swift API to other storage systems without reinventing the wheel.

Swift-Ceph backend uses Ceph as a backend for Swift which is especially useful for Ceph operators that want to offer a native Swift API to their users. github.com/enovance/swift-ceph-backend
Swift-on-File is formerly known as gluster-swift and allows accessing files on any filesystem supporting extended attributes as objects using the Swift API. github.com/swiftonfile/swiftonfile
Seagate Kinetics are drives that are accessed directly over Ethernet as a key/value store, eliminating the need for dedicated storage servers. github.com/swiftstack/kinetic-swift
ZeroVM is a virtualization technology that can be embedded into storage systems. Doing this the application is moved closely to the data and not reverse. github.com/zerovm/zerocloud/

While the diskfile API allows storing objects on other backends, the account and container
databases still need local storages. There is another patch ready for review that is useful if one wants to use different backends for storing these databases.

Swift and Python 3

A lot of the recent work is devoted to support Python 3 within OpenStack.

Thanks to the work of the OpenStack community python-swiftclient now supports Python 3. The next step is to focus on Swift itself, and for this Python 2.6 support will be dropped in the future. A non-voting Python 3 linter gate job will be added to support the development to be able to start fixing the trivial SyntaxError with python3. We are still actively monitoring the eventlet port to python3.

More efficient Swift replication

In a global distributed Swift cluster data might be replicated multiple times across WAN links. Let’s take a cluster with two datacenters and three replicas as an example: in that case it might happen that data is replicated from one site to the other two times (if there are two remote replicas). A better approach is to replicate data only once using the WAN link and replicate to the second copy locally. This lowers bandwidth cost and required replication time.
http://www.slideshare.net/tsuyuzaki/open-stacks-ummitjuno

Swift devops

This session was dedicated to the operators of Swift clusters and their needs. Discussions covered logging (and if Swift should use oslo.logging instead of syslog), places to discuss operations (openstack-operators at lists.openstack.org and #openstack-swift at freenode are good places to start) as well as cluster sizes. Operators that are using global clusters reported that using more replicas than regions works well, while using more regions than replicas might need some tweaking.

An important aspect doing operations is monitoring and benchmarking, as well as ring management. A separate tool for ring management might be very useful for operators and simplifies operations.

Tracing and benchmarking

The possibility to trace and analyze requests is very helpful for operation teams and developers as well to identify bottlenecks, middlewares that increase the latency and nodes that are slowing down the cluster.

There are different ways to analyze requests, and one way is to add the parent transaction ID (txid) and timing information to the logfiles. Doing so operators and developers can use the existing logfiles to analyze Swift and ensure a stable cluster performance.

Benchmarking Swift and especially production clusters can also help to identify problems. Mark Seger of HP gave some insights in his work benchmarking Swift clusters and the tools he used for this (github.com/markseger/getput/ and collectl.sourceforge.net/). He observed some interesting behaviors, for example a huge difference in IOPS for objects sizes of 7887 and 7888 bytes that is related to the Nagle algorithm.

It was also shown that it might be very useful for operators to execute some small benchmarks continuously to make sure their cluster is operating in a reasonable state at all times.

Improving Swift documentation

Interestingly swift.openstack.org is the most visited page outside of the official OpenStack documentation. While there are many topics already covered the user experience could be improved a lot. The idea is to create several landing pages for application developers, operators and developers.

We also discussed on other parts of the documentation and how to improve them. Especially the documentation on multiple-node setups needs to be updated or even rewritten, while other currently uncovered topics like debugging and operating tools need to be added.