OpenStack Homebrew Club: Swift in the closet

OpenStack powers more than giant global data centers. This is the first in a series that highlights how Stackers are using it at home.

Here we talk to John Dickinson, one of OpenStack’s longest-running project team leads. In addition to his work with Swift was recently given the “Keys to Stack City” Community Contributor Award in recognition for his dedication over the years.

He talks about why a Swift cluster is taking up valuable closet space in his San Francisco home and what he plans to do with it next.

When did you start using OpenStack at home and what are you doing with it?

I first started running OpenStack Swift at home about a year ago.

I’ve got a small setup with five nodes in a Swift cluster. I use it to store backups from a bunch of different computers and store movies that it can then stream to my TV. And I use it for some sharing between people and machines. My kids like to record videos of themselves playing games and stuff like that. They can put it on the Swift cluster, use their own account and then make a little home web page and watch themselves stream.

I’m going to start a new project on making some air sensor monitors and put it throughout the house. And I’ll store that data in Swift.

So it’s part smarthome, part family media vault. Where do you keep it?

I did take over a closet. Not everyone in the family was happy about that, but that’s where it is.

Do you have any plans to do anything with it next?

The next thing I’m going to start is a smarthome experiment. You can store lot of time-series sensor data in a Swift cluster.

And I need to expand the usage, especially backup usage across several different more computers, so instead of doing something like Time Machine I can back up directly to Swift using a couple of different backup clients on Windows, Linux or Mac computers.

What advice would you give to somebody trying to do this? “Don’t try this at home? Or do this at home?”

It’s absolutely possible to do it at home. And it doesn’t take much; it’s not expensive compared to an equivalent file server or something like that.

In fact, the reason I started using Swift is because my old file server finally died after about 10 years. I wanted something that was a little bit better. The main problem I had with the old file server was the lack of expandability over time. I couldn’t just add a new server to the file server cluster or add a hard drive when there are no more available ports on the motherboard.

I know how Swift works, so I figured I’d use it to store all of that data. It’s absolutely possible to get started. I’m using really cheap low-powered ARM single-board computers but you can go bigger than that. I chose to have several individual servers that are each connected to just one hard drive. It’s super lightweight, but super flexible. You could do something with a slightly bigger server that has several hard drives. And if that’s what you have, that would totally work as well.

What’s the ballpark cost?

Well, of course it depends on what size hard drive you get, I think I ended up spending about $70 each on a hard drive. So four of those plus roughly $50 for each of the single board computers, on average. So you’re looking at around $500 or $600 total. With the current setup, I’ve got a fully redundant scalable storage system that has about two and a half terabytes of usable storage and can store over 2,500 movies.

How did you figure out the right scale?

It was primarily budget and then thinking about what I needed. I didn’t want to spend $1,000 or $2,000 on a huge storage system. Starting from there, I researched the computers and hard drives that would meet my storage needs in the price range I was looking for.

What else do people need to know?

The hard part about any kind of storage system is figuring out operations. I’m just monitoring it manually right now. I’m still in the process of building some of that stuff out, but it doesn’t take lots of daily care and feeding or anything like that. I forget it for awhile and then pick it up later. And it works just fine.

The idea of a smart home/spy terrifies me. What are you tracking with it?

I’m not going to put microphones or anything like that all over my house. I’m going to install air sensors to start with because I’m curious to know more about where we live — it’s a 100-year-old house without central AC. I’m interested in finding out what happens when at night you’re sleeping with all the doors and windows closed. What’s the co2 level? What’s the temperature, humidity? Do we need to add air purifiers or something?

Any other plans?

Once I have that infrastructure setup, I want to talk more stuff, expose a lot more about more the clusters, especially cluster health through the same kind of visualization. I want to expose a lot of network activity, I’ve got fairly complicated network setup with different networks — my kids, myself — and I want different networks for different smart devices. No reason my washer and dryer need to be on the same network as my work computer.

My goal is to track and monitor all that with pretty graphs on the screen and then maybe make some more decisions — in terms of energy use and trying to figure out where things are.

I do have a another kind of dream pet project, a continual video camera that points at the sky…to track the constellations and see how the weather changes, the fog, the city lights… It’s just kind of a fun thing to play with.

You brought Swift into your home. Do you ever stop working?

I don’t spend a lot of time on it! I just knew it was something that I could do, and then you run with it for six months. At the same time, it’s cool because especially with the files and stuff, the idea that you lose all of your family photos, or some video that your kids made, that’s always a fear when you keep it on a single hard drive somewhere. I’m less worried about that now because I’ve got Swift. I know how that works and that I can trust it.

Got an OpenStack homebrew story? Get in touch: editorATopenstack.org

Photo // CC BY NC

Tags: OpenStack Homebrew Club, Swift