How service optimization keeps ViaSat flying

Managing systems like the ones satellite-based broadband provider ViaSat provides can be a tough assignment. It requires a ton of code, knowledge, and a patching together a wide variety of disparate systems and client needs into a coherent whole.

Even once that herculean effort is concluded, however, monitoring the reliability and performance is a key factor in keeping users happy and trouble-tickets low.

Now imagine doing all that with a team of just six people.

ViaSat chose OpenStack to support its developers who build the apps that support demanding satellite broadband customers across the globe, including the broadband company’s new contract with American Airlines, who chose ViaSat over rival GoGo Wireless for its new fleet of planes.

@travis_newhouse @crafty_house @brianeckblad ON NOW @openstack Austin. Service optimization keeps @ViaSatInc flying pic.twitter.com/becU4a05VH

— AppFormix (@AppFormix) April 27, 2016

Mike Craft of ViaSat took to the stage at the OpenStack Summit in Austin recently to walk attendees through the process of how ViaSat uses service optimization with OpenStack to keep the company flying high.

Why OpenStack

ViaSat provides consumer, commercial and government customers with communications services and systems. The company hopes to exceed expectations for performance anywhere in the world. That’s a tall order, especially when serving such a diverse group of customers and users.

The company reached a point where they needed to scale their operations up in a big way. They needed an on-demand infrastructure that clients could service themselves while at the same time reducing capital and operational costs. The team also wanted to be transparent about how capacity was scaled across their customer base.

ViaSat also had a range of applications they needed to maintain and monitor, from customer-facing apps in the airline, government, residential and commercial spaces to employee-facing enterprise applications as well as internal development and test environments.

OpenStack met all these needs for ViaSat. The team began its use of the cloud platform in December of 2014 and continues today. ViaSat partnered with Rackspace at first, yet continues to evolve to a more in-house supported deployment model.

“That’s the end goal for any application team,” said Craft, “you really want to own it end-to-end.”

OpenStack Ansible (OSA) allows ViaSat to run five private clouds with over 200 hosts and more than 7,000 instances for their customers, utilizing over two petabytes of storage (between OpenStack Cinder and Swift) with a Linux bridge and ML2 plugin. The team manages all of this with a six-member devops team to keep track of over 300 internal customers.

Operating a lean team

ViaSat can manage such a lean operations team as long as it focuses on availability, visibility, cost management, capacity planning and allowing its customers to do their own self-service.

“We try to be very very visible about who’s using what,” says Craft. “Is there really capacity left, is your neighbor using too much?”

The availability of the services, combined with a high visibility and transparency, allow ViaSat and its customers to manage costs with much more information than is common in the cloud space world. Empowering customers with real-time as well as historical data about resource availability allows everyone to manage costs at a much more real level.

Teams must also plan capacity; thanks to OpenStack monitoring, ViaSat can provide clear, precise data on what it is able to provide to any potential customers as well as those who want to upgrade their services.

Finally, ViaSat uses its transparency in terms of data to remove themselves from customer project planning. If a customer can create its own apps and projects without ViaSat’s direct involvement, says Craft, the customer is happier and it requires less of ViaSat’s time.

Manage infrastructure costs

The private cloud is different than the typical pay-per-use public cloud in that the private sector is much more budget-constrained. You have to make sure your customers aren’t wasteful of what are, essentially, resources that could be used for other customers. If they’re wasteful, says Craft, you’ll need to empower them with data to help them determine what’s worth paying for and what isn’t. Once that determination is made, you can reclaim unused and under-utilized resources to best manage costs for yourself and your users.

Self-Service for users

ViaSat partnered with AppFormix to provide monitoring-as-a- service for its customers’ projects and instances. This partnership allowed ViaSat to expose underlying infrastructure data points, enabled users to answer their own questions about available resources and how they’re being used, and lets users understand issues around hypervisor and storage health.

Transparency is key to empower users, says Craft. It allows them to create their own alarms and event triggers without the need for ViaSat to step in and do it for them.

“We not only have the data there,” he says, “but we also educate them to help them understand what the data means.”

Once customers understand what the issues are, they can work more intelligently with ViaSat to solve any capacity or tech issues that arrive.

Challenges

It hasn’t all been smooth sailing, says Craft, noting that ViaSat started off by offering custom “flavors” of instance size, CPU utilization, and other variables when new customers came to purchase services. This ended up not being very scalable for such a small operations team, and ended up causing inefficient workload placements that could lead to CPU or disk exhaustion.

Now, the team uses standardized flavor sizes for all new users, which helps them avoid resource fragmentation and improves capacity planning. Of course, their users need data to choose the correct standard size for their needs.

“All of the smallest flavors fit into the largest one,” says Craft. “We can fit 32 of our smallest flavors into our largest ones, so it all fits together nicely.”

The data customers need is easily accessed by the customer themselves, which both helps users make the right decisions for their own needs as well as the ops team at ViaSat make the right recommendations when problems arise, like asking them to resize their flavors down dramatically when resources are being underutilized.

“We’re not forcing our customers to do this,” says Craft, “but we’re empowering them to get the data, be cost conscious, be aware of what they’re doing, and how they’re actually using their environment.”

Overcoming these challenges, as well as others like maintaining virtual management (hypervisor) systems when clients use older interfacing systems, has been simply a matter of providing everyone with the data needed to see the problem, and then deciding on the best course to fix it.

Ultimately, all the self-service and transparency that ViaSat uses in its OpenStack system helps everyone do the best work possible. And isn’t that what it’s all about?

You can catch the entire 33-minute session on the OpenStack Foundation video page.

Cover Photo // CC BY NC