Monitoring a multi-region cloud based on OpenStack

When you’re running a large, federated cloud across multiple regions in Europe, each with its own funding sources and institutional governance, you need to make sure your monitoring system is top-notch.

To monitor and manage this diverse group of users is why Silvio Cretti of create-net and his team at FIWARE Labs chose OpenStack.

Cretti came to the OpenStack Summit in Austin, TX this past April to discuss the FIWARE Lab’s monitoring architecture and implementation.

What is FIWARE and FIWARE Lab?

FIWARE provides OpenStack-based cloud capabilities along with a library of open source APIs tha make developing smart apps much easier for developers. It has five areas of operation, or pillars: FIWARE, a generic open standard platform which serves the needs of developers in multiple domains, FIWARE Lab, a digital sandbox for developers to try out new technologies, as well as a funding arm (FIWARE Accelerate), a global outreach program (FIWARE Mundus), and a local support community (FIWARE iHubs).

FIWARE Lab, the focus of Cretti’s talk, is a playground that lets innovative developers experiment with innovative applications. It’s an open, working instance of FIWARE where innovation is encouraged.

“Infrastructure owners, data providers, entrepreneurs, developers and technology providers are all stakeholders who can contribute to this open instance,” says Cretti.

yf5ghoqw9rblbvfigyvn

Funded in part by the European Commission (EC,) the multi-regional, federated cloud called FIWARE Lab is based on OpenStack. It serves 15 regions spread across Europe, as well as a presence in Latin America. It offers over more than 3,000 cores, 10 terabytes of RAM and 600 terabytes of hard disk space. It serves more than 5,000 public IP addresses with over 1,500 users and the same number of virtual machines, putting FIWARE Lab in the top 30 percent of OpenStack users in the world.

Cretti and his team leveraged Ceilometer, a monitoring component that can be used to provide customer billing, resource tracking and alarm capabilities for OpenStack-based systems and Monasca, an open-source monitoring-as-a-service solution that integrates with OpenStack, to allow FIWARE Lab to collect, process, analyze and share relevant data to all interested users.

Cretti points out that FIWARE Lab must have a highly dynamic infrastructure, as well, with various regions joining, leaving and re-joining FIWARE Lab as funding is secured or evaporates. FIWARE Ops, a suite of tool that ease the creation and operation of FIWARE instances, is the process by which the various stakeholders can join and leave the digital federation, as well as to manage and monitor the services in real time, including deployment, federation management and monitoring, connectivity management and service offer management.

The high-level architecture for FIWARE Labs is shown below.

y4oofmpw3m7cuxpq1s76

Requirements for monitoring FIWARE Lab

Because FIWARE Lab is a multi-region environment and managed by different owners who might have different objectives, FIWARE Lab is a sort of “federation” of OpenStack nodes.

“Multi-tenancy is a must, not related to end-users, however, but in terms of administration,” says Cretti.

The FIWARE Lab team has to manage errors and monitor different servers, processes and services under different perspectives to know if the services offered are running correctly from the point of vie of the end-users, and then be able to notify infrastructure owners and administrators about any detected problems.

Root-cause analysis is important, as is performance management, Cretti adds. The monitoring system must give exhaustive information about the performance of the whole Lab, in terms of resource allocation and consumption (CPU/disk/memory) per region, computing node or instance. It also needs to analyze capacity and trends, and easily integrate with existing monitoring tools that are already installed, like Ceilometer and other data collectors that are configured and running in most nodes.

“We (also) have a Monasca agent installed because of the monitoring of specific processes and services,” says Cretti, “but also the needed integration with pre-existing monitoring systems.”

scdnco4bfvcygshp7py4

The team also uses Ceilosca, a codebase that gathers data from Ceilometer and sends to Monasca, which helps translating the different data to various preferred monitoring systems. The team developed custom pollsters to collect data from the varying sources not covered by the default OpenStack tools, as well.

“All this is what we did architecturally,” says Cretti, “then deployed this system into production into FIWARE Lab.”

Everything went well during installation, though there was a small problem installing Monasca, but the team solved it quickly.

Issues and solutions

The team had a couple of smaller issues in publishing metrics and filtering metrics, but were able to work around the problem with some added code. Both solutions involve some massaging of data before submission to the larger monitoring processes as below.

wjk7j7xevdgh8ilbfmuj

For publishing metrics, Ceilosca (version 2015.1) gets a string deserialization error when publishing into the Monasca API those Ceilometer metrics whose metadata includes non-string items, like the “properties” dict of Glance images. The team solved this by marshaling the “value_meta” (stringification) before posting those metrics to the Monasca API.

With regards to filtering metrics, Ceilosca processes a configuration file “monasca_field_definitions.yaml” that lets FIWARE filter the dimensions and metadata to be included when publishing metrics, but there is no flexibility in filtering the metrics to be posted to Monasca.

“We solved this by introducing a new storage driver (impl_monasca_filtered.py) which only processes metrics explicitly listed in a new item ‘metrics,’” Cretti adds.

What’s next?

The team would like to integrate more powerful analytics into FIWARE Lab, like Apache Spark. They are also looking into open source dashboards like grafana, using root cause analysis more often, adding monitoring and analysis for logging, adding automatic deployment via OpenStack FUEL of Ceilometer Pollster, adding agents and other custom components, and verifying potential contributions to the OpenStack community.

You can find code repositories and catalogs for FIWARE at the following links:

FIWARE catalog
FIWARE Ceilometer Pollsters
Ceilosca 2015.1.3-FIWARE

Ultimately, Cretti credits the ease of implementation and current monitoring to OpenStack and the various open source tools like Ceilometer and Monasca with making FIWARE Lab a success across the regions of Europe and into the future.

You can catch his entire 34-minute talk on the [Foundation video page.]
(https://www.openstack.org/videos/video/monitoring-a-multi-region-cloud-based-on-openstack-the-fiware-lab-case-study)