How to integrate Ceph with OpenStack

In this article, we’ll discuss why Ceph is perfect fit for OpenStack.

We’ll also show you how to integrate three prominent OpenStack use cases with Ceph: Cinder (block storage), Glance (images) and Nova (VM virtual disks).

Ceph provides unified scale-out storage, using commodity x86 hardware that is self-healing and intelligently anticipates failures. It has become the de facto standard for software-defined storage. Because Ceph is open source, it has enabled many vendors the ability to provide Ceph based software-defined storage systems. Ceph is not just limited to companies like Red Hat, Suse, Mirantis, Ubuntu, etc. — SanDisk, Fujitsu, HP, Dell, Samsung and many more now offer integrated solutions, too. There are even large-scale community built environments — CERN comes to mind — that provide storage services for 10,000s of VMs.

Ceph is by no means limited to OpenStack, however this is where Ceph started gaining traction. Looking at latest OpenStack user survey, Ceph is the clear leader for OpenStack storage. Page 42 in OpenStack April 2016 User Survey reveals Ceph is 57 percent of OpenStack storage. The next is LVM (local storage) with 28 percent followed by NetApp with 9 percent. If we remove LVM, Ceph leads any other storage company by 48 percent, that is incredible. Why is that?

There are several reasons, but here are my top three:

Ceph is a scale-out unified storage platform. OpenStack needs two things from storage: ability to scale with OpenStack itself and do so regardless of block (Cinder), File (Manila) or Object (Swift). Traditional storage vendors need to provide two or three different storage systems to achieve this. They don’t scale the same and in most cases only scale-up in never-ending migration cycles. Their management capabilities are never truly integrated across broad spectrum of storage use-cases.
Ceph is cost-effective. Ceph leverages Linux as an operating system instead of something proprietary. You can choose not only whom you purchase Ceph from, but also where you get your hardware. It can be same vendor or different. You can purchase commodity hardware, or even buy integrated solution of Ceph + Hardware from single vendor. There are even hyper-converged options for Ceph that are emerging (running Ceph services on compute nodes).
Ceph is OpenSource project just like OpenStack. This allows for a much tighter integration and cross-project development. Proprietary vendors are always playing catch-up since they have secrets to protect and their influence is usually limited in Opensource communities, especially in OpenStack context.

Here is an architecture diagram that shows all the different OpenStack components that need storage. It shows how they integrate with Ceph and how Ceph provides a unified storage system that scales to fill all these use cases.

If you are interested in more topics relating to Ceph and OpenStack, I recommend this.

Ok enough talking about why Ceph and OpenStack are so great, let’s get our hands dirty and see how to hook it up!

If you don’t have a Ceph environment you can follow this article on how to set one up quickly.

Glance Integration

Glance is the image service within OpenStack. By default images are stored locally on controllers and then copied to compute hosts when requested. The compute hosts cache the images but they need to be copied again, every time an image is updated.

Ceph provides back end for Glance allowing images to be stored in Ceph, instead of locally on controller and compute nodes. This greatly reduces network traffic for pulling images and increases performance since Ceph can clone images instead of copying them. In addition, it makes migrating between OpenStack deployments or concepts like multi-site OpenStack much simpler.

Install the Ceph client used by Glance.

  
    [root@osp9 ~]# yum install -y python-rbd

Create Ceph user and set home directory to /etc/ceph.

  
    [root@osp9 ~]# mkdir /etc/ceph
    [root@osp9 ~]# useradd ceph
    [root@osp9 ~]# passwd ceph

Add Ceph user to sudoers.

  
    cat << EOF >/etc/sudoers.d/ceph
    ceph ALL = (root) NOPASSWD:ALL
    Defaults:ceph !requiretty
    EOF

On Ceph admin node.
Create Ceph RBD Pool for Glance images.

  [ceph@ceph1 ~]$ sudo ceph osd pool create images 128

Create keyring that will allow Glance access to pool.

  [ceph@ceph1 ~]$ sudo ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rdb_children, allow rwx pool=images' -o /etc/ceph/ceph.client.images.keyring

Copy the keyring to /etc/ceph on OpenStack controller.

  [ceph@ceph1 ~]$ scp /etc/ceph/ceph.conf [email protected]:/etc/ceph

Set permissions so Glance can access Ceph keyring.

  [root@osp9 ~(keystone_admin)]# chgrp glance /etc/ceph/ceph.client.images.keyring
[root@osp9 ~(keystone_admin)]#chmod 0640 /etc/ceph/ceph.client.images.keyring

Add keyring file to Ceph configuration.

  [root@osp9 ~(keystone_admin)]# vi /etc/ceph/ceph.conf
[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

Create a backup of original Glance configuration.

  [root@osp9 ~(keystone_admin)]# cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.orig

Update Glace configuration

  
  [root@osp9 ~]# vi /etc/glance/glance-api.conf
  [glance_store]
  stores = glance.store.rbd.Store
  default_store = rbd
  rbd_store_pool = images
  rbd_store_user = images
  rbd_store_ceph_conf = /etc/ceph/ceph.conf

Restart Glance.

  [root@osp9 ~(keystone_admin)]# systemctl restart openstack-glance-api

Download Cirros images and add it into Glance.

  [root@osp9 ~(keystone_admin)]# wget http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img

Convert QCOW2 to RAW. It is recommended for Ceph to always use RAW format.

  [root@osp9 ~(keystone_admin)]#qemu-img convert cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.raw

Add image to Glance.

  [root@osp9 ~(keystone_admin)]#glance image-create --name "Cirros 0.3.4" --disk-format raw --container-format bare --visibility public --file cirros-0.3.4-x86_64-disk.raw

  +------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2016-09-07T12:29:23Z |
| disk_format | qcow2 |
| id | a55e9417-67af-43c5-a342-85d2c4c483f7 |
| min_disk | 0 |
| min_ram | 0 |
| name | Cirros 0.3.4 |
| owner | dd6a4ed994d04255a451da66b68a8057 |
| protected | False |
| size | 13287936 |
| status | active |
| tags | [] |
| updated_at | 2016-09-07T12:29:27Z |
| virtual_size | None |
| visibility | public |
+------------------+--------------------------------------+

Check that a Glance image exists in Ceph.

  [ceph@ceph1 ceph-config]$ sudo rbd ls images
a55e9417-67af-43c5-a342-85d2c4c483f7

  [ceph@ceph1 ceph-config]$ sudo rbd info images/a55e9417-67af-43c5-a342-85d2c4c483f7
rbd image 'a55e9417-67af-43c5-a342-85d2c4c483f7':
 size 12976 kB in 2 objects
 order 23 (8192 kB objects)
 block_name_prefix: rbd_data.183e54fd29b46
 format: 2
 features: layering, striping
 flags:
 stripe unit: 8192 kB
 stripe count: 1

Cinder Integration

Cinder is the block storage service in OpenStack. Cinder provides an abstraction around block storage and allows vendors to integrate by providing a driver. In Ceph, each storage pool can be mapped to a different Cinder back end. This allows for creating storage services such as gold, silver or bronze. You can decide for example that gold should be fast SSD disks that are replicated three times, while silver only should be replicated two times and bronze should use slower disks with erasure coding.

Create a Ceph pool for cinder volumes.

  [ceph@ceph1 ~]$ sudo ceph osd pool create  128

Create a keyring to grant Cinder access.

  [ceph@ceph1 ~]$ sudo ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.volumes.keyring

Copy the keyring to OpenStack controllers.

  [ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.volumes.keyring [email protected]:/etc/ceph

Create a file that contains just the authentication key on OpenStack controllers.

  [ceph@ceph1 ~]$ sudo ceph auth get-key client.volumes |ssh osp9.lab tee client.volumes.key

Set permissions on the keyring file so it can be accessed by Cinder.

  [root@osp9 ~(keystone_admin)]# chgrp cinder /etc/ceph/ceph.client.volumes.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.volumes.keyring

Add the keyring to Ceph configuration file on OpenStack controllers.

  [root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

Give KVM Hypervisor access to Ceph.

  [root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/cinder.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

 [root@osp9 ~(keystone_admin)]#vi /etc/ceph/cinder.xml


 ce6d1549-4d63-476b-afb6-88f0b196414f
 
 client.volumes secret

  [root@osp9 ceph]# virsh secret-define --file /etc/ceph/cinder.xml

  [root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret ce6d1549-4d63-476b-afb6-88f0b196414f --base64 $(cat /etc/ceph/client.volumes.key)

Add a Ceph backend for Cinder

[root@osp9 ~(keystone_admin)]#vi /etc/cinder/cinder.conf

[rbd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = volumes
rbd_secret_uuid = ce6d1549-4d63-476b-afb6-88f0b196414f

Restart Cinder service on all controllers.

  [root@osp9 ~(keystone_admin)]# openstack-service restart cinder

Create a Cinder volume.

  
  [root@osp9 ~(keystone_admin)]# cinder create --display-name="test" 1
  +--------------------------------+--------------------------------------+
  | Property | Value |
  +--------------------------------+--------------------------------------+
  | attachments | [] |
  | availability_zone | nova |
  | bootable | false |
  | consistencygroup_id | None |
  | created_at | 2016-09-08T10:58:17.000000 |
  | description | None |
  | encrypted | False |
  | id | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b |
  | metadata | {} |
  | migration_status | None |
  | multiattach | False |
  | name | test |
  | os-vol-host-attr:host | None |
  | os-vol-mig-status-attr:migstat | None |
  | os-vol-mig-status-attr:name_id | None |
  | os-vol-tenant-attr:tenant_id | dd6a4ed994d04255a451da66b68a8057 |
  | replication_status | disabled |
  | size | 1 |
  | snapshot_id | None |
  | source_volid | None |
  | status | creating |
  | updated_at | None |
  | user_id | 783d6e51e611400c80458de5d735897e |
  | volume_type | None |
  +--------------------------------+--------------------------------------+
  
  
  List new cinder volume
  
  [root@osp9 ~(keystone_admin)]# cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | ID | Status | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b | available | test | 1 | - | false | |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+

List a Cinder volume in Ceph.

  [ceph@ceph1 ~]$ sudo rbd ls volumes
volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b

  
   [ceph@ceph1 ~]$ sudo rbd info volumes/volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
  rbd image 'volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b':
   size 1024 MB in 256 objects
   order 22 (4096 kB objects)
   block_name_prefix: rbd_data.2033b50c26d41
   format: 2
   features: layering, striping
   flags:
   stripe unit: 4096 kB
   stripe count: 1

Integrating Ceph with Nova Compute

Nova is the compute service within OpenStack. Nova stores virtual disks images associated with running VMs by default, locally on Hypervisor under /var/lib/nova/instances. There are a few drawbacks to using local storage on compute nodes for virtual disk images:

Images are stored under root filesystem. Large images can cause filesystem to fill up, thus crashing compute nodes.
A disk crash on compute node could cause loss of virtual disk and as such a VM recovery would be impossible.

Ceph is one of the storage backends that can integrate directly with Nova. In this section we will see how to configure that.

  [ceph@ceph1 ~]$ sudo ceph osd pool create vms 128

Create an authentication keyring for Nova.

  [ceph@ceph1 ~]$ sudo ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring

Copy the keyring to OpenStack controllers.

  [ceph@ceph1 ~]$ scp /etc/ceph/ceph.client.nova.keyring [email protected]:/etc/ceph

Create key file on OpenStack controllers.

  [ceph@ceph1 ~]$ sudo ceph auth get-key client.nova |ssh osp9.lab tee client.nova.key

Set permissions on the keyring file so it can be accessed by Nova service.

  [root@osp9 ~(keystone_admin)]# chgrp nova /etc/ceph/ceph.client.nova.keyring
[root@osp9 ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.nova.keyring

Ensure the required rpm packages are installed.

  
  [root@osp9 ~(keystone_admin)]# yum list installed python-rbd ceph-common
  Loaded plugins: product-id, search-disabled-repos, subscription-manager
  Installed Packages
  ceph-common.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms
  python-rbd.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms

Update the Ceph configuration.

  [root@osp9 ~(keystone_admin)]#vi /etc/ceph/ceph.conf

[client.nova]
keyring = /etc/ceph/ceph.client.nova.keyring

Give KVM access to Ceph.

  [root@osp9 ~(keystone_admin)]# uuidgen |tee /etc/ceph/nova.uuid.txt

Create a secret in virsh so KVM can access the Ceph pool for cinder volumes.

  
  [root@osp9 ~(keystone_admin)]#vi /etc/ceph/nova.xml
  
  
  c89c0a90-9648-49eb-b443-c97adb538f23
  
  client.volumes secret

  [root@osp9 ~(keystone_admin)]# virsh secret-define --file /etc/ceph/nova.xml

  [root@osp9 ~(keystone_admin)]# virsh secret-set-value --secret c89c0a90-9648-49eb-b443-c97adb538f23 --base64 $(cat /etc/ceph/client.nova.key)

Make a backup of Nova configuration

  [root@osp9 ~(keystone_admin)]# cp /etc/nova/nova.conf /etc/nova/nova.conf.orig

Update the Nova configuration to use Ceph back end.

  [root@osp9 ~(keystone_admin)]#vi /etc/nova/nova.conf
force_raw_images = True
disk_cachemodes = writeback

[libvirt]
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
rbd_secret_uuid = c89c0a90-9648-49eb-b443-c97adb538f23

Restart Nova services.

  [root@osp9 ~(keystone_admin)]# systemctl restart openstack-nova-compute

List Neutron networks.

  [root@osp9 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-----------------------------------------------------+
| 4683d03d-30fc-4dd1-9b5f-eccd87340e70 | private | ef909061-34ee-4b67-80a9-829ea0a862d0 10.10.1.0/24 |
| 8d35a938-5e4f-46a2-8948-b8c4d752e81e | public | bb2b65e7-ab41-4792-8591-91507784b8d8 192.168.0.0/24 |
+--------------------------------------+---------+-----------------------------------------------------+

Start an ephemeral VM instance using Cirros image that was added in the steps for Glance.

  [root@osp9 ~(keystone_admin)]# nova boot --flavor m1.small --nic net-id=4683d03d-30fc-4dd1-9b5f-eccd87340e70 --image='Cirros 0.3.4' cephvm
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | wzKrvK3miVJ3 |
| config_drive | |
| created | 2016-09-08T11:41:29Z |
| flavor | m1.small (2) |
| hostId | |
| id | 85c66004-e8c6-497e-b5d3-b949a1666c90 |
| image | Cirros 0.3.4 (a55e9417-67af-43c5-a342-85d2c4c483f7) |
| key_name | - |
| metadata | {} |
| name | cephvm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| updated | 2016-09-08T11:41:33Z |
| user_id | 783d6e51e611400c80458de5d735897e |
+--------------------------------------+-----------------------------------------------------+

Wait until the VM is active.

  [root@osp9 ceph(keystone_admin)]# nova list
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+---------------------+
| 8ca3e74e-cd52-42a6-acec-13a5b8bda53c | cephvm | ACTIVE | - | Running | private=10.10.1.106 |
+--------------------------------------+--------+--------+------------+-------------+---------------------+

List images in Ceph vms pool. We should now see the image is stored in Ceph.

  [ceph@ceph1 ~]$ sudo rbd -p vms ls
8ca3e74e-cd52-42a6-acec-13a5b8bda53c_disk

Troubleshooting

UNABLE TO DELETE GLANCE IMAGES STORED IN CEPH RBD

  [root@osp9 ceph(keystone_admin)]# nova image-list
+--------------------------------------+--------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+--------------+--------+--------+
| a55e9417-67af-43c5-a342-85d2c4c483f7 | Cirros 0.3.4 | ACTIVE | |
| 34510bb3-da95-4cb1-8a66-59f572ec0a5d | test123 | ACTIVE | |
| cf56345e-1454-4775-84f6-781912ce242b | test456 | ACTIVE | |
+--------------------------------------+--------------+--------+--------+

  [root@osp9 ceph(keystone_admin)]# rbd -p images snap unprotect cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# rbd -p images snap rm cf56345e-1454-4775-84f6-781912ce242b@snap
[root@osp9 ceph(keystone_admin)]# glance image-delete cf56345e-1454-4775-84f6-781912ce242b

Summary

In this article, we discussed how OpenStack and Ceph fit perfectly together. We discussed some of the use cases around Glance, Cinder and Nova. Finally we went through steps to integrate Ceph with those use cases. I hope you enjoyed the article and found the information useful. Please share your feedback.

Happy Cephing!

This post first appeared on Keith Tenzer’s blog. Superuser is always interested in community content, email: [email protected].

Cover Photo // CC BY NC

Tags: Ceph, Glance, Nova, tutorial, User Survey

Author
Recent Posts

Superuser

Superuser Magazine is the Open Infrastructure Foundation's official online publication. It covers open infrastructure ecosystem news, case studies, event recaps, product updates and announcements, project releases, and more.