)
OpenStack and Kubernetes are currently the most popular open infrastructure solutions, so it’s worthwhile to provide users access to a platform that provides both services, using a single personal account. Currently this is hardly possible, since the two systems provide different authentication mechanisms. OpenStack uses its own identity system, Keystone, while Kubernetes delegates authentication to external providers through a mechanism of plug-ins.
Previous attempts to integrate assumed password-based authentication for OpenStack and enabled Kubernetes users to authenticate with their OpenStack passwords through Keystone.
However, there other means to authenticate to Keystone, for example through federated authentication, which is a more secure and scalable solution, redirecting to an external trusted identity provider. In particular, the federated GARR cloud platform uses federated authentication through EduGain, enabling SSO for all users of the worldwide research community.
To provide a general solution, we developed an innovative technique, based on a novel feature of OpenStack called application credentials that became fully available with the Rocky release.
The proposed solution requires a modified version of one of the official SDK libraries for OpenStack. This change has been approved by the project maintainers and will be released as part of the next official distribution of the library.
The implementation of Keystone authentication for Kubernetes relies on a WebHook, one of the authentication modules provided by Kubernetes authentication. When WebHook authentication is enabled, the Kubernetes API redirects requests to a RESTful service. In our case, we use an OpenStack-Keystone service as authentication provider.
To simplify usage, we’ve extended the OpenStack dashboard by adding a button for downloading a config file, ready to use with kubectl, that includes the application credentials.
GARR deployed a multi-tenant Kubernetes cluster on bare metal to reduce management overhead resource fragmentation and delays in cluster creation. We used the same declarative modeling tools for deploying the cluster with MaaS and Juju, side by side with our OpenStack infrastructure. This facilitates maintenance and scaling of the infrastructure with a single set of tools. Tough multi-tenancy limits restrict the rights for users to access common extensions. Therefore, we provide a set of specific roles and bindings to give normal users without privileges on namespace kube-system but with rights to perform installations, for example through Helm.
What follows is the architecture of the solution, its components and their implementation in a real-world production environment as well as an installation and configuration guide for users. We conclude with suggestions for future extensions in order to deal with role-based access control (RBAC.)
Introduction
We’ll start with how to integrate authentication between OpenStack, an infrastructure-as-a-service (IaaS) provider, and Kubernetes, a container deployment service to allow OpenStack users to seamlessly access Kubernetes services.
In Kubernetes, processing of a request goes through the following stages:
- Authentication
- Authorization
- Admission control
Kubernetes supports several authentication strategies that may invoke external authenticator providers (e.g. LDAP or OpenID Connect) available through plug-ins, as shown in step one in the following diagram:
Each plug-in, which is invoked as an external command through the library k8s.io/client-go, implements its protocol-specific logic, then returns opaque credentials. Credential plug-ins typically require a server-side component with support for WebHook token authentication to interpret the credential format produced by the client plug-in.
The solution we developed to provide Keystone authentication for Kubernetes consists of the following modules:
- A credential plug-in for Keystone authentication
- A service implementing a WebHook for authentication
- A service implementing a WebHook for authorization (currently the same as module two)
Here are the steps for the authentication process:
- A user issues a kubectl command or issues an API call, which is handled by client-go.
- The credential plug-in obtains the user’s Keystone credential, either from the kubeconfig file or by prompting the user and requests a token to Keystone using the aforementioned credentials.
- The token from Keystone returns to the client through the credential plug-in.
- The client uses this token as a bearer token against the Kubernetes API server.
- The Kubernetes API server uses the WebHook token authenticator to validate the token against the Keystone service.
- The Keystone service verifies the token and returns the user’s username and groups.
The solution we present is of general interest, since it allows cloud providers to offer both a container deployment platform based on Kubernetes and IaaS services provided by OpenStack, both accessible through a single set of credentials.
An earlier solution for integrating Kubernetes authentication with OpenStack relied on password authentication. But OpenStack can be configured to use federated authentication, like the one used in the GARR Federated Cloud Platform, provided by Idem or EduGain. Consequently, password authentication isn’t available for normal users in this scenario.
A development team from SWITCH and GARR worked jointly to find a more general solution. The recent Queens release for Keystone introduced the mechanism of application credentials. Through this mechanism, an application can request a token that can be used thereafter to validate user requests before performing operations on his behalf. Furthermore, in the Rocky release of the Horizon dashboard, a panel has been added allowing users to create application credentials.
The key idea of this solution is to use an application credential obtained from Keystone and pass it to Kubernetes for validating user requests. This requires exploiting the plug-in architecture provided by Kubernetes to insert suitable steps in the authentication process. In particular, Kubernetes needs to convert credentials into a token and later use that token whenever needed to validate each individual request before performing it.
The ability to obtain credentials directly from the dashboard allows users to be completely autonomous in setting up integrated Kubernetes/Keystone authentication. For example, the given credentials can be inserted in the user configuration file for kubectl, the standard command-line interface for operating on Kubernetes. Afterwards, the user can access Kubernetes without any further complications.
A limitation of the current solution is that it requires installing a plug-in on the user’s machine, which has these drawbacks:
- Binary versions for each machine architecture and for each Kubernetes release must be maintained
- Mobile devices are not supported
Keystone authentication with application credentials for Kubernetes
Since the Queens release of OpenStack, Keystone has supported application credentials. These credentials can be used by applications to authenticate through Keystone with the assigned privileges by the user who created them. In particular, such credentials can be used by the Kubernetes API to authenticate and authorize operations.
In the solution presented here, authentication is performed by a plugin (kubectl-keystone-auth), while authorization is delegated by the Kubernetes API through a WebHook to a RESTful web service (k8s-keystone-auth).
In the next section, we describe how to use Keystone Application Credentials for authenticating to Kubrernetes and use them for Kubernetes services.
Create application credentials with Horizon
The following screenshots illustrate the steps needed to an application credential through the OpenStack Horizon dashboard.
Select Application Credentials in the Identity Panel:
Fill out the form to create an application credential:
Download both an openrc file to set OpenStack environment variables for using the generated application credential and a configuration file for kubectl:
The button “Download kubeconfig file” is an extension that we developed for the Horizon dashboard, which creates a a preconfigured ./kube/config file ready to use to work on Kubernetes. It contains the application credentials and other parameters for connecting to the Kubernetes API server.
The code for this extension is available on GitLab and mirrored on GitHub.
Enable Kubernetes authentication via application credentials
Once the application credential is created, you can download the kubectl config file with the button “Download kube config file”.
The credential plugin kubectl-keystone-auth is required in order to enable application credentials authentication. It can be either downloaded or compiled from sources.
Download the credential plugin
Download kubectl-keystone-auth for your architecture from:
Install it in a folder accessible by kubectl, for example:
$ mkdir -p ~/.kube/bin
$ cp -p kubectl-keystone-auth ~/.kube/binBuild the credential plugin
A working installation of Golang is needed to build the plugin. Follow the instructions at: https://golang.org/doc/install#install.
Clone the repository for cloud-provider-openstack:
$ git clone https://github.com/kubernetes/cloud-provider-openstack
$GOPATH/src/kubernetes/cloud-provider-openstack
$ cd $GOPATH/src/kubernetes/cloud-provider-openstackBuild the plugin with:
$ sudo make client-keystone-authInstall it in a folder accessible by kubectl, for example:
$ mkdir -p ~/.kube/bin
$ cp -p client-keystone-auth ~/.kube/bin/kubectl-keystone-authSetting up Keystone authentication
This section describes the steps that a cloud administrator needs to perform to setup Keystone authentication in a Kubernetes cluster.
The Kubernetes API server must be configured with WebHook token authentication to invoke an authenticator service for validating tokens with Keystone. The service to be invoked cannot be Keystone itself, since the payload produced by the WebHook has a different format than the requests expected by the Keystone API for application credentials.
Here’s an example of a WebHook payload:
{
"apiVersion": "authorization.k8s.io/v1beta1",
"kind": "SubjectAccessReview",
"spec": {
"resourceAttributes": {
"namespace": "kittensandponies",
"verb": "get",
"group": "unicorn.example.org",
"resource": "pods"
},
"user": "jane",
"group": ["group1"]
}
}
While this token validation request to Keystone has an empty payload and parameters are passed as follows: token of authorized user in the X-Auth-Token request header, token to validate in the X-Subject-Token request header. The response has the following form:
{
"token": {
"audit_ids": [
"mAjXQhiYRyKwkB4qygdLVg"
],
"expires_at": "2015-11-05T22:00:11.000000Z",
"issued_at": "2015-11-05T21:00:33.819948Z",
"methods": [
"password"
],
"user": {
"domain": {
"id": "default",
"name": "Default"
},
"id": "10a2e6e717a245d9acad3e5f97aeca3d",
"name": "admin",
"password_expires_at": null
}
}
}
The program that implements the authenticator service is called k8s-keystone-auth. Steps to obtaining it are described below.
Configure the Kubernetes API server
The Kubernetes API receives a request including a Keystone token. In the Kubernetes language, this is a Bearer Token. To validate the Keystone token the Kubernetes API server will use a WebHook. The service invoked through the WebHook will in turn contact the Keystone service that generated the token in order to validate it.
Here we describe how to configure the Kubernetes API server to invoke the k8s-keystone-auth authenticator through a WebHook.
Create the following file in /path/to/webhook.kubeconfig:
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: KEYSTONE_URL
name: webhook
contexts:
- context:
cluster: webhook
user: webhook
name: webhook
current-context: webhook
kind: Config
preferences: {}
users:
- name: webhook
where KEYSTONE_URL is the endpoint of the Keystone service.
Execute the following command in the master Kubernetes API node to configure it:
$ sudo snap set kube-apiserver authentication-token-webhook-config-file = /path/to/webhook.kubeconfigIf you do not used snap edit file in /etc/kubernetes/manifests/kube-apiserver.yaml and add this line as parameter to the kubectl command:
- --authentication-token-webhook-config-file=webhook.kubeconfigInstall the Keystone authenticator service
The Keystone authenticator service is the component in charge of validating requests containing bearer tokens.
The Keystone authenticator service is implemented by the program k8s-keystone-auth. You can either download a pre-compiled version or build it from various sources.
Download the Keystone authenticator
You can find pre-compiled versions of k8s-keystone-auth for different architectures in the following repository:
https://git.garr.it/cloud/charms/kubernetes-keystone/raw/master/bin/
Deploy via Juju
In order to deploy the Keystone authorization service on a cluster managed through Juju, we provide a charm that automates its deployment. The service will be automatically replicated on all the Kubernetes Master units, ensuring high availability. The charm is available on the public repository: https://git.garr.it/cloud/charms/kubernetes-keystone.
The k8s-keystone-auth service can be deployed by doing:
$ juju deploy cs:~csd-garr/kubernetes-keystone
--config keystone-url='KEYSTONE_URL'
--config k8s-keystone-auth-url='DOWNLOAD_URL'
--config authn-server-url='AUTHN_URL'
--config authz-server-url='AUTHZ_URL'
$ juju add-relation kubernetes-master kubernetes-keystoneThe configuration parameters are:
KEYSTONE_URL URL of the Keystone endpoint.
DOWNLOAD_URL URL for downloading the Keystone authenticator server program.
AUTHN_URL URL of the WebHook authentication service.
AUTHZ_URL URL for the WebHook authorization service.
Configuration parameters can also be passed through a YAML file as explained here: https://docs.jujucharms.com/2.4/en/charms-config.
Alternatively, the WebHook authenticator service can be deployed as a Kubernetes pod. This requires a Docker image for k8s-keystone-auth to be deployed within a Docker container
The steps for building the Docker image are described in the section “Build the Keystone authenticator,” including the following:
$ make image-k8s-keystone-authThe following deployment file is used for deploying the WebHook authenticator service on Kubernetes itself.
——————————————————————————–
kind: Deployment
metadata:
name: k8s-keystone-auth
namespace: kube-system
labels:
app: k8s-keystone-auth
spec:
replicas: 1
selector:
matchLabels:
app: k8s-keystone-auth
template:
metadata:
labels:
app: k8s-keystone-auth
spec:
nodeSelector:
dedicated: k8s-master
containers:
- name: k8s-keystone-auth
image: rdil/k8s-keystone-auth:latest
imagePullPolicy: Always
args:
- ./bin/k8s-keystone-auth
- --tls-cert-file
- /etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file
- /etc/kubernetes/pki/apiserver.key
- --keystone-url
- KEYSTONE_URL
- k8s-auth-policy
- --sync-config-file
- /etc/kubernetes/pki/identity/keystone/syncconfig.yaml
volumeMounts:
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
ports:
- containerPort: 844
volumes:
- name: k8s-certs
hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
- name: ca-certs
hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
---
kind: Service
apiVersion: v1
metadata:
name: k8s-keystone-auth-service
namespace: kube-system
spec:
selector:
app: k8s-keystone-auth
ports:
- protocol: TCP
port: 8443
targetPort: 8443
----------------------------------------------------------------------------------
where KEYSTONE_URL is https://keystone.cloud.garr.it:5000/v3 for the GARR Cloud Platform.
In order to deploy the component on the master node (because the master node contains the sync config file and the WebHook that forward the auth request to the localhost), we labeled the master node (kubectl label nodes name_of_your_node dedicated=k8s-master).
The pod can be deployed on the master node (kubectl taint nodes –all node-role.kubernetes.io/master-) and we added to the deployment specification a section to choose the master node as node for the deployment.
We also added “hostNetwork: true” in order to put the pod on the same network as the master node in order to make the communication with it possible.
OpenStack client using application credentials
Application Credentials can be used also with the OpenStack Client for authenticating to Keystone.
In order to use application credentials, the following variables must be set:
export OS_AUTH_TYPE=v3applicationcredential
export OS_AUTH_URL=KEYSTONE_URL
export OS_APPLICATION_CREDENTIAL_NAME=CREDENTIAL_NAME
export OS_APPLICATION_CREDENTIAL_SECRET=CREDENTIAL_SECRETwhere KEYSTONE_URL for the GARR Cloud Platform is https://keystone.cloud.garr.it:5000/v3, while CREDENTIAL_NAME and CREDENTIAL_SECRET are the values from the application credential.
The openstack command, so configured, will then pass authentication:
$ openstack token issue
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2018-09-04T09:58:33+0000 |
| id | 99e74e7a8ec14b1bb945672662580ea7 |
| project_id | daadb4bcc9704054b108de8ed263dfc2 |
| user_id | 4ae5e9b91b1446408523cb01e5da46d5 |
+------------+----------------------------------+
Conclusions
We’ve presented a solution for integrating Kubernetes with OpenStack, exploiting Keystone as a common authentication service and using application credentials provided by Keystone.
In a follow-up post, we’ll describe how we set up a multi-tenant Kubernetes cluster on bare metal using automation tools MaaS and Juju. The cluster is shared among users of the GARR comunity, providing better performance and reduced costs. However, we needed to provide some common functionality in order to enable users to install services that are normally installed n the system namespace kube-system. In particular, we’ll show how to deal with the creation of Kubernetes dashboard and to exploit Helm for installing packaged containerized applications.
About the authors
The work was carried out by GARR‘s Giuseppe Attardi, Alberto Colla, Alex Barchiesi, Roberto Di Lallo, Fulvio Galeazzi Claudio Pisa and Saverio Proto at SWITCH as part of the GÉANT project (GN4-2.)
Saverio Proto
Lingxian Kong
Kendall Waters
