Architecture and Main Components of Keystone

The Keystone or OpenStack Identity service acts as a catalog of all OpenStack services and provides the ability for authenticating and managing user accounts and role information for the cloud environment. If you are familiar with the Microsoft Windows Server environment, you can think of Keystone as the “Active Directory analog” for your OpenStack cloud. Usually Keystone is the first component to be installed when starting an OpenStack cloud. Keystone supports multiple forms of authentication, including login name and password, token-based credentials, and REST API log ins.

First, let’s define some terms which Keystone operates with:

• Service: OpenStack cloud component listed in Keystone catalog. Examples of the services are Nova, Neutron, Glance, Keystone itself, etc. Service provides one or more endpoints through which users can access service’s API.

• Endpoint: URL from which the service is available. Service can have three endpoints: internal, public, and administration. They can have different subsets of API calls. Endpoint can look like https://controller.my-domain.com:35357/v2.0 . At this URL you would find that the service is listening to incoming calls on port number 35357 and the API version is 2.0. Common port numbers for OpenStack services are shown in Table 3-1.

  Table 3-1. Common Port Numbers for OpenStack Services

  Network Port Number	OpenStack Service
  5000	Public API endpoint port for Keystone
  35357	Admin API endpoint port for Keystone
  8776	Cinder Block Storage service
  9292	Image service Glance
  9191	Glance Registry
  8774	Compute service Nova
  8080 and 6001-6003	Object Storage services Swift
  9696	Networking service Neutron
  8777	Telemetry service Ceilometer
  8004	Orchestration service Heat

• Project: Represents the base unit of ownership in OpenStack. Networks, VMs, users, roles, and so on belong to a particular project. For administrative operations in OpenStack, an environment special administrative project “admin” exists.

• Domain: Represents a collection of projects, groups, and users that defines administrative boundaries for managing OpenStack Identity entities.

• Region: Separates the OpenStack environment with dedicated API endpoints but with common Keystone service.

• Token: Issued by Keystone service then passed to API requests and used by OpenStack to verify that the client is authorized to run the requested operation. The token is issued for a limited time and, if necessary, may be withdrawn prior to the expiration. In order to get the user token, the user must either provide a name and password, or the name and the key to access the API (API key). The token also contains a list of roles that defines the roles available to the user.

• User: Individual API consumer. User can be associated with roles, projects, or both.

• Role: Specific set of operations associated with a user. A role includes a set of rights and privileges.

From an architectural point of view, Keystone is the simplest service in the cloud. As for many other OpenStack services, OpenStack Identity service uses the MariaDB/MySQL database. As an alternative, it is possible to store information in the LDAP (Lightweight Directory Access Protocol) server or Microsoft Active Directory. Starting from the Miaka release, Keystone uses the Apache web server as the front end, so you no longer need to start openstack-keystone.service. Prior to the Mitaka release, Keystone worked under the built-in Eventlet Python service by default.

Let’s have a quick look through the Keystone main configuration file /etc/keystone/keystone.conf. Table 3-2 summarizes the main configuration options from config.

Managing Keystone Catalog Services and Endpoints

Before initiating something with OpenStack, you need to go through the authorization and authentication processes. You can use the CLI commands options, but it is better and easier to create a file with global variables for GNU/Linux environment and to process this file with the source command. You need to create in any text editor something like the following code:

unset OS_SERVICE_TOKEN
export OS_AUTH_URL=http://10.0.2.15:5000/v2.0
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_IDENTITY_API_VERSION=3

You need to use your correct IP address and correct admin password for your environment. Now you can execute the script:

$ source keystonerc_admin

Let’s try to create a service record in Keystone for the OpenStack image service Glance (the Glance service is described in Chapter 4):

$ openstack service create --name glance --description "OpenStack Image service" image
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | OpenStack Image service          |
| enabled     | True                             |
| id          | 9d33c464f61749cd9f5811cda1ae5444 |
| name        | glance                           |
| type        | image                            |
+-------------+----------------------------------+

Only two mandatory options exist in this particular command. First, the name of the service, and second, the type of the service. The name is glance and the type is image. You can check the existing services with their types by using the openstack service list command:

$ openstack service list
+----------------------------------+------------+---------------+
| ID                               | Name       | Type          |
+----------------------------------+------------+---------------+
| 1b3b63218f1042a4994b51e8d20078ec | cinderv2   | volumev2      |
| 49b256b46a0f4052acee768b5b0bbe65 | cinder     | volume        |
| 4b815b6d85474c70a449326b6bf4b4ea | ceilometer | metering      |
| 7d5da91499224026a21efdf84300381a | nova_ec2   | ec2           |
| 7e621a56c3aa41f78ed6d5bddaba3a92 | swift      | object-store  |
| 9d33c464f61749cd9f5811cda1ae5444 | glance     | image         |
| b0763e843e0e4e7284e14e76f4dd702c | keystone   | identity      |
| b8367ddba94248cfa16451390684f89c | heat       | orchestration |
| c81578d4864349e1b29a04e2554556bc | nova       | compute       |
| d83dacc916434390b3557c4ff0e893a4 | neutron    | network       |
| e4851946adb14ee481660bd45b76496f | novav3     | computev3     |
| ea9433eec76d49ebb11ed47645b5765b | swift_s3   | s3            |
+----------------------------------+------------+---------------+

If you made a mistake in service creation, you can easily delete it with the openstack service delete command. After creating the service record in the Keystone catalog, you need to create three endpoints for this service. This can be done with the next command:

$ openstack endpoint create 
>   --publicurl http://10.0.2.15:9292 
>   --internalurl http://10.0.2.15:9292 
>   --adminurl http://10.0.2.15:9292
>   --region RegionOne image
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| adminurl     | http://10.0.2.15:9292            |
| id           | 5ae58266319446a4837ce0c212c5ad1a |
| internalurl  | http://10.0.2.15:9292            |
| publicurl    | http://10.0.2.15:9292            |
| region       | RegionOne                        |
| service_id   | 9d33c464f61749cd9f5811cda1ae5444 |
| service_name | glance                           |
| service_type | image                            |
+--------------+----------------------------------+

You can run a check of all of the endpoints:

$ openstack endpoint list
+----------------------------------+-----------+--------------+---------------+
| ID                               | Region    | Service Name | Service Type  |
+----------------------------------+-----------+--------------+---------------+
| f312043049e04056a793c16fd4b81bc5 | RegionOne | ceilometer   | metering      |
| 6af17d75bdec498cbc2af32e6625b1b2 | RegionOne | keystone     | identity      |
| 82c5b56f536e446189abbef7c114e9c4 | RegionOne | neutron      | network       |
| 9d6a6b9d9ee744e3a2991dbb39cec995 | RegionOne | cinderv2     | volumev2      |
| ecfb03318bcb4bd588ee7a02833aae31 | RegionOne | nova         | compute       |
| 93591bd7d0394abc8a1e624e5be2f284 | RegionOne | cinder       | volume        |
| 73598ea8c3e8480a965f83f50fbf92bb | RegionOne | nova_ec2     | ec2           |
| bc52befa27da44cfbd709b5c67fc44fe | RegionOne | swift        | object-store  |
| d5d7afeaf892470bac9fc587bb413cb3 | RegionOne | heat         | orchestration |
| 37316205c43746ca96ca6435fd2f4b7a | RegionOne | swift_s3     | s3            |
| 2d8fb2f861a24f5f8964df225a7961cd | RegionOne | novav3       | computev3     |
| 5ae58266319446a4837ce0c212c5ad1a | RegionOne | glance       | image         |
+----------------------------------+-----------+--------------+---------------+

And you can show the details about a particular endpoint:

$ openstack endpoint show glance
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| adminurl     | http://10.0.2.15:9292            |
| enabled      | True                             |
| id           | 5ae58266319446a4837ce0c212c5ad1a |
| internalurl  | http://10.0.2.15:9292            |
| publicurl    | http://10.0.2.15:9292            |
| region       | RegionOne                        |
| service_id   | 9d33c464f61749cd9f5811cda1ae5444 |
| service_name | glance                           |
| service_type | image                            |
+--------------+----------------------------------+

Horizon can handle approximately 70% of the overall available OpenStack functionality. So you can’t create services and endpoints through the web client, although you can check a list of services and their current statuses. Log in as admin and go to Admin ➤ System Information (see Figure 3-1).

Figure 3-1. List of services in Horizon

Managing/Creating Projects, Users, and Roles

You can easily create projects, users, and roles with the help of the openstack command. Let’s start from a new project. You need to use admin credentials for this operation:

$ source keystonerc_admin
$ openstack project create --description "Test project" apress
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Test project                     |
| enabled     | True                             |
| id          | ee6dbb7d8b5e420da8e8bd1b5e23953b |
| name        | apress                           |
+-------------+----------------------------------+

You can get a list of all projects and details about each project as well:

$ openstack project list
+----------------------------------+----------+
| ID                               | Name     |
+----------------------------------+----------+
| 1542af2b20d349d29710d8c4019ba202 | demo     |
| 233d4bfa02ee46e69194a7594570da45 | services |
| 560a3e76bdc64ea2bee9316038b12793 | admin    |
| ee6dbb7d8b5e420da8e8bd1b5e23953b | apress   |
+----------------------------------+----------+
$ openstack project show apress
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Test project                     |
| enabled     | True                             |
| id          | ee6dbb7d8b5e420da8e8bd1b5e23953b |
| name        | apress                           |
+-------------+----------------------------------+

Now you can create a new user—apressadmin—granting _member_ a role in the Apress project:

$ openstack user create --password-prompt apressuser
User Password:
Repeat User Password:
+----------+----------------------------------+
| Field    | Value                            |
+----------+----------------------------------+
| email    | None                             |
| enabled  | True                             |
| id       | 639a67455b474a9eae2a9f048ee811b1 |
| name     | apressuser                       |
| username | apressuser                       |
+----------+----------------------------------+
$ openstack role add --project apress --user apressuser _member_
+-------+----------------------------------+
| Field | Value                            |
+-------+----------------------------------+
| id    | 9fe2ff9ee4384b1894a90878d3e92bab |
| name  | _member_                         |
+-------+----------------------------------+

If you want to get a list of all of the roles in OpenStack cloud, you can use the command:

$ openstack role list
+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 7f8760410d94476c81fa77589cf7f6e2 | heat_stack_user  |
| 9120dbbe6c324f96978f34ae8e386c36 | heat_stack_owner |
| 9d70fe84f7524503aeb69dfa9a2b987e | admin            |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_         |
| e39b1852b0674392a4c56a48e37fa7d7 | SwiftOperator    |
+----------------------------------+------------------+

The file policy.json exists in each service /etc/service_name/ directory. In such files, policy definitions for roles are described. For example, this is a part of /etc/keystone/policy.json:

{
    "admin_required": "role:admin or is_admin:1",
...
    "identity:create_region": "rule:admin_required",
...
    "identity:create_domain": "rule:admin_required",
...
   "identity:list_groups_for_user": "rule:admin_or_owner",
}

The first line is the admin role definition and the next three lines are the policy rules, which are represented by the following syntax:

"<service>:<action>" : "<subject conditions>"

As you can see, to create a region or domain in the identity service, you need an admin role. You will get an HTTP 403 error code if the current policy doesn’t allow the command to be performed.

After creating a new user, you may want to create a new keystonerc file for it. You may use the keystonerc_admin file as a template. In this case, you need to change the OS_TENANT_NAME, OS_USERNAME and OS_PASSWORD variables.

If you need to delete a user or project, you can use the same openstack command but with the delete subcommand. For example:

$ openstack user delete apressuser
$ openstack project delete apress

It is possible to create, delete, and edit users and projects in OpenStack in web interface (Horizon). Go to Identity ➤ Users or Identity ➤ Projects. Examples of editing project and creating user are shown in Figures 3-2 and 3-3, respectively.

Figure 3-2. Managing project members in Horizon

Figure 3-3. Creating a new user in Horizon

Managing and Verifying Operation of the Identity Service

For performance, a modern OpenStack installation deploys the Apache HTTP server with mod_wsgi package to handle requests and Memcached fronfend to store tokens. In CentOS, Scientific Linux, Oracle Linux, and other Red Hat Enterprise Linux derivatives, distribution for service management systemd is used. You can check whether Memcached and Apache servers are started and active:

# systemctl status memcached.service
● memcached.service - Memcached
   Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2016-04-10 13:13:45 MSK; 1h 34min ago
...
# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─openstack-dashboard.conf
   Active: active (running) since Sun 2016-04-10 13:14:50 MSK; 1h 33min ago
...

The standard GNU/Linux command ps can show you two processes with names containing keystone:

# ps aux | grep keystone
root     22377  0.0  0.0 112644   960 pts/0    R+   15:42   0:00 grep --color=auto keystone
keystone 23767  0.0  1.9 595200 78144 ?        Sl   14:14   0:05 keystone-admin  -DFOREGROUND
keystone 23768  0.0  0.2 397220 10088 ?        Sl   14:14   0:00 keystone-main   -DFOREGROUND

There are two distinct daemon processes created for running through the WSGI module of Apache HTTP server. You can find their configuration files in the /etc/httpd/conf.d/ directory:

# ls /etc/httpd/conf.d/*keystone*
/etc/httpd/conf.d/10-keystone_wsgi_admin.conf  /etc/httpd/conf.d/10-keystone_wsgi_main.conf

Here is a shorted example of the configuration file for admin’s virtual host:

<VirtualHost *:35357>
  ServerName centos7.test.local
  DocumentRoot "/var/www/cgi-bin/keystone"

  <Directory "/var/www/cgi-bin/keystone">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride None
    Require all granted
  </Directory>

 ErrorLog /var/log/httpd/keystone-error.log
 CustomLog /var/log/httpd/keystone-access.log combined
  WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=1 threads=1 user=keystone
  WSGIProcessGroup keystone_admin
  WSGIScriptAlias / "/var/www/cgi-bin/keystone/admin"
  WSGIPassAuthorization On
</VirtualHost>

In case there is a need for troubleshooting, you may also want to check the endpoints by name or by ID:

$ openstack endpoint show identity
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| adminurl     | http://10.0.2.15:35357/v2.0      |
| enabled      | True                             |
| id           | 6af17d75bdec498cbc2af32e6625b1b2 |
| internalurl  | http://10.0.2.15:5000/v2.0       |
| publicurl    | http://10.0.2.15:5000/v2.0       |
| region       | RegionOne                        |
| service_id   | b0763e843e0e4e7284e14e76f4dd702c |
| service_name | keystone                         |
| service_type | identity                         |
+--------------+----------------------------------+

As you see, the internal and public URLs are the same, but the admin endpoint uses port 35357, as discussed earlier in this chapter. In case of debugging, you may want to check the log ins using /var/log/httpd/keystone_* and /var/log/leystone/keystone.log.

Review Questions

1. How would you add the user apressuser with a Member role to the apress project?

   1. openstack role add --project apress --user apressuser _member_

   2. openstack role add --project apress --user apressuser member

   3. openstack role add --project apress --user _member_ apressuser

   4. openstack role add --project apress --user member apressuser

2. Which two system services should be started for a proper Keystone functioning (choose two)?

   1. keystone-main

   2. keystone-admin

   3. memcached

   4. httpd

3. How would you define a new role in OpenStack cloud (choose all applicable)?

   1. Enter command openstack role create newrole.

   2. Restart httpd service.

   3. Create new "keystonerc" file.

   4. Add definition to policy.json files.

4. How would you separate two or more cloud instances but manage them with one Keystone instance?

   1. Use Domains feature.

   2. Use Regions feature.

   3. Use availability zones.

   4. Each cloud instance should use own Keystone instance feature.

5. Which HTTP error code would you get if the Keystone token has expired?

   1. ERROR 404.

   2. ERROR 403.

   3. ERROR 401.

   4. All of them.

Answers to Review Questions

1. A

2. C, D

3. A, D

4. A

5. C