3.1 History of Keystone Token Formats

Keystone offers several token formats, and users may wonder why there are so many. To help with understanding why, we provide a brief history of how the Keystone token formats have evolved. In the early days, Keystone supported a UUID token. This token was a 32-character string bearer token used for authentication and authorization. The advantage of this token format was that the token was small and very easy to use, and it was simple enough to add to a cURL command. The disadvantage of this token is it did not carry with it enough information to locally do authorization. OpenStack services would always have to send this token back to the Keystone server to determine if an operation was authorized. This resulted in Keystone being pinged for any OpenStack action, and becoming a bottleneck for all of OpenStack.

In an attempt to address the issues encountered with UUID tokens, the Keystone team created a new token format called the PKI token. This token contained enough information to perform local authorization and also contained the service catalog in the token as well. In addition, the token was signed and services could cache the token and use it until it expired or was revoked. The PKI token resulted in less traffic to the Keystone server, but the disadvantage of this token was that they were huge. PKI tokens could easily be 8K in size, and this made them difficult to fit in HTTP headers. Many web servers could not handle an 8K entry in an HTTP header without reconfiguration. In addition, these tokens were more difficult to use in cURL commands and offered poor user experience. Each of these drawbacks could be mitigated, but each took extra effort. The Keystone team tried a variant of this token that was compressed. This token format was called PKIz. Unfortunately, the feedback from the OpenStack community was that this token format was still too big.

The Keystone team went back to the drawing board and came up with a new token format known as the Fernet token. The Fernet token was small (around 255 characters) but contained enough information in it to be used for local authorization. The token had another critical advantage. It contained enough information that Keystone did not have to store token data in a token database. One of the most annoying aspects of Keystone’s early token formats was that they needed to be persisted in a database. The database would fill up and Keystone’s performance would suffer. OpenStack operators were constantly having to flush the Keystone token database to keep their OpenStack environments running. The Fernet token did have the drawback that the symmetric keys used to sign the tokens needed to be distributed and rotated. The OpenStack operators needed to come up with a way to handle these issues. Nonetheless, the operators we have worked with have said that they’d much rather deal with the key distribution associated Fernet tokens than have to use the other token formats and be responsible for flushing the Keystone token database.

With so many token choices, you may be wondering which tokens are used by OpenStack operators. Fortunately, the OpenStack community performs user surveys to gain these types of insights. As shown in the chart below, the UUID token is currently the most popular format. Please note, however, that this survey was performed in the OpenStack Juno release time frame, and does not include Fernet tokens because they were not added until the Kilo release.

Figure 3-1. A survey conducted by the OpenStack User committee indicates that UUID tokens are still the preferred token format (as of Juno)

In this chapter, we describe Keystone’s UUID, PKI, and Fernet tokens in greater detail to provide the reader with a much better understanding of the token formats that are available to them in Keystone.

For all three token formats, it is important to note that all are bearer tokens. This means all three tokens need to be protected from unnecessary disclosure to prevent unauthorized access.

3.2 UUID Tokens

Keystone’s first token format was the UUID token format. The UUID token is simply a randomly generated UUID 32-character string. These are issued and validated by the identity service. A hexdigest() method is used, which ensures the tokens are made up of solely hexadecimal digits. This makes the tokens URL friendly and safe for transfer in any non-binary environment. A UUID token must be saved in a persistent backend (typically a database) in order to be available for subsequent validation. A UUID token can be revoked by simply issuing a DELETE request with the token ID. Note that the token is not actually removed from the backend, but rather marked as revoked. Since the token is only 32 characters, its size in an HTTP header is 32 bytes.

A typical UUID token would look like the following: 468da447bd1c4821bbc5def0498fd441.

The token is extremely small and easy to use when accessing Keystone through a cURL command. As previously mentioned, the disadvantage with this token format is that Keystone can become a bottleneck due to the tremendous amount of communication that occurs when Keystone is needed to validate the token.

The method for returning a UUID token is relatively simple. It is shown below and can be found on GitHub.

def _get_token_id(self, token_data):
    return uuid.uuid4().hex

3.3 PKI Tokens

Keystone’s second supported format is the PKI token format. In the PKI format, the token contains the entire validation response that would be received from Keystone. What this means is that the token has a large amount of information in it, such as: when it was issued, when it expires, user identification, project, domain and role information for this user, a service catalog, and other information as well. All of this information is represented in a JSON document payload, and the payload is then signed using cryptographic message syntax (CMS). With the PKIz format, the signed payload is then compressed using zlib compression.

An example of what the JSON token payload would look like can be found in Keystone’s documentation and is repeated here:

{
    "token": {
        "domain": {
            "id": "default",
            "name": "Default"
        },
        "methods": [
            "password"
        ],
        "roles": [
            {
                "id": "c703057be878458588961ce9a0ce686b",
                "name": "admin"
            }
        ],
        "expires_at": "2014-06-10T21:52:58.852167Z",
        "catalog": [
            {
                "endpoints": [
                    {
                        "url": "http://localhost:35357/v2.0",
                        "region": "RegionOne",
                        "interface": "admin",
                        "id": "29beb2f1567642eb810b042b6719ea88"
                    },
                    {
                        "url": "http://localhost:5000/v2.0",
                        "region": "RegionOne",
                        "interface": "internal",
                        "id": "87057e3735d4415c97ae231b4841eb1c"
                    },
                    {
                        "url": "http://localhost:5000/v2.0",
                        "region": "RegionOne",
                        "interface": "public",
                        "id": "ef303187fc8d41668f25199c298396a5"
                    }
                ],
                "type": "identity",
                "id": "bd7397d2c0e14fb69bae8ff76e112a90",
                "name": "keystone"
            }
        ],
        "extras": {},
        "user": {
            "domain": {
                "id": "default",
                "name": "Default"
            },
            "id": "3ec3164f750146be97f21559ee4d9c51",
            "name": "admin"
        },
        "audit_ids": [
            "Xpa6Uyn-T9S6mTREudUH3w"
        ],
        "issued_at": "2014-06-10T20:52:58.852194Z"
    }
}

As shown in the JSON above, the token has everything a service needs with respect to user, domain, project, and role information to determine if the user is authorized for a particular OpenStack service operation. The service is thus able to cache this token and make authorization decisions locally without having to contact the Keystone server on every authorization request. In order to transport the token via HTTP, the signed JSON document is base64 encoded with some minor modifications. An example of what the token would look like in transport is shown here in an abbreviated form:

MIIDsAYCCAokGCSqGSIb3DQEHAaCCAnoEggJ2ew0KICAgICJhY2QogICAgICAgI...EBMFwwVzELMAkGA
1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MCoIIDoTCCA50CAQExCTAHBgUrDgMQ4wDAYDVQQHEwVVbnNldD
EOMAwGA1UEChM7r0iosFscpnfCuc8jGMobyfApz/dZqJnsk4lt1ahlNTpXQeVFxNK/ydKL+tzEjg

Note that even a basic token with a single endpoint in the catalog approaches a size of approximately 1,700 bytes. With larger deployments with several endpoints and services, the size of a PKI token quickly exceeds that of a default HTTP header limit for most web servers, which is 8KB. Even when compressed, the size of PKIz tokens does not decrease enough to mitigate these problems. In practice, the resultant token is approximately 10% smaller.

Although PKI and PKIz tokens can be cached, they have several disadvantages. It can be difficult to configure Keystone to use this type of token, as they really should use certificates issued by a trusted certificate authority. Moreover, they are extremely large and their size may cause other OpenStack services to have issues. Their size can also break web performance tools, and they are quite difficult to use in a cURL command. In addition, in practice the Keystone service typically must persist these tokens in its persistence backend for purposes such as creating revocation lists. As a result the user must continue to worry about flushing the Keystone token database frequently or performance will suffer.

Once the difficult part of having all the certificates created is complete, generating PKI tokens is not that complex. As an illustration, the key aspects of the method for returning a PKI token are shown below.¹

def _get_token_id(self, token_data):
        try:
            token_json = jsonutils.dumps(token_data, cls=utils.PKIEncoder)
            token_id = str(cms.cms_sign_token(token_json,
                                              CONF.signing.certfile,
                                              CONF.signing.keyfile))
            return token_id
        except environment.subprocess.CalledProcessError:
            LOG.exception(_LE('Unable to sign token'))
            raise exception.UnexpectedError(_('Unable to sign token.')) 

3.4 Fernet Tokens

The newest Keystone token format is the Fernet token format. The Fernet token attempts to improve on previous token formats in a variety of ways. First, it is quite small—it typically comes in around 255 characters, and as such are larger than UUID tokens, but significantly smaller than PKI.² The token also contains just enough information to enable the token to not have to be stored in a persistent Keystone token database. Instead, the token has enough information that the rest of the needed information such as the roles a user has on a project can be generated. In large-scale OpenStack deployments, having to store massive amounts of token data has been identified as a key contributor to performance degradation, not to mention the need to constantly flush the persistent token database.³

Fernet tokens contain a small amount of information, such as a user identifier, a project identifier, token expiration information, and other auditing information. Fernet tokens are signed using a symmetric key to prevent tampering. The use of a Fernet token follows the same workflow as a UUID token. Thus, in contrast to a PKI token, they must be validated by Keystone similar to how this must be done with UUID tokens.

One complicating factor is that Fernet tokens use symmetric keys to sign the token, and these keys need to be distributed to the various OpenStack regions. Additionally, these keys need to be rotated. Because Fernet tokens are being revised in the current release, we are hesitant to provide a large amount of detail on these tokens at this point in time. Instead, we refer the reader to some online sources we anticipate will be updated based on real-world experience using Fernet tokens and symmetric key rotation.⁴

Keystone provides a configuration tool for setting up symmetric encryption keys. This can be performed by invoking the following command:

$ keystone-manage fernet_setup

In addition, the keys should be periodically rotated using the following command:

$ keystone-manage fernet_rotate

As an illustration, the key aspects of creating a project-scoped Fernet token is shown below. The example deviates from the source code to improve readability. The fundamental step to creating a Fernet token is to safely convert any data to bytes, then MessagePack the token contents, sign it, and make it URL safe.⁵

def create_token(self, user_id, expires_at, audit_ids, methods=None,
                 domain_id=None, project_id=None, trust_id=None,
                 federated_info=None):

    b_user_id = cls.attempt_convert_uuid_hex_to_bytes(user_id)
    methods = auth_plugins.convert_method_list_to_integer(methods)
    b_project_id = cls.attempt_convert_uuid_hex_to_bytes(project_id)
    expires_at_int = cls._convert_time_string_to_int(expires_at)
    b_audit_ids = list(map(provider.random_urlsafe_str_to_bytes, audit_ids))

    payload = (b_user_id, methods, b_project_id, expires_at_int, b_audit_ids)
    serialized_payload = msgpack.packb(payload)
    token = urllib.parse.quote(self.crypto.encrypt(serialized_payload))
    return token

A typical Fernet token looks like:

gAAAAABU7roWGiCuOvgFcckec-0ytpGnMZDBLG9hA7Hr9qfvdZDHjsak39YN98HXxoYLIqVm19Egku5YR
3wyI7heVrOmPNEtmr-fIM1rtahudEdEAPM4HCiMrBmiA1Lw6SU8jc2rPLC7FK7nBCia_BGhG17NVHuQu0
S7waA306jyKNhHwUnpsBQ%3D

Fernet tokens are validated by simply performing the same steps to create the token, but in reverse. We begin by unquoting the URL-safe bits, decrypting the value, and unpacking the payload. From there, it’s simply a matter of checking the different sections of the payload, specifically the expires_at field.

def validate_token(self, token):

    token = urllib.parse.unquote(six.binary_type(token))
    serialized_payload = self.crypto.decrypt(token)
    payload = msgpack.unpackb(serialized_payload)
       
    user_id = cls.attempt_convert_uuid_bytes_to_hex(payload[0])
    methods = auth_plugins.convert_integer_to_method_list(payload[1])
    project_id = cls.attempt_convert_uuid_bytes_to_hex(payload[2])
    expires_at_str = cls._convert_int_to_time_string(payload[3])
    audit_ids = list(map(provider.base64_encode, payload[4]))

    return (user_id, methods, project_id, expires_at_str, audit_ids)

3.5 Tips, Common Pitfalls, and Troubleshooting

In this section, we describe a few commonly experienced pitfalls that occur when using the different types of Keystone token formats and provide troubleshooting tips to address these issues.

$ keystone-manage token_flush

[token]
provider = keystone.token.providers.pkiz.Provider