Using Chef encrypted data bags and Hiera-eyaml with Puppet

Some information in data bags can be safely stored in the Chef server in plain text, but under some circumstances, sensitive information might be safer if encrypted. Companies might not like production API keys, private keys, or similar sensitive content to be stored in plain text on the Chef server or on third-party services, such as GitHub. We'll see how to encrypt and decrypt data in the command line and from inside a Chef recipe.

Getting ready

To step through this recipe, you will need:

  • A working Chef DK installation on the workstation
  • A working Vagrant installation on the workstation
  • The Chef code (optionally) from Chapter 6, Fundamentals of Managing Servers with Chef and Puppet, Chapter 7, Testing and Writing Better Infrastructure Code with Chef and Puppet, or any custom Chef code

How to do it…

Our goal is to create a configuration file containing our AWS credentials for the us-east-1 region, and it's not acceptable that you store the credentials in clear text on the Chef server. We'd like to use a data bag, as it can be encrypted:

  1. Create a data bag folder aws to store the credentials for the us-east-1 region:
    $ mkdir data_bags/aws
  2. Create the data bag on the Chef server while we're at it:
    $ knife data bag create aws
Created data_bag[aws]
  3. Inside this aws data bag folder, create a sample us-east-1.json file containing the credentials:
    {
  "id": "us-east-1",
  "aws_access_key": "AKIAJWTIBGE3NFDB4HOB",
  "aws_secret_key": "h77/xZt/5NUafuE+q5Mte2RhGcjY4zbJ3V0cTnAc"
}

This is the standard procedure for a normal data bag. If we upload it now as is, it won't be encrypted.

Encrypting data bags with a shared secret

The solution to use an encrypted data bag is to send it encrypted from our workstation. The encryption is done through a shared secret, the secret being either a file or a string. Let's use the string s3cr3t as an encryption key (weak). To simply send the encrypted version of the data bag, let's use the encryption feature of the knife command:

$ knife data bag from file --encrypt --secret s3cr3t aws us-east-1.json
Updated data_bag_item[aws::us-east-1]

If we request the data without providing a decryption key, we'll get the encrypted data from the Chef server:

$ knife data bag show aws us-east-1
WARNING: Encrypted data bag detected, but no secret provided for decoding. Displaying encrypted data.
aws_access_key:
  cipher:         aes-256-cbc
  encrypted_data: RwbfsWgKk16sSCkMD38tXKGHmT1AHFGHRm/7fyzppye7wSS0kk19Zml0VuhQ
  XxxI

  iv:             iRRgrKfz6Ou2qdpYLkUA+w==

  version:        1
aws_secret_key:
  cipher:         aes-256-cbc
  encrypted_data: uSppKMYrRbEYn/njDYo3CIGC5tY+pptN1Z7LiARtNIU/zsllBNdSVENC1XwX
  QksifE6g00sdcHTGlHlVU0WJ0Q==

  iv:             ppjeAJcegZ9Yyn9rXgHRBQ==

  version:        1
id:             us-east-1

It looks like we got what we wanted: data is stored encrypted on the Chef server!

As it may not be a secure move to store unencrypted data bags on version control systems, such as Git, we can ask for a JSON-formatted encrypted version, such as the following, and redirect the output to a JSON file for storage purposes:

$ knife data bag show aws us-east-1 -Fj
{
  "id": "us-east-1",
  "aws_access_key": {
    "encrypted_data": "RwbfsWgKk16sSCkMD38tXKGHmT1AHFGHRm/7fyzppye7wSS0kk19Zml0VuhQ
XxxI
",
    "iv": "iRRgrKfz6Ou2qdpYLkUA+w==
",
    "version": 1,
    "cipher": "aes-256-cbc"
  },
  "aws_secret_key": {
    "encrypted_data": "uSppKMYrRbEYn/njDYo3CIGC5tY+pptN1Z7LiARtNIU/zsllBNdSVENC1XwX
QksifE6g00sdcHTGlHlVU0WJ0Q==
",
    "iv": "ppjeAJcegZ9Yyn9rXgHRBQ==
",
    "version": 1,
    "cipher": "aes-256-cbc"
  }
}

This might be the content you'd like to store on Git!

Accessing an encrypted data bag in the CLI

To access unencrypted data from the knife CLI, the process is as easy as encrypting data—pass the shared secret as an argument:

$ knife data bag show aws us-east-1 --secret s3cr3t
Encrypted data bag detected, decrypting with provided secret.
aws_access_key: AKIAJWTIBGE3NFDB4HOB
aws_secret_key: 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
id:             us-east-1

Now we have access to our data but in an unencrypted form.

Using an encrypted data bag from a recipe

Now that the data is safely stored on the Chef server, how do we access it from inside a Chef recipe? Let's say our objective is to create a file named /etc/aws/credentials that will contain the unencrypted value from the encrypted version on the Chef server. The final file should look like this:

[region_name]
aws_access_key_id = the_access_key
aws_secret_access_key = the_secret_key
  1. To do so, create a new recipe named aws inside the mysite cookbook:
    $ chef generate recipe aws

    Note

    Don't forget to bump the cookbook version and environment constraints accordingly.

  2. Start by creating the /etc/aws folder using the directory resource:
    directory "/etc/aws" do
  owner 'root'
  group 'root'
  mode '0755'
  action :create
end
  3. Here's a templates/aws.erb ERB template file for our destination, namely /etc/aws/credentials:
    [<%= @aws_region %>]
aws_access_key_id = <%= @aws_access_key %>
aws_secret_access_key = <%= @aws_secret_key %>

    We see the template is expecting the aws_region, aws_access_key, and aws_secret_key variables. Let's write the code to inject these values to the aws.rb recipe. To begin with, let's access our encrypted data bag item us-east-1 from the aws data bag, using the inline shared secret s3cr3t:

    aws = Chef::EncryptedDataBagItem.load("aws", "us-east-1", 's3cr3t')

    Note

    All of this information can be set as attributes if we like. If the file method is chosen for the shared secret, the final argument will be the path to the secret key file to decrypt the data.

  4. Now let's create the template, writing the decrypted credentials to the /etc/aws/credentials file:
    template "/etc/aws/credentials" do
  source 'aws.erb'
  owner 'root'
  group 'root'
  mode '0600'
  variables(
    aws_region: aws['id'],
    aws_access_key: aws['aws_access_key'],
    aws_secret_key: aws['aws_secret_key']
  )
end

Here we are! The Chef server is now safely storing encrypted data. For added security, it's better to not hardcode the shared key—use the key file that is sent separately (but this creates an added layer of complexity in the deployment system).

There's more…

While using Puppet, it is a good practice to store the credentials and site information in Hiera, as we saw in Chapter 6, Fundamentals of Managing Servers with Chef and Puppet. Using hiera-eyaml, it is possible to encrypt sensitive data. Using our previous LAMP setup with Vagrant, let's try to encrypt the root password for MySQL.

Preparing the Puppet server

We need to install a new backend for Hiera. We have not discussed a lot about Hiera yet, and it's time to do so. Hiera is used to store data out of manifests, and is based on a hierarchy to look up data. A default configuration is provided with the Puppet server installation and is located at /etc/puppetlabs/puppet/hiera.yaml:

---
:backends:
  - yaml
:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:yaml:
:datadir:

Here, a yaml backend is defined, allowing us to use yaml files in the hieradata directory of our environments. Then, a hierarchy is defined. Puppet will first try to look up data in a yaml file with the name matching the name of the client certificate (that is, the FQDN node) and located under the nodes subdirectory. If no data is found, Puppet will try to look up a common.yaml file.

With hiera-eyaml, we need to declare a new backend to look up data in encrypted files. This backend is eyaml, and by default, we will look for files with the .eyaml extension. This backend relies on a key pair to read data, so we need to generate these keys.

Fortunately, a Puppet module, named puppet/hiera, exists to handle all of this for us. So we just need to add it to Puppetfile with its dependencies (do not forget to run r10k puppetfile install):

mod 'puppetlabs/inifile'
mod 'puppet/hiera'
mod 'puppetlabs/puppetserver_gem'

With this module, it is now very easy to prepare the Puppet server using the following:

node 'puppet.pomes.pro' {

  # Create a service resource for the puppetserver
  # This is needed by the hiera module, in order
  # to restart the server once hiera-eyaml is installed
  service {'puppetserver':
    ensure => running,
  }
  # Configure hiera
  class { 'hiera':
    hierarchy => [
      'nodes/%{::trusted.certname}',
    ],
    eyaml          => true,
    manage_package => true,
    provider       => 'puppetserver_gem',
    master_service => 'puppetserver',
  }
}

This piece of code will:

  • Declare a service resource for the Puppet server. This is needed by the puppet/hiera module (see parameter master_service)
  • Install the eyaml backend in the Puppet server
  • Update the Hiera configuration in order to use this backend
  • Generate the private and public keys
  • Restart the Puppet server

Private and public keys will be respectively placed in /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem and /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem.

Preparing the workstation

To prepare the workstation, follow these steps:

  1. To create and edit encrypted data, we need eyaml. Let's install it using the following:
    $ sudo puppet resource package hiera-eyaml provider=puppet_gem
  2. Let's copy the keys from the Puppet server and store them in a keys folder under $HOME:
    $ ls ~/keys/
private_key.pkcs7.pem  public_key.pkcs7.pem
  3. For security reasons, it is a good idea to restrict access to the private key:
    $ chmod 500 keys
$ chmod 400 keys/private_key.pkcs7.pem
  4. We also need an eyaml configuration file, located in ~/.eyaml/config.yaml, with this content (do not forget to adjust the path of your $HOME directory):
    ---
pkcs7_public_key: "/Users/me/keys/public_key.pkcs7.pem"
pkcs7_private_key: "/Users/me/keys/private_key.pkcs7.pem"

We are now ready to encrypt sensitive data.

Securing the MySQL root password

From the command line, eyaml can encrypt values. Here is a session example:

$ eyaml encrypt -s 'super_secure_password'
[hiera-eyaml-core] Loaded config from /Users/me/.eyaml/config.yaml
string: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEALjJ2a9uZ04lk2V5xKqEd0n3BtA4OLe1B6rA2iVruJRKxWJdevuGvJ55DDedRwBMZmqbvSMO1cgMUyPbfEy54i3SXw4x3LEuxc1R31ILoOspBgzU4OLuepCotuhBASA/pI/xu40y66AZAcCQ4CtD9SZJYjiWNtUA91rcARy/xYQGK39QievxT2eq5De89qIn2w/5fIRIkJBRyNqnwyYCWKcKSRwaiLbimpwmarOP+dxGHEFRrD/FiM4NfoV1WNNVr1UkPEFuNrWBzwBpvyZUnMbGHN676Rg5vq9sS6aWI6zPxTrJyLtssZm1f4GsfhmE+anFmuxrcWtEH6C82wKMOoTBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBC3MhSP09yUw8XTj0XdlG1VgCCDCGhqIFdUmORYKlq0Pn5CE/cDZKTO+bhHxdBw5amAGQ==]

OR

block: >
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEALjJ2a9uZ04lk2V5xKqEd0n3BtA4OLe1B6rA2
    iVruJRKxWJdevuGvJ55DDedRwBMZmqbvSMO1cgMUyPbfEy54i3SXw4x3LEux
    c1R31ILoOspBgzU4OLuepCotuhBASA/pI/xu40y66AZAcCQ4CtD9SZJYjiWN
    tUA91rcARy/xYQGK39QievxT2eq5De89qIn2w/5fIRIkJBRyNqnwyYCWKcKS
    RwaiLbimpwmarOP+dxGHEFRrD/FiM4NfoV1WNNVr1UkPEFuNrWBzwBpvyZUn
    MbGHN676Rg5vq9sS6aWI6zPxTrJyLtssZm1f4GsfhmE+anFmuxrcWtEH6C82
    wKMOoTBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBC3MhSP09yUw8XTj0Xd
    lG1VgCCDCGhqIFdUmORYKlq0Pn5CE/cDZKTO+bhHxdBw5amAGQ==]

Since the eyaml backend is looking for files with the .eyaml extension, we just need to create a hieradata/nodes/web.pomes.pro.eyaml file with the following content:

---
root_password: >
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEw
    DQYJKoZIhvcNAQEBBQAEggEALjJ2a9uZ04lk2V5xKqEd0n3BtA4OLe1B6rA2
    iVruJRKxWJdevuGvJ55DDedRwBMZmqbvSMO1cgMUyPbfEy54i3SXw4x3LEux
    c1R31ILoOspBgzU4OLuepCotuhBASA/pI/xu40y66AZAcCQ4CtD9SZJYjiWN
    tUA91rcARy/xYQGK39QievxT2eq5De89qIn2w/5fIRIkJBRyNqnwyYCWKcKS
    RwaiLbimpwmarOP+dxGHEFRrD/FiM4NfoV1WNNVr1UkPEFuNrWBzwBpvyZUn
    MbGHN676Rg5vq9sS6aWI6zPxTrJyLtssZm1f4GsfhmE+anFmuxrcWtEH6C82
    wKMOoTBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBC3MhSP09yUw8XTj0Xd
    lG1VgCCDCGhqIFdUmORYKlq0Pn5CE/cDZKTO+bhHxdBw5amAGQ==]

However, eyaml has a very handy feature—the edit mode—allowing us to create and edit encrypted values in plain text, based on the keys stored in the $HOME directory:

$ eyaml edit hieradata/nodes/web.pomes.pro.eyaml

This command will launch an editor, and we just need to enter the following content:

---
root_password: >
  DEC::PKCS7[super_secure_password]!

While saving, eyaml will write the file with the encrypted content for root_password. If needed, we can edit the file again and all the encrypted values will be automatically decrypted.

Note

When editing with eyaml edit, all the new values should be contained in a DEC::PKCS7[value]! block. For existing values, eyaml will add an index called num to DEC(<num>)::PKCS7[value]! blocks. This index must remain unchanged.

As our last step, we need to modify the main manifest to do a Hiera lookup in order to get the password:

node 'web.pomes.pro' {
...
    $pass=hiera('root_password');
...
    class { 'mysql::server':
      root_password => $pass;
    }
...
}

Now the root password is encrypted in Hiera, and only people having the keys can recover it.

See also

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.188.218.157