Puppet Directory Environments

Puppet allows you to create, modify, and remove Puppet environments without making configuration changes to your servers or agents. You do this by adding or removing directories from Puppet’s $environmentpath directory. Although you’d never do it manually like this, the following steps are all that is required to create a test environment:

$ cd /etc/puppetlabs/code/environments
$ cp -r production test

The creation and removal of Puppet environments can be easily automated so that changes approved in source code management can be automatically deployed for testing. This ability to quickly create, update, and remove Puppet environments provides the foundation for enabling Puppet testing, release, deployment, and upgrade strategies.

Selectable Blocks for Catalog Building

Each Puppet environment has the components (code and data) necessary to build a node’s catalog. All of the following can be distinct between environments:

• Manifests

• Modules, including:

  • Classes

  • Defined types

  • Custom facts

  • Functions

  • Resource types

  • Resource providers

• Hiera data, including:

  • Customized hierarchy

  • Custom lookup and merge policies on a per-key basis

  • Environment data providers

  • Encryption keys

• Files

• Caching (for performance)

• Catalog version provisioning

It’s not required that each of these things differ: most of your environments will likely be very similar. But each of these things can be tuned distinctly for any environment.

Environment Configuration

Each environment contains a configuration file (environment.conf) that specifies the configuration of the environment. At the time this book was last updated, this file can specify the following:

• A directory containing Puppet manifests (replacing the historic site.pp)

• The directories from which Puppet modules (and their facts, functions, types, and providers) are loaded

• Whether the data should be cached for performance

• An optional program to provide the catalog version number

Every parameter in this configuration is optional and falls back to the global default.

Environment Independence and Isolation

One of the often-overlooked features of Puppet environments is that they operate independently. When you deploy a new environment, you don’t need to modify any other environment to enable its existence. When you remove an environment, no changes are required in anything that the other environments use. This complete independence greatly simplifies testing and deployment without any risk of impact to uninvolved environments.

Because each environment can have a unique module path, this means that each environment can have different versions of functions, resource types, and resource providers of the same name. And because Puppet 4.6 environments can safely utilize these conflicting versions by generating type metadata for each environment, this greatly simplifies development and testing of improved resource types and providers.

What Does r10k Actually Do?

Although our introduction touched on the benefits of deploying r10k, we didn’t really explain how it works to support those goals. r10k has two primary commands:

deploy

Deploy Puppet environments based on code branches in one or more control repositories.

puppetfile

Install modules and data in an environment from a Puppetfile specification.

The most common use of r10k uses both commands together to deploy a complete environment in two phases. Let’s review how to configure each of these phases.

The Control Repository

r10k deploys environments based on a configuration stored in what is commonly called a control repository. A control repository contains multiple branches, one for each Puppet environment available to be deployed. Each branch contains the minimum necessary files and metadata used to deploy the entire environment.

The concept of a control repository is not specific to r10k. This is actually a design pattern around which r10k’s features and functionality continue to evolve.

The control repository needs to have a branch named for each Puppet environment. Even though it would not provide much value, the simple steps presented in Example 9-1 would create a fully functional control repository.

$ mkdir environments
$ cd environments
$ git init
Initialized empty Git repository in /home/you/environments/.git/
$ git checkout -b production
Switched to a new branch 'production'
$ git commit --allow-empty -m 'empty production environment'
[production (root-commit) cd1cde6] empty production environment
$ git remote add origin https://git.example.com/puppet/environments
$ git push -u origin production

You might have noticed that this creates a default branch of production rather than master. Every Puppet environment must have a production environment (whether you use it or not) so this is commonly used as the default branch. The default branch can be anything you want, so long as you keep it mind that name will be deployed as a Puppet environment.

Control Repository Branch Contents

There is no requirement for any file to exist in a branch of the control repository. An empty branch (as created in Example 9-1) will simply deploy an empty environment directory. However, there are a number of files within a Puppet environment and design patterns that enhance and empower usage of r10k. The following are the most common default files.

r10k Configuration File

r10k has a configuration file, /etc/puppetlabs/r10k/r10k.yaml, that contains only the things required to access the control repository and Puppet module sources, including the following:

cachedir

The directory where r10k caches repositories it has downloaded.

proxy

The proxy required to access the control repository.

forge

Configuration required to access the Puppet Forge (or a mirror).

git

Configuration required to access the Git server hosting the control repositories.

sources

A hash of control repositories that should be checked for each environment deployed.

The use of r10k and the parameters in this file are well documented by Puppet at https://github.com/puppetlabs/r10k, although we do refer to some of these settings when we discuss multiteam cooperation in “Enabling Multiteam Coordination”.

Puppetfile

The Puppetfile contains the specification for Puppet modules and data to be installed in the environment.

# Get released version 3.1.0 from the Puppet Forge
mod 'puppetlabs/apache', '3.1.0'

# Install puppetlabs/apache from Github and track the 'docs_experiment' branch
mod 'apache',
  :git    => 'https://github.com/puppetlabs/puppetlabs-apache',
  :branch => 'docs_experiment'

# Install puppetlabs/apache from GitHub and pin to the '83401079' commit
mod 'apache',
  :git    => 'https://github.com/puppetlabs/puppetlabs-apache',
  :commit => '83401079053dca11d61945bd9beef9ecf7576cbf'

# Install Hiera data
mod 'data',
  :git => 'https://git.example.com/hiera_data.git',
  :branch => :control_branch,       # Track control branch name
  :default_branch => 'production',  # Default if no matching branch
  :install_base => ''               # Install in the root of the environment

r10k Deployment Walkthrough

r10k’s installation and configuration is quite elegant, despite appearing to be fairly complex at a glance. Lets take a look at a very simple configuration that you can use for learning, experimentation, and testing. For this example, we use the default user environment for Puppet, allowing you to experiment in your home directory.

The most basic use case of r10k involves deploying modules using a Puppetfile. This is commonly used during testing and development and requires minimal effort to use. Here are the steps for this deployment:

1. Install r10k and create an r10k configuration file.

2. Create a control repository as shown in Example 9-1.

3. Create a new test environment branch.

4. Populate it with an environment.conf and and Puppetfile.

5. Use r10k deploy to deploy the test environment.

$ sudo puppet resource package r10k ensure=present provider=puppet_gem

sources:
  main:
    # where will the control repo be stored?
    remote: 'https://git.example.com/puppet/environments.git'

    # run Puppet as normal user for this exercise
    basedir: '/home/username/.puppetlabs/etc/code/environments'

$ cd environments
$ git checkout -b test
Switched to a new branch 'test'

$ cp /etc/puppetlabs/code/environments/production/environment.conf ./

mod 'puppetlabs/stdlib', '4.25.1'
mod 'ipcrm/echo', '0.1.5'

$ /opt/puppetlabs/puppet/bin/r10k puppetfile check
Syntax OK
$ git commit -am 'small test environment'
[test (root-commit) 641da544] small test environment
$ git push -u origin test

$ alias r10k=/opt/puppetlabs/puppet/bin/r10k
$ r10k deploy display
---
:sources:
- :name: :main
  :basedir: "/home/user/.puppetlabs/etc/code/environments"
  :remote: https://git.example.com/puppet/environments
  :environments:
  - production
  - test

$ r10k deploy environment test --v info
INFO   -> Deploying environment /home/user/.puppetlabs/etc/code/environments/test
INFO   -> Environment test is now at 641da544b4df4bb14a48928ae71065f6736de4c7
INFO   -> Deploying Puppetfile content
          /home/user/.puppetlabs/etc/code/environments/test/modules/stdlib
INFO   -> Deploying Puppetfile content
          /home/user/.puppetlabs/etc/code/environments/test/modules/echo

$ r10k deploy environment test --verbose info
INFO   -> Deploying environment /home/user/.puppetlabs/etc/code/environments/test
INFO   -> Environment test is now at 641da544b4df4bb14a48928ae71065f6736de4c7

$ /opt/puppetlabs/puppet/bin/r10k deploy environment test --puppetfile --v info
INFO   -> Deploying environment /home/user/.puppetlabs/etc/code/environments/test
INFO   -> Environment test is now at 641da544b4df4bb14a48928ae71065f6736de4c7
INFO   -> Deploying Puppetfile content
          /home/user/.puppetlabs/etc/code/environments/test/modules/stdlib
INFO   -> Deploying Puppetfile content
          /home/user/.puppetlabs/etc/code/environments/test/modules/echo

$ puppet apply --verbose --environment test --execute 'echo { $environment: }'
Info: Loading facts
Notice: /Echo[test]/message: test
Notice: Compiled catalog for testnode.example.com in environment test
Info: Applying configuration version '1523171606'
Notice: Applied catalog in 0.02 seconds

Build Development Environments

One of r10k’s strengths is its ability to build and deploy development and test environments. You can use this to reduce iteration time, reduce the risk associated with new deployments to production, and perform small-scale test upgrades to production infrastructure.

Introducing new, potentially breaking, features can be a huge risk. The common three-way split of dev/stage/prod does not provide sufficient testing to minimize this risk. Every one of us who work in production environments has seen the drift problems created by too simple of a split. If the dev environment is preparing for next month’s release, and the stage environment is load-testing this month’s release, but a critical break-fix needs to be deployed to production ASAP—you need something more flexible for testing.

With on-demand creation of Puppet directory environments, you can introduce changes as separate feature branches. The ability to create arbitrary Puppet environments provides a lot of flexibility when implementing dangerous changes, significantly increasing the agility of the development team. Instead of testing against whatever is deployed in stage today, an exact replica of production with only the single change to be tested is made available. This ability to introduce one change at a time to a subset of nodes implements and simplifies CI practices.

The benefit of this approach is that you can apply new Puppet environments quickly and easily, allowing developers to test changes to environments as needed. The ability to quickly deploy new releases will encourage testing in live nonproduction environments and will tend to increase development agility. This approach is also somewhat simpler to implement than a package-based solution.

For testing purposes, you can think of feature branches as versions of code. This concept is very important to understand. Puppet directory environments allow the safe release of new code without applying it. Only when a node is configured to use that branch/version/environment will the new version be applied.

Simplifying Acceptance Testing

In acceptance tests, all of the modules and data are brought together to build out the complete solution (as opposed to unit tests, which are intended to be run in isolation). This kind of testing needs a complete Puppet environment mirroring what you have today (to provide a starting point) and another environment providing what you intend to deploy.

r10k is an invaluable tool to deploy environments for acceptance, integration, and end-to-end testing. If you are using r10k to build your production environments, you need to do nothing more than copy the production branch and insert the one change you’d like to test. The following example shows how to change one module’s version number to create a test environment:

$ cd environments
$ git checkout production
$ git checkout -b try_new_stdlib
Switched to a new branch 'try_new_stdlib'
$ sed -i'' -e 's/(^mod 'puppetlabs/stdlib',).*$/1 '4.26.0'/" Puppetfile
$ git commit Puppetfile -m 'stdlib version 4.26.0'
[production 7d1c3e6] stdlib version 4.26.0
$ git push -u origin try_new_stdlib

Your testing framework could deploy the new environment using r10k, and perform existing acceptance tests to ensure that nothing is broken by the change.

Implement Continuous Integration, Delivery, and Deployment

Continuous integration (CI), continuous delivery (CD), and continuous deployment refer to the process of automating the testing and deployment of your code. Each of them builds upon the previous one as follows.

CI involves automating your unit and acceptance testing in such a way that it can be triggered every time new feature code is available. The value of such a process is that it can catch a lot of problems before the code reaches a live environment. This immediate feedback improves the quality of code and can help improve the deployment process.

CD is more of a release mindset: it involves ensuring that every feature being merged back to the main branch is ready for deployment. r10k simplifies this process by making each branch available as a Puppet environment for testing by the admin, developer, or user waiting for that change. The fast roundtrip for testing the change and integrating it to the main branch improves the quality of the main branch. We find this to be crucial to avoiding drift between development and production environments.

Continuous deployment is a release methodology that might not be suitable for all environments. It involves immediately testing a CD branch (usually after a new feature is merged) and deploying it immediately if the test succeeds. Although this isn’t usually suitable for application development, it works rather well with website deployments, and you guessed it, Puppet configuration management changes.

Development and release processes are usually broken down into a number of deployment stages, from feature branch creation to release candidate creation to final production deployment. Automating these steps individually and independently creates a short-circuit feedback loop on integration problems. This in turn results in much faster turnaround times for overall and combined delivery efforts. A huge benefit of r10k is that it provides a simple, unified way to deploy development, test, and production environments. This helps improve consistency and simplifies the process of setting up a CI solution.

As CI/CD processes are general-purpose release management strategies well documented elsewhere, we refer to them here only when applicable to Puppet and related technologies. We cover a number of these deployment methodologies and how they relate to Puppet and r10k in “Release Management Strategies with r10k”.

Deploy Production Environments

Production deployment is the final step of a release-management process, and typically takes place only after the build has passed all other validation steps.

You can use r10k on production nodes to deploy changes. How you use it depends on how catalogs are built in your environment.

Puppet Server

r10k is invoked on the Puppet server to deploy the new or updated environment for all agents using that server.

Puppet apply

r10k is invoked on a destination node to deploy the new or updated environment in the local filesystem.

Regardless of the approach, r10k checks the control repository with each invocation, and populates one or more environments based on the branches available in the control repositories. Repositories are locally cached, thus limiting the load created by new deployments to only the differences (e.g., git pull).

The requirement of this approach is that each node invoking r10k must be authorized to fetch your control repository and every module referenced by the control repository. A failure to fetch any repository or module can cause the update to fail. This is most commonly used in Puppet server environments where a relatively small number of servers need to invoke r10k to service a large number of Puppet agents. Standalone environments might need to scale out their Git infrastructure to support a large number of nodes.

All of the standard best practices for deployment apply to Puppet. None of the following are specific to Puppet, but r10k makes a lot of them easier to implement:

• Employ a canary deployment to evaluate problems whenever possible.

• Spin up a production clone automatically to compare scaling characteristics.

• Deploy changes in pods, allowing instaneous cutover and rollback from one version to another.

• Implement error-tracking variance to signal or initiate an automated rollback.

• Utilize parallel orchestration tools (e.g., MCollective/Choria) to minimize version drift when deploying to large clusters.

• Prototype resource changes in a release to tag whether the change is low, medium, or high risk.

• Automate creation and testing of a rollback commit that reverts the state of every resource modified by the change.

Build and Package

You can use r10k as a build tool. Puppet code can be bundled together, versioned, and packaged for use in detached or embedded environments. A major benefit is that you can use your existing package management for the release of Puppet code.

This process is nothing more than combining the steps we just outlined with a testing framework and packaging solution. The process would generally look like this:

1. The new release changes are placed on a new branch in the control repository.

2. r10k is invoked to deploy the new version.

3. r10k or another tool kicks off acceptance testing of the solution.

4. If testing is successful, the r10k-deployed environment is packaged up.

We recommend this approach with standalone, black-box, or embedded Puppet nodes, especially when included with published software or black-box deployments in other environments. It scales very well, and requires that only your CI infrastructure be authorized to pull from your code repositories.

Stage/Production Branches

With this strategy, multiple branches are maintained, allowing flexibility in what code is released to staging hosts, and then when that code moves to production hosts. Production follows stage, lagging behind as needed. Changes are initially merged into the stage branch, which is deployed to stage nodes for testing. After testing, the changes on stage are merged onto the production branch.

In theory, this approach has the following benefits:

• All changes are tested in a working environment before going to production.

• Stage and product have their own current HEAD and revision history.

• It’s easy to test with postcommit hooks to a testing framework that utilizes the existing stage nodes.

In practice, this theory falls down due to the following real-life implementation issues:

• It doesn’t cope with overlapping short-term and long-term deliverables.

  • Locking stage for release testing holds up hotfix testing, and vice versa.

  • Hotfix changes might be merged to production without implicit dependencies left on the stage branch.

• It requires a strong skill set with source code tools to manage production merges.

  • Identifying and cherry-picking codependent changes is a highly skilled practice.

  • Any mistake would merge changes being tested in stage to production early.

• Drift over time and abandoned efforts left in stage must be cleaned up manually.

• Long-lived testing branches invariably are abused by developers for conditional execution.

That last bullet item can be far more risky than it might seem. if stage: do X, if prod: do Y leads to testing different code than what is deployed in production. When all tests are done on (effectively) random branch names, these kind of mistakes can be caught and removed.

Nutshell version: managing stage branches in this fashion is much more complicated than it would seem, and will require an expert release management skill set to manage. Consider using the layered approach provided by “GitFlow”.

Single Branch (GitHub Flow)

This release strategy avoids the complications of managing multiple development and release branches. Each and every change flows back to a single branch for use in deployment. Although this is likely unsuitable for large-scale application release strategies that have overlapping periods of new and maintenance releases, this is often perfect for configuration management purposes.

The single branch deployment model is often called branch-per-task or branch-per-issue but the best guide we’ve seen is the GitHub Flow introduction.

This model has the following benefits:

• All deployed code is committed on a single master branch.

• All other branches are feature branches for testing purposes.

• It is easy to learn by team members who have minimal experience with revision control systems.

• It is easy to test with precommit hooks, does not require a testing framework and infrastructure (although it works well with them, too).

• Changes implemented in single, whole pieces are easy to revert.

• Abandoned efforts aren’t left in a shared branch.

The major advantage of this approach is simplicity; teams need only understand the fundamentals of creating a new branch, and are less likely to make mistakes when merging efforts together. This makes it suitable to employ in a team of system administrators unfamiliar with complex release management practices.

Even though the production infrastructure follows a single branch, you still create and deploy feature branches with r10k. This allows an opportunity to test dangerous changes before merging into the master branch.

GitFlow

The GitFlow strategy takes advantage of Git’s branching and merging capabilities to maximize flexibility. You can view it as an implementation of the single-branch practices on a dual-branch (stage/prod) model. This deployment model is well documented at A successful Git branching model.

With GitFlow, each feature change starts with its own feature branch. From that point the process diverges:

• Feature branches are forked from and merged back to the develop branch.

• Automatic nightly builds and acceptance tests are built from develop branch.

• Some infrastructure might consistently run from develop (e.g., stage).

• Release branches fork from develop.

• You can deploy release branches in specific environments.

• Release branch changes are merged back to develop for ongoing integration.

• Release branches are merged to master as a complete change.

In a GitFlow model, production nodes utilize one of the release branches. This model has the following benefits:

• All deployed code is merged to a single master branch.

• People with limited release management experience manage only their own feature branches.

• Release management or QA practices (whether manual or automated) control the creation of release branches.

• Automated test frameworks can be triggered based on the GitFlow branch names.

• Changes that involve multiple dependent changes can be safely tested in isolation from other release efforts.

• Release branches can be used to test alternate implementations, with only one merging to master after a successful deployment.

The major advantage of this approach compared to GitHub Flow is that it enables layered processes:

1. Developers create and test the concepts on feature branches before merging back to develop.

2. Release management or QA (automated or manual) runs tests and creates release branches.

3. Hotfixes can be created from a release branch, and merged back to develop after testing.

4. The concept of a release branch and a Puppet environment are effectively identical. Changing the Puppet environment of the node changes the release version.

5. Flexible and partial upgrades are possible, because you can roll out changes node by node, or group by group.

If you are already familiar with GitFlow, you might ask why we maintain release branches rather than rely on tags. This is an intentional design decision; tags are not intended to be removed. Puppet environments are created only for branches. Tags can be used for multirelease association and long-term archival, allowing your release branches to be pruned without the risk of losing history.

There are only two disadvantages to this approach:

• It requires an experienced release management skill set to perform the release and master branch fork and merge tasks. This works well in organizations that already have significant release management talent and infrastructure, but it can be difficult to implement without those skills and tools.

• The GitFlow model was built around Git’s strengths in decentralized development. Although you can apply this workflow using other revision control systems, it requires one with a low cost of effort for branching and merging.

Puppet Prerun Command

It is possible to invoke r10k as a prerun command using the prerun_command configuration setting. Serverless nodes can use this to update the code stored locally, thus ensuring that each Puppet run is performed with the latest available release of code. The infrastructure affected by this benefits from Puppet’s built-in concurrency protection and splay timing.

This is suitable only for on-demand test configurations that would update a single environment prior to running a test in that environment.

Deploying on Receipt of a WebHook

By far the most common implementation of r10k utilizes a WebHook from the source repository to indicate that new content is available in a given branch. Every source code management system contains this capability.

This approach is handy because it ensures that new code is deployed immediately after it has been pushed to the Git repository. This reduces the latency between pushing a change and the change becoming available to agents. This allows developers to push code and then immediately invoke Puppet to test that the changes worked as desired.

Another benefit of using a postreceive hook is that you can build intelligence into the hook. For example, the hook can do the following:

• Deploy only the branch that has changed instead of synchronizing all environments.

• Deploy appropriate branches: feature tests in dev, releases in production.

Orchestrating Deployments with MCollective/Choria

Among other useful tools, Vox Pupuli’s puppet/r10k module includes an MCollective plug-in that can invoke r10k on demand. This approach allows automation tools to run r10k on many remote nodes at the same time.

Here are just a few of the benefits of using MCollective to trigger r10k deployment actions:

• New nodes automatically subscribe to the channel (versus static WebHook list).

• Requests can be filtered by many different criteria.

• Execution can be ordered, timed, and handled serially or in parallel.

• Access is validated against signed Transport Layer Security (TLS) keys and logged for auditing purposes.

r10k deployment via MCollective requests is far more robust and secure than Secure Shell (SSH) or source code repository WebHook notifications.

Invoking r10k in Testing Frameworks

You can configure whatever testing framework you use to invoke r10k as part of its features and operations. Due to how r10k derives its actions from the existence and content of source-code branches, few if any options are necessary to be passed when invoking r10k. This makes it easy to use in multipurpose test scripts.

The following are just a few of the testing framework plugins available to utilize r10k:

• The Puppet Development Kit

• Rake tasks related to r10k and Puppetfile

• A puppet provisioner for Test Kitchen

• Autogenerate Puppetfile for r10k and .fixtures for rspec testing

• Use AWS CodeCommit as a Puppet r10k Remote Control Repository

• (Gitlab) Workflow automations for deploying Puppet modules with r10k

• Jenkins Plugin: Puppet Enterprise Pipeline

Combining Multiple Invocation Methods

None of these processes are mutually exclusive. It’s not uncommon for a wide variety of implementations to be used within a single organization, like so:

• Local r10k deployments on developer workstations

• Automated r10k invocation by the testing framework on feature branch update

• WebHook notification to dev Puppet servers for r10k invocation on merge

• MCollective-based deployment to all production sites on release branch cut

The possibilities are endless. The most important aspect of your invocation process is that it works well with your development strategies. r10k should facilitate development rather than impede it. Because r10k works quickly and immediately, any blockage will come from delay of code changes reaching it. If your team is routinely blocked waiting for changes to arrive, you should improve or streamline your code release pipeline.

Repository-per-Module Benefits

Repository-per-module means that each module resides in its own source-code repository. The opposite approach is to create a monolithic repository, containing all of your modules, manifests, and Hiera data. There are major benefits to the modular approach:

SoC

The repository-per-module design strategy encourages modular development. Rather than pulling your entire site for development, you develop in the context of a single module. This approach encourages modular design and discourages violation of important design principles.

History of changes in each Puppet module

A clean history for each module makes it easy to develop, deploy, and revert each module independently. This means that change logs will pertain to the module at hand rather than being spread across every module in your code. This also means that there will be less simultaneous development in each module’s repository, reducing the likelihood of a merge conflict.

Tight specification of dependencies

A specified version for each module makes it possible to easily test and revert small changes. If all modules’ changes happen in a large group, rolling forward or back one change requires rolling back all changes.

Simplifies use and development of public modules

A huge benefit of maintaining separate histories comes into play when you need to use and contribute back to a public module. With a repository-per-module structure, your changes can simply be a fork of the public module. If you maintain your changes as an independent branch, you can often rebase those changes onto upstream improvements. This approach also makes it very simple to see how your fork differs from the upstream repository, which can be invaluable in troubleshooting and upgrades.

Allows common core upgrades; avoids forced collective upgrades

When every role and profile needs to share a single common core, an upgrade to the common core forces every profile to upgrade immediately. In practice, this means that improvements in the core require agreement and investment from every team that uses them. With unique versioning of shared components some teams can upgrade to the latest version, whereas other teams can stay with their last tested version until ready to update.

Configuring an Environment in the Control Repository

If a monolithic repository contains environment configuration or other files in the environment directory, you should add these to the branch of the control repository named for the environment, as demonstrated here:

$ cd environments/
$ git checkout stage
$ cp /monolithic/environments/stage/environment.conf ./
$ cp /monolithic/environments/stage/hiera.yaml ./
$ git add environment.conf hiera.yaml
$ git commit
...
$ git checkout production
$ cp /monolithic/environments/production/environment.conf ./
$ cp /monolithic/environments/production/hiera.yaml ./
...

You should try to avoid copying other files to the control repository, for the reasons outlined in this section.

Enabling Monolithic and Per-module Hybrid Deployment

It’s entirely possible to deploy r10k with an existing monolithic repository. Once it is deployed, r10k simplifies the process of moving your modules into their own repositories. This section outlines the steps to enable a hybrid deployment that will make it easier to iteratively move other modules as needed.

# old repo with many modules
mod 'dist'
  :git => 'git://github.example.com:puppet/monolithic.git'
  :install_path => ''

modulepath = modules:dist/modules:$basemodulepath

Moving Modules to their Own Repositories

After you use r10k for deployment, you can begin the process of moving your modules to their own repositories. You do not need to move all modules at once; you can perform this process iteratively.

The easiest moves to migrate are those that have no dependencies and are not a dependency of any other module.

mod 'puppetlabs/stdlib', '4.12.0' # last version before deprecations

fixtures:
  repositories:
    depends1: git://git.example.com/puppet/depends1.git
    depends2: git://git.example.com/puppet/depends2.git

fixtures:
  repositories:
    monolithic: git://git.example.com/puppet/monolithic.git
  symlinks:
    dependent: "#{source_dir}"
    depends1: "#{source_dir}/spec/fixtures/modules/monolithic/modules/depends1"
    depends2: "#{source_dir}/spec/fixtures/modules/monolithic/modules/depends2"

Placing Roles and Profiles in the site/ Module Directory

The site/ module directory is often used for site-specific modules, most commonly your roles and profiles. It differs from dist/ in that it contains modules that are very specific to your organization and will never be shared with the outside world. Site modules are typically tightly coupled to the module set deployed at your site and to your site-specific data.

The most common layout for your roles and profiles within the site directory is to create a module called roles and another called profiles. The actual roles and profiles are child classes within these modules. Using this design pattern avoids the risk of a collision between your roles, profiles, and other modules. This is important because your roles and profiles are often named after the services they manage. Here’s an example of creating the site modulepath structure:

$ cd /etc/puppetlabs/code/environment/environment
$ mkdir site
$ cd site/
$ pdk new module profiles --skip-interview 
$ cd profiles
$ pdk new class apache 
repeat for each profile
$ cd ../
$ pdk new module roles --skip-interview 
$ cd roles
$ pdk new class wordpress 
repeat for each profile

You would then need to prepend the site/ path to the environment’s modulepath, like so:

modulepath = site:modules:dist/modules:$basemodulepath

If you use this design pattern, your webserver role would be named roles::webserver, and your mysql profile would be named profiles::mysql. This avoids a namespace collision with the mysql service module.

You might instead prefer to create individual modules for each role or each profile. In that situation, prefix the role and/or profile with role_ or profile_, respectively. This will avoid name-space collisions with other modules. For this design pattern, your webserver role would be named role_webserver, and your mysql profile would be named profile_mysql. Using a prefix groups your roles and profiles together when viewing a list of site modules.

This isn’t an either/or decision: you can mix and match these two approaches as necessary. Use whichever method or combination thereof is clear and understandable for your team.

Remove Fully Qualified Paths

The key to the r10k adaption process is that Puppet uses relative paths and module search paths to locate module components. For example, when you include a class, the Puppet autoloader searches $modulepath for the appropriate manifest. You do not need to specify the location of the class file. So long as the class is in your modulepath and its module has the appropriate structure, it will be located and loaded. Files and templates use the same basic process; templates are referenced by module name and sourced from the module’s directory. Because all of these objects are located using configurable reference points, you can reorganize the underlying file structure at will.

It’s a good idea to scan your codebase for the following:

• import statements that might use absolute paths

• prerun or postrun scripts in the Puppet configuration file

• generate statements

Even though this cleanup requires some effort, it will pay down technical debt and make future upgrades much easier.

Moving Shared Tools to Their Own Repository

With monolithic repositories, it’s fairly common for modules to centralize and share tools in the large repository. These tools might be used by many modules and might be included or invoked from qualified paths that need to be addressed.

We recommend packaging any shared tools as a versionable component that can be installed with the module:

• Modules should be given their own repository and version.

• Ruby tools should be packaged as gems.

• Scripts can be given their own repository and version.

• Compiled tools should be added to the native package system (RPM, dpkg, etc.).

Implementing Test Cases

With the hybrid deployment described in the previous section, the process of moving your modules is fairly low risk. As you move modules to their own repositories, take a moment to refactor the modules to modern practices, including doing the following:

• pdk convert the module for use with the PDK

• Ensure that the module uses modern standards by validating the module code.

• Add units tests if they don’t exist already, and test the module.

Unit tests are small and easy to write. Acceptance tests can be more difficult but can catch significantly more real-life problems. Take the time to create or improve the acceptance tests.

Using Repository Access Control to Enforce Deployment Policy

As a general rule, anyone with the ability to modify your configuration management repositories also has the ability to modify and control the hosts managed by those configurations.

It is much more difficult, if not impossible, to limit access to code and data that has been widely shared. It is thus very important to set up appropriate controls early in the process.

A code collaboration system will provide tools that simplify peer-review, commentary, bug tracking, change control, and other workflows. Each collaboration system has its own access control and release management models. Puppet is no different from any other code tree in this sense, and you should give Puppet code the same careful code review process that you would apply to any other development project. The choice of source code management and release process has the same needs. Workflows for Puppet and r10k require only a source code system that enables easy creation and merge back of short-term feature branches.

Don’t get too wrapped up in security controls. The security of code branches should ensure that release management processes are followed. It should not inhibit developers from deploying feature branches to development and test hosts.

Enabling Multiteam Coordination

No matter how small the team using Puppet is today, you should prepare and plan for coordination between multiple teams. In larger organizations, it’s not unusual for Puppet code to be written and maintained by multiple teams: infrastructure, security, and application teams will all have a part to play, and we recommend that you plan for this from the start.

Such an approach is valuable because it allows your subject matter experts to automate their own applications. This is especially valuable when teams are horizontally layered: one team is responsible for node provisioning, another for security policy, and another for application deployment. Allow these teams to maintain their own code and data while sharing access to modules developed for common use throughout the organization.

It’s not uncommon for a tools, release, or automation team to take on the role of a service provider to delivery-oriented teams. These organizations can still develop, review, and approve modules, but a lot of the individual delivery goals might be handled by teams directly involved in delivery.

r10k facilitates multiteam development by allowing each team to control their own modules and even their own environments. If necessary, each team can utilize distinct control repositories to provide a secure method to control what is ultimately deployed.

# Puppet Forge Modules
mod 'apache',
  :git => 'https://github.com/puppetlabs/puppetlabs-apache'

# Security team
mod 'security'
  :git => 'git://github.example.com:security/profiles.git'
  :install_path => 'security'

# Platform team
mod 'role_webserver'
  :git => 'git://github.example.com:platform/webserver_role.git'
  :install_path => 'site'

mod 'role_database'
  :git => 'git://github.example.com:platform/webserver_role.git'
  :install_path => 'site'

Pinning Module Versions

Modules listed in the Puppetfile can optionally specify a tag, branch, or ref that should be checked out. If not supplied, r10k will deploy the latest version of a module, or the HEAD revision of a Git repository.

Using r10k with pinned versions guarantees consistent tests and deployment. When you deploy the same versions of every component, they will operate in the same way. You can pin versions of modules to packaged versions, repository tags, and Git hashes. When you pin a module to a commit hash or release tag, you can be confident that you are getting exactly the code you expect.

This guarantee of consistency can be the cornerstone of your change review and release process. By pinning dependencies to a specific version, the team who originally developed that module is free to continue development with minimal risk to your production infrastructure. Conversely, you can easily test new versions simply by changing the version number.

For testing purposes, it is possible to select a branch name or :latest as the version to deploy. This is useful for automatically testing the latest updates to public modules, but it introduces considerable risk in deployed environments. A release-ready branch should always specify explicit versions or Git hashes.

Always pin each repository-sourced module to a specific commit hash. Unlike refs and tags, or even short commit hashes, a full commit hash is cryptographically secure; you get exactly the code you expect, and there’s no way for the repository owner to accidentally break you. Pinning dependency modules this way prevents both malicious and nonmalicious changes from reaching your production environment. The following code shows you how to pin a module:

mod 'apache',
  :git    => 'https://github.com/puppetlabs/puppetlabs-apache',
  :commit => 'f7946501ddb68e0057b5dc3272657bea891639e5' # Pin to specific bugfix

It’s a good practice to document inline the criteria used for the selection of tag or version used. This will make life much simpler for anyone attempting to update the Puppetfile, debug a problem, or investigate version-specific features. This also simplifies the process of identifying out-of-date modules.

Being able to pin to a commit hash is a Git-specific feature; other revision control systems might not provide equivalent functionality. In those cases, you should pin to the specific branch or commit that is likely to remain unchanged. Pinning this way allows your team to continue improving the head revision of the module without the risk of breaking production nodes. Alternatively, mirror from the other repository into Git and thus make Git hashes available.

Isolating Puppet Extensions

Native Ruby extensions to Puppet are handled using Ruby’s built-in autoloader. New resource types and functions can be encapsulated in modules because Puppet adds the module’s lib directory to Ruby’s load path. Without enabling environment isolation, after a native extension is loaded from one environment, Ruby will not attempt to load the same extension from another environment.

This hasn’t generally been a problem for short-lived, ad hoc Puppet applications, but it can prevent testing of improvements in long-lived Puppet server environments. This is not an issue for agent-side extensions such as facts and resource providers, because the client process terminates at the end of each application—even when the agent is daemonized.

For years it was necessary to work around this by using a long list of environment separation practices, including physical separation of development Puppet servers. Depending on how far you were willing to go, crazy monkey-patches that attempted to adjust the Ruby load path of active instances were available. If you have any of that hackery in your Puppet deployment today, get rid of it.

After an environment is deployed, run the following command in the environment to enable environment isolation:

$ puppet generate types --environment production

That’s it. No separation of development components. No crazy hackery. Just unique metadata within each environment identifying a unique load path for each extension.

Utilizing Standard Environment Configuration Practices

The following are all practices that have become so commonly used that the tools all assume you’ll be using the same approach. Take advantage of these conventions unless you have a really good reason for not doing so.

Git Best Practices

There are no Puppet-specific best practices with regards to Git or any other version control system. That doesn’t mean you don’t need to apply them, but you should employ the established best practices in your Puppet release management strategy.

Deployment Practices

Each control repository branch contains the contents of an individual environment. Do not attempt to overload this practice to supply configuration files to copy down to the node.

The Puppet server, Puppet agent, and r10k configurations should be managed by Puppet using modules and data, the same as any other application. The advantage of this approach is that it will allow you to apply your change control procedure to Puppet configuration updates.

If you are a Puppet Enterprise user, a number of bundled modules are included to manage the Puppet infrastructure. They are installed in /opt/puppetlabs/puppet/modules and should not be moved into the control repository. These modules are managed as part of the Puppet agent or server all-in-one installation.

flock -w 360 /var/lock/r10k -c '/opt/puppet/bin/r10k deploy environment -v warn'