We've introduced you to installing and configuring Puppet. In this chapter, we show how you might integrate Puppet into your organization's workflow. This will allow you to use Puppet to make changes and manage your infrastructure in a logical and stable way.
To do this, we introduce a Puppet concept called "environments." Environments allow you to define, maintain and separate your infrastructure into appropriate divisions. In most organizations, you already have some of these divisions: development, testing, staging, pre-production and others. Just like a set of production, testing, and development systems, which are separated from one another to effectively isolate risky changes from production services, Puppet environments are designed to isolate changes to the configuration from impacting critical production infrastructure.
In this chapter we also build upon the concept of modules, which we introduced in Chapters 1 and 2. We show you how to configure environments on your Puppet masters and how to control which agents connect to which environment. Each agent can connect to a specific environment that will contain a specific set of configuration.
Finally, we exercise the workflow of making changes using our version control system, testing those changes in a safe and easy way using environments, then promoting the tested changes to the production environment in Puppet.
In order to demonstrate all of this to you, we create another host for the Example.com Pty Ltd organization we first introduced in Chapter 1. This new host is called mailtest.example.com. This host has been introduced to allow Example.com to test changes to their email server without impacting the production mail server. You can see the new node in Figure 3-1.
To get started, we've installed the Red Hat Enterprise Linux operating system on mailtest.example.com in order to match the operating system Puppet already manages on mail.example.com. As we already have configuration to manage the mail.example.com host, we don't need to create any new manifests - we can re-use the existing ones to configure our new mailtest.example.com host.
Tip
This chapter starts to demonstrate the power of Puppet for re-using configuration: Rather than starting a new configuration from scratch, we can use existing Puppet manifests to create a new mail server.
To configure Puppet environments, you need to add them to the Puppet master's configuration. If you add each environment to the Puppet master, then each Puppet agent can request a specific environment when requesting a catalog from the master.
The first step to configure your Puppet master and agents to use environments is to add a stanza in the /etc/puppet.conf configuration file on the Puppet master for each environment you want to support. Let's do this now, by creating the three environments shown in Listing 3-1.
Example 3-1. Puppet Master environments in puppet.conf
[main] modulepath = $confdir/modules manifest = $confdir/manifests/site.pp [development] modulepath = $confdir/environments/development/modules manifest = $confdir/environments/development/manifests/site.pp [testing] modulepath = $confdir/environments/testing/modules manifest = $confdir/environments/testing/manifests/site.pp
As you can see, each environment section of the puppet configuration file defines two settings, modulepath and manifest. The modulepath setting defines the path to the modules that will apply to each environment, and the manifest option specifies the site.pp file that applies to that environment. Recall from Chapter 1 that site.pp is the file that tells Puppet which configuration to load for our clients. These settings allow each environment to have a distinct set of modules and configuration.
Tip
When setting up environments, the Puppet master process should be restarted in order to activate configuration changes. As described in Chapter 1, the restart process depends on how Puppet is installed on the master. Most systems include an init script to accomplish this task.
In Chapters 1 and 2, we introduced you to building modules to store your Puppet configuration. In order to fully utilize environments, your Puppet manifests should be organized into modules. In this chapter, we use the modules we've created to manage our production environment, the main environment defined in Listing 3-1.
Once you've defined the multiple environments on the Puppet master server, you need to populate these new search paths with the Puppet modules and manifests you've already created in production. In the "Version Controlling Your Modules" section of Chapter 2, our hypothetical company configured Puppet modules using the Git version control system. We'll expand on the file organization and introduce a strategy to manage and migrate changes between Puppet environments.
Note
If you have not yet installed Git and would like to do so now, please refer back to the Git installation information in Chapter 2.
We will use Git to make sure each of our three new environments; main (or production), development and testing will receive an identical copy of our production environment. The version control system will also allow us to easily keep these three environments synchronized when necessary, while also allowing them to diverge when we want to try out new changes. Three environments with identical modules and manifests will allow us to quickly make changes in the development or testing environment without impacting the production environment. If we're satisfied, we can easily merge the changes into production.
Note
Many organizations with multiple people committing changes to the Puppet configuration will benefit from a code review process. Information about the code review process used by the Puppet development community is available at: http://projects.puppetlabs.com/projects/puppet/wiki/Development_Development_Lifecycle.
In Chapter 2, we initialized the /etc/puppet/modules directory as a Git repository. Once a Git repository exists, it may be cloned one or more times. Once there are multiple clones, changes to any of the repositories may be fetched and merged into any other repository.
Let's create a clone of the /etc/puppet/modules Git repository for the development and testing environments now.
First, you need to create the directory structure necessary to contain the new module search path:
$ cd /etc/puppet
$ mkdir -p environments/{development,testing}Next, clone the original module repository you created in Chapter 2 into your development environment:
$ cd /etc/puppet/environments/development $ git clone ../../modules Initialized empty Git repository in /etc/puppet/environments/development/modules/.git/
This command makes a new copy of the Git repository, called a "clone," and automatically sets up a reference to the repository we cloned from. This reference, named "origin," refers to the original repository this repository was cloned from. The origin is actually the repository in the production Puppet environment, so you can add another name to be clear when you fetch updates:
$ cd /etc/puppet/environments/development/modules $ git remote add production /etc/puppet/modules $ git remote -v production /etc/puppet/modules (fetch) production /etc/puppet/modules (push)
As you can see, we've added a remote reference to the production environment module repository in the development environment's module repository. This remote reference allows Git to fetch changes.
Similar to the development environment you just set up, you'll also clone the production environment modules into a testing environment.
$ cd /etc/puppet/environments/testing $ git clone ../../modules Initialized empty Git repository in /etc/puppet/environments/testing/modules/.git/ $ cd modules $ git remote add production /etc/puppet/modules $ git remote add development /etc/puppet/environments/development/modules
Notice how we've also added the development repository as a remote in the testing environment repository. This will allow you to fetch changes you make in the development repository to the testing repository.
Now that you have your three environments populated with the same Puppet modules, you can make changes without affecting the production environment. We're going to use a basic workflow of editing and committing changes in the development branch first. This mirrors the common development life cycle of moving from development to testing and finally to production. We'll start with running a Puppet agent in the development environment to test the change we've made. Then, if everything goes well in the development environment, you can merge this change into testing or into production.
Tip
In large Puppet setups where changes from multiple groups of people need to be managed, it is common to run a selection of hosts against the testing environment. Periodically, the production environment repository will be synchronized against the testing environment.
We're going to edit the Postfix configuration file template we created in Chapter 2 to explore how Puppet isolates the three environments we've created. We'll edit the file main.cf.erb in the development environment and then run the Puppet agent in this environment to see the change. We'll also run the Puppet agent in the production environment, which we have not changed yet, and make sure our changes do not have any effect on production.
To start, edit the file main.cf.erb in /etc/puppet/environments/development/modules/postfix
/templates/ using your favorite text editor and add a new line at the very top of the file to look like:
# This file managed by puppet: <%= this_will_fail %> soft_bounce = no command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix mail_owner = postfix myhostname = <%= hostname %> mydomain = <%= domain %> myorigin = $mydomain mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain unknown_local_recipient_reject_code = 550 relay_domains = $mydestination smtpd_reject_unlisted_recipient = yes unverified_recipient_reject_code = 550 smtpd_banner = $myhostname ESMTP setgid_group = postdrop
Now that you've made a change to the development environment, Git will let you know that the status of the repository has changed:
$ git status # On branch master # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: main.cf.erb # no changes added to commit (use "git add" and/or "git commit -a")
Git has noticed that you've made a change to the main.cf.erb file and tells you this on the "modified" line. As we learned in Chapter 2, we must add files changed in the working directory to the index, and then commit the index to the repository. Before you do this, you should double-check to make sure the line you modified is what will actually be added in the new commit.
$ git diff diff --git a/postfix/templates/main.cf.erb b/postfix/templates/main.cf.erb index 3331237..2be61e0 100644 --- a/postfix/templates/main.cf.erb +++ b/postfix/templates/main.cf.erb @@ −1,3 +1,4 @@ +# This file managed by puppet: <%= this_will_fail %> soft_bounce = no command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix
Notice the line beginning with the single plus sign. This indicates that you've added one line and this addition will be recorded when we commit the change, as we will with the git commit command:
$ git commit -a -m 'Updated postfix configuration template' [master 0fb0463] Updated postfix configuration template 1 files changed, 1 insertions(+), 1 deletions(-)
You've now successfully changed the development environment. But before testing the change on our mailtest.example.com system, let's review the environment configuration changes you've made to the Puppet Master.
puppet.confon the master now contains adevelopmentandtestingsection.The Puppet master process has been restarted to activate the change to
puppet.conf.You updated
modulepathandmanifestin the development and testing section.You cloned the modules VCS repository to /etc/puppet/environments/{testing,development}/modules.
You updated the
postfixmodule and committed the change to the development repository.
Now that you have multiple environments configured on the Puppet master system and have made a change to the development environment, you're able to test this change using the Puppet agent.
In order to tell Puppet to use an environment other than production, use the environment configuration parameter or command line option:
$ puppet agent --noop --test --environment testing
Tip
Up through Puppet 2.6, the Puppet configuration on a node configures the environment that the node uses. The Puppet master does not directly control which environment a machine connects to. This may change in the future once issue #2834 is resolved; please watch http://projects.puppetlabs.com/issues/2834 for up-to-date information. If you would like to manage the environment from the Puppet master, we recommend having Puppet manage the node's puppet.conf file and specify the environment parameter in the managed configuration file.
Running the Puppet agent on mailtest.example.com in the testing environment should produce the same results as running the agent in the production environment.
Tip
We recommend developing a habit of testing changes to Puppet using the --noop command line option. As mentioned in Chapter 1, the --noop option tells Puppet to check the current state of the system against the configuration catalog, but does it not manage the resources on the node. This provides a safe way determine if Puppet is going to make a change. It's also a unique feature of Puppet, compared to other tools.
You can switch between the production and testing environments by simply removing the environment command line option. The default environment is production (defined in the main stanza in the puppet.conf file); therefore, you need only leave the environment unspecified to switch back to the production environment.
$ puppet agent --noop --verbose -test
Notice how no resources are changing when switching between the two environments. This is because the testing environment is a clone of the production environment, and you have not made any changes to either of these two environments. In the last section, however, you made a change to Postfix module in the development environment, and we expect the Puppet agent to update the main.cf postfix configuration file with this change. Let's check the development environment now:
$ puppet agent --noop --verbose --test --environment development err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parsetemplate postfix/main.cf.erb: Could not find value for 'this_will_fail' at/etc/puppet/environments/development/modules/postfix/manifests/config.pp:17 onnode mailtest.example.com warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run
Unlike the testing and production environment we ran the Puppet agent in, this run in the development environment resulted in an error. Such a bad error, in fact, that we didn't even receive a valid configuration catalog from the Puppet master. So what happened?
Notice that the error message returned by the Puppet master provides the exact line number in the manifest the error occurred on. On this line we're using the template we modified when we made a change to the development environment, and this change references a variable that we have not defined in the Puppet manifests. If we run the Puppet agent against the production environment, we can see everything is still OK:
$ puppet agent --test --noop ... notice: Finished catalog run in 0.68 seconds
Let's go back and fix the problem with the ERB template by removing the reference to the undefined puppet variable this_will_fail. As you can see in the following file difference, we've fixed the problem in the first line of the template:
diff --git a/postfix/templates/main.cf.erb b/postfix/templates/main.cf.erb index 3331237..241b4bb 100644 --- a/postfix/templates/main.cf.erb +++ b/postfix/templates/main.cf.erb @@ −1,3 +1,4 @@ +# This file managed by puppet. soft_bounce = no command_directory = /usr/sbin daemon_directory = /usr/libexec/postfix
Now, when we run Puppet agent in the development environment, we're no longer getting the error:
$ puppet agent --test --noop --environment development
This verification step allowed us to make changes and test them in an isolated environment without impacting Puppet nodes with their agent running against the production environment. Now that you're confident our change will not break production, you can commit the changes:
$ git add /etc/puppet/environments/development/modules/postfix/templates/main.cf.erb $ git commit -m 'Added comment header, postfix main.cf is managed by puppet.' Created commit d69bc30: Added comment header, postfix main.cf is managed by puppet. 1 files changed, 2 insertions(+), 1 deletions(-)
In the next section, we examine the workflow of merging changes like this into the testing and production environments. This workflow helps teams of developers and system administrators work together while making changes to the system, without impacting production systems, through the use of Puppet environments.
As you saw in the previous section, configuring multiple environments in Puppet requires three things:
One of the key benefits of version control systems is the ability to manage and organize the contributions from a group of people. In this section, we'll explore how a group of three people may use Puppet Environments, version control, and the concept of a "branch" to effectively coordinate and manage their changes to the configuration system. Branches are lines of independent development in a repository that share a common history. A branch could be a copy of our development environment with changes made to it; it shares a common history with the development environment but has a history of its own too. Branches allow multiple people to maintain copies of an environment, work on them independently and potentially combine changes between branches or back into the main line of development.
Expanding on our hypothetical company, imagine we have a small team of people working together: a system administrator, a developer and an operator. In this exercise, we'll explore how this team effectively makes changes that do not impact one another, can be merged into the main development and testing branch, and ultimately make their way to the production infrastructure.
Before the small group is able to work together in harmony, you'll need to make a few slight changes to the version control system. Git is unique compared to other version control systems, such as Subversion, in that each repository stands apart and is complete without the need to perform a checkout from a central repository. When working with a team, however, it is convenient to have a central place to store and track changes over time.
In this section, you'll clone a copy of the /etc/puppet/modules repository into /var/lib/puppet/git/modules.git and use this location as the "central" repository. It is central by convention only; there is technically nothing different about the repository that makes it any different from the other Git repositories we've been working with in this chapter. Once you have a repository designated as the central location, everyone will clone this repository and submit their changes back to it for review and testing. Let's go through this process now.
First, you need to create a "bare" repository containing your Puppet modules. A bare repository in Git is a repository with the history of commits, but no working copy. We want to create a bare repository to help make sure files aren't accidentally directly modified in the central location. Modifications should only happen through commits pushed to this location. We're going to perform these steps as the Puppet user, who is usually running as puppet, in order to help ensure file permissions and ownership remain consistent when different users are modifying the repository.
$ cd /var/lib/puppet $ mkdir git $ chown puppet:puppet git $ sudo -H -u puppet -s $ cd /var/lib/puppet/git $ git clone --bare /etc/puppet/modules modules.git Initialized empty Git repository in /var/lib/puppet/git/modules.git/
Once you have a central repository, it's time for everyone in the group to check out their own personal copies to work on. We recommend they do this in their home directories. Changes will be made there and submitted to the central repository for review. Let's first clone a repository for our system administrator, hereafter sysadmin:
sysadmin:~$ git clone [email protected]:git/modules.git Initialized empty Git repository in ~/modules/.git/ remote: Counting objects: 36, done. remote: Compressing objects: 100% (33/33), done. remote: Total 36 (delta 0), reused 0 (delta 0) Receiving objects: 100% (36/36), 5.58 KiB, done.
After cloning the repository from the central location, you can begin to make changes. In order to make sure you have the same changes you made to the main.cf.erb file in the previous section, pull the change made to the main.cf.erb file from the repository in /etc/puppet/environments/development/modules. You could directly fetch the change from the repository Puppet is using in /etc/puppet, but it may become confusing to manage what changes are located in which repositories.
To help coordinate with the rest of the team, instead push the change from the development repository into the central repository. This should be done using the puppet user account:
puppet:~$ cd /etc/puppet/environments/development/modules puppet:development/modules$ git remote rm origin puppet:development/modules$ git remote add origin [email protected]:git/modules.git puppet:development/modules$ git push origin master:master Counting objects: 9, done. Compressing objects: 100% (4/4), done. Writing objects: 100% (5/5), 499 bytes, done. Total 5 (delta 2), reused 0 (delta 0) To [email protected]:git/modules.git a13c3d8..d69bc30 master -> master puppet:~$ cd /etc/puppet/environments/testing/modules puppet:testing/modules$ git remote rm origin puppet:testing/modules$ git remote add origin [email protected]:git/modules.git puppet:~$ cd /etc/puppet/modules puppet:/etc/puppet/modules$ git remote rm origin puppet:/etc/puppet/modules$ git remote add origin [email protected]:git/modules.git
After executing these commands, you've updated each of the three Git repositories containing the production, testing, and development working copies to point at your fourth, central repository. The systems administrator now has a personal working copy which points to the central repository.
In order to make a change, each team member should create a new Git branch for the topic he or she is working on and make their changes in this branch. A topic branch will allow other team members to easily fetch all of their work as a self-contained bundle, rather than requiring them to sort through each commit or set of commits. This will also make it easier to merge each team member's contributions into the master branch when necessary, as you can see in Listing 3-2.
Example 3-2. Merging in changes
sysadmin:~$ cd modules sysadmin:~/modules$ git fetch origin From [email protected]:git/modules a13c3d8..d69bc30 master -> origin/master sysadmin:~/modules$ git checkout master Already on "master" Your branch is behind the tracked remote branch 'origin/master' by 1 commit, and can be fast-forwarded. sysadmin:~/modules$ git merge origin/master Updating a13c3d8..d69bc30 Fast forward postfix/templates/main.cf.erb | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
As you can see, we've pushed the change to main.cf.erb into the central repository. The sysadmin was able to update her personal copy with this change.
The sysadmin now has her copy and is able to push and pull changes in the central repository, but what about the developer and operator? They should each clone a copy of the central repository URL, [email protected]:git/modules.git, into their home directory. We'll run through the situation where the operator needs to make and test a change to the sshd configuration file, while the developer needs to make and test a change to the Postfix configuration files. These two changes will be tested independently in the development environment and then merged together in the testing environment.
Tip
SSH Keys and Agent Forwarding should be employed when using Git in order to increase security, keep file ownership consistent, and manage the central code using the Puppet user. To accomplish this, people with authorization to change Puppet could have their public key added to ~puppet/.ssh/authorized_keys. For more information about SSH public keys, please see: http://www.debian-administration.org/articles/530
We'll go through the changes to Secure Shell or SSH the operator needs to make first. The operator is working specifically to make sure only members of certain groups are allowed to log in to the system using SSH.
To begin, you should create a topic branch to work on this problem. In Git, unlike other version control systems, a branch does not create a new directory path in the working directory of the repository. Instead, Git checks out the branch into the base directory of the repository.
Let's create a topic branch based on the current master branch in our central "origin" repository, like so:
operator:~/modules $ git checkout -b operator/ssh origin/master Branch operator/ssh set up to track remote branch refs/remotes/origin/master. Switched to a new branch "operator/ssh" operator:~/modules $ git branch * operator/ssh Master
Notice that the operator now has two branches in their personal ~/modules/ Git repository. Using a topic branch, we are free to modify things without worrying about impacting the work of the rest of the team. The branch provides a reference point to revert any of the changes we make to the Puppet configuration. Similarly, the development, production, and testing environments in the /etc/puppet directory on the Puppet master must explicitly check out this new branch in order for our changes to affect any of the Puppet agent systems. This strategy is much less risky and easier to coordinate with team members than directly editing the files contained in the /etc/puppet directory.
Now that the operator has his or her own branch, we're ready to make a change. We're going to add two lines using two commits to illustrate the history tracking features of a version control system.
First, add the groups who should have access to the machine. To start, only the wheel group should be allowed to log in, so add the following lines to the sshd_config template:
operator:~/modules $ git diff diff --git a/ssh/files/sshd_config b/ssh/files/sshd_config index 7d7f4b4..1fd84e5 100644 --- a/ssh/files/sshd_config +++ b/ssh/files/sshd_config @@ −3,4 +3,5 @@ Protocol 2 SyslogFacility AUTHPRIV PermitRootLogin no PasswordAuthentication yes +AllowGroups wheel adm UsePAM yes
As you can see, we've added a single line to the file ~/modules/ssh/files/sshd_config in the personal clone of the repository in the operator's home directory. We must commit and push this change into the central repository, but we haven't tested it yet so we should be careful and not merge the branch we're working on, operator/ssh, into the master branch yet.
operator:~/modules $ git commit -a -m 'Added AllowGroups to sshd_config' Created commit eea4fbb: Added AllowGroups to sshd_config 1 files changed, 1 insertions(+), 0 deletions(-) operator:~/modules $ git push origin operator/ssh:operator/ssh Counting objects: 9, done. Compressing objects: 100% (4/4), done. Writing objects: 100% (5/5), 454 bytes, done. Total 5 (delta 2), reused 0 (delta 0)
To [email protected]:git/modules.git * [new branch] operator/ssh -> operator/ssh
The git push command the operator used creates a new branch in the central repository with the same name as the topic branch the operator is working on in his or her home directory. This is important to prevent untested changes from making their way into the master branch. Once we've pushed the new branch to the central repository, we should test the new branch in the development environment.
Note
There is no limit to the number of environments you can configure on the central Puppet master. Many large teams find it beneficial to create per-contributor environments in addition to the standard development, testing and production environments. Per-contributor environments allow each person to test their own branches without interfering with the development environments of other individuals.
puppet:~ $ cd /etc/puppet/environments/development/modules puppet:modules $ git fetch origin remote: Counting objects: 14, done. remote: Compressing objects: 100% (8/8), done. remote: Total 10 (delta 4), reused 0 (delta 0remote: ) Unpacking objects: 100% (10/10), done. From [email protected]:git/modules d69bc30..fa9812f master -> origin/master * [new branch] operator/ssh -> origin/operator/ssh puppet:modules $ git checkout -b operator/ssh origin/operator/ssh Branch operator/ssh set up to track remote branch refs/remotes/origin/operator/ssh. Switched to a new branch "operator/ssh"
Now that we've switched to our new topic branch in the development environment, we're able to test the Puppet agent against the development environment.
puppet:~ $ puppet agent --test --environment development --noop info: Caching catalog for scd.puppetlabs.vm info: Applying configuration version '1289751259' --- /etc/ssh/sshd_config 2010-11-14 08:16:45.000000000 -0800 +++ /tmp/puppet-file.13997.0 2010-11-14 08:16:57.000000000 -0800 @@ -3,4 +3,5 @@ SyslogFacility AUTHPRIV PermitRootLogin no PasswordAuthentication yes +AllowGroups wheel adm UsePAM yes notice: /Stage[main]/Ssh::Config/File[/etc/ssh/sshd_config]/content: is{md5}9d4c3fba3434a46528b41a49b70b60e4, should be {md5}da54f2cdc309faf6d813a080783a31f6 (noop)
info: /Stage[main]/Ssh::Config/File[/etc/ssh/sshd_config]: Scheduling refresh of Service[sshd] notice: /Stage[main]/Ssh::Service/Service[sshd]: Would have triggered 'refresh' from 1 events notice: Finished catalog run in 0.39 seconds
Notice that this Puppet agent run is running in noop mode, and that the agent tells us it would have changed /etc/ssh/sshd_config by inserting the line we just committed to the branch operator/ssh and checked out in the development environment's repository on the Puppet master.
You're able to verify that the production environment remains unchanged, just like we did in the "Making Changes to the Development Environment" section when we updated the Postfix configuration file. Simply remove the environment command line option to cause the agent to execute in the default production environment again:
puppet:~ $ puppet agent --test --noop info: Caching catalog for scd.puppetlabs.vm info: Applying configuration version '1289752071' notice: Finished catalog run in 0.33 seconds
While the system operator is working on the change to the sshd_config file, the developer in our hypothetical company is working on a change to the Postfix configuration file. Just like the operator, he'll need a personal copy of the central repository we set up at [email protected]:git/modules.git in his home directory.
Once the developer has cloned his personal copy of the central repository, he's able to make his change to the Postfix configuration file. He'll also use a branch to track his changes and make it easy to merge into the testing branch for use in the testing environment. Finally, after testing, he'll use the tag feature of the version control system to cut a new release of the configuration used in production, then check out this tag in the repository used by the production Puppet environment.
To start, the developer creates his topic branch from the development branch named master. Note that the changes his teammate, the operator, has made have not yet been merged into the master branch, so the developer does not have them. We cover the process of merging multiple changes together when we merge both of these changes into the testing branch in the next section.
developer:~ $ cd ~/modules developer:~/modules $ git checkout -b developer/postfix master Switched to a new branch "developer/postfix"
Now that the developer has his own topic branch, he's free to change the code without impacting the work of anyone else on the team. His changes can be discarded or merged at a later point in time. Let's look at his changes to the Postfix configuration file and how he committed them into the version control system:
$ git log --abbrev-commit --pretty=oneline master..HEAD 7acf23d... Updated config.pp to use $module_name 0c164f6... Added manual change warning to postfix config
Using the git log command, you're able to see the developer has made two commits since he created his topic from the main master development branch. This specific command displays the series of commits from the master development branch to the tip of the current topic branch. You're able to use the git log command again to see exactly what the developer changed in these two commits, as shown in Listing 3-3.
Example 3-3. Listing Git changes
developer:~/modules $ git log --summary -p --stat master.. commit 7acf23dc50774aee1139e43aec5b1e8f60fa9da9 Author: Devevloper <[email protected]> Date: Sun Nov 14 09:45:16 2010 -0800 Updated config.pp to use $module_name The $module_name variable has been introduced in Puppet 2.6 and makes for easily renamed puppet modules without having to refactor much of the code. --- postfix/manifests/config.pp | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/postfix/manifests/config.pp b/postfix/manifests/config.pp index 9feb947..471822c 100644 --- a/postfix/manifests/config.pp +++ b/postfix/manifests/config.pp @@ −7,14 +7,14 @@ class postfix::config { file { "/etc/postfix/master.cf": ensure => present, - source => "puppet:///modules/postfix/master.cf", + source => "puppet:///modules/${module_name}/master.cf", require => Class["postfix::install"], notify => Class["postfix::service"], } file { "/etc/postfix/main.cf": ensure => present, - content => template("postfix/main.cf.erb"), + content => template("${module_name/main.cf.erb"), require => Class["postfix::install"], notify => Class["postfix::service"], } commit 0c164f676da64cec5e6d02ac5cb8a60229e60219 Author: Developer <[email protected]> Date: Sun Nov 14 09:42:17 2010 -0800 Added manual change warning to postfix config --- postfix/files/master.cf | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/postfix/files/master.cf b/postfix/files/master.cf index 280f3da..7482d4c 100644
--- a/postfix/files/master.cf +++ b/postfix/files/master.cf @@ −1,3 +1,5 @@ +# This file managed by puppet. Manual changes will be reverted. +# # # Postfix master process configuration file. For details on the format # of the file, see the master(5) manual page (command: "man 5 master").
Reviewing his changes, the developer notices he made a typographical mistake in the postfix configuration file and decides to fix this problem. In the second section of the diff output in the Postfix configuration file, the line containing template("${module_name/main.cf.erb") is missing a closing curly brace around the variable module_name. He decides to fix this and make a third commit to his topic branch. The output of git log now shows:
developer:~/modules $ git log --abbrev-commit --pretty=oneline master.. 6b9f2b5... Fixup missing closing curly brace 7acf23d... Updated config.pp to use $module_name 0c164f6... Added manual change warning to postfix config
Tip
In order to help prevent typographical errors from being accepted into the repository, it is a good idea to execute puppet --parseonly as a pre-commit hook in your version control system. Most version control systems support hook scripts to accept or deny a commit. If you use Subversion or Git, example pre-commit hooks are available online at http://projects.puppetlabs.com/projects/1/wiki/Puppet_Version_Control.
The developer is satisfied with his changes to Postfix, and he would like to try them out in the development environment in a similar way the operator tested out her changes. The overall workflow the developer follows is to push their topic branch to the central repository, fetch the changes in the development environment's repository, check out the topic branch, then run the Puppet agent against the development environment.
Before publishing his topic branch to a different repository, he decides to clean up his commit history to remove the entire commit that he created simply to fix a single character mistake he introduced. The git rebase command allows him to quickly and easily modify his topic branch to clean up this mistake.
developer:~modules/ $ git rebase -i master
This command will open, in your default text editor, a list of commits to the topic branch since it diverged from the master development branch. In order to clean up the commit history, the developer replaces "pick" with "squash" in the line listing his commit to add the missing curly brace. This will effectively combine this commit with the commit above it, where the curly brace should have been present in the first place.
pick 0c164f6 Added manual change warning to postfix config pick 7acf23d Updated config.pp to use $module_name squash 6b9f2b5 Fixup missing closing curly brace
# Rebase fa9812f..6b9f2b5 onto fa9812f # # Commands: # pick = use commit # edit = use commit, but stop for amending # squash = use commit, but meld into previous commit # # If you remove a line here THAT COMMIT WILL BE LOST. # However, if you remove everything, the rebase will be aborted.
Once the developer makes this change, he saves the file and quits the editor. git rewrites history for him, giving him the option to change the commit message of the freshly cleaned commit:
".git/COMMIT_EDITMSG" 17L, 524C written Created commit e4e27c7: Updated config.pp to use $module_name 1 files changed, 2 insertions(+), 2 deletions(-) Successfully rebased and updated refs/heads/developer/postfix
The developer is now ready to publish his topic branch to the rest of his colleagues and to the puppet master system itself, in order to check out the topic branch in the /etc/puppet/environments/development/modules repository.
developer:~/modules $ git push origin developer/postfix:developer/postfix Counting objects: 21, done. Compressing objects: 100% (14/14), done. Writing objects: 100% (15/15), 1.79 KiB, done. Total 15 (delta 3), reused 0 (delta 0) To [email protected]:git/modules.git * [new branch] developer/postfix -> developer/postfix
Next, he logs into the puppet master system as the user puppet, fetches his topic branch from the central repository, and then checks out his topic branch in the development environment. This process will switch the current development environment away from whatever branch it was previously on. This could potentially interfere with the work of the operator. If this becomes a common problem, it is possible to set up more environments to ensure each contributor has their own location to test their changes without interfering with others.
puppet:~ $ cd /etc/puppet/environments/development/modules ppuppet:modules $ git fetch origin remote: Counting objects: 21, done. remote: Compressing objects: 100% (14/14), done. remote: Total 15 (delta 3), reused 0 (delta 0) Unpacking objects: 100% (15/15), done. From [email protected]:git/modules * [new branch] developer/postfix -> origin/developer/postfix puppet:modules $ git checkout -b developer/postfix origin/developer/postfix Branch developer/postfix set up to track remote branch refs/remotes/origin/developer/postfix. Switched to a new branch "developer/postfix"
The developer's topic branch has now been checked out in the location the Puppet master is using for the development environment.
You can run the Puppet agent against the development environment, as we have previously and as shown in Listing 3-4, to verify your changes.
Example 3-4. Testing the Puppet
agentroot:~ # puppet agent --test --noop --environment development
info: Caching catalog for scd.puppetlabs.vm
info: Applying configuration version '1289764649'
--- /etc/ssh/sshd_config 2010-11-14 12:11:28.000000000 -0800
+++ /tmp/puppet-file.25961.0 2010-11-14 12:11:40.000000000 -0800
@@ −3,5 +3,4 @@
SyslogFacility AUTHPRIV
PermitRootLogin no
PasswordAuthentication yes
-AllowGroups wheel adm
UsePAM yes
notice: /Stage[main]/Ssh::Config/File[/etc/ssh/sshd_config]/content: is {md5}da5
4f2cdc309faf6d813a080783a31f6, should be {md5}9d4c3fba3434a46528b41a49b70b60e4 (
noop)
info: /Stage[main]/Ssh::Config/File[/etc/ssh/sshd_config]: Scheduling refresh of
Service[sshd]
notice: /Stage[main]/Ssh::Service/Service[sshd]: Would have triggered 'refresh'
from 1 events
--- /etc/postfix/master.cf 2010-11-14 11:54:37.000000000 -0800
+++ /tmp/puppet-file.22317.0 2010-11-14 11:58:15.000000000 -0800
@@ −1,3 +1,5 @@
+# This file managed by puppet. Manual changes will be reverted.
+#
#
# Postfix master process configuration file. For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
notice: /Stage[main]/Postfix::Config/File[/etc/postfix/master.cf]/content: is
{md5}3b4d069fa7e4eb6570743261990a0d97, should be {md5}710171facd4980c2802a354ee4cb4a4e (noop)
info: /Stage[main]/Postfix::Config/File[/etc/postfix/master.cf]: Scheduling refresh of
Service[postfix] notice: /Stage[main]/Postfix::Service/Service[postfix]: Would have
triggered 'refresh' from 1 events
notice: Finished catalog run in 0.61 secondsThe Puppet agent run against the development environment shows us that Puppet will update the Postfix configuration file and notify the Postfix service as a result. Notice how the changes we've made to this system by trying out the operator/ssh branch will now be reverted. This is because the developer created his branch from the master branch and the operator has not yet merged her operator/ssh branch back into master, therefore her changes are not present.
At this point, both the changes of the operator and the developer have been tried in using the development environment. It's now time to merge both change lists into a testing branch and make them both available in the testing Puppet environment.
Unlike the development Puppet environment, where anything goes and people may perform a checkout on their branches to quickly try out their changes and topic branches, the testing environment should change less frequently. The process of merging topic branches from the master development branch into a testing branch periodically, once every two weeks for example, has worked well for many projects and companies. In this section, we work through the process of merging change lists into the testing branch with the goal of ultimately promoting the testing branch to a production release.
First, our system administrator will create a new branch, called "testing," based on the current master branch we started with. When starting out with Puppet, this testing branch and the process of merging change lists should be set early on in order to provide a good reference point. It also provides and staging area that's not quite as risky as the development environment, and does not require a release process like the production environment does.
The system administrator creates the new testing branch in a manner similar to how the operator and developer created their topic branches. This should be done in the personal repository the system administrator has in her home directory:
sysadmin:~modules/ $ git checkout testing master Switched to a new branch "testing"
Tip
There is no technical difference between a topic branch and a testing branch the system administrator creates for the testing environment. The team is simply using a convention of treating the testing branch as a long-lived branch to merge change lists into. Similarly, the master branch is the branch where current development happens.
Before checking out the testing branch on the Puppet master, the system administrator decides to merge the change lists from the operator and the developer into the main development branch. This keeps the main development branch in sync with the testing branch and allows the system administrator to advance the master development branch with additional changes without affecting the testing environment, which will only be updated on the Puppet master periodically.
sysadmin:~modules/ $ git fetch origin From [email protected]:git/modules * [new branch] developer/postfix -> origin/developer/postfix * [new branch] operator/ssh -> origin/operator/ssh sysadmin:~modules/ $ git merge --no-ff origin/developer/postfix Merge made by recursive. postfix/files/master.cf | 4 +++- postfix/manifests/config.pp | 4 ++--
2 files changed, 5 insertions(+), 3 deletions(-) sysadmin:~modules/ $ git merge --no-ff origin/operator/ssh Merge made by recursive. ssh/files/sshd_config | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
Tip
It is a good idea to perform a git fetch origin to see if there are any changes in the central repository prior to merging topic branches. If there are, then performing git merge origin/master while on the master branch will bring those changes into the local repository.
The system administrator has merged the changes using the --no-ff option in order to create a merge commit for each of the two topic branches. In the future, this merge commit will allow the team to refer back to the change list as a whole rather than having to tease apart which commit is associated with which topic. We're able to verify that both the changes from the operator and the developer are now in the master branch of the system administrator's repository, by using the git log command:
sysadmin:~modules/ $ git log --abbrev-commit --pretty=oneline origin/master.. 1bbda50... Merge commit 'origin/operator/ssh' 9b41d49... Merge commit 'origin/developer/postfix' e4e27c7... Updated config.pp to use $module_name 0c164f6... Added manual change warning to postfix config eea4fbb... Added AllowGroups to sshd_config
Notice that this time, the system administrator has used the git log command to display abbreviated log messages from the current head of the origin/master branch to the current head of the local checked out branch. He chose origin/master because he has not pushed the newly merged changes to the central repository and this command therefore shows a list of changes that will be pushed if he decides to do so.
Everything looks good, as he expected. He also doesn't see the commit the developer made on his own topic branch to add the missing curly brace, because the developer chose to rebase his topic branch against the master branch before publishing his change list.
The team members decide to make the newly merged master branch the first testing branch, and they decide to continue developing on the master branch over the next couple of weeks. In a few days or weeks, the team will come together and decide on which of the change lists that each member has contributed are ready for merging into the testing branch. The system administrator starts this process by merging the changes he just made to the master branch into the testing branch, then pushing all of these changes to the central repository:
sysadmin:~modules/ $ git checkout testing Switched to branch "testing"
sysadmin:~modules/ $ git merge master Updating fa9812f..1bbda50 Fast forward postfix/files/master.cf | 4 +++- postfix/manifests/config.pp | 4 ++-- ssh/files/sshd_config | 1 + 3 files changed, 6 insertions(+), 3 deletions(-) sysadmin:~modules/ $ git push origin Counting objects: 6, done. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 494 bytes, done. Total 3 (delta 1), reused 0 (delta 0) To [email protected]:git/modules.git fa9812f..1bbda50 master -> master sysadmin:~modules/ $ git push origin testing:testing Total 0 (delta 0), reused 0 (delta 0) To [email protected]:git/modules.git * [new branch] testing -> testing
Notice that the system administrator executes two different push commands: one plain git push origin, and one git push origin testing:testing. This is because git push, by default, will only push the changes made to local branches into a remote repository if there is a branch with the same name in both locations.
Previously, the operator and developer logged into the puppet master and activated their changes by checking out their code in /etc/puppet/environments/development/modules. Similarly, the system administrator needs to fetch and checkout the new testing branch into the /etc/puppet/environments/testing/modules repository to activate the new configuration in the testing environment. Before doing so, he verifies that the remote named "origin" is configured to connect to the central repository at [email protected]:git/modules.git:
puppet:~ $ cd /etc/puppet/environments/testing/modules/ puppet:modules/ $ git remote -v origin /etc/puppet/modules/.git puppet:modules/ $ git remote rm origin puppet:modules/ $ git remote add origin [email protected]:git/modules.git puppet:modules/ $ git fetch origin remote: Counting objects: 39, done. remote: Compressing objects: 100% (24/24), done. remote: Total 28 (delta 9), reused 0 (delta 0) Unpacking objects: 100% (28/28), done. From [email protected]:git/modules * [new branch] developer/postfix -> origin/developer/postfix * [new branch] master -> origin/master
* [new branch] operator/ssh -> origin/operator/ssh * [new branch] testing -> origin/testing
Now that the testing environment repository has an up-to-date list of the branches, including the new testing branch, the system administrator performs a git checkout to activate the new changes on the system:
puppet:modules/ $ git checkout -b testing --track origin/testing Branch testing set up to track remote branch refs/remotes/origin/testing. Switched to a new branch "testing"
The system administrator is finally able to test a Puppet agent against the new testing environment, which now contains both changes: the SSH contribution from the operator, and the Postfix contribution from the developer. The testing environment is the only place where both changes are currently active in the configuration management system.
root:~ # puppet agent --test --noop --environment testing info: Caching catalog for scd.puppetlabs.vm info: Applying configuration version '1289770137' ... info: /Stage[main]/Ssh::Config/File[/etc/ssh/sshd_config]: Scheduling refresh of Service[sshd] notice: /Stage[main]/Ssh::Service/Service[sshd]: Triggered 'refresh' from 1 events notice: Finished catalog run in 2.77 seconds
Our team of Puppet contributors at Example.com has been effectively making changes to the configuration management system. Using Puppet Environments and a version control system, they're able to work efficiently and independently of one another without creating conflicts or obstructing another person's work. We've seen how the operator and the developer were able to make two changes in parallel, publishing those changes in a branch in the central version control repository for the system administrator to merge into a testing branch.
The team has also tested a number of machines using the Puppet agent in the testing environment, and is now ready to release the configuration to the production systems. This section covers how the team creates their first release, and provides a process to follow for subsequent releases.
You'll also see how a Git feature called "tagging" is useful to provide a method of referring to a specific point in time when the production configuration was active. You'll see how tags provide the ability to quickly roll back changes that might not be desirable in the production environment.
First, the team decides to release the current testing branch into production. Before doing so, the system administrator creates a tag so this release can be easily referred back to in the future. The system administrator does this in his own personal repository in his home directory:
sysadmin:~ $ cd ~/modules/ sysadmin:~modules/ $ git checkout testing Switched to branch "testing" sysadmin:~modules/ $ git tag -m 'First release to production' 1.0.0 sysadmin:~modules/ $ git push --tags origin Counting objects: 1, done.
Writing objects: 100% (1/1), 177 bytes, done. Total 1 (delta 0), reused 0 (delta 0) To [email protected]:git/modules.git * [new tag] 1.0.0 -> 1.0.0
The process of creating a tag is often called "cutting a release." The system administrator has done just this, tagged the current testing branch as a release to production, and then published the new tagged release into the central repository.
New branches, such as the testing or topic branches, were activated in the development and testing environments in the previous section. The process of activating a new production release is very similar, except instead of checking out a branch, which may change over time, a specific tag is checked out, which is static and refers to a very specific point in the history of configuration changes.
To activate the new production release, the system administrator logs into the Puppet master system as the user puppet, fetches the new tag from the central repository, and then checks out the tagged production release. Unlike the development and testing environments, Example.com has chosen to configure the production environment to use the working copy at /etc/puppet/modules rather than as a sub directory of /etc/puppet/environments where the development and testing active working copies reside.
puppet:~ $ cd /etc/puppet/modules puppet:modules/ $ git fetch origin remote: Counting objects: 21, done. remote: Compressing remote: objects: 100% (13/13), done. remote: Total 14 (delta 3), reused 0 (delta 0) Unpacking objects: 100% (14/14), done. From [email protected]:git/modules * [new branch] developer/postfix -> origin/developer/postfix fa9812f..1bbda50 master -> origin/master * [new branch] testing -> origin/testing * [new tag] 1.0.0 -> 1.0.0
Remember that the git fetch command does not affect the currently checked out configuration; it only updates the internal git index of data. The system administrator then checks out the newly-released production environment using the same familiar syntax we've seen so far:
puppet:modules/ $ git checkout tags/1.0.0 git checkout tags/1.0.0 Note: moving to "tags/1.0.0" which isn't a local branch If you want to create a new branch from this checkout, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new_branch_name> HEAD is now at 1bbda50... Merge commit 'origin/operator/ssh'
The note about moving to a non-local branch may be safely ignored. A tag is static reference, and the team should not have any need to directly modify the files in /etc/puppet/modules or make commits from the active production environment repository.
After executing the git checkout command to place the 1.0.0 release of the configuration into the production environment, everything is now active for the puppet agents. The system administrator verifies this by executing puppet agent in the default environment:
root:~ # puppet agent --test --noop info: Caching catalog for scd.puppetlabs.vm info: Applying configuration version '1289772102' notice: Finished catalog run in 0.53 seconds
You will also remember that the default environment is the production environment, and as such, the system administrator did not need to set the --environment command line option. If something were to have gone wrong in the production environment, a previous tag may be activated quickly, rolling back the changes introduced by the release of a new production configuration. One of the team members simply needs to execute git checkout tags/x.y.z to roll back the configuration.
The changes and workflow we've seen the operator, developer, and system administrator undertake in this chapter may now be repeated in a cycle. This development, testing, and release cycle provides an effective method to make changes to the configuration management system in a safe and predictable manner. Changes to the production system can be made with confidence: They've been vetted through the development and testing phases of the release process, they've been explicitly tagged in a release, and they can be quickly and easily backed out if things go awry.
You've seen how Puppet environments enable a team of contributors to work effectively and efficiently. Puppet environments, combined with a modern version control system, enable three people to make changes simultaneously and in parallel without obstructing each other's work. Furthermore, the tagging and branching features of modern version control systems provide an effective release management strategy. The process a single team member may follow in order to make changes is summarized as:
Develop changes in a local topic branch
Rebase against the master branch to remove any unnecessary commits
Publish the topic branch to the central repository
Activate and try the changes in the development puppet environment
Periodically merge and activate change lists from multiple people into a testing branch
Periodically cut a release of the testing branch using version control tags.
Debian stable, testing, unstable releases and distributions -
http://www.debian.org/doc/FAQ/ch-ftparchives.en.htmlPuppet Labs Environments Curated Documentation -
http://docs.puppetlabs.com/guides/environment.htmlPuppet Labs Environments Wiki Article -
http://projects.puppetlabs.com/projects/1/wiki/Using_Multiple_Environments