Chapter 16. DevOps tools: Deploying a scripted server environment using Ansible

This chapter covers

• Using orchestration tools to automate tiered Linux deployments
• Managing Linux servers using Ansible playbooks
• Organizing deployment-related data in a modular architecture

You’ve seen how scripts can automate complicated and boring processes to ensure they’re done regularly and done right. You wrote scripts to help with backup jobs back in chapter 5. You also saw how virtual Linux servers can be provisioned and launched in seconds in chapter 2. Is there any reason why you shouldn’t be able to put those tools together and automate the creation of entire virtual infrastructure environments? Nope. No reason at all.

Should you want to? Well, if you and your team are involved in an IT project involving multiple developers regularly pushing software versions to multiple servers, then you should probably give it some serious consideration, especially if you’ve got plans to adopt some variation of the DevOps approach to project management illustrated in figure 16.1.

Figure 16.1. A typical DevOps cycle incorporates feedback, testing, and monitoring into the development process.

What’s DevOps? It’s a way to organize the workflow used by technology companies and organizations through close collaboration among a project’s development, quality assurance (QA), and system administration teams. The goal is to use templates (infrastructure as code) to speed up time-to-deployment and software update cycles, and to allow greater levels of process automation and monitoring.

Many of the automation dividends will come through the smart implementation of orchestration tools like Ansible. Being able to plug new or updated code into a kind of virtual assembly line with all the underlying infrastructure and compatibility details invisibly taken care of can certainly speed things up. But it can also greatly improve quality and reduce errors.

Because most of the DevOps action is built on Linux infrastructure, and because sysadmins are as important to the process as developers, there’s a good chance that sooner or later your Linux career will touch DevOps. Before wrapping up this book, it would be a good idea to get a bit of a taste of the world of DevOps and orchestration.

Imagine you’re responsible for a complicated platform like the one illustrated in figure 16.2. That includes separate application, database, and authentication servers, all replicated in development, production, and backup environments. The development servers give you a safe place to test your code before pushing it out to production, and the backup servers can be called into service should the production servers crash. Your developers are constantly working to add features and squash bugs. And they’re regularly pushing their new code through the production cycle. In addition, the number of servers you run is constantly changing to meet rising and falling user demand.

Figure 16.2. A typical application development environment fed by regular software updates from the development team

With so much code flying back and forth in so many directions, you’re going to need some help keeping it all straight.

16.1. What deployment orchestrators can do for you

Deployment orchestrators will be perfectly happy working their magic on old-fashioned bare-metal servers, but you’ll only enjoy their full power when you incorporate them into virtualized deployments. Given how easy it is to script the creation of virtual servers, whether on your own hardware or using the resources of a cloud provider like AWS, being able to automate the creation of software stacks for your VMs will only add speed and efficiency.

The idea is that you compose one or more text files whose contents declare the precise state you want for all the system and application software on a specified machine (usually known as a host). When run, the orchestrator will read those files, log on to the appropriate host or hosts, and execute all the commands needed to achieve the desired state. Rather than having to go through the tedious and error-prone process manually on each of the hosts you’re launching, you tell the orchestrator to do it all for you. Once your infrastructure grows to dozens or even thousands of hosts, this kind of automation isn’t just convenient, it’s essential.

But if this is all about automating file system actions, why not use the Bash scripting skills you already have? Well, you probably could, but once you start trying to incorporate things like remote authentication and conflicting software stacks into those scripts, your life will quickly become insanely complicated.

Orchestrators will safely and reliably manage variables and passwords for you, and apply them within the proper context as often and in as many ways as necessary. You don’t need to track all the fine details on your own. Because there are all kinds of orchestration tools, the one you choose will largely depend on the specifics of your project, organization, and background. You’ll need to ask yourself some basic questions: “Are most of the people involved going to be developers or IT professionals?” “Will you be using a continuous integration methodology?” Table 16.1 provides some quick and dirty profiles of four of the main players.

Table 16.1. Popular deployment orchestrators

Tool	Features
Puppet	Broad community support
	Some coding skills recommended
	Extensible using Ruby
	Requires agents installed on all clients
Chef	Integrated with Git
	Some coding skills recommended
	Extensible using Ruby
	High learning curve
	Broad community support
	Requires chef-client installed on all clients
Ansible	Sysadmin friendly
	Python-based
	No code needed, no host-based agents
	Simple, fast connections work via SSH
	Run via text-based files (called playbooks)
	Minimal learning curve
Salt	Works through agents (called minions)
	Highly scalable
	Sysadmin friendly

As a sysadmin, Ansible sounds like a winner for me, so that’s what we’ll focus on for the rest of this chapter. But the needs and expectations of your specific project may differ.

16.2. Ansible: Installation and setup

Before starting, you’ll need a recent version of Python on your Ansible server and on all the machines you plan to use as hosts. Either apt install python or yum install python will do that job. Whichever version of Python you use (meaning Python 2 or 3), make sure python --version works from the command line.

Note

As of the time of this writing, Ansible often works better using the older 2.7 version of Python. That, however, will probably not be a long-term condition.

In order to install Ansible on the server (or control machine), you’ll need to enable the EPEL repository (for CentOS 6), the Extras repository (for CentOS 7), or the ppa:ansible repository for Ubuntu. Before you can enable that repository on Ubuntu using the add-apt-repository command, however, you may need to install the software-properties-common package. That’ll go like this:

# apt install software-properties-common
# add-apt-repository ppa:ansible/ansible
# apt update
# apt install ansible

Finally, fire up two or three Python-ready LXC containers to serve as hosts (or nodes), the creatures that do all the work. There’s no need to install Ansible on any of the hosts, just on the control machine.

16.2.1. Setting up passwordless access to hosts

Let’s look at how to set up passwordless access to hosts. Ansible prefers to do its work over SSH connections. Although it’s possible to handle authentication from the command line, it’s far better to send SSH keys to enable passwordless access with your hosts. You remember how that works from chapter 3, but here it is again:

$ ssh-keygen                                                           1
$ ssh-copy-id -i .ssh/id_rsa.pub [email protected]                     2
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s),
    to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed --
    if you are prompted now it is to install the new keys
[email protected]'s password:                                          3

• 1 Run this if you don’t already have a key pair on your local system.
• 2 The login and IP address of the host machine to which you’re copying your server’s SSH key
• 3 You’ll need to enter a password for the host machine user account to authorize the key transfer.

Now that Ansible is properly installed and connected to your hosts, it’s time to configure your environment.

16.2.2. Organizing Ansible hosts

Ansible gets its information about which hosts to manage from an inventory file called hosts in the /etc/ansible/ directory. The file can be a simple list of IP addresses or domain names, or a combination thereof.

Listing 16.1. A simple example of a /etc/ansible/hosts file

10.0.3.45
192.168.2.78
database.mydomain.com

But as the number of hosts you’re expecting Ansible to administer grows, along with the complexity of your overall environment, you’ll want to organize things a bit better. One way to do that is by dividing your hosts into host groups, which can then be targeted for precise Ansible actions.

Listing 16.2. An example of a /etc/ansible/hosts file organized into host groups

[webservers]
10.0.3.45
192.168.2.78

[databases]
database1.mydomain.com

Using host groups, Ansible tasks can be configured to run against only a well-defined subset of your hosts, perhaps sending updated public-facing web pages to only the web servers and new configuration files to the databases (illustrated in figure 16.3).

Figure 16.3. Task-specific updates being pushed to servers organized in host groups

There’s a lot more control that can be applied to the hosts file. You’ll find that the default hosts file created in /etc/ansible/ during installation will already include a nice selection of syntax suggestions, like how you can reference multiple host names in a single line: www[001:006].example.com.

Note

So you’ll be able to follow along with the demos in this chapter, add the IP address of your Python-ready LXC (or other) hosts to the hosts file.

16.2.3. Testing connectivity

To test that things are set up properly, Ansible can try to contact the hosts listed in the hosts file. This command runs Ansible from the command line in what’s known as ad hoc mode. The -m tells Ansible to load and run the ping module to send a simple “Are You There?” request to all the hosts listed in the hosts file:

$ ansible all -m ping
10.0.3.103 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

The all condition in that command means you want this action performed on all the hosts listed in the hosts file. If you only wanted to ping a specific host group, you would use the group name instead of all:

$ ansible webservers -m ping

Now that you’re connected, you can run simple commands remotely. This example copies the /etc/group file to the home directory of each of your hosts. Remember, the reason you’re able to do this without providing authentication is because you previously used ssh-keygen to save your SSH key to the remote host:

$ ansible all -a "cp /etc/group /home/ubuntu"

You can confirm the operation worked by running ls over SSH:

$ ssh [email protected] "ls /home/ubuntu"

If the username of the account you’re logged in to on your Ansible server is not the same as the usernames on your hosts, you’ll need to tell Ansible about it. You can do that from the command line using the --user argument, which, assuming the host usernames are ubuntu, would look like this: ansible --user ubuntu all -m ping.

16.3. Authentication

Now suppose you need to execute a command on your remote hosts that requires sudo powers. Imagine you want to push an updated .html file to all of the dozens of web servers toiling away tirelessly behind your load balancer. It sure would make a lot of sense to do it in one go, rather than to repeat the operation individually for each host.

Why not try it yourself? See what happens when you try to use the copy module to copy a file in your local home directory (perhaps the group file you copied there earlier) to the /var/www/html/ directory on your remote host. If your host doesn’t happen to have a /var/www/html/ directory already, you can produce the same effect by substituting any system directory (like /etc/) that’s not owned by your user:

$ ansible webservers -m copy -a "src=/home/ubuntu/group 
    dest=/var/www/html/"                                         1
10.0.3.103 | FAILED! => {
    "changed": false,
    "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "failed": true,
    "msg": "Destination /var/www/html not writable"              2
}

• 1 src= points to the location of the source file on the local machine; dest= points to the target location on the host.
• 2 A descriptive error message explaining what went wrong

Whoops. “Destination /var/www/html not writable” sounds like a permissions issue. Looks like you’ll have to find a way to escalate your privileges. The best way to do that is through settings in the /etc/ansible/ansible.cfg file. As you can see from the following example, I edited the [privilege_escalation] section of ansible.cfg by uncommenting its four lines.

Listing 16.3. Changed settings in /etc/ansible/ansible.cfg

[privilege_escalation]
become=True
become_method=sudo
become_user=root
become_ask_pass=True

When you run the copy operation once again, this time adding the --ask-become-pass argument, Ansible reads the updated configuration file and prompts for the remote ubuntu user’s sudo password. This time you’ll be successful:

$ ansible --ask-become-pass webservers -m copy -a "src=/home/ubuntu/group 
    dest=/var/www/html/"
SUDO password:
10.0.3.103 | SUCCESS => {
    "changed": true,
    "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
    "dest": "/var/www/html/stuff.html",
    "gid": 0,
    "group": "root",
    "md5sum": "d41d8cd98f00b204e9800998ecf8427e",
    "mode": "0644",
    "owner": "root",
    "size": 0,
    "src": "/home/ubuntu/.ansible/tmp/
        ansible-tmp-1509549729.02-40979945256057/source",
    "state": "file",
    "uid": 0
}

Log in to your remote server to confirm that the file has been copied. By the way, from a security perspective, it’d be a terrible idea to leave a copy of your group file in the web root. This was just an example. Please don’t leave it there.

16.4. Ansible playbooks

As you’ve seen, you can be up and running with some basic Ansible activities in a minute or two. But those basics won’t get you far. If you want to exploit the real power of the tool so it can orchestrate the kind of automated multi-tier infrastructure I described in the chapter introduction, you’ll need to learn to use playbooks. Playbooks are the way you closely define the policies and actions you want Ansible to trigger. They’re also an easy way to share working configuration profiles. Here are two ways you can use a playbook:

• As a simple, standalone script
• As a reference that points to resources spread across a specially structured directory tree (for more complicated environments)

16.4.1. Writing a simple playbook

Let’s learn how to create a simple playbook that can all in one go provision a relatively straightforward web server. To do this, you’ll use modules (like the copy module you saw previously), tasks for running Linux system actions, and handlers to dynamically respond to system events. First, make sure your hosts file in /etc/ansible/ is up to date.

Listing 16.4. A simple /etc/ansible/hosts file

10.0.3.103
10.0.3.96

Next, you’ll need to create a YAML-formatted file called site.yml. YAML is a text-formatting language related to the more widely used JavaScript Object Notation (JSON). Although you’ll need to be careful getting the indentation right, the YAML format does produce configuration profiles that are easy to read, understand, and edit.

After starting with a line containing three dashes (---), your file will include three sections: hosts, tasks, and handlers. In this case, the hosts section tells Ansible to apply the playbook’s actions to all the addresses from the webservers group in the hosts file. The tasks section (indented the same number of spaces as hosts) introduces three tasks (or modules): apt to install the Apache web server, copy to copy a local file to the web document root, and service, much like systemctl in a systemd environment, to make sure Apache is running.

Listing 16.5. A simple Ansible playbook called site.yml

- hosts: webservers                                                        1

  tasks:
  - name: install the latest version of apache
    apt:
      name: apache2                                                        2
      state: latest                                                        3
      update_cache: yes
  - name: copy an index.html file to the web root and rename it index2.html
    copy: src=/home/ubuntu/project/index.html                              4
        dest=/var/www/html/index2.html
    notify:
    - restart apache
  - name: ensure apache is running
    service: name=apache2 state=started

  handlers:
   - name: restart apache                                                  5
     service: name=apache2 state=restarted

• 1 Runs the tasks only on those hosts listed in the webservers host group
• 2 The apt module installs the apache2 package on an Ubuntu machine
• 3 Requiring the latest value for the state property ensures that the latest version of the software is installed.
• 4 Requires the existence of an index.html file in a local directory called project/
• 5 The handler name; even though it looks more like a description, it can be used as a reference from within a task.

To test this yourself, you could create a simple file called index.html and save it to a directory on your Ansible server. (I used an LXC container for my Ansible lab.) Make sure to properly reference the file location in the playbook (the way it was in the copy: src= line from the previous playbook example). The file can, if you like, contain nothing more complicated than the words Hello World. It’s only there to confirm the playbook worked. Once the playbook has run, a copy of that file should exist in the web document root.

Also, note the notify: line within the copy task in the previous example. Once the copy task is complete, notify triggers the handler with the name restart apache that will, in turn, make sure that Apache is restarted and running properly.

As you build your own playbooks, you’ll definitely need more syntax and feature information. Running ansible-doc and the name of a particular module will get you going:

$ ansible-doc apt

Running your playbook

Assuming your /etc/ansible/ansible.cfg file is still properly configured to handle host authentication, you’re ready to use the ansible-playbook command to run your playbook. By default, the command will use the hosts listed in /etc/ansible/hosts, but you can use -i to point it to a different file. Here’s an example:

$ ansible-playbook site.yml
SUDO password:
PLAY ****************************************************
TASK [setup] ********************************************
ok: [10.0.3.96]
TASK [ensure apache is at the latest version] ***             1
changed: [10.0.3.96]
TASK [copy an index.html file to the root directory] ****
changed: [10.0.3.96]
TASK [ensure apache is running] *************************
ok: [10.0.3.96]                                               2
RUNNING HANDLER [restart apache] ************************
changed: [10.0.3.96]
PLAY RECAP **********************************************
10.0.3.96 : ok=5 changed=3 unreachable=0 failed=0             3

• 1 A brief summary of the task’s purpose
• 2 The ok message tells you that a task has successfully completed.
• 3 A summary of the results of running the playbook

Success! With that single command you’ve built a working web server on all the hosts you listed in your hosts file. Don’t believe me? Point your browser to the URL that should be used by the index2.html file you copied (10.0.3.96/index2.html, in my case). You should see your index2.html file displayed.

16.4.2. Creating multi-tiered, role-powered playbooks

Once your Ansible-managed infrastructure becomes weighted down with layers of elements, each with its own detailed parameters, keeping them all in a single playbook script is impractical. Try to imagine what it might be like to manage the kind of platform illustrated earlier in figure 16.2.

Breaking out the tasks, handlers, and other data types into separate directories and files will make things much more readable. This kind of modular organization also makes it possible to build new playbooks without having to reinvent any wheels: you’ll always have full and easy access to everything you’ve created.

Ansible organizes its modular elements into roles and even provides its own command-line tool, ansible-galaxy, to manage existing roles and generate the necessary file system framework for starting new roles. Figure 16.4 illustrates the basic Ansible topology.

Figure 16.4. Ansible roles shown as self-contained groupings of resources, including access to system dependencies

Generating an Ansible role

Choose a directory to use as your Ansible root. If you’re working on a container or VM whose whole purpose is to act as an Ansible server, this might as well be your main user’s document root (/home/username/). From the Ansible root, you’ll create a directory called roles and then move to the new directory.

Once there, initialize the directory using ansible-galaxy init followed by the name you want to use for your role:

$ mkdir roles
$ cd roles
$ ansible-galaxy init web-app
- web-app was created successfully

A new directory called web-app is created. Run ls -R to recursively list the new subdirectories and their contents that ansible-galaxy created for you:

$ cd web-app
$ ls -R
.:
defaults  files  handlers  meta  README.md  tasks  templates  tests  vars
./defaults:                                                               1
main.yml                                                                  2
./files:
./handlers:
main.yml
./meta:
main.yml
./tasks:
main.yml
./templates:
./tests:
inventory  test.yml
./vars:
main.yml

• 1 Each subdirectory within web-app is displayed with a leading ./.
• 2 Some directories are already populated with largely empty playbook files of their own.

How does Ansible consume the data contained in those directories? Here’s something to chew on: the variable values and parameters set in those files will often control the way Ansible manages the resources launched using that particular role.

Read that one or two more times. Done? OK, there are two things that should stand out: often control (but not always?) and that particular role (you mean I could have others?).

Right and right. The settings added to files beneath your web-app role directory can be invoked either from a top-level playbook or through an ad hoc command-line action. By way of example, you might have defined a web document root location in the roles/web-app/defaults/main.yml file as webroot_location: /var/www/myroot/. Invoking the webroot_location variable will always return the value /var/www/myroot/.

Except when it doesn’t. You see, Ansible was designed for environments encompassing multiple projects. You might have a separate playbook for each of a handful of separate applications and others for internal company services. There’s nothing stopping you from managing more than one application from a single Ansible server. This will probably mean that you want a particular variable to mean one thing for application x and another for application y.

Which brings us to the second notable point: each application or service can be defined by its own Ansible role. But, so that multiple roles can happily coexist on a single system, you’ll need a way to prioritize their overlapping variables. The way Ansible does that is quite complicated, but I can summarize it by saying that values found in a role’s vars/ directory override those from /defaults, and values explicitly set using -e (or --extra-vars=) beat everything else.

What might go into each of your roles/web-app/ directories? Here’s a short list:

• The vars/ directory is likely to contain information on the file system locations for encryption keys.
• The templates/ directory will hold templates that are meant to be installed as, say, Apache configuration files in Python’s .j2 format.
• The files/ directory could contain other files (like that .html file you copied in the previous example) that are used for host-based data.

16.4.3. Managing passwords in Ansible

Although you’ll probably need to include host passwords in your Ansible infrastructure, you should never store them in plain text documents. Ever. Rather, Ansible provides a tool called Vault that stores sensitive data in encrypted files that can, when necessary, be safely called by a playbook. This snippet opens an editor into which you can enter a new Vault password:

$ export EDITOR=nano                        1
$ ansible-vault create mypasswordfile
New Vault password:                         2
Confirm New Vault password:

• 1 If you don’t want Vault to open the password file in Vim, you can export the editor variable as Nano.
• 2 You’ll be prompted to enter a new Vault password before adding the password you want to use for host access.

Assuming your hosts are all using only a single password, this works by adding the --ask-vault-pass argument to the ansible-playbook command:

$ ansible-playbook site.yml --ask-vault-pass
Vault password:

For your information, since Ansible version 2.3, it’s also possible to make use of what Ansible calls a vaulted variable, which is essentially an encrypted password stored in a plain text YAML file. This makes it possible to manage multiple passwords.

Summary

• Orchestration tools let you automate server infrastructure at scale, whether it’s a single host or thousands. The tool you choose will depend on your team’s skill set, project needs, and company culture.
• Ansible requires no coding, runs on SSH, and has a light footprint.
• Ansible playbooks, especially playbooks that run resources through roles, are effective and efficient ways to manage security and resources.

Key terms

• DevOps is a project organization structure to help development and admin teams speed up and automate product development cycles.
• Orchestration deployment tools let you precisely script infrastructure behavior to achieve desired states through automation.
• A host group is a way to organize hosts so that Ansible can be directed to manage well-defined subsets of your host fleet.
• In Ansible playbooks, modules are predefined command sequences run on a host system, handlers are actions triggered by events, and roles are bundled resources organized to serve a single project.

Security best practices

• Directing Ansible to access your hosts using passwordless, key-pair-based SSH is preferred to having to enter passwords at the command line for each operation.
• Never include passwords in Ansible playbooks or other plain text scripts. Use Ansible Vault instead.

Command-line review

• add-apt-repository ppa:ansible/ansible adds the Debian Ansible software repository to allow apt to install Ansible on an Ubuntu/Debian machine.
• ansible webservers -m ping tests all the hosts in the webservers host group for network connectivity.
• ansible webservers -m copy -a "src=/home/ubuntu/stuff.html dest=/var/ www/html/" copies a local file to the specified file location on all the hosts in the webservers group.
• ansible-doc apt displays syntax and usage information on the apt module.
• ansible-playbook site.yml launches an operation based on the site.yml playbook.
• ansible-playbook site.yml --ask-vault-pass uses a Vault password to authenticate and perform playbook operations.

Test yourself

1

Which of the following orchestration tools would work best for a team of developers with little experience in DevOps who are building a large and complex platform?

1. Ansible
2. Chef
3. Puppet
4. Salt

2

Which of the following packages must be installed on each host for Ansible to work?

1. Ansible
2. Python
3. software-properties-common
4. Ansible and Python

3

Which of the following design considerations is primarily a security concern?

1. Organizing your hosts into host groups
2. Scheduling regular connectivity testing for all hosts
3. Separating environment variables
4. Storing data in Ansible Vault

4

What command tells Ansible to automatically populate the default web document root with a local file on only those hosts running Apache?

1. ansible all -i copy -a "src=/var/www/html/ dest=/home/ubuntu/stuff.html"
2. ansible all webservers -m copy -a "/home/ubuntu/stuff.html /var/www/html/"
3. ansible webservers -m copy -a "src=/home/ubuntu/stuff.html dest=/var/www/html/"
4. ansible webservers -m copy -a src=/home/ubuntu/stuff.html dest=/var/www/html/

5

Which of the following commands will create the directories and files you need for a new Ansible role?

1. ansible-root-directory/roles/ansible-galaxy init rolename
2. ansible-root-directory/ansible-galaxy rolename
3. ansible-root-directory/roles/ansible-init rolename
4. ansible-root-directory/roles/ansible init rolename

Answer key

1.

b

2.

b

3.

d

4.

c

5.

a