Listing Tasks in a Playbook

Before we dive into the guts of our playbook, let’s get a high-level view. The ansible-playbook command-line tool supports a flag called --list-tasks. This flag prints out the names of all the tasks in a playbook. Here’s how you use it:

$ ansible-playbook --list-tasks mezzanine.yml

Example 6-1 shows the output for the mezzanine.yml playbook in Example 6-27.

 playbook: mezzanine.yml
  play #1 (web): Deploy mezzanine	TAGS: []
    tasks:
      install apt packages	TAGS: []
      create project path	TAGS: []
      create a logs directory	TAGS: []
      check out the repository on the host	TAGS: []
      create python3 virtualenv	TAGS: []
      copy requirements.txt to home directory	TAGS: []
      install packages listed in requirements.txt	TAGS: []
      create project locale	TAGS: []
      create a DB user	TAGS: []
      create the database	TAGS: []
      ensure config path exists	TAGS: []
      create tls certificates	TAGS: []
      remove the default nginx config file	TAGS: []
      set the nginx config file	TAGS: []
      enable the nginx config file	TAGS: []
      set the supervisor config file	TAGS: []
      install poll twitter cron job	TAGS: []
      set the gunicorn config file	TAGS: []
      generate the settings file	TAGS: []
      apply migrations to create the database, collect static content	TAGS: []
      set the site id	TAGS: []
      set the admin password	TAGS: []

It’s a handy way to summarize what a playbook is going to do.

Organization of Deployed Files

As we discussed earlier, Mezzanine is built atop Django. In Django, a web app is called a project. We get to choose what to name our project, and we’ve chosen to name this one mezzanine_example.

Our playbook deploys into a Vagrant machine and will deploy the files into the home directory of the Vagrant user’s account.

.
|---- logs
|---- mezzanine
|     |___ mezzanine_example
|____ .virtualenvs
      |___ mezzanine_example

Example 6-2 shows the relevant directories underneath /home/vagrant:

• /home/vagrant/mezzanine/mezzanine-example will contain the source code that will be cloned from a source code repository on GitHub.

• /home/vagrant/.virtualenvs/mezzanine_example is the virtualenv directory, which means that we’re going to install all of the Python packages into that directory.

• /home/vagrant/logs will contain log files generated by Mezzanine.

Variables and Secret Variables

As you can see in Example 6-3, this playbook defines quite a few variables.

vars:
  user: "{{ ansible_user }}"
  proj_app: mezzanine_example
  proj_name: "{{ proj_app }}"
  venv_home: "{{ ansible_env.HOME }}/.virtualenvs"
  venv_path: "{{ venv_home }}/{{ proj_name }}"
  proj_path: "{{ ansible_env.HOME }}/mezzanine/{{ proj_name }}"
  settings_path: "{{ proj_path }}/{{ proj_name }}"
  reqs_path: requirements.txt
  manage: "{{ python }} {{ proj_path }}/manage.py"
  live_hostname: 192.168.33.10.nip.io
  domains:
    - 192.168.33.10.nip.io
    - www.192.168.33.10.nip.io
  repo_url: [email protected]:ansiblebook/mezzanine_example.git
  locale: en_US.UTF-8
  # Variables below don't appear in Mezzanine's fabfile.py
  # but I've added them for convenience
  conf_path: /etc/nginx/conf
  tls_enabled: True
  python: "{{ venv_path }}/bin/python"
  database_name: "{{ proj_name }}"
  database_user: "{{ proj_name }}"
  database_host: localhost
  database_port: 5432
  gunicorn_procname: gunicorn_mezzanine

vars_files:
  - secrets.yml

We’ve tried for the most part to use the same variable names that the Mezzanine Fabric script uses. I’ve also added some extra variables to make things a little clearer. For example, the Fabric scripts directly use proj_name as the database name and database username. For clarity Lorin prefers to define intermediate variables named database_name and database_user and define these in terms of proj_name.

It’s worth noting a few things here. First off, we can define one variable in terms of another. For example, we define venv_path in terms of venv_home and proj_name.

Also, we can reference Ansible facts in these variables. For example, venv_home is defined in terms of the ansible_env fact collected from each host.

Finally, we have specified some of our variables in a separate file, called secrets.yml, by doing this:

vars_files:
  - secrets.yml

This file contains credentials such as passwords and tokens that need to remain private. The GitHub repository does not actually contain this file. Instead, it contains a file called secrets.yml.example that looks like this:

db_pass: e79c9761d0b54698a83ff3f93769e309
admin_pass: 46041386be534591ad24902bf72071B
secret_key: b495a05c396843b6b47ac944a72c92ed
nevercache_key: b5d87bb4e17c483093296fa321056bdc# You need to create a Twitter application at https://dev.twitter.com
# in order to get the credentials required for Mezzanine's
# twitter integration.
## See http://mezzanine.jupo.org/docs/twitter-integration.html
# for details on Twitter integration
twitter_access_token_key: 80b557a3a8d14cb7a2b91d60398fb8ce
twitter_access_token_secret: 1974cf8419114bdd9d4ea3db7a210d90
twitter_consumer_key: 1f1c627530b34bb58701ac81ac3fad51
twitter_consumer_secret: 36515c2b60ee4ffb9d33d972a7ec350a

To use this repo, copy secrets.yml.example to secrets.yml and edit it so that it contains the credentials specific to your site.

Installing Multiple Packages

We’re going to need to install two types of packages for our Mezzanine deployment: some system-level packages and some Python packages. Because we’re going to deploy on Ubuntu, we’ll use apt as our package manager for the system packages. We’ll use pip to install the Python packages.

System-level packages are generally easier to deal with than Python packages because they’re designed specifically to work with the operating system. However, the system package repositories often don’t have the newest versions of the Python libraries we need, so we turn to the Python packages to install those. It’s a trade-off between stability and running the latest and greatest.

Example 6-4 shows the task we’ll use to install the system packages.

    - name: install apt packages
      become: true
      apt:
        update_cache: true
        cache_valid_time: 3600
        pkg:
          - acl
          - git
          - libjpeg-dev
          - libpq-dev
          - memcached
          - nginx
          - postgresql
          - python3-dev
          - python3-pip
          - python3-venv
          - python3-psycopg2
          - supervisor

Because we’re installing multiple packages, Ansible will pass the entire list to the apt module, and the module will invoke the apt program only once, passing it the entire list of packages to be installed. The apt module has been designed to handle this list entirely.

Adding the Become Clause to a Task

In the playbook examples you read in Chapter 2, we wanted the whole playbook to run as root, so we added the become: true clause to the play. When we deploy Mezzanine, most of the tasks will be run as the user who is SSHing to the host, rather than root. Therefore, we don’t want to run as root for the entire play, only for select tasks.

We can accomplish this by adding become: true to the tasks that do need to run as root, such as Example 6-4. For auditing purposes, Bas prefers to add become: true right under the - name:.

Updating the Apt Cache

Ubuntu maintains a cache with the names of all of the apt packages that are available in the Ubuntu package archive. Let’s say you try to install the package named libssl-dev. We can use the apt-cache program to query the local cache to see what version it knows about:

$ apt-cache policy libssl-dev

The output is shown in Example 6-5.

libssl-dev:
  Installed: (none)
  Candidate: 1.1.1f-1ubuntu2.4
  Version table:
     1.1.1f-1ubuntu2.4 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1.1.1f-1ubuntu2.3 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     1.1.1f-1ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

As you can see, this package is not installed locally. According to the local cache, the latest version is 1.1.1f-1ubuntu2.4. It also tells us the location of the package archive.

In some cases, when the Ubuntu project releases a new version of a package, it removes the old version from the package archive. If the local apt cache of an Ubuntu server hasn’t been updated, then it will attempt to install a package that doesn’t exist in the package archive.

To continue with our example, let’s say we attempt to install the libssl-dev package:

$ sudo apt-get install libssl-dev

If version 1.1.1f-1ubuntu2.4 is no longer available in the package archive, we’ll see an error.

On the command line, the way to bring the local apt cache up to date is to run apt-get update. When using the apt Ansible module, however, you’ll do this update by passing the update_cache: true argument when invoking the module, as shown in Example 6-4.

Because updating the cache takes additional time, and because we might be running a playbook multiple times in quick succession to debug it, we can avoid paying the cache update penalty by using the cache_valid_time argument to the module. This instructs to update the cache only if it’s older than a certain threshold. The example in Example 6-4 uses cache_valid_time: 3600, which updates the cache only if it’s older than 3,600 seconds (1 hour).

Checking Out the Project Using Git

Although Mezzanine can be used without writing any custom code, one of its strengths is that it is written on top of the Django web application platform, which is great if you know Python. If you just want a CMS, you’ll likely just use something like WordPress. But if you’re writing a custom application that incorporates CMS functionality, Mezzanine is a good way to go.

As part of the deployment, you need to check out the Git repository that contains your Django applications. In Django terminology, this repository must contain a project. We’ve created a repository on GitHub (https://github.com/ansiblebook/mezzanine_example) that contains a Django project with the expected files. That’s the project that gets deployed in this playbook.

We created these files using the mezzanine-project program that ships with Mezzanine, like this:

$ mezzanine-project mezzanine_example
$ chmod +x mezzanine_example/manage.py

Note that we don’t have any custom Django applications in my repository, just the files that are required for the project. In a real Django deployment, this repository would contain subdirectories with additional Django applications.

Example 6-6 shows how to use the git module to check out a Git repository onto a remote host.

    - name: check out the repository on the host
      git:
        repo: "{{ repo_url }}"
        dest: "{{ proj_path }}"
        version: master
        accept_hostkey: true

We’ve made the project repository public so that you can access it, but in general, you’ll be checking out private Git repositories over SSH. For this reason, we’ve set the repo_url variable to use the scheme that will clone the repository over SSH:

repo_url: [email protected]:ansiblebook/mezzanine_example.git

If you’re following along at home, to run this playbook, you must have the following:

• A GitHub account

• A public SSH key associated with your GitHub account

• An SSH agent running on your control machine, with agent forwarding enabled

• Your SSH key added to your SSH agent

Once your SSH agent is running, add your key:

$ ssh-add <path to the private key>

If successful, the following command will output the public key of the SSH you just added:

$ ssh-add -L

The output should look like something this:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN1/YRlI7Oc+KyM6NFZt7fb7pY+btItKHMLbZhdbwhj2 Bas 

To enable agent forwarding, add the following to your ansible.cfg:

[ssh_connection]
ssh_args = -o ForwardAgent=yes

You can verify that agent forwarding is working by using Ansible to list the known keys:

$ ansible web -a "ssh-add -L"

You should see the same output as when you run ssh-add -L on your local machine.

Another useful check is to verify that you can reach GitHub’s SSH server:

$ ansible web -a "ssh -T [email protected]"

If successful, the output should look like this:

web | FAILED | rc=1 >>
Hi bbaassssiiee! You've successfully authenticated, but GitHub does not provide shell
access.

Even though the word FAILED appears in the output (we cannot log into a bash shell on github), if this message from GitHub appears, then it was successful.

In addition to specifying the repository URL with the repo parameter and the destination path of the repository as the dest parameter, we also pass an additional parameter, accept_hostkey, which is related to host-key checking. (We discuss SSH agent forwarding and host-key checking in more detail in Chapter 20.)

Installing Mezzanine and Other Packages into a Virtual Environment

We can install Python packages systemwide as the root user, but it’s better practice to install these packages in an isolated environment to avoid polluting the system-level Python packages. In Python, these types of isolated package environments are called virtual environments, or virtualenvs. A user can create multiple virtualenvs and can install Python packages into a virtualenv without needing root access. (Remember, we’re installing some Python packages to get more recent versions.)

Ansible’s pip module has support for installing packages into a virtualenv, as well as for creating the virtualenv if it is not available.

Example 6-7 shows how to use pip to install a Python 3 virtualenv with the latest package tools.

    - name: create python3 virtualenv
      pip:
        name:
          - pip
          - wheel
          - setuptools
        state: latest
        virtualenv: "{{ venv_path }}"
        virtualenv_command: /usr/bin/python3 -m venv

Example 6-8 shows the two tasks that we use to install Python packages into the virtualenv. A common pattern in Python projects is to specify the package dependencies in a file called requirements.txt.

    - name: copy requirements.txt to home directory
      copy:
        src: requirements.txt
        dest: "{{ reqs_path }}"
        mode: 0644
    - name: install packages listed in requirements.txt
      pip:
        virtualenv: "{{ venv_path }}"
        requirements: "{{ reqs_path }}"

Indeed, the repository in our Mezzanine example contains a requirements.txt file. It looks like Example 6-9.

Mezzanine==4.3.1

Note that the Mezzanine Python package in requirements.txt is pinned to a specific version (4.3.1). That requirements.txt file is missing several other Python packages that we need for the deployment, so we explicitly specify these in a requirements.txt file in the playbooks directory that we then copy to the host.

We just take the latest available version of the other dependencies.

Alternately, if you wanted to pin all of the packages, you’d have several options: for example, you could specify all the packages in the requirements.txt file, for repeatability. This file contains information about the packages and the dependencies. An example file looks like Example 6-10.

beautifulsoup4==4.9.3
bleach==3.3.0
certifi==2021.5.30
chardet==4.0.0
Django==1.11.29
django-appconf==1.0.4
django-compressor==2.4.1
django-contrib-comments==2.0.0
filebrowser-safe==0.5.0
future==0.18.2
grappelli-safe==0.5.2
gunicorn==20.1.0
idna==2.10
Mezzanine==4.3.1
oauthlib==3.1.1
packaging==21.0
Pillow==8.3.1
pkg-resources==0.0.0
psycopg2==2.9.1
pyparsing==2.4.7
python-memcached==1.59
pytz==2021.1
rcssmin==1.0.6
requests==2.25.1
requests-oauthlib==1.3.0
rjsmin==1.1.0
setproctitle==1.2.2
six==1.16.0
soupsieve==2.2.1
tzlocal==2.1
urllib3==1.26.6
webencodings==0.5.1

If you have an existing virtualenv with the packages installed, you can use the pip freeze command to print out a list of installed packages. For example, if your virtualenv is in ~/.virtualenvs/mezzanine_example, then you can activate your virtualenv and save the packages in the virtualenv into a requirements.txt file:

$ source .virtualenvs/mezzanine_example/bin/activate
$ pip freeze > requirements.txt

Example 6-11 shows how to specify both the package names and their versions in the list. with_items passes a list of dictionaries, to dereference the elements with item.name and item.version when the pip module iterates.

- name: install python packages with pip
  pip: 
    virtualenv: "{{ venv_path }}"
    name: "{{ item.name }}"
    version: "{{ item.version }}" 
  with_items:
    - {name: mezzanine, version: '4.3.1' }
    - {name: gunicorn, version: '20.1.0' }
    - {name: setproctitle, version: '1.2.2' }
    - {name: psycopg2, version: '2.9.1' }
    - {name: django-compressor, version: '2.4.1' }
    - {name: python-memcached, version: '1.59' }

Please note the single quotes around version numbers: this ensures they are treated as literals and are not rounded off in edge cases.

Complex Arguments in Tasks: A Brief Digression

When you invoke a module, you can pass the argument as a string (great for ad-hoc use). Taking the pip example from Example 6-11, we could have passed the pip module a string as an argument:

- name: install package with pip
  pip: virtualenv={{ venv_path }} name={{ item.name }} version={{ item.version }}

If you don’t like long lines in your files, you could break up the argument string across multiple lines by using YAML’s line folding:

- name: install package with pip
  pip: >
    virtualenv={{ venv_path }}
    name={{ item.name }}
    version={{ item.version }}

Ansible provides another option for breaking up a module invocation across multiple lines. Instead of passing a string, you can pass a dictionary in which the keys are the variable names. This means you could invoke Example 6-11 like this instead:

- name: install package with pip
  pip:
    virtualenv: "{{ venv_path }}"
    name: "{{ item.name }}"
    version: "{{ item.version }}"

The dictionary-based approach to passing arguments is also useful when invoking modules that take complex argument, or arguments to a module that is a list or a dictionary. The uri module, which sends web requests, is a good example. Example 6-12 shows how to call a module that takes a list as an argument for the body parameter.

- name: Login to a form based webpage
  uri:
    url: https://your.form.based.auth.example.com/login.php
    method: POST
    body_format: form-urlencoded
    body:
      name: your_username
      password: your_password
      enter: Sign in
    status_code: 302
  register: login

Passing module arguments as dictionaries instead of strings is a practice that can avoid the whitespace bugs that can arise when using optional arguments, and it works really well in version control systems.

If you want to break your arguments across multiple lines and you aren’t passing complex arguments, which form you choose is a matter of taste. Bas generally prefers dictionaries to multiline strings, but in this book we use both forms.

Configuring the Database

When Django runs in development mode, it uses the SQLite backend. This backend will create the database file if the file does not exist.

When using a database management system such as Postgres, we need to first create the database inside Postgres and then create the user account that owns the database. Later, we will configure Mezzanine with the credentials of this user.

Ansible ships with the postgresql_user and postgresql_db modules for creating users and databases inside Postgres. Example 6-13 shows how we invoke these modules in our playbook.

When creating the database, we specify locale information through the lc_ctype and lc_collate parameters. We use the locale_gen module to ensure that the locale we are using is installed in the operating system.

    - name: create project locale
      become: true
      locale_gen:
        name: "{{ locale }}"
    - name: create a DB user
      become: true
      become_user: postgres
      postgresql_user:
        name: "{{ database_user }}"
        password: "{{ db_pass }}"
    - name: create the database
      become: true
      become_user: postgres
      postgresql_db:
        name: "{{ database_name }}"
        owner: "{{ database_user }}"
        encoding: UTF8
        lc_ctype: "{{ locale }}"
        lc_collate: "{{ locale }}"
        template: template0

Note the use of become: true and become_user: postgres on the last two tasks. When you install Postgres on Ubuntu, the installation process creates a user named postgres that has administrative privileges for the Postgres installation. Since the root account does not have administrative privileges in Postgres by default, we need to become the Postgres user in the playbook in order to perform administrative tasks, such as creating users and databases.

When we create the database, we set the encoding (UTF8) and locale categories (LC_CTYPE, LC_COLLATE) associated with the database. Because we are setting locale information, we use templateO as the template.²

Generating the local_settings.py File from a Template

Django expects to find project-specific settings in a file called settings.py. Mezzanine follows the common Django idiom of breaking these settings into two groups:

• Settings that are the same for all deployments (settings.py)

• Settings that vary by deployment (local_settings.py)

We define the settings that are the same for all deployments in the settings.py file in our project repository. You can find that file on GitHub (http://bit.ly/2jaw4zf).

The settings.py file contains a Python snippet that loads a local_settings.py file that contains deployment-specific settings. The .gitignore file is configured to ignore the local_settings.py file, since developers will commonly create this file and configure it for local development.

As part of our deployment, we need to create a local_settings.py file and upload it to the remote host. Example 6-14 shows the Jinja2 template that we use.

# Make these unique, and don't share it with anybody.
SECRET_KEY = "{{ secret_key }}"
NEVERCACHE_KEY = "{{ nevercache_key }}"
ALLOWED_HOSTS = [{% for domain in domains %}"{{ domain }}",{% endfor %}]

DATABASES = {
    "default": {
         # Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
         "ENGINE": "django.db.backends.postgresql_psycopg2",
         # DB name or path to database file if using sqlite3.
         "NAME": "{{ proj_name }}",
         # Not used with sqlite3.
         "USER": "{{ proj_name }}",
         # Not used with sqlite3.
         "PASSWORD": "{{ db_pass }}",
         # Set to empty string for localhost. Not used with sqlite3.
         "HOST": "127.0.0.1",
         # Set to empty string for default. Not used with sqlite3.
         "PORT": "",
    }
}

CACHE_MIDDLEWARE_KEY_PREFIX = "{{ proj_name }}"
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
        "LOCATION": "127.0.0.1:11211",
    }
}
SESSION_ENGINE = "django.contrib.sessions.backends.cache"

Most of this template is straightforward; it uses the {{ variable }} syntax to insert the values of variables such as secret_key, nevercache_key, proj_name, and db_pass. The only nontrivial bit of logic is the line shown in Example 6-15.

ALLOWED_HOSTS = [{% for domain in domains %}"{{ domain }}",{% endfor %}]

If you look back at our variable definition, you’ll see we have a variable called domains that’s defined like this:

domains:
  - 192.168.33.10.nip.io
  - www.192.168.33.10.nip.io

Our Mezzanine app is only going to respond to requests that are for one of the hostnames listed in the domains variable: http://192.168.33.10.nip.io or http://www.192.168.33.10.nip.io in our case. If a request reaches Mezzanine but the host header is something other than those two domains, the site will return “Bad Request (400)”

We want this line in the generated file to look like this:

ALLOWED_HOSTS = ["192.168.33.10.nip.io", "www.192.168.33.10.nip.io"]

We can achieve this by using a for loop, as shown in Example 6-15. Note that it doesn’t do exactly what we want. Instead, it will have a trailing comma, like this:

ALLOWED_HOSTS = ["192.168.33.10.nip.io", "www.192.168.33.10.nip.io",]

However, Python is perfectly happy with trailing commas in lists, so we can leave it like this.

Let’s examine the Jinja2 for loop syntax. To make things a little easier to read, we’ll break it up across multiple lines, like this:

ALLOWED_HOSTS = [
{% for domain in domains %}
                 "{{ domain }}",
{% endfor %}
                ]

The generated config file looks like this, which is still valid Python.

ALLOWED_HOSTS = [
                 "192.168.33.10.nip.io",
                 "www.192.168.33.10.nip.io",
                ]

Note that the for loop has to be terminated by an {% endfor %} statement. Furthermore, the for statement and the endfor statement are surrounded by {% %} delimiters, which are different from the {{ }} delimiters that we use for variable substitution.

All variables and facts that have been defined in a playbook are available inside Jinja2 templates, so we never need to explicitly pass variables to templates.

Running django-manage Commands

Django applications use a special script called manage.py (http://bit.ly/2iica5a) that performs administrative actions for Django applications such as the following:

• Creating database tables

• Applying database migrations

• Loading fixtures from files into the database

• Dumping fixtures from the database to files

• Copying static assets to the appropriate directory

In addition to the built-in commands that manage.py supports, Django applications can add custom commands. Mezzanine adds a custom command called createdb that is used to initialize the database and copy the static assets to the appropriate place. The official Fabric scripts do the equivalent of this:

$ manage.py createdb --noinput --nodata

Ansible ships with a django_manage module that invokes manage.py commands. We could invoke it like this:

- name: initialize the database
  django_manage:
    command: createdb --noinput --nodata
    app_path: "{{ proj_path }}"
    virtualenv: "{{ venv_path }}"

Unfortunately, the custom createdb command that Mezzanine adds isn’t idempotent. If invoked a second time, it will fail like this:

TASK [initialize the database] *************************************************
fatal: [web]: FAILED! => {"changed": false, "cmd": "./manage.py createdb --noinput --nodata", "msg": "
:stderr: CommandError: Database already created, you probably want the migrate command
", "path": "/home/vagrant/.virtualenvs/mezzanine_example/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "syspath": ["/tmp/ansible_django_manage_payload_4xfy5e7i/ansible_django_manage_payload.zip", "/usr/lib/python38.zip", "/usr/lib/python3.8", "/usr/lib/python3.8/lib-dynload", "/usr/local/lib/python3.8/dist-packages", "/usr/lib/python3/dist-packages"]}

Fortunately, the custom createdb command is effectively equivalent to two idempotent built-in manage.py commands:

migrate

Create and update database tables for Django models

collectstatic

Copy the static assets to the appropriate directories

By invoking these commands, we get an idempotent task:

- name: apply migrations to create the database, collect static content
  django_manage:
    command: "{{ item }}"
    app_path: "{{ proj_path }}"
    virtualenv: "{{ venv_path }}"
  with_items:
    - syncdb
    - collectstatic

Running Custom Python Scripts in the Context of the Application

To initialize our application, we need to make two changes to our database:

• We need to create a Site model object that contains the domain name of our site (in our case, that’s 192.168.33.10.nip.io).

• We need to set the administrator username and password.

Although we could make these changes with raw SQL commands or Django data migrations, the Mezzanine Fabric scripts use Python scripts, so that’s how we’ll do it.

There are two tricky parts here. The Python scripts need to run in the context of the virtualenv that we’ve created, and the Python environment needs to be set up properly so that the script will import the settings.py file that’s in ~/mezzanine/mezzanine_example/mezzanine_example.

In most cases, if we needed some custom Python code, I’d write a custom Ansible module. However, as far as I know, Ansible doesn’t let you execute a module in the context of a virtualenv, so that’s out.

We used the script module instead. This will copy over a custom script and execute it. Lorin wrote two scripts: one to set the Site record, and the other to set the admin username and password.

You can pass command-line arguments to script modules and parse them out, but I decided to pass the arguments as environment variables instead. I didn’t want to pass passwords via command-line argument (those show up in the process list when you run the ps command), and it’s easier to parse out environment variables in the scripts than it is to parse command-line arguments.

In order to run these scripts in the context of the virtualenv, I also needed to set the path variable so that the first Python executable in the path would be the one inside the virtualenv. Example 6-16 shows how I invoked the two scripts.

- name: set the site id
  script: scripts/setsite.py
  environment:
    PATH: "{{ venv_path }}/bin"
    PROJECT_DIR: "{{ proj_path }}"
    PROJECT_APP: "{{ proj_app }}"
    WEBSITE_DOMAIN: "{{ Uve_hostname }}"
- name: set the admin password
  script: scripts/setadmin.py
  environment:
    PATH: "{{ venv_path }}/bin"
    PROJECT_DIR: "{{ proj_path }}"
    PROJECT_APP: "{{ proj_app }}"
    ADMIN_PASSWORD: "{{ admin_pass }}"

The scripts themselves are shown in Example 6-17 and Example 6-18. You can find them in the scripts subdirectory.

#!/usr/bin/env python3
""" A script to set the site domain """
# Assumes three environment variables
#
# PROJECT_DIR: root directory of the project
# PROJECT_APP: name of the project app
# WEBSITE_DOMAIN: the domain of the site (e.g., www.example.com)
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
proj_app = os.environ['PROJECT_APP']
os.environ['DJANGO_SETTINGS_MODULE'] = proj_app + '.settings'
import django
django.setup()
from django.conf import settings
from django.contrib.sites.models import Site
domain = os.environ['WEBSITE_DOMAIN']
Site.objects.filter(id=settings.SITE_ID).update(domain=domain)
Site.objects.get_or_create(domain=domain)

#!/usr/bin/env python3
""" A script to set the admin credentials """
# Assumes three environment variables
#
# PROJECT_DIR: root directory of the project
# PROJECT_APP: name of the project app
# ADMIN_PASSWORD: admin user's password
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
proj_app = os.environ['PROJECT_APP']
os.environ['DJANGO_SETTINGS_MODULE'] = proj_app + '.settings'
import django
django.setup()
from django.contrib.auth import get_user_model
User = get_user_model()
u, _ = User.objects.get_or_create(username='admin')
u.is_staff = u.is_superuser = True
u.set_password(os.environ['ADMIN_PASSWORD'])
u.save()

Note: The environment variable DJANGO_SETTINGS_MODULE needs to be set before importing django.

- name: set the gunicorn config file
  template:
    src: templates/gunicorn.conf.py.j2
    dest: "{{ proj_path }}/gunicorn.conf.py"
    mode: 0750
- name: set the supervisor config file
  become: true
  template:
    src: templates/supervisor.conf.j2
    dest: /etc/supervisor/conf.d/mezzanine.conf
    mode: 0640
  notify: restart supervisor
- name: set the nginx config file
  become: true
  template:
    src: templates/nginx.conf.j2
    dest: /etc/nginx/sites-available/mezzanine.conf
    mode: 0640
  notify: restart nginx

handlers:
  - name: restart supervisor
    become: true
    supervisorctl:
      name: "{{ gunicorn_procname }}"
      state: restarted
  - name: restart nginx
    become: true
    service:
      name: nginx
      state: restarted 

from multiprocessing import cpu_count
bind = "unix:{{ proj_path }}/gunicorn.sock"
workers = cpu_count() * 2 + 1
errorlog = "/home/{{ user }}/logs/{{ proj_name }}_error.log"
loglevel = "error"
proc_name = "{{ proj_name }}"

[program:{{ gunicorn_procname }}]
command={{ venv_path }}/bin/gunicorn -c gunicorn.conf.py -p gunicorn.pid {{ proj_app }}.wsgi:application
directory={{ proj_path }}
user={{ user }}
autostart=true
stdout_logfile = /home/{{ user }}/logs/{{ proj_name }}_supervisor
autorestart=true
redirect_stderr=true
environment=LANG="{{ locale }}",LC_ALL="{{ locale }}",LC_LANG="{{ locale }}"

{% if tls_enabled %}
...
{% endif %}

server_name {{ domains|join(", ") }};

domains:
  - 192.168.33.10.nip.io
  - www.192.168.33.10.nip.io

server_name 192.168.33.10.nip.io, www.192.168.33.10.nip.io;

upstream {{ proj_name }} {
    server unix:{{ proj_path }}/gunicorn.sock fail_timeout=0;
}
server {
    listen 80;
    {% if tls_enabled %}
    listen 443 ssl;
    {% endif %}
    server_name {{ domains|join(", ") }};
    server_tokens off;
    client_max_body_size 10M;
    keepalive_timeout    15;
    {% if tls_enabled %}
    ssl_certificate      conf/{{ proj_name }}.crt;
    ssl_certificate_key  conf/{{ proj_name }}.key;
    ssl_session_tickets off;
    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  10m;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
    ssl_prefer_server_ciphers on;
    {% endif %}
    location / {
        proxy_redirect      off;
        proxy_set_header    Host                    $host;
        proxy_set_header    X-Real-IP               $remote_addr;
        proxy_set_header    X-Forwarded-For         $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Protocol    $scheme;
        proxy_pass          http://{{ proj_name }};
    }
    location /static/ {
        root            {{ proj_path }};
        access_log      off;
        log_not_found   off;
    }
    location /robots.txt {
        root            {{ proj_path }}/static;
        access_log      off;
        log_not_found   off;
    }
    location /favicon.ico {
        root            {{ proj_path }}/static/img;
        access_log      off;
        log_not_found   off;
    }
}

Enabling the Nginx Configuration

The convention with Nginx configuration files is to put your configuration files in /etc/nginx/sites-available and enable them by creating a symbolic link to /etc/nginx/sites-enabled.

The Mezzanine Fabric scripts just copy the configuration file directly into sites-enabled, but I’m going to deviate from how Mezzanine does it because it gives me an excuse to use the file module to create a symlink. We also need to remove the default configuration file that the Nginx package sets up in /etc/nginx/sites-enabled/default.

- name: remove the default nginx config file
  become: true
  file:
    path: /etc/nginx/sites-enabled/default
    state: absent
  notify: restart nginx
- name: set the nginx config file
  become: true
  template:
    src: templates/nginx.conf.j2
    dest: /etc/nginx/sites-available/mezzanine.conf
    mode: 0640
  notify: restart nginx-54t5`5555
- name: enable the nginx config file
  become: true
    file:
    src: /etc/nginx/sites-available/mezzanine.conf
    dest: /etc/nginx/sites-enabled/mezzanine.conf
    state: link
    mode: 0777
  notify: restart nginx

As shown in Example 6-24, we use the file module to create the symlink and to remove the default config file. This module is useful for creating directories, symlinks, and empty files; deleting files, directories, and symlinks; and setting properties such as permissions and ownership.

Installing TLS Certificates

Our playbook defines a variable named tls_enabled. If this variable is set to true, the playbook will install TLS certificates. In our example, we use self-signed certificates, so the playbook will create the certificate if it doesn’t exist. In a production deployment, you would copy an existing TLS certificate that you obtained from a certificate authority.

Example 6-25 shows the two tasks involved in configuring for TLS certificates. We use the file module to ensure that the directory that will house the TLS certificates exists.

- name: ensure config path exists
  become: true
  file:
    path: "{{ conf_path }}"
    state: directory
    mode: 0755
- name: create tls certificates
  become: true
  command: >
    openssl req -new -x509 -nodes -out {{ proj_name }}.crt
    -keyout {{ proj_name }}.key -subj '/CN={{ domains[0] }}' -days 365
    chdir={{ conf_path }}
    creates={{ conf_path }}/{{ proj_name }}.crt
  when: tls_enabled
  notify: restart nginx

Note that both tasks contain this clause:

when: tls_enabled

If tls_enabled evaluates to false, Ansible will skip the task.

Ansible doesn’t ship with modules for creating TLS certificates, so we use the command module to invoke the openssl command in order to create the self-signed certificate. Since the command is very long, we use YAML line-folding syntax, with the '>' character, so that we can break the command across multiple lines.

These two lines at the end of the command are additional parameters that are passed to the module; they are not passed to the command line.

chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt

The chdir parameter changes the directory before running the command. The creates parameter implements idempotence: Ansible will first check whether the file {{ conf_path }}/{{ proj_name }}.crt exists on the host. If it already exists, Ansible will skip this task.

Installing Twitter Cron Job

If you run manage.py poll_twitter, Mezzanine will retrieve tweets associated with the configured accounts and show them on the home page. The Fabric scripts that ship with Mezzanine keep these tweets up-to-date by installing a cron job that runs every five minutes.

If we followed the Fabric scripts exactly, we’d copy a cron script into the /etc/cron.d directory that had the cron job. We could use the template module to do this. However, Ansible ships with a cron module that allows us to create or delete cron jobs, which I find more elegant. Example 6-26 shows the task that installs the cron job.

- name: install poll twitter cron job
  cron:
    name: "poll twitter"
    minute: "*/5"
    user: "{{ user }}"
    job: "{{ manage }} poll_twitter"

If you manually SSH to the box, you can see the cron job that gets installed by using crontab -l to list the jobs. Here’s what it looks like for me when I deploy as the Vagrant user:

#Ansible: poll twitter
*/5 * * * * /home/vagrant/.virtualenvs/mezzanine_example/bin/python3 /home/vagrant/mezzanine/mezzanine_example/manage.py poll_twitter

Notice the comment at the first line. That’s how the Ansible module supports deleting cron jobs by name. For example:

- name: remove cron job
  cron: 
    name: "poll twitter" 
    state: absent

If you were to do this, the cron module would look for the comment line that matches the name and delete the job associated with that comment.

The Full Playbook

Example 6-27 shows the complete playbook in all its glory.

#!/usr/bin/env ansible-playbook
---
- name: Deploy mezzanine
  hosts: web
  vars:
    user: "{{ ansible_user }}"
    proj_app: 'mezzanine_example'
    proj_name: "{{ proj_app }}"
    venv_home: "{{ ansible_env.HOME }}/.virtualenvs"
    venv_path: "{{ venv_home }}/{{ proj_name }}"
    proj_path: "{{ ansible_env.HOME }}/mezzanine/{{ proj_name }}"
    settings_path: "{{ proj_path }}/{{ proj_name }}"
    reqs_path: '~/requirements.txt'
    manage: "{{ python }} {{ proj_path }}/manage.py"
    live_hostname: 192.168.33.10.nip.io
    domains:
      - 192.168.33.10.nip.io
      - www.192.168.33.10.nip.io
    repo_url: [email protected]:ansiblebook/mezzanine_example.git
    locale: en_US.UTF-8
    # Variables below don't appear in Mezannine's fabfile.py
    # but I've added them for convenience
    conf_path: /etc/nginx/conf
    tls_enabled: true
    python: "{{ venv_path }}/bin/python3"
    database_name: "{{ proj_name }}"
    database_user: "{{ proj_name }}"
    database_host: localhost
    database_port: 5432
    gunicorn_procname: gunicorn_mezzanine
  vars_files:
    - secrets.yml
  tasks:
    - name: install apt packages
      become: true
      apt:
        update_cache: true
        cache_valid_time: 3600
        pkg:
          - acl
          - git
          - libjpeg-dev
          - libpq-dev
          - memcached
          - nginx
          - postgresql
          - python3-dev
          - python3-pip
          - python3-venv
          - python3-psycopg2
          - supervisor
    - name: create project path
      file:
        path: "{{ proj_path }}"
        state: directory
        mode: 0755
    - name: create a logs directory
      file:
        path: "{{ ansible_env.HOME }}/logs"
        state: directory
        mode: 0755
    - name: check out the repository on the host
      git:
        repo: "{{ repo_url }}"
        dest: "{{ proj_path }}"
        version: master
        accept_hostkey: true
    - name: create python3 virtualenv
      pip:
        name:
          - pip
          - wheel
          - setuptools
        state: latest
        virtualenv: "{{ venv_path }}"
        virtualenv_command: /usr/bin/python3 -m venv
    - name: copy requirements.txt to home directory
      copy:
        src: requirements.txt
        dest: "{{ reqs_path }}"
        mode: 0644
    - name: install packages listed in requirements.txt
      pip:
        virtualenv: "{{ venv_path }}"
        requirements: "{{ reqs_path }}"
    - name: create project locale
      become: true
      locale_gen:
        name: "{{ locale }}"
    - name: create a DB user
      become: true
      become_user: postgres
      postgresql_user:
        name: "{{ database_user }}"
        password: "{{ db_pass }}"
    - name: create the database
      become: true
      become_user: postgres
      postgresql_db:
        name: "{{ database_name }}"
        owner: "{{ database_user }}"
        encoding: UTF8
        lc_ctype: "{{ locale }}"
        lc_collate: "{{ locale }}"
        template: template0
    - name: ensure config path exists
      become: true
      file:
        path: "{{ conf_path }}"
        state: directory
        mode: 0755
    - name: create tls certificates
      become: true
      command: >
        openssl req -new -x509 -nodes -out {{ proj_name }}.crt
        -keyout {{ proj_name }}.key -subj '/CN={{ domains[0] }}' -days 365
        chdir={{ conf_path }}
        creates={{ conf_path }}/{{ proj_name }}.crt
      when: tls_enabled
      notify: restart nginx
    - name: remove the default nginx config file
      become: true
      file:
        path: /etc/nginx/sites-enabled/default
        state: absent
      notify: restart nginx
    - name: set the nginx config file
      become: true
      template:
        src: templates/nginx.conf.j2
        dest: /etc/nginx/sites-available/mezzanine.conf
        mode: 0640
      notify: restart nginx-54t5`5555
    - name: enable the nginx config file
      become: true
      file:
        src: /etc/nginx/sites-available/mezzanine.conf
        dest: /etc/nginx/sites-enabled/mezzanine.conf
        state: link
        mode: 0777
      notify: restart nginx
    - name: set the supervisor config file
      become: true
      template:
        src: templates/supervisor.conf.j2
        dest: /etc/supervisor/conf.d/mezzanine.conf
        mode: 0640
      notify: restart supervisor
    - name: install poll twitter cron job
      cron:
        name: "poll twitter"
        minute: "*/5"
        user: "{{ user }}"
        job: "{{ manage }} poll_twitter"
    - name: set the gunicorn config file
      template:
        src: templates/gunicorn.conf.py.j2
        dest: "{{ proj_path }}/gunicorn.conf.py"
        mode: 0750
    - name: generate the settings file
      template:
        src: templates/local_settings.py.j2
        dest: "{{ settings_path }}/local_settings.py"
        mode: 0750
    - name: apply migrations to create the database, collect static content
      django_manage:
        command: "{{ item }}"
        app_path: "{{ proj_path }}"
        virtualenv: "{{ venv_path }}"
      with_items:
        - migrate
        - collectstatic
    - name: set the site id
      script: scripts/setsite.py
      environment:
        PATH: "{{ venv_path }}/bin"
        PROJECT_DIR: "{{ proj_path }}"
        PROJECT_APP: "{{ proj_app }}"
        DJANGO_SETTINGS_MODULE: "{{ proj_app }}.settings"
        WEBSITE_DOMAIN: "{{ live_hostname }}"
    - name: set the admin password
      script: scripts/setadmin.py
      environment:
        PATH: "{{ venv_path }}/bin"
        PROJECT_DIR: "{{ proj_path }}"
        PROJECT_APP: "{{ proj_app }}"
        ADMIN_PASSWORD: "{{ admin_pass }}"
  handlers:
    - name: restart supervisor
      become: true
      supervisorctl:
        name: "{{ gunicorn_procname }}"
        state: restarted
    - name: restart nginx
      become: true
      service:
        name: nginx
        state: restarted

Playbooks can become longer than needed, and harder to maintain, when all actions and variables are listed in one file. So this playbook should be considered as a step in your education on Ansible. We’ll discuss a better way to structure this in the next chapter.

Running the Playbook Against a Vagrant Machine

The live_hostname and domains variables in our playbook assume that the host we are going to deploy to is accessible at 192.168.33.10. The Vagrantfile shown in Example 6-28 configures a Vagrant machine with that IP address.

Vagrant.configure("2") do |this|
  # Forward ssh-agent for cloning from Github.com
  this.ssh.forward_agent = true
  this.vm.define "web" do |web|
    web.vm.box = "ubuntu/focal64"
    web.vm.hostname = "web"
    # This IP is used in the playbook
    web.vm.network "private_network", ip: "192.168.33.10"
    web.vm.network "forwarded_port", guest: 80, host: 8000
    web.vm.network "forwarded_port", guest: 443, host: 8443
    web.vm.provider "virtualbox" do |virtualbox|
      virtualbox.name = "web"
    end
  end
  this.vm.provision "ansible" do |ansible|
    ansible.playbook = "mezzanine.yml"
    ansible.verbose = "v"
    ansible.compatibility_mode = "2.0"
    ansible.host_key_checking = false
  end
end

Deploying Mezzanine into a new Vagrant machine is fully automated with the provision block:

$ vagrant up

You can then reach your newly deployed Mezzanine site at any of the following URLs:

• http://192.168.33.10.nip.io

• https://192.168.33.10.nip.io

• http://www.192.168.33.10.nip.io

• https://www.192.168.33.10.nip.io

Troubleshooting

You might hit a few speed bumps when trying to run this playbook on your local machine. This section describes how to overcome some common obstacles.

fatal: Could not read from remote repository.

dig +short 192.168.33.10.nip.io

192.168.33.10

192.168.33.10 192.168.33.10.nip.io

domains:
  - 192.168.33.10.nip.io
  - www.192.168.33.10.nip.io

Deploying Mezzanine on Multiple Machines

In this scenario, we’ve deployed Mezzanine entirely on a single machine. You’ve now seen what it’s like to deploy a real application with Mezzanine.

The next chapter covers some more advanced features of Ansible that didn’t come up in our example. We’ll show a playbook that deploys across the database and web services on separate hosts, which is common in real-world deployments.