In one of the previous chapters on making various tools play well with Terraform, we already took a quick look at running infrastructure tests. Back then we used Inspec to run a test against the single EC2 instance. A few chapters forward, and now we have much more complex Terraform setup on our hands, one that is split across four repositories.

If we would consider ourselves old-fashioned traditional system administrators, we would be quite happy with what we have achieved by now. But a good software developer (and if we are doing Infrastructure as Code, then we are already software developers, regardless of our previous experience) would never leave any code without proper tests. And what we wrote in the past is nothing like a proper test.

But what should we test? We must not run tests against the production environment (the one we just configured GitLab CI for), and it is meaningless to test VPC and IAM repositories in isolation. So the only good (really good) candidate for integration tests is the application module we wrote earlier. How about we create an integration test, located in the application module repository, that would spin up an instance of this module, connect it to existing VPC and IAM configuration, and verify that it really does start a web application (the base Apache web server, in this case)?

Perhaps the most popular tool to run infrastructure tests these days is TestKitchen. The idea behind TestKitchen is to make running this kind of tests dead simple: you only need to write a single YAML configuration file that defines how to create machines (using driver) and how to test them (using verifier). After configuring, you can create, test, and destroy servers with a single kitchen test command. Initially, it was built to work with Chef, but now it has many instance of driver and verifier, distributed as Ruby gems. And you guessed it right; there is a kitchen-terraform plugin. Let's learn how to use it.

Navigate to the packt-terraform-app-module repository and create Gemfile over there with these contents:

source 'https://rubygems.org/' 
ruby '2.3.1' 
gem 'test-kitchen' 
gem 'kitchen-terraform' 

You need Ruby, rubygems and bundler gem installed before you proceed. Once you have them, simply run the bundle install command to install test-kitchen and its Terraform plugin. kitchen-terraform handles all the terraform get, terraform apply, and terraform destroy for us, we only need to create a template that it will handle. Create a test directory inside a module repository and add .kitchen.yml inside it. Be careful: it's important not to add it to the root folder of the module because, in that case, TestKitchen will try to apply the module template itself, and not a template that uses module inside it.

First of all, we need to define a driver inside this file:

--- 
driver: 
  name: terraform 

It's Terraform. This means that kitchen-terraform will be used. Next goes the provisioner. Normally, driver is responsible for creating machines (Docker, EC2, and others) and the provisioner for how to provision them (Chef, Puppet, and Ansible). In case of Terraform, we are interested only in creating, and provisioning is configured somewhere inside the Terraform templates. Because of this, the only provisioner we configure is Terraform itself:

provisioner: 
  name: terraform 

Normally, if you test a server configuration, you would want to test it for multiple platforms, such as Red Hat Linux, Debian, and so on. Again, it makes little sense in the context of Terraform. Still, TestKitchen requires us to define a platform, so let's make it happy:

platforms: 
  - name: centos 

The transport section is responsible for connecting to the machines created by TestKitchen. Generate a dummy key-pair inside the test folder and configure transport as follows:

transport: 
  name: ssh 
  ssh_key: ./test/id_rsa 

suites is a set of tests to run. We will define just one:

suites: 
  - name: default 

Finally, verifier is what TestKitchen will run to make sure that the server (or infrastructure, in our case) was created and configured correctly:

verifier: 
  name: shell 
  command: ./test.sh 
  sleep: 180 

There are multiple verifiers available, including one already familiar to us, Inspec. Lots of them are focused on testing one particular server. But when we talk about the whole infrastructure, we can't test just one server: we need to somehow verify all of it. It is especially true for our application module because it doesn't create the server directly: instead it creates an Auto-scaling groups and exposes only an endpoint that all the servers hide behind. That's why we are using shell verifier which invokes a script on the machine you run tests from: inside this script, we have full flexibility of what to test. If the script returns 1, the test has failed; if 0, it succeeded. In the case of an application module that only creates a bunch of statelesss web servers running Apache in default configuration, the whole test can consist of a check to see whether the ELB endpoint returns a standard Apache page or not:

#!/bin/bash
cd .kitchen/kitchen-terraform/$KITCHEN_SUITE-$KITCHEN_PLATFORM
hostname=$(terraform output app_endpoint)
res=$(curl $hostname | grep "Testing 123" | wc -c)
if [[ $res = "0" ]]
then
  exit 1
else
  exit 0
fi

This example is kept intentionally simple, of course. It's up to you how complex you make this test. It can even, for example, run the complete Selenium-based set of tests for a complex web application. It all depends on what exactly Terraform creates. Note this weird line:

cd .kitchen/kitchen-terraform/$KITCHEN_SUITE-$KITCHEN_PLATFORM

When you run TestKitchen, it stores the Terraform state file in a local .kitchen/kitchen-terraform directory, divided by suites and platforms. The name of the suite and a platform is available via the environment variable in any script configured for shell verifier.

The only things missing now is an actual test Terraform template. It's not much different from what we used in a production repository earlier, except that behind the curtains some of variables were removed. As an exercise, modify the module itself to fit the following template:

# ...
module "test_app" {
  source = "../"
  vpc_id = "${data.terraform_remote_state.vpc.vpc_id}"
  subnets = [
             "${data.terraform_remote_state.vpc.public-subnet-1-id}",
             "${data.terraform_remote_state.vpc.public-subnet-2-id}"
            ]
  name = "TestApp"
  keypair = "${aws_key_pair.terraform.key_name}"
  environment = "${var.environment}"
  extra_sgs = ["${aws_security_group.default.id}"]
  instance_count = 1
  iam_role = "${data.terraform_remote_state.iam.base-role-name}"
}
output "app_endpoint" {
  value = "${module.test_app.app_address}"
}

It still uses the same remote state files for IAM and VPC configuration; no need to change anything there. instance_count was changed to just 1 : we don't want our tests to be too expensive. To run these tests, we only need to run the bundle exec kitchen test command. This command will do this:

For a small template that we have will take roughly three minutes to create all of the infrastructure and then another few minutes for Puppet to do its job: hence sleep: 180 option for verifier. Clearly you want to run these tests inside a Continuous Integration server instead of doing it manually.

We looked at shell verifier, but kitchen-terraform has an extra verifier built-in. This verifier is nothing but a wrapper around Inspec, and it expects you to provide a set of IP addresses or DNS names to SSH to. For shell verifier, we wrote that we don't really need transport section, but it was left in intentionally as the first step for you to try terraform verifier out.

As a final, more difficult exercise for you to verify that you've learned Terraform and all its related tools, do the following:

If you can do this, then you've mastered Terraform and Infrastructure as Code.