6.1.1 Creating a directory structure

Let’s start by creating a directory and file structure to hold our Terraform configuration. We’ll create it under our home directory.

$ cd ~
$ mkdir -p dc/{development,production,modules/{web,api,vpc}}
$ cd dc
$ git init

We’ve created a base directory called dc and a directory for each environment we want to create—development, production—as well as a modules directory. The modules directory will contain each of the modules we’ll use in our environment. These directories will contain the module source code. We’ll upload the modules to GitHub and use that as their source so we can more easily reuse them.

Our new directory structure looks like this:

Each environment will contain environment configuration, and then each service’s module will be declared and configured. We’ll add variables and outputs for the environment, as well as for each service.

Let’s add a .gitignore file too. We’ll exclude any state files to ensure we don’t commit any potentially sensitive variable values.

$ echo "terraform.tfstate*" >> .gitignore
$ echo ".terraform/" >> .gitignore
$ git add .gitignore
$ git commit -m "Adding .gitignore file"

We’ve also created a Git repository for our data center environment. You can find it on GitHub here.

6.2.1 Develop

We develop our infrastructure in the development environment. We’ll see some patterns in this chapter for how to build each environment and how to package services using modules to ensure appropriate isolation for each environment and each service we wish to deploy.

We should ensure our code is clean, simple, and well documented. There are some good guidelines to work within to help with this:

• All code should be in version control.
• Always comment code when it requires explanation.
• Add description fields to all variables.
• Include README files or documentation for your module and their interfaces.
• Running terraform fmt and terraform validate prior to committing or as a commit hook is strongly recommended to ensure your code is well formatted and valid.

6.2.2 Plan

When you’re ready to test your infrastructure, always run terraform plan first. Use this plan to ensure the development version of your infrastructure is accurate and that any changes you’re proposing are viable. Between validation and the planning process you are likely to catch the vast majority of potential errors.

Remember from Chapter 2 that the terraform plan also allows you to save the generated plan from each run. To do this we run the terraform plan command with the -out flag.

$ terraform plan -out development-`date +'%s'`.plan

This saves the generated plan, including any differences, in a file—here, development-epochtime.plan.

We can now use this file as a blueprint for our change. This ensures that even if the configuration or state has drifted or changed since we ran the plan command, the changes applied will be the changes we expect.

$ terraform apply development-epochtime.plan

6.2.3 Apply in development

Even with a plan things can still go wrong. The real world is always ready to disappoint us. Before you apply your configuration in your production environment, apply it in your development environment. Apply it using plan output files in iterative pieces to confirm it is working correctly.

You can use scaled-down versions of infrastructure—for example, in AWS we often use smaller instance types to build infrastructure in development, in order to save money and capacity. Actually deploying your configuration further validates it and likely reduces the risk of potential errors.

6.2.4 Automation and testing

If things have worked in your development environment, then it’s time to consider promoting it to your production environment. This doesn’t mean, however, that you shouldn’t make your changes jump through a few more hoops.

You should have your code in version control so it becomes easy to pass changes through an appropriate workflow. Many people use the GitHub Flow to review and promote code. Broadly, this entails:

1. Creating a branch.
2. Developing your changes.
3. Creating a pull request with your changes.
4. Reviewing your code changes.
5. Merging your changes to master and deploying them.

As part of Step 4 above, you can also run tests, often automatically, using continuous integration tools like Jenkins or third-party services like TravisCI or CircleCI.

6.2.5 Deploy to production

Once we’re satisfied that our changes are okay, any tests have passed, and our colleagues have reviewed and approved our code, then it’s time to merge and deploy. This is another opportunity to again run terraform plan. You can never be too sure that what you’re deploying is correct.

You might even automate this process upon the merge of your code into master—for example, by using one of the CI tools we’ve mentioned or a Git hook of some kind. This is an excellent way to combine plan output files. Each commit generates a plan output file that is applied iteratively to your environments.

If this planning reveals that everything is still okay, then it’s time to run the terraform apply command and push your changes live!

6.3.1 Getting our VPC module

Before we use it we’ll need to initialize our environment and get the vpc module from GitHub. Inside the ~/dc/development directory we need to run the terraform init command and then the terraform get command to download our modules.

$ cd ~/dc/development
$ terraform init

. . .

$ terraform get
Get: git::https://github.com/turnbullpress/tf_vpc.git?ref=v0.0.4

This has downloaded the v0.0.4 tag of our vpc module and installed it in the .terraform/modules directory.

6.3.2 Adding some outputs for the development environment

Let’s add some outputs, all based on outputs of the vpc module, to the end of the main.tf file in our development environment. This will expose them for us for later use.

. . .

output "public_subnet_ids" {
  value = [module.vpc.public_subnet_ids]
}
output "private_subnet_ids" {
  value = [module.vpc.private_subnet_ids]
}
output "bastion_host_dns" {
  value = module.vpc.bastion_host_dns
}
output "bastion_host_ip" {
  value = module.vpc.bastion_host_ip
}

We’ve added four outputs from the vpc module. For example, we’ve added an output called bastion_host_dns with a value of module.vpc.bastion_host_dns to expose the bastion host’s fully qualified domain name from the vpc module.

6.3.3 Planning the development environment

We’ve passed all of our current variables into the module. If we now run plan, we should see our proposed VPC and environment. Let’s try that now.

$ terraform plan
. . .

+ module.vpc.aws_eip.environment.0
    allocation_id:     "<computed>"
    association_id:    "<computed>"
    domain:            "<computed>"
    instance:          "<computed>"
    network_interface: "<computed>"
    private_ip:        "<computed>"
    public_ip:         "<computed>"
    vpc:               "true"

. . .

Plan: 23 to add, 0 to change, 0 to destroy.

We can see that we’ll create 23 resources.

6.3.4 Applying the development environment

Now let’s apply our resources.

$ terraform apply
module.vpc.aws_vpc.environment: Creating...
  cidr_block:                "" => "10.0.0.0/16"
  default_network_acl_id:    "" => "<computed>"

. . .

Apply complete! Resources: 23 added, 0 changed, 0 destroyed.

. . .

Outputs:

bastion_host_dns = ec2-52-90-119-131.compute-1.amazonaws.com
bastion_host_ip = 52.90.119.131
private_subnet_ids = [
    subnet-5056af6c,
    subnet-f6bb74ad
]
public_subnet_ids = [
    subnet-f5bb74ae,
    subnet-f4bb74af
]

We can see that our 23 resources have been added and that Terraform has outputted some useful data about our new VPC, including details for the bastion host that will allow us to connect to any instances that we launch in the VPC.

6.3.5 Configuring remote state

Let’s configure remote state. We enable the remote state like so:

terraform {
  backend "s3" {
    region = "us-east-1"
    bucket = "examplecom-remote-state-development"
    key    = "terraform.tfstate"
  }
}

We’ve specified our remote state in an S3 bucket and can now run the terraform init command to initialize our remote state backend.

6.4.1 The web module

We’re going to build the web module inside our ~/dc/modules directory.

$ cd ~/dc/modules
$ mkdir -p web
$ cd web
$ touch {interface,main}.tf
$ mkdir -p files
$ git init

We’ve created a directory, web, and a sub-directory, files, in our modules directory. We’ve also touch’ed the base files—interface.tf and main.tf—for our module, and initialized the directory as a Git repository.

Let’s start with our module’s variables in interface.tf. We know we need to define a variable for each incoming variable used in our module block.

variable "region" {}
variable "ami" {
  default = {
    "us-east-1" = "ami-f652979b"
    "us-west-1" = "ami-7c4b331c"
  }
}
variable "instance_type" {
  default = "t2.micro"
}
variable "key_name" {
  default = "james"
}
variable "environment" {}
variable "vpc_id" {}
variable "public_subnet_ids" {
  type = list(string)
}
variable "private_subnet_ids" {
  type = list(string)
}
variable "domain" {}
variable "web_instance_count" {}
variable "app_instance_count" {}

You can see we’ve defined the inputs from our module block plus some additional variables for the Ubuntu 16.04 AMIs to use to create any EC2 instances and the instance type.

We also need to define some outputs for our module to match the outputs we specified in web.tf. We’ll add these to end of the interface.tf file.

output "web_elb_address" {
  value = aws_elb.web.dns_name
}
output "web_host_addresses" {
  value = aws_instance.web[*].private_ip
}
output "app_host_addresses" {
  value = aws_instance.app[*].private_ip
}

Finally, we’ve gotten to the heart of our module: the resources we want to create. These are contained in the main.tf file. It’s too big to show here in its entirety, but you can find it in the book’s source code. For now, let’s look at a couple specific resources.

6.4.2 Using a data source in the web module

The first resource we’re going to look at will show us a useful pattern for using data sources in modules.

data "aws_vpc" "environment" {
  id = var.vpc_id
}

. . .

resource "aws_security_group" "web_host_sg" {
  name        = "${var.environment}-web_host"
  description = "Allow SSH and HTTP to web hosts"
  vpc_id      = "${data.aws_vpc.environment.id}"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.environment.cidr_block}]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [data.aws_vpc.environment.cidr_block}]
  }

. . .

}

You’ll first note we haven’t specified an AWS provider or credentials. This is because the module will inherit the existing provider configuration provided by our development environment.

We instead have the aws_vpc data source defined as our first resource. The aws_vpc data source returns data from a specific VPC. In our web module we’ve used the VPC ID we received from the vpc module and passed into the web module.

data "aws_vpc" "environment" {
  filter {
    name   = "tag:Name"
    values = [var.environment]
  }
}

. . .
  tags = {
    Name = var.environment
  }
. . .

module.web.data.aws_vpc.environment:
  id = vpc-c96f7eae
  cidr_block = 10.0.0.0/16
  default = false
  dhcp_options_id = dopt-1ef3017b
  instance_tenancy = default
  state = available
  tags.% = 1
  tags.Name = development
. . .

cidr_block = [data.aws_vpc.environment.cidr_block]

6.4.3 Web instances

Our next resources are the AWS instances that will run our web servers.

resource "aws_instance" "web" {
  ami           = var.ami[var.region]
  instance_type = var.instance_type
  key_name      = var.key_name
  subnet_id     = var.public_subnet_ids[0]
  user_data     = file("${path.module}/files/web_bootstrap.sh")
  vpc_security_group_ids = [
    aws_security_group.web_host_sg.id
  ]
  tags = {
    Name = "${var.environment}-web-${count.index}"
  }
  count = var.web_instance_count
}

This is very similar to the EC2 resources we created earlier in the book. There are a couple of interesting features we should note.

The first feature is the selection of the subnet in which to run the instance. We’re taking advantage of a sorting quirk of Terraform to select a specific subnet for our instances. We’re passing in the list of public and private subnet IDs into the web module. These are generated in the vpc module and emitted as an output of that module. We’ve assigned them in the web module to a variable called var.public_subnet_ids.

The original subnets were created using values from our development environment’s terraform.tfvars file:

public_subnets = [ "10.0.1.0/24", "10.0.2.0/24" ]
private_subnets = [ "10.0.101.0/24", "10.0.102.0/24" ]

Each subnet is created in the order specified—here 10.0.1.0/24 and then 10.0.2.0/24. The resulting subnet IDs are also outputted in the order they are created from the vpc module.

public_subnet_ids = [
    subnet-f5bb74ae,
    subnet-f4bb74af
]

We select the first element in the var.public_subnet_ids and know we’re getting the 10.0.1.0/24 subnet and so on.

The second interesting feature is contained in the user_data attribute for our instance:

user_data = file("${path.module}/files/web_bootstrap.sh")

The user_data uses the file function to point to a file inside our module. To make sure we find the right path, we’ve used the path variable to help us locate this file. We briefly saw the path variable in Chapter 3.

The path variable can be suffixed with a variety of methods to select specific paths. For example:

• path.cwd for current working directory.
• path.root for the root directory of the root module.
• path.module for the root directory of the current module.

Here we’re using path.module, which is the root directory of the web module. Hence our web_bootstrap.sh script is located in the files directory inside the root of the web module.

6.4.4 A Cloudflare record

We’ve also specified a resource to manage our Cloudflare record. Let’s take a look at that now.

resource "cloudflare_record" "web" {
    domain = var.domain
    name   = "${var.environment}.${var.domain}"
    value  = aws_elb.web.dns_name
    type   = "CNAME"
    ttl    = 3600
}

The cloudflare_record resource creates a DNS record—in our case a CNAME record. We’ve created our CNAME record by joining our environment name, development, with the domain name, turnbullpress.com, we specified for the var.domain variable in our ~/dc/development/web.tf file. Our CNAME record points to the DNS name of our AWS load balancer provided by the variable aws_elb.web.dns_name. Specifying this variable guarantees our Cloudflare record will be created after our load balancer in the dependency graph.

6.4.5 Committing our module

Finally, we commit and add our module to GitHub.

$ git add .
$ git commit -a -m "Adding initial module"

We would then create a GitHub repository to store our module and push it.

6.4.6 Getting our web module

Now that we’ve added our module to the development environment, we need to get it.

$ cd ~/dc/development
$ terraform get
Get: git::https://github.com/turnbullpress/tf_vpc.git?ref=v0.0.4
Get: git::https://github.com/turnbullpress/tf_web.git

We’ve now downloaded the web module to join our vpc module.

6.4.7 Planning our web service

If we again run terraform plan, we’ll see some new resources to be added.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

module.vpc.aws_vpc.environment: Refreshing state... (ID: vpc-782c281f)
module.vpc.aws_eip.environment[0]: Refreshing state... (ID: eipalloc-52336f6d)
module.vpc.aws_eip.environment[1]: Refreshing state... (ID: eipalloc-bf366a80)

. . .

Plan: 9 to add, 0 to change, 0 to destroy.

Terraform has refreshed our state to check what’s new and has identified nine resources, the contents of our web module, that will be added to the development environment.

6.4.8 Applying the web service

When we run terraform apply we’ll get our first service installed in the development environment.

$ terraform apply
module.vpc.aws_vpc.environment: Refreshing state... (ID: vpc-782c281f)

. . .

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

. . .

Outputs:

app_host_addresses = [
    10.0.101.238,
    10.0.101.174
]
bastion_host_dns = ec2-52-90-119-131.compute-1.amazonaws.com
bastion_host_ip = 52.90.119.131
private_subnet_ids = [
    subnet-5056af6c,
    subnet-f6bb74ad
]
public_subnet_ids = [
    subnet-f5bb74ae,
    subnet-f4bb74af
]
web_elb_address = development-web-elb-1020554483.us-east-1.elb.amazonaws.com
web_host_addresses = [
    10.0.1.56,
    10.0.1.233
]

Terraform has created our new resources and outputted their information, in addition to the previous vpc outputs. We can also see our new Cloudflare domain record has been created.

module.web.cloudflare_record.web: Creating...
  domain:   "" => "turnbullpress.com"
  hostname: "" => "<computed>"
  name:     "" => "development.turnbullpress.com"
  proxied:  "" => "false"
  ttl:      "" => "3600"
  type:     "" => "CNAME"
  value:    "" => "development-web-elb-1020554483.us-east-1.elb.amazonaws.com"
  zone_id:  "" => "<computed>"
module.web.cloudflare_record.web: Creation complete

This shows us that our second provider is working too.

6.4.9 Testing our web service

Let’s try to use a couple of those resources now. We’ll browse to our load balancer first.

Let’s also sign in to our bastion host, 52.90.119.131, and then bounce into one of our web server hosts.

$ ssh -A [email protected]
The authenticity of host '52.90.119.131 (52.90.119.131)' can't be established.
ECDSA key fingerprint is SHA256:g/Jfap5CgjZVEQoxDsAVDMILToEHfY/mQ13mzLjXJe8.
Are you sure you want to continue connecting (yes/no)? yes

. . .

ubuntu@ip-10-0-1-224:~$

We’ve connected to the bastion host, specifying the -A flag on the SSH command to enable agent forwarding so we can use the james key pair on subsequent hosts.

From the bastion host we should be able to ping one of our web hosts and SSH into it by using its private IP address in the 10.0.1.0/24 subnet.

ubuntu@ip-10-0-1-224:~$ ping 10.0.1.56
PING 10.0.1.56 (10.0.1.56) 56(84) bytes of data.
64 bytes from 10.0.1.56: icmp_seq=1 ttl=64 time=1.17 ms
64 bytes from 10.0.1.56: icmp_seq=2 ttl=64 time=0.613 ms
64 bytes from 10.0.1.56: icmp_seq=3 ttl=64 time=0.598 ms
^C
ubuntu@ip-10-0-1-224:~$ ssh [email protected]

. . .

ubuntu@ip-10-0-1-56:~$

Excellent! It all works and we’re connected. We now have a running service inside a custom built VPC that’s secured and can be managed via a bastion host. What’s even more awesome is that this is totally repeatable in other environments.

6.4.10 Removing our web service

If we want to remove the web service, all we need to do is remove the web.tf file from the ~/dc/development directory and run terraform apply to remove the web service resources. If we can remove the file and run terraform plan we can see the proposed deletions.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

. . .

- module.web.aws_elb.web
- module.web.aws_instance.app[0]
- module.web.aws_instance.app[1]
- module.web.aws_instance.web[0]
- module.web.aws_instance.web[1]
- module.web.aws_security_group.app_host_sg
- module.web.aws_security_group.web_host_sg
- module.web.aws_security_group.web_inbound_sg

Plan: 0 to add, 0 to change, 8 to destroy.

But instead of removing these resources, let’s finish our development environment by adding our second service, an API service.

6.5.1 The API module

Our api module will create five AWS instances, a load balancer in front of our instances, and appropriate security groups to allow connectivity. We’re not going to create each individual file; rather we’re going to check out the existing module from GitHub.

$ cd ~/dc/modules
$ git clone https://github.com/turnbullpress/tf_api.git api

The resulting file tree will look like:

6.5.2 API instances

Let’s take a peak at the aws_instance resource in our api module’s main.tf file to see how we’ve used our new count variable.

resource "aws_instance" "api" {
  ami           = var.ami[var.region]
  instance_type = var.instance_type
  key_name      = var.key_name
  subnet_id     = var.public_subnet_ids[1]
  user_data     = file("${path.module}/files/api_bootstrap.sh")

  vpc_security_group_ids = [
    aws_security_group.api_host_sg.id,
  ]  ]

  tags gs {
    Name = "${var.environment}-api-${count.index}"
  }

  count = var.api_instance_count
}

Our aws_instance.api resource looks very much like our earlier instance resources from the web module. We’ve specified a different subnet, the second public subnet in our VPC: 10.0.2.0/24.

We’ve also used the var.api_instance_count variable as the value of the count meta-argument. This allows us to avoid hard-coding a number of API servers in our configuration. Instead we can set the value of this variable, either in an terraform.tfvars file, or via one of the variable population methods we explored in Chapter 3. This means we can raise and lower the number of instances in our API cluster simply and quickly.

6.5.3 The API service outputs

Back in our ~/dc/development/api.tf file we’ve also specified two outputs:

output "api_elb_address" {
  value = module.api.api_elb_address
}
output "api_host_addresses" {
  value = [module.api.api_host_addresses]
}

One will return the DNS address of the API service load balancer, and one will return the host addresses for the API servers.

6.5.4 Getting our module

In order to use our API module, we need to get it.

$ cd ~/dc/development
$ terraform get -update
Get: git::https://github.com/turnbullpress/tf_api.git (update)
Get: git::https://github.com/turnbullpress/tf_vpc.git?ref=v0.0.4 (update)
Get: git::https://github.com/turnbullpress/tf_web.git (update)

Like our web module, we’re pulling the HEAD of the Git repository, rather than a tag or commit.

6.5.5 Planning the API service

With the api module in place we can then plan the service.

$ terraform plan

. . .

+ module.web.aws_elb.web
    availability_zones.#:                   "<computed>"
    connection_draining:                    "false"
    connection_draining_timeout:            "300"
    cross_zone_load_balancing:              "true"

. . .

Plan: 8 to add, 0 to change, 0 to destroy.

You can see we’ll add eight resources to our existing infrastructure.

6.5.6 Applying the API service

Let’s do that now by applying our configuration.

$ terraform apply
module.vpc.aws_vpc.environment: Refreshing state... (ID: vpc-782c281f)
module.vpc.aws_eip.environment.1: Refreshing state... (ID: eipalloc-bf366a80)

. . .

Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

. . .

Outputs:

api_elb_address = development-api-elb-1764248438.us-east-1.elb.amazonaws.com
api_host_addresses = [
    10.0.2.120,
    10.0.2.172,
    10.0.2.64,
    10.0.2.72,
    10.0.2.239
]

. . .

Now we have two services running in our development environment. We can see that our API service has created a new load balancer and five API hosts located behind it in our 10.0.2.0/24 public subnet.

6.5.7 Testing our API service

Let’s test our API is working by curl’ing an API endpoint on the load balancer.

$ curl development-api-elb-1764248438.us-east-1.elb.amazonaws.com/api/users/1
{
  "id": 1,
  "name": "user1"
}

Our API is very simple and only has one endpoint: /api/users. With /api/users/1 we can return a single user entry, or with /api/users we can return all of the users in our API’s database.

We can also test the API is working by visiting the load balancer via browser.

Finally, we can easily remove our API service by removing the api.tf file and running terraform apply to adjust our configuration. We can also adjust the API instance count to grow or shrink our pool of API instances.

6.6.1 Adding services to production

We can now add services, such as our web service, to the environment by duplicating—and potentially adjusting—the web.tf file from our development environment. With the core of our modules driven by variables, we’re likely to only need to adjust the scale and size of the various services rather than their core configuration.

$ cd ~/dc
$ cp development/web.tf production/
$ vi production/web.tf
. . .

We would wrap this whole process in the workflow we introduced at the beginning of the chapter to ensure our changes are properly managed and tested.