Immutable Infrastructure

In technology, an object is immutable if it can’t be changed after it’s created. The only way to update an immutable object is to destroy it and create a new one. Things that are immutable contain behaviour and structures that are easier to predict and reproduce because they don’t change. This can make activities like testing and replication of these objects easier.

The immutable infrastructure principle is an application of the immutability property to infrastructure components. For example, suppose we were to setup and install a network load balancer with a set of defined routes. Subscribing to the property of immutability means that once it’s created, we can’t change the routes that have been setup. Instead, we’ll need to destroy the load balancer and create a new one.

The main advantage of adhering to the immutability principle for our infrastructure design is to create predicable and reproducible infrastructures that are easy to stand up. Years ago, it was common place for operators to mutate or patch systems in order to make deployed applications work. Servers and devices would be kept continually running and alive. The operator’s job was to shape the environment of a server to create a state that an application could safely run in. As long as all the servers had the same running state, the application should run the same across all of them.

But, over time, as more changes are applied (often inconsistently) the state of these systems drift. It becomes increasingly difficult to keep all of the servers running in the same state. It also becomes increasingly difficult to introduce new servers or make changes as it becomes increasingly challenging to predict how the environment actually work. Instead, we can use an immutable infrastructure to ensure that all of our environments are easy to replicate and change, because we know exactly how they have been created and the configuration required to create their run-time state.

Our decision to build our Microservices architecture in the cloud is what makes it possible for us to embrace this principle. In the past, the cost of hardware acquisition, server management and software procurement would make this approach unfeasible. But, a cloud deployment means that we don’t need to perform these activities ourselves. Instead, all of our infrastructure components are virtual. Virtuality means we can treat the infrastructure the same way we treat software. It gives us the freedom to create and destroy servers and devices in the same way as we might do with a software component or an object in a object oriented system.

Immutable infrastructure will help us avoid server drift and improve our ability to replicate and instantiate new environments with a production-like state. But we’ll still need a way of defining all of our infrastructure with a manageable set of configurations. That’s where the principle of Infrastructure as Code can help.

Infrastructure as Code

The principle of Infrastructure as Code (or IaC) is based on one powerful constraint: all of the changes you want to make to your infrastructure must be managed as a set of machine-readable files (or code). Teams that embrace IaC can point to a group of files that define the target state for their infrastructure. That means they can re-create an environment by re-applying the code that created it. Managing the infrastructure code becomes a way of managing the infrastructure state. Ultimately adopting the principle of Infrastructure as Code means we can manage changes to our environments by managing the way we write, test and deploy our infrastructure code.

The great thing about implementing an IaC practice is that we get to apply the practices we use for managing application changes to our infrastructure. This is important for the Microservices solution we are building. Managing changes to infrastructure components is difficult work. You need to deal with a diverse collection of management consoles, products, configuration formats, hardware connectors and estate planning. But, if we can treat all of those components as if they are parts of a software system, we can minimize the complexity of our change management work.

Adopting an Infrastructure as Code practice can also help us deploy a smaller team to manage the system. Turning the task of infrastructure management into a software-like discipline radically changes the skill-set requirements for operations work. There is still a need to understand the complexities of network design, server and resource management and security. But, by removing the operational complexities of managing a diverse set of devices and machines, that job can be performed by people in our team who understand both software development. It becomes another software based domain for an engineer to add to their repertoire.

To get going with an IaC approach, we’ll need a tool that will allow us to define the changes we want to make as machine readable code files. That tool wil also need to interpret our infrastructure as code files and apply them to a target environment. Years ago, we might have had to build this tooling ourselves, but now there are lots of tools available that can do this work for us. For our example project, we are going to use Hashicorp’s Terraform to define our changes and apply them to our cloud-based environment.

Continuous Integration and Continuous Delivery

Immutability and Infrastructure as Code give us a powerful way of reducing uncertainty in our environment builds. But, they don’t specifically address the problems of managing the risks that we will break the environment with a bad change. For example, what happens if a small change to our network inadvertently brings down a load balancer in production and our Microservices are no longer available? Or a change intended only for development accidentally makes its way to production and causes an outage?

One way to mitigate these risks is to do a lot of checking (and double-checking) of every change that we want to make in production. But, the problem with this approach is that it slows down our rate of change because of all the validation work we’d need to do. It can also lead to the late discovery of problems that we should have found a lot earlier in our infrastructure design and development work. We end up spending a lot of time in a testing phase where we need to fix a large batch of problems that could fundamentally alter our infrastructure plan.

Instead, we can apply the DevOps software practices of Continuous Integration and Continuous Delivery. Instead of scheduling a big testing effort right before we change production, we’ll continuously integrate our changes into our repository, test that those changes work and automatically deliver safe changes to a target environment. Ideally, we should get into a cadence of releasing small, testable changes instead of a big batch at once.

CI/CD practices rely heavily on tooling and automation - that’s because automation allows teams to run tests against their code faster, better and safer than doing it by hand (or by eyeballing). Teams that deliver software applications typically use lots of different tools across the domains of code management, testing, security, issue tracking, release management and workflow management. There are lots of different activities that can take place as part of a CI/CD process. These activities are typically defined as steps, within a “pipeline” that is triggered when a code change is made. Most teams use a CI/CD tool to define, manage and monitor their pipelines.

We’ll be defining a CI/CD pipeline for our infrastructure build. Later in Chapter {X} we’ll also define a CI/CD pipeline for our Microservices. For this book, we’ve chose to use Github’s Actions as our CI/CD pipeline. We’ve chosen to use Github as a platform because it’s a popular choice for open source projects and most developers already have a github account. Github’s “Actions” feature is relatively new and are not as feature rich as more established options such as Jenkins or Gitlab’s. But, being able to use a single tool to manage our code and setup our CI/CD pipeline is an attractive feature - especially when we are constrained by the real estate of a printed page.

When we build our Infrastructure as Code Pipeline, we’ll need to automate the work of testing our Terraform based “code” before we apply it to an environment. We’ll be using a Terraform’s in-built testing features to validate and test our Terraform files before we apply them. We can use these tools to progressively increase our test coverage as we move through the IaC Pipeline. Terraform will allow us to automatically validate the format, syntax and grammar of a proposed change. These tests are relatively quick and will not require us to make any changes to an environment.

Once we’ve applied the Terraform tests, our code changes will be ready to be deployed into a real Cloud environment. The final step in our IaC Pipeline will be to release a new version of our Microservices environment. By the end of this chapter, we’ll have setup an IaC pipeline that can spin up a simple Terraform-based “sandbox” environment in the Cloud. But, the first step will be to install Terraform and the other tools we’ll need for infrastructure as code development.

Set up Github

The first thing we’ll need is a way to manage and release our code. We’ll be using git for code management and Github as a host. There are plenty of great options available for git hosting, GitLab being one of the most popular alternatives. We’re using Github for our up and running solution so that we can also take advantage of Github’s bundled DevOps pipeline called Actions. But, as always, we strongly encourage you to choose the tools that make sense for you when you build your own Microservices architecture.

In order to work with our examples, you’ll need to register for a Github account and you’ll also need a local copy of a Git client. Git is an incredibly popular source control tool, so chances are that you already have it installed in your machine and you are familiar with how to use it.

But, just in case Git is new to you, we recommend that you start by taking a look at Scott Chacon and Ben Straub’s book called “Pro Git” which they’ve graciously made available to read for free on the web. You can also visit Github’s “Git Handbook” if you are just looking for quick overview of what Git is and why it’s useful.

If you don’t have the Git client installed, visit https://git-scm.com/downloads and follow the instructions to download the appropriate version.

In addition to the Git client, you’ll also need to register for an account at Github if you don’t already have one. Github is one of many Git hosting providers who provide a place to store Git repositories as well as a lot of additional functions to manage Git based projects and releases. As we mentioned earlier, we’ll be using Github to host our code repositories and also as our CI/CD tool.

If you don’t have a Github account already, you can register for a free account at https://github.com/join.

We’ll be using Git repositories to manage our infrastructure as code, just like we manage code for an application. We’ll be writing our code in a language that Hashicorp’s Terraform tool will understand. So, we’ll need to install a copy of the Terraform client in our workspace.

Install Terraform

Terraform and its declarative way of describing changes gives us an effective way of achieving our principles of infrastructure as code. We’ll be using Terraform from inside of a Github Action pipeline, so technically, you don’t need a local copy of terraform running on your machine. In theory you could just let Github run terraform whenever you commit a new infrastructure file into the repository.

But, in practice this is a difficult way to write good code and troubleshoot problems. The feedback loop for doing a commit and waiting for a pipeline to automatically test your changes is too long. Ideally, you should be able to create and perform early testing of your Terraform code before you commit it into the repository and continuous integration pipeline. So, its worthwhile installing a local copy of Terraform for your local environment.

At the time of writing, Terraform is available to run on the following platforms: * OS/X * FreeBSD * Linux * OpenBSD * Solaris * Windows

Visit https://www.terraform.io to download the client of your choice and install it on your machine. We used version 0.12.20 for all of the examples in this book. We’ll leave it to you to follow the instructions for the platform you’ve chosen.

Once you have completed the installation, run the following command to make sure Terraform is setup correctly:

$ terraform version

You should see something like the following:

Terraform v0.12.20

We’ll be using Terraform based code to provision resource in a Cloud infrastructure. In particular, we’ll be creating resources in an Amazon Web Services (AWS) cloud instance. But, before we can do that, we’ll need to create an AWS account.

Set up an AWS Operations Account

By the end of this chapter, we’ll have a pipeline that can deploy infrastructure into AWS automatically. Sticking to our principles of automation and immutability, we should never have to manage our AWS infrastructure by making changes directly through the browser. But, to start off with, we’ll need to perform a few steps manually to get our automation system up and running. We need to configure a set of credentials and permissions to allow our tools to work with our AWS objects.

In AWS, users, groups and permissions are all managed within the Identity and Access Management (IAM) service. We’ll need to create a special user that represents our tooling and define a set of permissions for what our tools can do. We’ll use this user identity whenever we are making calls from our CI/CD pipeline platform. As we mentioned earlier, we’ll be using Terraform as our primary IaC tool. Follow the steps in this section to create a Terraform user in AWS that will allow us to make the kinds of changes we’ll need for our Microservices environment.

Login to your AWS management console at http://aws.amazon.com. Once you’ve logged in, you should be presented with a list of AWS services. Find and select the IAM service - it’s usually found in the Security, Identity & Compliance section.

Figure 4-2. Select IAM

Select the Users link from the IAM navigation menu on the left-hand side of the screen. Click the “Add user” button to start the IAM user creation process.

Figure 4-3. Add User

Enter ops-account as the AIM user name in the User name field. We want our tools to access AWS through the platofrm’s management API, so select Programmatic Access as the AWS Access type. When you are done, click the Next: Permissions button to start defining access permissions.

Figure 4-4. Enter User Details

Click the Next: Tags button and then click the Next: Review button (we don’t need to define any tags at the moment.) On the review screen if everything looks OK, click the Create user button.

If everything has gone correctly, you should now see a screen that looks something like this:

Figure 4-5. Review User

Before we do anything else, we’ll need to make a note of our new user’s secret access key. Click the Show link and copy and paste both the “Access key ID” and the “Secret access key” into a temporary file. We’ll use both of these later in this section with our automated pipeline. Be careful with this key material as it will give whoever has it an opportunity to create resources in your AWS environment at your expense.

We have now created a user called ops-account with permission to make IAM changes. That gives us all that we need to transition from using the browser based console over to the AWS CLI application that we installed earlier. The first thing we’ll need to do is configure the CLI to use the Ops user we’ve just created.

Configure The AWS CLI

There are three ways to manage major cloud provider configurations: web browser, web based APIs and a command line interface. We’ve already used the web browser to create our operator account, later we’ll be using Terraform to configure changes via the AWS APIs. But, we’ll need to make some more changes before Terraform can make AWS API calls on our behalf. For that we’ll use the AWS command line interface (or CLI).

Using the CLI makes it a lot easier for us to describe the changes you need to make. It’s also less prone to the changes that user interfaces go through. But, to use the CLI, the first thing we’ll need to do is install it into our local working environment. Navigate to https://aws.amazon.com/cli/ and follow the instructions on the page to install the CLI onto your local system.

Once it’s ready, the first thing we’ll do is configure the CLI so it can access our instance. Run the aws configure command as shown in Example 4-1. You can replace the default region name with an AWS region that is closer to you. A full list of AWS regions is available at https://docs.aws.amazon.com/general/latest/gr/rande.html.

$ aws configure
AWS Access Key ID [****************AMCK]: AMIB3IIUDHKPENIBWUVGR
AWS Secret Access Key [****************t+ND]: /xd5QWmsqRsM1Lj4ISUmKoqV7/qr4u4ZrpP0KOLK
Default region name [None]: eu-west-2
Default output format [None]: json

You can test that you’ve configured the CLI correctly by listing the user accounts that have been created. Run the iam list-users command to test your setup:

$ aws iam list-users
{
    "Users": [
        {
            "Path": "/",
            "UserName": "admin",
            "UserId": "AYURIGDYE7PXW3QCYYEWM",
            "Arn": "arn:aws:iam::842218941332:user/admin",
            "CreateDate": "2019-03-21T14:01:03+00:00"
        },
        {
            "Path": "/",
            "UserName": "ops-account",
            "UserId": "AYUR4IGBHKZTE3YVBO2OB",
            "Arn": "arn:aws:iam::842218941332:user/ops-account",
            "CreateDate": "2020-07-06T15:15:31+00:00"
        }
    ]
}

If you’ve done everything correctly you should see a list of your AWS user accounts. That means AWS CLI is working properly and has access to your instance. Now, we can move on to setting up the permissions that our operations account will need.

Setup AWS Permissions

When we created our ops-account user we attached an IAM policy to it that only gives it permission to modify IAM settings. But, our ops account will need a lot more permissions than that to manage the AWS resources we’ll need for our infrastructure build. In this section, we’ll use the AWS command line tool to create and attach additional permission policies to the ops account.

The first thing we’ll do is make the ops-account part of a new group called Ops-Accounts. That way we’ll be able to assign new users to the group if we want them to have the same permissions. Use the following command to create a new group called Ops-Accounts

$ aws iam create-group --group-name Ops-Accounts

On success, the AWS CLI will display the group that has been created:

{

    "Group": {
        "Path": "/",
        "GroupName": "Ops-Accounts",
        "GroupId": "AGPA4IGBHKZTGWGQWW67X",
        "Arn": "arn:aws:iam::842218941332:group/Ops-Accounts",
        "CreateDate": "2020-07-06T15:29:14+00:00"
    }
}

Now, we just need to add our user to the new group. Use the following command to do that:

$ aws iam add-user-to-group --user-name ops-account --group-name Ops-Accounts

Next, we need to associate a set of permissions with our Ops Account group so that our user will be able to work with AWS resources in the way we expect. In practice, you’d likely need to change permissions for your Ops user as you go through the process of designing your infrastructure. In this book, we’ve already done the design work ahead of time, so we know exactly which policies we’ll need to set.

Run the following command to attach all the policies we’ll need to the Ops-Accounts group:

$ aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/IAMFullAccess &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonEKSClusterPolicy &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonEKSServicePolicy &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess &&
aws iam attach-group-policy --group-name Ops-Accounts
 --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess &&

In addition to the out of the box policies that AWS provides, we’ll also need some special permissions to work with the AWS Kubernetes service called EKS. There isn’t an existing policy that we can attach for the permissions we need, so we’ll need to create our own customer policy and attach it to our user group.

To do this, create a file called custom-eks-policy.json and populate it with the code in listing Example 4-2.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "eks:DescribeNodegroup",
        "eks:DeleteNodegroup",
        "eks:ListClusters",
        "eks:CreateCluster"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "eks:*",
      "Resource": "arn:aws:eks:*:*:cluster/*"
    }
  ]
}

Now, run the following command to create a new policy named “EKS-Managment” based on the JSON we’ve just created:

$ aws iam create-policy --policy-name EKS-Management
 --policy-document file://custom-eks-policy.json

If the command was succesfull, you’ll see a JSON representation of the new policy:

{
    "Policy": {
        "PolicyName": "EKS-Management",
        "PolicyId": "ANPA4IGBHKZTP3CFK4FAW",
        "Arn": "arn:aws:iam::[some_number]:policy/EKS-Management",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2020-07-06T15:50:26+00:00",
        "UpdateDate": "2020-07-06T15:50:26+00:00"
    }
}

With the new policy created, all that’s left is to attach it to our user group. Run the following command, substituting your new policy’s ARN for the token we’ve called {YOUR_POLICY_ARN}:

$ aws iam attach-group-policy --group-name Ops-Accounts --policy-arn {YOUR_POLICY_ARN}

You now have an ops-account user that has the permissions needed to automatically create AWS infrastructure resources for us. We’ll be using this user account when we write our Terraform code and when we configure the infrastructure pipeline. Make sure you keep the access key and secret somewhere handy (and safe) as we’ll need it later.

We have one last bit of setup to take care of before we can get to work building the pipeline: the creation of an AWS S3 storage bucket for Terraform to store state.

Create an S3 Backend for Terraform

Terraform is powerful because it allows us to declare what an infrastructure should look like, rather than defining the specific steps needed to reach that state. Terraform works it’s magic by making the right changes to an environment to make it look the way we’ve described. But, in order to do that, Terraform needs to keep track of what the environment looks like and the last operations it’s performed. Terraform keeps track of all that information in a JSON based state file that is read and updated every time it is run.

By default, Terraform will keep this state file in your local filesystem. But, in practice storing the state file locally is problematic. State often needs to be shared across machines and users so that an environment can be managed in multiple places. However, local state files are difficult to share and you can easily find yourself dealing with state conflicts and synchronisation issues.

Instead, we’ll use the AWS S3 service to store the Terraform state file. Terraform comes with out of the box support for using S3 as a state backend. All we’ll need to do is create a new “bucket” for the data and make sure we have the correct permissions set for our ops user account.

To create a bucket, you’ll need to come up with a unique name for it and pick the region that it should reside in. You should have already selected a default region when you configured the AWS CLI and we suggest that you use the same region for the S3 bucket. You can find more information about S3 bucket regions in the AWS documentation at https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro.

Once you’ve decide on a name and region, run the following command (replacing the tokenized parameters with your values):

$ aws s3api create-bucket --bucket {YOUR_S3_BUCKET_NAME} 
> --region {YOUR_AWS_REGION} --create-bucket-configuration 
>  LocationConstraint={YOUR_AWS_REGION}

If everything has gone well, you should see something that looks like this indicating that the bucket has been created:

{
    "Location": "http://my-msur-test-bucket.s3.amazonaws.com/"
}

This means the bucket has been successfully created and it’s been assigned it’s own unique URL. By default, S3 buckets aren’t publicly accessible. That’s a good thing because we don’t want just anyone to be able to see and change our Terraform state file. However, we’ve already given our ops account user full permissions to the S3 service, so it is ready for use.

With this final step complete, we now have an AWS user called ops-account configured to create, edit and delete resources in AWS. We’ve also given it permissions to store objects in a special S3 bucket we’ve created just for managing Terraform state. This should be the last time we make manual operator changes to our AWS instance - from here on out we’ll only make changes through code and with an automated pipeline!

Create the Sandbox Repository

We’ve already mentioned in the beginning of this chapter that we’ll be using Git and Github to manage our infrastructure code. If you’ve been following along, you’ll already have a local copy of Git installed and a Github account ready to be used. We’re going to use both of those tools to create a new repository for our Sandbox environment.

We’ve made a decision to give each environment is its own repository with the code and pipeline bundled within it. We like this approach because it gives teams more independence in how they manage the environments they want to create, while still keeping the pipeline configuration and code together for easier management.

We’ll use Github’s browser based interface to create the sandbox repository. Although there is a Github CLI application available, It will be quicker and easier to use the web based interface to create our new repository. Later, we’ll also be using Github’s browser interface to run and monitor the pipeline.

To create the repository, open your browser and navigate to https//github.com/new. If you haven’t already logged in to your Github account, you’ll be prompted to enter your login credentials. Once that is done, you’ll be presented with a form to create a new repository. Give your new repository the name env-sandbox and make sure to choose “Terraform” from the “Add .gitignore” drop down at the bottom of the screen, as shown in Figure 4-6.

Figure 4-6. Create Github Sandbox Repository

It’s important that we ask Github to add a .gitignore for Terraform to the module because it will make sure we don’t accidentally commit Terraform’s hidden working files to our module. If you’ve missed this step you can always add this file later by copying the source from https://raw.githubusercontent.com/github/gitignore/master/Terraform.gitignore.

It’s possible to write code using Github’s browser based text editor, but it’s not a great idea if you are doing real work. Instead, we’ll clone this repository into a local development environment so we can use our own tools. We’ll leave it to you to create a clone of your env-sandbox repository in your own local development environment.

Thats all we need to do with Github for now. We’ll need to come back to this browser based interface later when we work on the pipeline activities. But with the local clone you’ve created, we can move on to working the Terraform code that belongs inside of it.

Understanding Terraform

We mentioned earlier that we’ll be using Terraform as our tool of choice for declaratively coding our infrastructure foundation. The terraform does a lot of complicated work to make changes that match a declared state. But, it’s surprisingly easy to get started with and the language it uses is fairly intuitive. That makes it a great fit for our architecture and our goal of getting a system running as quickly as possible.

Terraform files are configured in a data format called HCL, which was invented by HashiCorp (the company that created Terraform). HCL is similar to JSON, with a few adaptations and improvements. If you’re used to JSON, the biggest difference you’ll notice is that HCL doesn’t use a “:” delimiter between key and value pairs. Instead, key and values are just separated by a white space or an “=”, depending on the context. There are some other minor improvements such as comments and multi-line strings as well. In our experience, it’s an easy language with a very low learning curve if you’ve used JSON or YAML format in the past.

In addition to understanding HCL, it’s useful to understand four of the key Terraform concepts: backends, providers, resources and modules:

Backend

Terraform needs to maintain a state file so that it knows what kinds of changes to make to the infrastructure environment. A backend is the location of that state file. By default this is located in the local file system. But, we’ll be using an AWS S3 bucket which we configured earlier.

Resource

A resource is an object that represents a thing that you are declaring a state for. Terraform does the work of making the changes to bring the resource to that state.

Provider

A Terraform provider is a packaged library of resources that you can use in your code. We’ll be using Terraform’s AWS provider for most of our work. The nice thing about Terraform is that you can use it for lots of different cloud platforms and infrastructure environments - you just need to specify the provider you plan to use.

Module

Terraform modules are similar to functions or procedures in a regular programming language. They give you a nice way of encapsulating your HCL code in a reusable, modular way.

There’s a lot more to Terraform that what we’ve described here, but this is enough knowledge for us to get started with our environment build work. If you want to go deeper, the Terraform’s documentation is a great place to start.

Our next step is to write some Terraform code that will help us build a Sandbox environment.

Writing the Sandbox Code

Our goal in this chapter is to setup the tooling and infrastructure for our environment build, so we won’t be writing a complete Terraform file that defines our infrastructure until the next chapter. For now, we’ll need to create a simple, “starter” file to test our Terraform based tool chain.

The Terraform CLI tool works by looking for files it recognizes in the working directory where it’s run. In particular, it looks for a file called main.tf and will parse that file and apply changes based on it’s contents. You can only have one main.tf file in a single directory, so that means we’ll need to have a directory dedicated to our sandbox environment and we’ll need to create a Terraform main.tf file that will describe its target state.

We’ve already created a Git repository for the sandbox environment, so that’s the directory we’ll use for the Terraform code. Let’s get started by creating a new file called main.tf in the local Sandbox git repository. Populate it with the HCL code in Example 4-3.

terraform {
  backend "s3" {
    bucket = "{YOUR_S3_BUCKET_NAME}"
    key    = "terraform/backend"
    region = "{YOUR_AWS_REGION}"
  }
}

locals {
  env_name         = "sandbox"
  aws_region       = "<strong>region code</strong>"
  k8s_cluster_name = "ms-cluster"
}

The HCL snippet you’ve just written let’s Terraform know that we are using an S3 bucket to store our backend state. It also defines a set of local variables using a Terraform construct called locals. These are values we’ll be using a lot as we start defining the actual infrastructure in the next chapter.

With our first Terraform code file written, we’re ready to try running some Terraform commands to make sure it works as expected. The Terraform CLI tool includes a lot of helpful features to improve the quality and safety of your infrastructure code. You can use it to format (or “lint”) the HCL that you’ve written, validate the syntax and do a “dry-run” of the changes that Terraform would run against your provider.

If you’ve followed the instructions earlier in this chapter, you should have a local copy of Terraform available in your working environment. Make sure you are in the same working directory as your main.tf file and try running the fmt command to format your code:

env-sandbox msur$ terraform fmt
main.tf

The fmt command is a formatter that will examine your HCL file and make changes to improve its consistency and readability. If any changes were made it will output the name of the file that it changed.

Next, we’ll validate that the syntax of the HCL we’ve written is valid. But, before we do that we’ll need to install the providers we’re using, otherwise Terraform will complain that it can’t do the syntax check. Run the following command to install the providers:

env-sandbox msur$ terraform init

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Now, we can run a validate command to ensure that we haven’t introduced any syntax errors:

env-sandbox msur$ terraform validate
Success! The configuration is valid.

Finally, we can run a plan to see what changes Terraform would make to create the environment we’ve specfied:

$ terraform plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Notice that our plan isn’t very interesting: “No changes.”. That’s because we haven’t actually defined any resources to create. But, the good new is that we now have a syntactically valid Terraform file to start building our Sandbox environment. This is a good time to commit and push the file into the Github repository so that the file is available for use:

$ git add .
$ git commit -m "The sandbox starter file"
$ git push origin

With our Terraform file working and ready to be used, we can shift our focus over to the pipeline that we’ll use to automatically apply it.

Build the Pipeline

In this section we’ll setup an automated CI/CD pipeline that will automatically apply the Terraform file that we’ve just created. We’ll be using Github’s built in DevOps tool called Github Actions to configure the pipeline activities. The nice thing about using Github Actions is that we can put our pipeline configuration in the same place as our infrastructure code.

The easiest way to use Github Actions is to configure it through the browser interface. So, go back to your browser and navigate to the sandbox repository you created in Github earlier.

Our plan is to create resources in the AWS account that we created earlier in this chapter. That means we’ll need to make sure that Github is able to use the AWS access key and secret that we provisioned when we created the operator account. There are lots of ways to manage secrets in a microservices architecture, but for our DevOps tooling, we’ll just use Github’s built-in secrets storage function.

Navigate to the Github secrets storage area by selecting Settings from the top navigation of your repository. Select Secrets from the menu of settings options on the left hand side of the screen as shown in Figure 4-7.

Figure 4-7. Github Secrets

Select Add a new secret and create a secret called AWS_ACCESS_KEY_ID. Enter the access key ID that you tucked away earlier in this chapter when you created your operator user. Repeat the process and create a secret named AWS_SECRET_ACCESS_KEY with the secret access key you generated earlier. When you are done you should have something that looks like Figure 4-8.

Figure 4-8. Add Your AWK ID and Key

Now that the secrets have been added, we can start working on the workflow for the pipeline. A workflow is the set of steps that we want to run whenever a pipeline is triggered. For our Microservices infrastructure pipeline, we’ll want a workflow that validates Terraform files and then applies them to our sandbox environment. But, in addition to testing and applying infrastructure changes, we’ll need to add a few steps before and after our Terraform apply.

The workflow will need to start with a trigger that lets Github know when the workflow should start. Github Actions gives us a few different options for triggers, but we’ll use Git’s tag mechanism as the trigger for our infrastructure builds. A tag is a way of giving a name or labelling a particular point in a Git repoisitory history. Using tagging as a trigger gives us a nice versioning history for the changes we are making to the environment. It also gives us a way of committing files to the repository without triggering a build.

When our pipeline workflow is triggered it will need to operate on the Terraform files that we’ve commited to the repository. But, we’ll need some setup steps to prepare the build environment. That means we’ll need to install Terraform and AWS just like we did in our local environment. Although we are running this in Github Actions, the actual build takes place in a virutal machine, so we’ll also need to grab a copy of the code from our code repository.

Finally, when the changes are applied to our sandbox environment, we’ll have a chance to do any cleanup or post-provisioning activities. In our case, we’ll be making a special configuration file available for download so that we can connect to the AWS based microservices environment from a local machine.

When it’s complete, the pipeline will look like Figure 4-9.

Figure 4-9. Infrastructure Pipeline Steps

We’ll be defining the step of the pipeline using the YAML language and Github Actions’ set of workflow commands. You can find the full documentation for Github Actions at https://docs.github.com/en/actions. Let’s dig into the YAML configuration by navigating to the Github Actions page for your repository. You should be able to do this by selecting Actions from the top navigation bar in your sandbox’s Github repository page. When you get there you should see a screen that looks similar to Figure 4-10.

Figure 4-10. Create a Github Action Workflow

Github Actions provides you with templates you can use to quickly get started with a workflow. But, we’re going to ignore the templates and setup a workflow ourselves from scratch. Click the Set up a workflow yourself button in the top right corner of the screen (or wherever it is in the latest version of the interface).

You’ll now find yourself editing a newly created YAML file for your workflow. Github keeps the Actions files in a hidden directory called /.github/workflows. When you clone a Github repository, you can edit these files in whatever editor you like, or create new Yaml files to define new Github Action workflows. But, the advantage of editing Actions on the Github website is that you can search for plug-ins from the marketplace. So, we’ll stick to the browser based editor for our initial workflow editing work.

The first thing we’ll do is configure a trigger for the workflow and setup a container environment to do the infrastructure build.

name: Sandbox Environment Build

on:
  create:
    tags:
      - v*
jobs:
  build:
    runs-on: ubuntu-latest
    env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

    steps:
    - uses: actions/checkout@v2

[...]

    steps:
    - uses: actions/checkout@v2 (TK use colour or italics to indicate this is already in the file)

	- name: Install aws-iam-authenticator
      run: |
        echo Installing aws-iam-authenticator...
        mkdir ~/aws
        curl -o ~/aws/aws-iam-authenticator 
        https://amazon-eks.s3.us-west-2.amazonaws.com/1.16.8/2020-04-16/bin/linux/amd64/aws-iam-authenticator
        chmod +x ~/aws/aws-iam-authenticator
        echo "::add-path::~/aws"

    - name: Install Istio CLI
      run: |
        curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.6.3 sh -
        echo "::add-path::./istio-1.6.3/bin"

	- uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 0.12.19

[...]
	- uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 0.12.19

    - name: Terraform fmt
      run: terraform fmt

    - name: Terraform Init
      run: terraform init

    - name: Terraform Validate
      run: terraform validate -no-color

    - name: Terraform Plan
      run: terraform plan -no-color

    - name: Terraform Apply
      run: terraform apply -no-color -auto-approve

Publish Assets and Commit Changes

When a Github Action workflow completes, the virtual machine that we used for our build is destroyed. But, sometimes we want to keep some of the state, files or results for later use. To help with that Github provides an upload-artifact action that gives us an easy way to make files available for us to download later.

In the next chapter, we’ll be setting up a Kubernetes cluster on AWS. When you work with Kubernetes, its useful to connect to the cluster from your remote machine. But, to do that you need a lot of connection and authentication details. So, to make that easier, we’ll introduce a final step that provisions a Kubernetes configuration file that can be downloaded to make it easy for us to connect to the cluster once it is created.

Add the code in Example 4-7 to the end of the workflow file to implement the final step of our job.

Example 4-7. Upload kubeconfig

    - name: Terraform Apply
      run: terraform apply -no-color -auto-approve


    - name: Upload kubeconfig file
      uses: actions/upload-artifact@v2
      with:
        name: kubeconfig
        path: kubeconfig

This action uploads a file called kubeconfig from the local working directory of the build environment to your Github Actions repository. It assumes that the file exists, so we’ll need to create that file in the next chapter when we get into the details of buildling our Sandbox infrastructure.

With this final addition, you now have a complete infrastructure pipeline for your sandbox environment. Github manages the workflow files the same way it manages code. So, we’ll need to commit our changes to save them. Click the Start commit button, give the commit a description and click the Commit changes button to finish commiting the change.

Figure 4-11. Commit Github Action

Taking your Pipeline Further

We couldn’t fit in all of the things that a production CI/CD IaC pipeline would have in this chapter. In particular, we had to omit integration testing from our pipeline activities. But, we highly recommend that you investigate and implement an integration test step for your Terraform code.The go based tool Terratest from Gruntworks.io is worth taking a look at when you start introducing this kind of functionality.

All that’s left is to try out our workflow to make sure that it runs correctly.

Test the Pipeline

To test the pipeline that we’ve created, we’ll need to fire the trigger for the job we defined. In our case that means we need to create a git tag in our repository with a label that starts with the letter “v”. We could do this in the browser based UI by using Github’s Releases feature. But, since we’ll be doing most of our work outside of Github on our local workstation, we’ll create the tag there instead.

The first thing we need to do is get the local clone of the repository up to date with the changes we’ve made. To do that, open a shell in your workstation and run the command git pull in your env-sandbox directory. You should see get a result that looks something like Example 4-8 indicating that we’ve pulled the new .github/workflows/main.yml file into the local repo.

env-sandbox msur$ git pull
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/msur/env-sandbox
   a6b706f..9923863  master     -> origin/master
Updating a6b706f..9923863
Fast-forward
 .github/workflows/main.yml | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 .github/workflows/main.yml

Now, that we are up to date with the Github hosted repository, we can create a tag. Since this is just a test, we’ll label our release “v0.1”. Use the git tag command as shown in Example 4-9 to create the new tag with a label.

env-sandbox msur$ git tag -a v0.1

Although we’ve created a tag, it only exists locally in our workstation clone of the repository. In order to trigger our workflow, we’ll need to push this tag to our Github hosted repo. Use the git push command with the name of the tag as shown in Example 4-10 to do this.

env-sandbox testuser$ git push origin v0.1
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 165 bytes | 165.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0)
To https://github.com/mitraman/env-sandbox.git
 * [new tag]         v0.1 -> v0.1

We’ll be doing this sequence of tagging and pushing a tag whenever we want the pipeline to run. Pushing the tag should have triggered the workflow we’ve created in Github Actions, all we need to do now is check to make sure it has run successfully.

To see the status of the run, go back to the browser based Github interface and navigate to Actions just like we did before. You should see something like Figure 4-12, indicating that our workflow job has completed successfully.

Figure 4-12. A successful run of the pipeline

You can also see more details of the job that has been run. This can be useful if your job hasn’t run as expected and you need to do some troubleshooting. To see job details, select the workflow you want more details on(ours is called Sandbox Environment Build), then select the job (in our case the job is called build.) In the detail screen you’ll be able to see what happened at each step of the job when the pipeline ran.

Figure 4-13. Details of a job

Github Actions is a relatively new product and Github changes the user interface frequently, so the exact steps to get to this screen may have changed by the time you read this. If you are having trouble getting to the steps of your job, refer to the documentation at https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run.

With our pipeline successfully tested, we’ve finished setting up the tooling we need to declaratively build our infrastructure.