Network Module Outputs

Let’s start by defining the resources that we expect the networking module to produce. We’ll do this by creating a Terraform file called output.tf in the root directory of the module-aws-network as follows:

output "vpc_id" {
  value = aws_vpc.main.id
}

output "subnet_ids" {
  value = [
    aws_subnet.public-subnet-a.id,
    aws_subnet.public-subnet-b.id,
    aws_subnet.private-subnet-a.id,
    aws_subnet.private-subnet-b.id]
}

output "public_subnet_ids" {
  value = [aws_subnet.public-subnet-a.id, aws_subnet.public-subnet-b.id]
}

output "private_subnet_ids" {
  value = [aws_subnet.private-subnet-a.id, aws_subnet.private-subnet-b.id]
}

Based on the Terraform module output file, we can see that the network module creates a Virtual Private Cloud (VPC) resource that represents the software defined network for our system. Within that network, our module will also create four logical subnets - these are bounded parts of our network (or subnetworks.) Two of these subnets will be public, meaning that they will be accessible over the Internet. Later, we’ll use all four subnets for our Kubernetes cluster setup and eventually we’ll deploy our Microservices into them.

Network Module Main Configuration

With the output of our module defined, we can start putting together the declarative code that builds it and creates the outputs we are expecting. In a Terraform module, that means we’ll be creating and editing a file named main.tf in the root directory of the module-aws-network repository.

We’ll start our module implementation by creating an AWS Virtual Private Cloud (VPC) resource. Terraform provides us with a special resource for defining AWS VPCs, so we’ll just need to fill in a few parameters to create our definition. When we create a resource in Terraform, we define the parameters and configuration details for it in the Terraform syntax. When we apply these changes, Terraform will make an AWS API call and create the resource if it doesn’t exist already.

Create a file called main.tf in the root of your network module’s repository and add the following Terraform code to your main.tf file to define a new AWS VPC resource:

provider "aws" {
  region = var.aws_region
}

locals {
  vpc_name = "${var.env_name} ${var.vpc_name}"
  cluster_name = "${var.cluster_name}-${var.env_name}"
}


## AWS VPC definition
resource "aws_vpc" "main" {
  cidr_block = var.main_vpc_cidr
  tags = {
    "Name"                                        = local.vpc_name,
    "kubernetes.io/cluster/${local.cluster_name}" = "shared",
  }
}

The network module starts with a declaration that it is using the AWS provider. This is a special instruction that lets Terraform know that it needs to download and install the libraries it will need in order to communicate with the AWS API and create resources on our behalf. When we validate or apply this file in Terraform it will attempt to connect to the AWS API using the credentials we’ve configured in the system on as environment variables. We’re also specifying an AWS region here so that Terraform knows which region it should be working in.

We’ve also specified two local variables using a Terraform locals block. These variables define a naming standard that will help us differentiate environment resources in the AWS console. This is especially important if we plan to create multiple environments in the same AWS account space as it will help us avoid naming collisions.

After the local variable declaration, you’ll find the code for creating a new AWS VPC. As you can see, there isn’t much to it, but it does define two important things: a CIDR block and a set of descriptive tags.

The CIDR (Classless Inter-Domain Routing) is a standard way of describing an IP address range for the network. It’s a shorthand string that defines which IP addresses are allowed inside of a network or a subnet. For example, a CIDR value of 10.0.0.0/16 would mean that you could bind to any IP address between 10.0.0.0 - 10.0.255.255 inside the VPC. We’ll be defining a pretty standard CIDR range for you when we build the sandbox environment, but if you want more details on how CIDRs work and why they exist you can read more about them in the RFC at https://tools.ietf.org/html/rfc4632.

We’ve also added some tag values to the VPC. Resource tags are useful because they give us a way of easily identifying groups of resources when we need to administrate them. Tags are also useful for automated tasks and for identifying resources that should be managed in specific ways. In our definition, we have defined a “Name” tag to make our VPC easier to identify. We have also defined a Kuberenetes tag that identifies this cluster as a target for our Kubernetes cluster (which we’ll define later in this chapter).

Also, notice that in a few cases we’ve referenced a variable instead of an actual value in our configuration. For example, our CIDR block is defined as var.main_vpc_cidr and the name of our VPC is tagged as var.project_name. These are Terraform variables and we’ll define their values later when we use this module as part of our sandbox environment. The variables are what makes the modules modular - by changing the variable values we can change the types of environments that we create.

With our main VPC defined, we can move on to configuring the subnets for the network. As we mentioned earlier in this chapter, we’ll be using Amazon’s managed Kubernetes service called EKS to run our workload. In order for EKS to function properly, we’ll need to have subnets defined in two different availability zones for availability and fault tolerance. Amazon also recommends that we configure our VPC with both public and private subnets, so that Kubernetes can create load balancers in the public subnets that manage traffic in the private subnets.

To meet those requirements, we’ll define a total of four subnets. Two of them will be designated as public subnets, which means that they will be accessible over the web. The other two subnets will be private. We’ll also split our public and private subnets up so that they are deployed in separate availability zones. When we are done, we’ll have a network that looks like the network depicted in Figure 5-1.

Figure 5-1. AWS Subnet Design

We’ve already specified a CIDR for the IP range in our VPC, now we’ll need to split those IP addresses up and for the subnets to use. Since the subnets are inside of the VPC, they’ll need to have a CIDR that is within the boundaries of the VPC IP range. We won’t actually be defining those IP addresses in our module though. Instead, we’ll use variables just like we did for the VPC.

In addition to the CIDR blocks, we’ll specify the availability zones for our subnets as a parameter. Rather than hardcoding the name of the availability zone, we’ll use a special terraform type called data that will let us dynamically choose the zone name. In this case, we’ll put public-subnet-a and private-subnet-a in data.aws.availability_zones.available.names[0] and public-subunet-b and private-subnet-b in data.aws.availability_zones.available.names[1]. Using dynamic data like this makes it easier for us to spin up this infrastructure in different regions.

Finally, we’ll add a name tag so that we can easily find our network resources through the admin and ops consoles. We’ll also add need to add some EKS tags to the subnet resources so that our AwS Kubernetes service will know which subnets we are using and what they are for. We’ll tag our public subnets with an elb role so that EKS knows it can use that subnet to create and deploy an elastic load balancer. We’ll tag the private subnets with an internal-elb role to indicate that our workloads will be deployed into it and can be load balanced. For more details on how AWS EKS uses load balancer tags, you can consult the documentation at https://docs.aws.amazon.com/eks/latest/userguide/load-balancing.html.

Add the following Terraform code to the end of your main.tf file in order to declare the subnet configuration:

# subnet definition

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "public-subnet-a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.public_subnet_a_cidr
  availability_zone = data.aws_availability_zones.available.names[0]

  tags = {
    "Name"                                        = "${local.vpc_name}-public-subnet-a"
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }
}


resource "aws_subnet" "public-subnet-b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.public_subnet_b_cidr
  availability_zone = data.aws_availability_zones.available.names[1]

  tags = {
    "Name"                                        = "${local.vpc_name}-public-subnet-b"
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }
}


resource "aws_subnet" "private-subnet-a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_a_cidr
  availability_zone = data.aws_availability_zones.available.names[0]


  tags = {
    "Name"                                        = "${local.vpc_name}-private-subnet-a"
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

resource "aws_subnet" "private-subnet-b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = var.private_subnet_b_cidr
  availability_zone = data.aws_availability_zones.available.names[1]

  tags = {
    "Name"                                        = "${local.vpc_name}-private-subnet-b"
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

Although we’ve configured four subnets and their IP ranges, we haven’t yet defined the network rules that AWS will need to manage traffic through them. To finish our network design, we’ll need to implement a set of routing tables that define what traffic sources we will allow into our subnets. For example, we’ll need to establish how traffic will be routed through our public subnets and how each of the subnets will be allowed to communicate with each other.

We’ll start by defining the routing rules for our two public subnets: public-subnet-a and public-subnet-b. To make these subnets accessible on the internet, we’ll need to add a special resource to our VPC called an Internet Gateway. This is an AWS network component that connects our private cloud to the public internet. Terraform gives us a resource definition for the gateway, so we’ll use that and tie it to our VPC with the vpc_id configuration parameter.

Once we’ve added the Internet Gateway, we’ll need to define routing rules that let AWS know how to route traffic from the gateway into our subnets. To do that, we’ll create an aws_route_table resource that allows all traffic from the Internet (which we’ll identify with CIDR block 0.0.0/0), through the gateway. Then we just need to create associations between our two public subnets and the table we’ve just defined.

Add the following Terraform code to main.tf to define routing instructions for our network:

# Internet gateway and routing tables for public subnets
resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = "${local.vpc_name}-igw"
  }
}

resource "aws_route_table" "public-route" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    "Name" = "${local.vpc_name}-public-route"
  }
}

resource "aws_route_table_association" "public-a-association" {
  subnet_id      = aws_subnet.public-subnet-a.id
  route_table_id = aws_route_table.public-route.id
}


resource "aws_route_table_association" "public-b-association" {
  subnet_id      = aws_subnet.public-subnet-b.id
  route_table_id = aws_route_table.public-route.id
}

With the routes for our public subnets defined, we can dive into the setup for our two private subnets. The route configuration for the private subnets will be a bit more complicated than what we’ve done so far. That’s because we’ll need to define a route from our private subnet out to the Internet to allow our Kubernetes pods to talk to the EKS service.

For that kind of route to work, we’ll need a way for nodes in our private subnet to talk to the Internet Gateway we’ve deployed in the public subnets. In AWS, that means we need to create a NAT (Network Address Translation) Gateway resource that gives us a path out. When we create the NAT, we’ll also need to assign it a special kind of IP address called an elastic IP address (or EIP). This is an AWS construct that means the IP is a real Internet accessible network address, unlike all of the other addresses in our network which are virtual and exist inside of AWS alone. Since real IP addresses aren’t unlimited, AWS limits the amount of these available. Unfortunately, we can’t create a NAT without one, so we’ll have to use two of them - one for each NAT we are creating.

Add the following Terraform code to implement a NAT gateway in our network.

resource "aws_eip" "nat-a" {
  vpc = true
  tags = {
    "Name" = "${local.vpc_name}-NAT-a"
  }
}

resource "aws_eip" "nat-b" {
  vpc = true
  tags = {
    "Name" = "${local.vpc_name}-NAT-b"
  }
}

resource "aws_nat_gateway" "nat-gw-a" {
  allocation_id = aws_eip.nat-a.id
  subnet_id     = aws_subnet.public-subnet-a.id
  depends_on    = [aws_internet_gateway.igw]


  tags = {
    "Name" = "${local.vpc_name}-NAT-gw-a"
  }
}

resource "aws_nat_gateway" "nat-gw-b" {
  allocation_id = aws_eip.nat-b.id
  subnet_id     = aws_subnet.public-subnet-b.id
  depends_on    = [aws_internet_gateway.igw]


  tags = {
    "Name" = "${local.vpc_name}-NAT-gw-b"
  }
}

In addition to the NAT gateway we’ve created, we’ll need to define routes for our private subnets. Add the following Terraform code to main.tf to complete the definition of our network routes.

modules/network/main.tf - route tables

resource "aws_route_table" "private-route-a" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat-gw-a.id
  }

  tags = {
    "Name" = "${local.vpc_name}-private-route-a"
  }
}


resource "aws_route_table" "private-route-b" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat-gw-b.id
  }

  tags = {
    "Name" = "${local.vpc_name}-private-route-b"
  }
}


resource "aws_route_table_association" "private-a-association" {
  subnet_id      = aws_subnet.private-subnet-a.id
  route_table_id = aws_route_table.private-route-a.id
}


resource "aws_route_table_association" "private-b-association" {
  subnet_id      = aws_subnet.private-subnet-b.id
  route_table_id = aws_route_table.private-route-b.id
}

That’s it for our main network definition. When we eventually run this Terraform file, we’ll end up with an AWS software defined network that is ready for Kubernetes and our Microservices. But, before we can use it, we’ll need to define all of the input variables that this module needs. Although, we’ve referenced a lot of var values in our code, Terraform modules require us to identify all of the input variables we’ll be using in a specific file called variables.tf. If we don’t do that, we won’t be able to pass variable values into our module.

Network Module Variables

Create a file in the root folder of the network module called variables.tf. Add the following Terraform code to variables.tf to define the inputs for the module:

variable "env_name" {
  type = string
}

variable "aws_region" {
  type = string
}

variable "vpc_name" {
  type    = string
  default = "ms-up-running"
}

variable "main_vpc_cidr" {
  type = string
}

variable "public_subnet_a_cidr" {
  type = string
}

variable "public_subnet_b_cidr" {
  type = string
}

variable "private_subnet_a_cidr" {
  type = string
}

variable "private_subnet_b_cidr" {
  type = string
}

variable "cluster_name" {
  type    = string
}

As you can see, the variable definitions are fairly self-explanatory. They describe a name, optional description and a type value. In our module we’re only using string values. In some cases, we’ve also provided a default value so that those inputs don’t always have to be defined for every environment. We’ll give the module values for those variables when we use it to create an environment.

The Terraform code for our network module is now complete. At this point, you should have a list of files that looks something like this in your module directory:

drwxr-xr-x   3 msur  staff   96 14 Jun 09:57 ..
drwxr-xr-x   7 msur  staff  224 14 Jun 09:58 .
-rw-r--r--   1 msur  staff   23 14 Jun 09:57 README.md
drwxr-xr-x  13 msur  staff  416 14 Jun 09:57 .git
-rw-r--r--   1 msur  staff    0 14 Jun 09:58 main.tf
-rw-r--r--   1 msur  staff  612 14 Jun 09:58 variables.tf
-rw-r--r--   1 msur  staff   72 14 Jun 09:58 outputs.tf

With the code all written, we’ll be testing the network module out by creating a sandbox environment network, but before we use the module we should make sure we haven’t made any syntax errors. The Terraform command line application includes some handy features to format and validate code. If you haven’t already installed the Terraform client in your local system, you can find a binary for your operating system at https://www.terraform.io/downloads.html.

Use the following Terraform command while you are in your module’s working directory to format the module’s code

module-aws-network$ terraform fmt

The fmt command will “lint” or format all Terraform code in the working directory and ensure that it conforms to a set of built-in style guidelines. It will automatically make those changes for you and will list any files that have been updated.

Next, run teraform init so that Terraform can install the AWS provider libraries. We need to do this so that we can validate the code. Note, that you’ll need to have AWS credentials defined for this to work. If you haven’t done that yet, follow the instructions in the previous chapter.

module-aws-network$ terraform init

Finally, you can run the validate command to make sure that our module is syntatically correct. If you run into any problems, try to fix those before you continue.

module-aws-network$ terraform validate
Success! The configuration is valid.

When you are satisfied that the code is formatted and valid, you can commit your changes to the Github repository. If you’ve been working in a local repository, you can use the following command to push your changes to the main repository:

module-aws-network$ git add .
module-aws-network$ git commit -m "network module created"
[master ddb7e41] network module created
 3 files changed, 226 insertions(+)
module-aws-network$ git push

Our Terraform based network module is now complete and available for use. It has a variables.tf file that describes the required and optional input variables to use it. It has a main.tf file that declaratively defines the resources for our network design. Finally, it has an outputs.tf that defines the significant resources that we’ve created in the module. Now, we can use the module to create a real network in our sandbox environment.

Create a Sandbox Network

The nice thing about using Terraform modules is that we can create our environments easily in a repeatable way. Outside of the specific values we’ve defined in the variables.tf file, any environment that we create with the module we’ve defined will operate with a network infrastructure that we know and understand. That means we can expect our microservices to work in a predictable way as we move them through testing and release environments since we’ve reduced the level of variation.

But, to apply the module we’ve defined and create a new environment, we’ll need to call it from a Terraform file that defines values for the module’s variables. To do that, we’ll create a “sandbox” environment that demonstrates a practical example of using a Terraform module. If you followed the steps in the previous chapter (TK link), you’ll already have a code repository for your sandbox environment with a single main.tf file in it.

In order to use the network module that we’ve created, we’ll use a special Terraform resource called module. It allows us to reference a Terraform module that we’ve created and pass in values for the variables that we’ve defined. In order to use our AWS network module to create a sandbox environment, you’ll need to determine values for the variables as shown in in Table 5-2:

Once you’ve decided on the values for your variable inputs, update your sandbox’s main.tf file so that it looks like the following code, with your values and S3 bucket name populated:

terraform {
  backend "s3" {
    bucket = "bucket name"
    key    = "terraform/backend"
    region = "region code"
  }
}

locals {
  env_name         = "sandbox"
  aws_region       = "region code"
  k8s_cluster_name = "ms-cluster"
}

module "aws-network" {
  source = "github.com/implementing-microservices/module-aws-network"
  aws_region = "region code"
}

Our infrastructure pipeline will apply Terraform changes, but before we kick it off we need to check to make sure that the Terraform code we’ve written will work. A good first step is to format and validate the code locally:

$ terraform fmt
[...]
$ terraform init
[...]
$ terraform validate
Success! The configuration is valid.

$ terraform get -update

If the code is valid, we can get a plan to validate the changes that Terraform will make when they are applied. It’s always a good idea to do a dry run and examine the changes that will be made before you actually change the environment, so make this step a part of your workflow. To get the terraform plan, run this command:

$ terraform plan

Terraform will provide you with a list of the resources that will be created, deleted and updated. Look through the plan and make sure the output makes sense. When you are happy that the plan make sense, you can push the code to the Github repository and tag it for release:

$ git add .
$ git tag -a v1.0 -m "network build"
$ git commit -m "initial network release"
$ git push origin v1.0

With the code tagged and pushed, our Github Action pipeline should take over and start building the network for our Sandbox environment. You’ll need to login to Github and check the Actions tab in your sandbox environment repository to make sure that everything goes according to plan. If you don’t remember how to do that, you’ll find instructions in the previous chapter. If everything goes well, you should now have an AWS network running and ready to use. That means it’s time to start writing the module for the Kubernetes service.

Kubernetes Module Outputs

We’ll start by declaring the outputs that our module provides. create a Terraform file called outputs.tf in the root directory of the module-aws-kubernets reopsitory and add the following code to it:

output "eks_cluster_id" {
  value = aws_eks_cluster.ms-up-running.id
}


output "eks_cluster_name" {
  value = aws_eks_cluster.ms-up-running.name
}


output "eks_cluster_certificate_data" {
  value = aws_eks_cluster.ms-up-running.certificate_authority.0.data
}


output "eks_cluster_endpoint" {
  value = aws_eks_cluster.ms-up-running.endpoint
}


output "eks_cluster_nodegroup_id" {
  value = aws_eks_node_group.ms-node-group.id
}

The main value we’re returning is the identifier for the EKS cluster that we’ll be creating in this module. The rest of the values need to be returned so that we can access the cluster from other modules once the cluster is stood up and operational. For example, we’ll need the endpoint and certificate data when we install the ArgoCD server into this EKS cluster at the end of the chapter.

While the output of our module is pretty simple, the work of getting our EKS based Kubernetes system setup is going to be a bit more complicated. Just like we did before, we’ll build the module’s main Terraform file in parts before we test it and apply it.

Defining the EKS Cluster

To begin with, create a Terraform file called main.tf in the root directory of your Kubernetes module and add an AWS provider definition:

provider "aws" {
  region = var.aws_region
}

Remember that we’ll be using the Terraform naming convention of var to indicate values that can be replaced by variables when our module is invoked.

As we mentioned earlier, we’re going to use Amazon’s EKS to create and manage our Kubernetes installation. But, EKS will need to create and modify AWS resources on our behalf in order to run. So, we’ll need to setup permissions in our AWS account so that it can do the work it needs to do. We’ll need to define policies and security rules at the overall cluster level and also for the virtual machines or nodes thatthat EKS will be spinning up for us to run Microservices on.

We’ll start by focusing on the rules and policies for the entire EKS cluster. Add the following Terraform code to your main.tf file to define a new cluster access management policy:

locals {
  cluster_name = "${var.cluster_name}-${var.env_name}"
}


resource "aws_iam_role" "ms-cluster" {
  name = local.cluster_name

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

resource "aws_iam_role_policy_attachment" "ms-cluster-AmazonEKSClusterPolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
  role       = aws_iam_role.ms-cluster.name
}

The snippet above establishes a trust policy that allows the AWS EKS service to act on your behalf. It defines a new identity and access management role for our EKS service and attaches a policy called AmazonEKSClusterPolicy to it. This policy has been defined by AWS for us and gives the EKS the permissions it needs to create virtual machines and make network changes as part of its Kubernetes management work. Notice that we are also defining and using a local variable for the name of the cluster. We’ll use that variable throughout the module.

Now that the cluster service’s role and policy are defined, add the following code to your module’s main.tf file to define a network security policy for the cluster:

resource "aws_security_group" "ms-cluster" {
  name        = "ms-up-running-cluster"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "ms-up-running"
  }
}

A VPC security group restricts the kind of traffic that can go into and out of the network. The Terraform code we’ve just written defines an “egress” rule that allows unrestricted outbound traffic, but doesn’t allow any inbound traffic because there is no “ingress” rule defined. Notice that we are applying this security group to a VPC that will be defined by an input variable. When we use this module, we can give it the ID of the VPC that our networking module has created.

With these policies and a security group defined for the EKS cluster, we can now add the declaration for the cluster itself to the main.tf Terraform file:

resource "aws_eks_cluster" "ms-up-running" {
  name     = local.cluster_name
  role_arn = aws_iam_role.ms-cluster.arn

  vpc_config {
    security_group_ids = [aws_security_group.ms-cluster.id]
    subnet_ids         = var.cluster_subnet_ids
  }

  depends_on = [
    aws_iam_role_policy_attachment.ms-cluster-AmazonEKSClusterPolicy
  ]
}

The EKS cluster definition we’ve just created is pretty simple. It simply references the name, role, policy and security group values we’ve defined earlier. It also references a set of subnets that the cluster will be managing. These subnets will be the ones that we created earlier in the networking module and we’ll be able to pass them in to this Kubernetes module as a variable.

When AWS creates an EKS cluster, it automatically sets up all of the management components that we need to run our Kubernetes cluster. This is called the “control plane” and it’s the brain of our Kubernetes system. But, in addition to the control plane, our Microservices need a place where they can run. In Kubernetes, that means we need to setup nodes - the physical or virtual machines that containerized workloads can run on.

One of the advantages of using a managed Kubernetes service like EKS is that we can off-load some of the work of managing the creation, removal and updates of Kubernetes nodes. Four our configuration, we’ll define a managed EKS node group and let AWS provision resources and interact with the Kubernetes system for us. But, to get a managed node group running, we’ll still need to define a few important configuration values.

Defining the EKS Node Group

Just like we did for our cluster, we’ll begin the node configuration by defining a role and some security policies. Add the following node group IAM definitions to our Kubernetes module’s main.tf file:

modules/eks/main.tf

# Node Role
resource "aws_iam_role" "ms-node" {
  name = "${local.cluster_name}.node"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
POLICY
}

# Node Policy
resource "aws_iam_role_policy_attachment" "ms-node-AmazonEKSWorkerNodePolicy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.ms-node.name
}

resource "aws_iam_role_policy_attachment" "ms-node-AmazonEKS_CNI_Policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.ms-node.name
}

resource "aws_iam_role_policy_attachment" "ms-node-AmazonEC2ContainerRegistryReadOnly" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.ms-node.name
}

The role and policies in the Terraform snippet above will allow any nodes that are created to communicate with Amazon’s container registries and virtual machine services. We need these policies, because the nodes in our Kubernetes system will need to be able to provision computing resources and access containers in order to run services. You can find more details on the IAM role for EKS worker nodes in the AWS EKS documentation at https://docs.aws.amazon.com/eks/latest/userguide/worker_node_IAM_role.html.

Now that we have our node’s role and policy resources defined, we can write the declaration for a node group that uses them. In EKS, a managed node group needs to specify the types of compute and storage resources it will use along with some defined limits for the number of individual nodes or VMs that can be created automatically. This is important because we are letting EKS automatically provision and scale our nodes. We don’t want to inadvertently consume massive amounts of AWS resources and end up with a correspondingly massive bill.

We could hardcode all of these parameters in our module, but instead we’ll use input variables as values for the size limits, disk size and CPU types. That way we’ll be able to use the same Kubernetes module to create different kinds of environments. For example, a development environment can be setup to use minimal resources, while a production environment can be more robust.

Add the following Terraform code to the end of the module’s main.tf file to define our EKS node group:

modules/eks/main.tf

resource "aws_eks_node_group" "ms-node-group" {
  cluster_name    = aws_eks_cluster.ms-up-running.name
  node_group_name = "microservices"
  node_role_arn   = aws_iam_role.ms-node.arn
  subnet_ids      = var.nodegroup_subnet_ids

  scaling_config {
    desired_size = var.nodegroup_desired_size
    max_size     = var.nodegroup_max_size
    min_size     = var.nodegroup_min_size
  }

  disk_size      = var.nodegroup_disk_size
  instance_types = var.nodegroup_instance_types

  depends_on = [
    aws_iam_role_policy_attachment.ms-node-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.ms-node-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.ms-node-AmazonEC2ContainerRegistryReadOnly,
  ]
}

The node group declaration is the last part of our EKS configuration. We have enough here to be able to call this module from our sandbox environment and instantiate a running Kubernetes cluster on the AWS EKS service. Our module’s outputs will return the values that are needed to connect to the nodegroup once it’s running. But, it’s also useful to provide those connection details in a configuration file for the kubectl CLI that most operators use for Kubernetes management.

To create a Kubernetes configuration file, append the following code to your module’s main.tf file:

# Create a kubeconfig file based on the cluster that has been created
resource "local_file" "kubeconfig" {
  content  = <<KUBECONFIG_END
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${aws_eks_cluster.ms-up-running.certificate_authority.0.data}
    server: ${aws_eks_cluster.ms-up-running.endpoint}
  name: ${aws_eks_cluster.ms-up-running.arn}
contexts:
- context:
    cluster: ${aws_eks_cluster.ms-up-running.arn}
    user: ${aws_eks_cluster.ms-up-running.arn}
  name: ${aws_eks_cluster.ms-up-running.arn}
current-context: ${aws_eks_cluster.ms-up-running.arn}
kind: Config
preferences: {}
users:
- name: ${aws_eks_cluster.ms-up-running.arn}
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "${aws_eks_cluster.ms-up-running.name}"
    KUBECONFIG_END
  filename = "kubeconfig"
}

This code looks complicated, but it’s actually fairly simple. We are using a special Terraform resource called local_file to create a file named “kubeconfig”. We are then populating kubeconfig with YAML content that defines connection parameters for our Kubernetes cluster. Notice that we are getting the values for the YAML file from the EKS resources that we created in the module.

When Terraform runs this block of code it will create a “kubeconfig” file in a local directory. We’ll be able to use that file to connect to the Kubernetes environment from CLI tools. We made a special provision for this file when we built our pipeline in the previous chapter. When you run the infrastructure pipeline you’ll be able to download this populated configuration file and use it to connect to the cluster.

The configuration file will make it a lot easier for us to connect to the cluster from your machine. But, we’re also going to use it for Istio installation. That’s the next bit of code we need to add to the Terraform file.

Installing Istio

If you recall, from the beginning of this chapter, Istio service mesh is a system of micro-proxies and management components that will help us secure, stabilize, shape and see all of the communication that is happening between our services. We need to bootstrap the Istio install now, so we can use the service mesh later in this book in Chapter TK, when we deploy Microservices.

Under the hood, Istio’s architecture is pretty complicated with a lot of moving parts. Fortunately, Istio provides an easy to use CLI application called istio_ctl to do the install with a single command line instruction. Unfortunately, as of this writing there isn’t a good, native Terraform way of installing the custom extensions that Isio needs for operation. Instead, we’ll have to do something that is a bit of a “hack” - we’ll need to run Istio’s command line application from our Terraform module.

The Istio CLI application is pretty straightforward, but it needs to be able to talk to the EKS cluster that we are creating in order to run. It needs to know the network address of our newly created cluster and it needs to have the right credentials to get access to the cluster’s management API. The good news is that it understands the kubeconfig format of the file we’ve just created, so we just need to call it and point it at the config file.

Add the following code to your main.tf file, to install Istio into the cluster:

resource "null_resource" "istio-install" {
  # Reinstall istio if the cluster is changed
  triggers = {
    cluster_id = aws_eks_cluster.ms-up-running.id
  }

  # Make sure that the EKS node group is running before we try to install Istio
  depends_on = [aws_eks_node_group.ms-node-group]

  provisioner "local-exec" {
    command = "istioctl install -y --kubeconfig kubeconfig"
  }
}

Since we are using a command line application application to do the Istio installation, there isn’t a specific Terraform resource defined for us to use. Instead, we’re using special Terraform resource called a “null_resource”. This means that the resource is generic and Terraform will run the expressions inside it without tying it to any specific provider’s API. Because this is a generic resource, we’ve had to define a “triggers” property that lets Terraform know that “istio-install” depends on the EKS cluster resource finishing before it gets called. It also means that if the EKS cluster is destroyed and recreated, this code will be run again.

The “local-exec” provisioner resource is a special object that instructs Terraform to run a command in the local operating system. Unlike the rest of the Terraform resources we’ve been using, a provisioner is not a declarative statement. That means, instead of identifying a target state and letting Terraform do the work of making the appropriate changes, provisioner statements just get blindly invoked. In this case, we are using it to call istioctl with arguments for installation and the location of our kubeconfig file.

Of course, for a local call to istioctl to work means that we’ll need it installed locally before the Terraform module is run. This isn’t ideal as it introduces a special pre-condition into our build, but it’s worth the tradeoff of getting Istio installed quickly and easily. In the previous chapter, we introduced a special installation step for installing istioctl as part of our infrastructure pipeline. Now, you know why!

There’s one last thing we’ll need to do to finish our Istio installation: create a namespace with a special Istio label..

Kubernetes is object based and most of the work you’ll do with it involves creating declarative objects in the system that let Kubernetes know what the system is supposed to look like. To keep those objects organised, Kubernetes provides a namespace management facility so that you can avoid naming conflicts and easily find and update them.

We’re going to create a namespace for the objects that we’ll be using to describe our Microservices deployments. We’ll use this namespace later when we deploy our services into the stack. But, we’ll also be adding a special bit of metadata that lets Kubernetes know the service should be part of Istio’s service mesh.

Add the following code to your Kubernetes module’s main.tf file to create the namespace with the Istio injection label:

provider "kubernetes" {
  load_config_file       = false
  cluster_ca_certificate = base64decode(aws_eks_cluster.ms-up-running.certificate_authority.0.data)
  host                   = aws_eks_cluster.ms-up-running.endpoint
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws-iam-authenticator"
    args        = ["token", "-i", "${aws_eks_cluster.ms-up-running.name}"]
  }
}

# Create a namespace for microservice pods and label it for automatic sidecar injection
resource "kubernetes_namespace" "ms-namespace" {

  # Make sure that the EKS node group is running before we try to install Istio
  depends_on = [aws_eks_node_group.ms-node-group]
  metadata {
    labels = {
      istio-injection = "enabled"
    }
    name = var.ms_namespace
  }
}

The code above declares a new namespace by using Terraform’s Kubernetes provider. Remember that a Terraform provider is like a plug-in that tells Terraform how to talk to a platform or system. In this case, Terraform provides a provider to perform declarative operations on Kubernetes clusters. We’re not going to use Terraform to manage our Kubernetes cluster once it’s up, but we are going to use it to bootstrap our namespace.

To configure the Kubernetes provider, we’re using the properties of the EKS cluster that we provisioned earlier in the module. This is quite similar to what we did earlier when we created the kubeconfig file. With the provider configured, we can use the provider’s “kubernetes_namespace” resource to declare our new namespace and our special istio-injection label. We’ve also use Terraform’s depends_on attribute to ensure that we don’t try to create the namespace before the EKS node group is running and ready to be used.

With the code we’ve written in main.tf, Terraform will instantiate an EKS cluster using AWS API calls and then it will call our local provisioners to install a default Istio profile into the cluster with a Kubernetes namespace. We’re almost done writing our module. All that’s left is to define the variables.

Kubernetes Module Variables

To declare the variables for our Kubernetes module, create a file called variables.tf in your moudle-aws-kubernetes repository and add the following code:

variable "aws_region" {
  type        = string
  default     = "eu-west-2"
}

variable "env_name" {
  type = string
}

variable "cluster_name" {
  type = string
}

variable "ms_namespace" {
  type    = string
  default = "microservices"
}

variable "vpc_id" {
  type = string
}

variable "cluster_subnet_ids" {
  type = list(string)
}

variable "nodegroup_subnet_ids" {
  type = list(string)
}

variable "nodegroup_desired_size" {
  type    = number
  default = 1
}

variable "nodegroup_min_size" {
  type    = number
  default = 1
}

variable "nodegroup_max_size" {
  type    = number
  default = 5
}

variable "nodegroup_disk_size" {
  type = string
}

variable "nodegroup_instance_types" {
  type = list(string)
}

Our AWS Kubernetes module is now fully written. As we did for our network module, we’ll take a moment to clean up the formatting and validate the syntax of the code by running the following Terraform commands:

module-aws-kubernetes$ terraform fmt
[...]
module-aws-kubernetes$ terraform init
[...]
module-aws-kubernetes$ terraform validate
Success! The configuration is valid.

When you are satisfied that the code is valid, commit your changes and push them to Github so that we can use this module in the Sandbox environment.

Create a Sandbox Kubernetes Cluster

Now that our complex Kubernetes system and Istio service mesh are wrapped up in a simple module, the work of setting it up in our Sandbox enviroment is pretty simple. All we’ll need to do is call our module with the input parameters that we want. Remember that our Sandbox environment is defined in it’s own code repository and has it’s own Terraform file called main.tf that we’ve used to setup the network. We’ll be editing that file again, but this time we’ll add a call to the terraform module.

If you recall, we gave some of our input variables default values. To keep things simple, we’ll just use those default values in our sandbox environment. We’ll also need to pass some of the output variables from our network module into this Kubernetes module so that it installs the cluster on the network we’ve just created. But, beyond those inputs, you’ll need to define the aws_region value for your installation. This should be the same as the value you used for the network module and the backend configuration. You’ll also need to set the source parameter to point to your Github hosted module.

Update the main.tf file of your sandbox environment so that it looks like the following, with the appropriate values replaced:

terraform {
  backend "s3" {
    bucket = "bucket name"
    key    = "terraform/backend"
    region = "region code"
  }
}

locals {
  env_name         = "sandbox"
  aws_region       = "eu-west-2"
  k8s_cluster_name = "ms-cluster"
}


module "aws-network" {
  source = "github.com/implementing-microservices/module-aws-network"


  env_name              = local.env_name
  vpc_name              = "msur-VPC"
  cluster_name          = local.k8s_cluster_name
  aws_region            = local.aws_region
  main_vpc_cidr         = "10.10.0.0/16"
  public_subnet_a_cidr  = "10.10.0.0/18"
  public_subnet_b_cidr  = "10.10.64.0/18"
  private_subnet_a_cidr = "10.10.128.0/18"
  private_subnet_b_cidr = "10.10.192.0/18"
}


module "aws-kubernetes-cluster" {
  source = "github.com/implementing-microservices/module-aws-kubernetes"

  ms_namespace       = "microservices"
  env_name           = local.env_name
  aws_region         = local.aws_region
  cluster_name       = local.k8s_cluster_name
  vpc_id             = module.aws-network.vpc_id
  cluster_subnet_ids = module.aws-network.subnet_ids


  nodegroup_subnet_ids     = module.aws-network.private_subnet_ids
  nodegroup_disk_size      = "20"
  nodegroup_instance_types = ["t3.medium"]
  nodegroup_desired_size   = 1
  nodegroup_min_size       = 1
  nodegroup_max_size       = 3
}

Now, you should be able to commit and push this file into your CI/CD infrastructure pipeline and create a working EKS cluster as well as an Istio service mesh installation. But, be prepared to wait for a few minutes for a result. Provisioning an EKS cluster can take up to 10-15 minutes. When it’s done you’ll have a powerful infrastructure ready to run your Microservices resiliently.

Our final step will be to install a deployment tool that will come in handy when it’s time to release our services into our environment’s Kubernetes cluster.

Variables for Argo CD

Create a file called variables.tf in your ArgoCD module repository and add the following code:

variable "kubernetes_cluster_id" {
  type = string
}

variable "kubernetes_cluster_cert_data" {
  type = string
}

variable "kubernetes_cluster_endpoint" {
  type = string
}

variable "kubernetes_cluster_name" {
  type = string
}

variable "eks_nodegroup_id" {
  type = string
}

We need to define these variables so that we can configure the Kubernetes and Helm providers in our code. That means we’ll need to grab them from the Kubernetes module’s output when we call it in our Sandbox’s Terraform file. Before we get to that step, let’s format and validate the code we’ve written in the same way as we did for our other modules:

module-argocd$ terraform fmt
[...]
module-argocd$ terraform init
[...]
module-argocd$ terraform validate
Success! The configuration is valid.

When you are satisfied that the code is valid, commit your code changes and push them to the Github repository so that we can use the module in our Sandbox environment.

Installing ArgoCD in the Sandbox

We want the ArgoCD installation to happen as part of our Sandbox environment bootstrapping, so we need to call the module from the Terraform definition in our sandbox environment. Add the following code to the end of your sandbox module’s main.tf file to install ArgoCD:

# Add this to the end of the file
module "argo-cd-server" {
  source = "{your_repository_location}/module-argo-cd"

  kubernetes_cluster_id        = module.aws-kubernetes-cluster.eks_cluster_id
  kubernetes_cluster_name      = module.aws-kubernetes-cluster.eks_cluster_name
  kubernetes_cluster_cert_data = module.aws-kubernetes-cluster.eks_cluster_certificate_data
  kubernetes_cluster_endpoint  = module.aws-kubernetes-cluster.eks_cluster_endpoint
  eks_nodegroup_id = module.aws-kubernetes-cluster.eks_cluster_nodegroup_id
}

Now, you can commit and push the Terraform file into your CI/CD pipeline or run it locally with Terraform. When it is done, you’ll have a continuous delivery server that will help to deploy microservices easier and more reliably. With that step completed, we’ve finished defining and provisioning the Sandbox environment. All that’s left is to test it and see if it works.