Templates and Stacks

To get started, you will create a minimal template that describes just a VPC in the us-west-2 region with a Classless Inter-Domain Routing (CIDR) range of 10.17.0.0/16. This template might look like the following using the YAML format:

AWSTemplateFormatVersion: "2010-09-09"
Description: VPC in Oregon
Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: "10.17.0.0/16"
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyVPC
        - Key: Environment
          Value: Testing

There are a few elements to call out in this example:

• AWSTemplateFormatVersion specifies which version of the template language was used. Currently, "2010-09-09" is the only valid value. Should the language ever require a backward-incompatible change, this field would be used to disambiguate how to interpret your template.
• All resources go into the Resources section of the template.
• In YAML, structure is identified by indentation. For example, MyVPC is a block within the Resources block, while Resources and Description are sibling blocks. If a block contains key: value pairs, it is a mapping; if it contains - value items (that is, values prefixed with a dash and space), it is a list.
• Each resource has a logical name. In this case, we gave the VPC the logical name of MyVPC.
• The type of the resource is specified within the Type field. AWS::EC2::VPC specifies a VPC. The AWS CloudFormation User Guide lists the valid resource types here:

• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ aws-template-resource-type-ref.html

• The resource-specific properties are always within the Properties block.
  • CidrBlock is required and specifies the CIDR block for MyVPC.
  • InstanceTenancy specifies the tenancy (default or dedicated) for new Amazon Elastic Compute Cloud (Amazon EC2) instances launched within MyVPC. This parameter is optional.
  • Tags is a list of tags to apply to the VPC. Each tag is a mapping with Key specifying the tag name and Value specifying the tag value. In this example, tags have been specified for Name and Environment keys.
  • We have omitted the other optional properties—EnableDnsSupport, EnableDnsHostnames, and Tags.

The template can be saved locally or to an Amazon Simple Storage Service (Amazon S3) bucket. In this example, you will save it locally with the filename my-network-template.yaml.

To instantiate the template, create a stack using either the AWS CloudFormation console or AWS Command Line Interface (CLI). Using the CLI, the process would be as follows:

> aws cloudformation create-stack --stack-name MyNetworkStack --template-body file://my-network-template.yaml

{
    "StackId": "arn:aws:cloudformation:us-west- 2:123456789012:stack/MyNetworkStack/4af622f0-8a24-11e7-8692-503ac9ec2435"
}

AWS CloudFormation will start creating your stack. You can watch its progress by calling the DescribeStack API. While it is being created, the output will look like this:

> aws cloudformation describe-stack --stack-name MyNetworkStack

{
    "Stacks": [
        {
            "StackId": "arn:aws:cloudformation:us-west- 2:123456789012:stack/MyNetworkStack/4af622f0-8a24-11e7-8692-503ac9ec2435",
            "Description": "VPC in Oregon",
            "Tags": [],
            "CreationTime": "2012-09-18T12:00:00.102Z",
            "StackName": "MyNetworkStack",
            "NotificationARNs": [],
            "StackStatus": "CREATE_IN_PROGRESS",
            "DisableRollback": false
        }
    ]
}

When the stack creation process has finished, CREATE_IN_PROGRESS will change to CREATE_COMPLETE. You can verify that the VPC has been created by calling DescribeVpcs:

> aws ec2 describe-vpcs --filters "Name=tag:Name,Values=MyVPC"
{
    "Vpcs": [
        {
            "VpcId": "vpc-1a2b3c4d",
            "InstanceTenancy": "default",
            "Tags": [
                {
                    "Value": "arn:aws:cloudformation:us-west-2:123456789012:stack/MyNetworkStack/4af622f0-8a24-11e7-8692-503ac9ec2435",
                    "Key": "aws:cloudformation:stack-id"
                },
                {
                    "Value": "MyVPC",
                    "Key": "aws:cloudformation:logical-id"
                },
                {
                    "Value": "MyVPC",
                    "Key": "Name"
                },
                {
                    "Value": "MyNetworkStack",
                    "Key": "aws:cloudformation:stack-name"
                },
                {
                    "Value": "Environment",
                    "Key": "Testing"
                }
            ],
            "State": "available",
            "DhcpOptionsId": "dopt-5e6f7a8b",
            "CidrBlock": "10.17.0.0/16",
            "IsDefault": false
        }
    ]
}

Note that AWS CloudFormation automatically applied three additional AWS-specific tags to the resource in addition to the Name tag. This can be useful for identifying resources in your account that are being managed by AWS CloudFormation. While you may be able to modify or delete these resources outside of AWS CloudFormation, this may impede the ability of AWS CloudFormation to manage these resources.

Stack Dependencies

The previous example does not create a usable VPC: It lacks subnets, a gateway, and routes that allow access to Amazon EC2 resources in the Amazon VPC. A minimally functional example is shown in Figure 10.1.

To create the subnet, you need to specify the Availability Zone, CIDR block, and VPC ID. The VPC is created elsewhere in the template, however, so you won’t know the ID beforehand. You can use the Ref intrinsic function to obtain the value.

Resources:
  # VPC definition omitted
  MySubnet2c:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: us-west-2c
      CidrBlock: "10.17.1.0/24"
      Tags:
        - Key: Name
          Value: MySubnet2c
      VpcId: !Ref MyVPC

Because the subnet resource references a property from the VPC, AWS CloudFormation automatically recognizes the dependency and will not start creating the subnet until the VPC has been created.

The route table also requires the VPC ID. Associating it with the subnet is done though a separate association resource that requires the subnet ID:

Resources:
  # Other resources omitted
  MyRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
  MySubnet2cRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref MyRouteTable
      SubnetId: !Ref MySubnet2c

The Internet gateway follows a similar pattern. Internet gateways are created and then attached to a VPC:

Resources:
  # Other resources omitted
  MyInternetGateway:
    Type: AWS::EC2::InternetGateway
  MyGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref MyInternetGateway
      VpcId: !Ref MyVPC

Lastly, you need to create the default route through the Internet gateway.

Resources:
  # Other resources omitted
  MyDefaultRoute:
    Type: AWS::EC2::Route
    DependsOn: MyGatewayAttachment
    Properties:
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref MyInternetGateway
      RouteTableId: !Ref MyRouteTable

Note the use of the DependsOn attribute. This instructs AWS CloudFormation to wait until the gateway has been attached to the VPC before creating this route. Without this attribute, AWS CloudFormation may attempt to create the route before the Internet gateway has been attached to the VPC, creating a race condition. Routes cannot refer to gateways that are not attached to the same VPC in which the route table resides.

The full template is shown here.

AWSTemplateFormatVersion: "2010-09-09"
Description: VPC in Oregon
Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: "10.17.0.0/16"
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyVPC
        - Key: Environment
          Value: Testing
  MySubnet2c:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: us-west-2c
      CidrBlock: "10.17.1.0/24"
      Tags:
        - Key: Name
          Value: MySubnet2c
      VpcId: !Ref MyVPC
  MyRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref MyVPC
  MySubnet2cRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref MyRouteTable
      SubnetId: !Ref MySubnet2c
  MyInternetGateway:
    Type: AWS::EC2::InternetGateway
  MyGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref MyInternetGateway
      VpcId: !Ref MyVPC
  MyDefaultRoute:
    Type: AWS::EC2::Route
    DependsOn: MyGatewayAttachment
    Properties:
      DestinationCidrBlock: "0.0.0.0/0"
      GatewayId: !Ref MyInternetGateway
      RouteTableId: !Ref MyRouteTable

To add these new resources to the existing stack, you call the UpdateStack API:

> aws cloudformation update-stack --stack-name MyNetworkStack --template-body file://my-network-template.yaml

{
    "StackId": "arn:aws:cloudformation:us-west- 2:123456789012:stack/MyNetworkStack/4af622f0-8a24-11e7-8692-503ac9ec2435"
}

Errors and Rollbacks

There are two classes of errors that can occur in a template: validation errors and semantic errors. A validation error is when AWS CloudFormation cannot parse the template. Forgetting a closing quotation mark, misspelling Resources as Resuorces, specifying an Amazon S3 URL that you do not have permission to access, and incorrectly indenting a YAML file are all examples of validation errors. AWS CloudFormation immediately returns an error when you attempt to call CreateStack or UpdateStack in the presence of a validation error, and the call is aborted before any operations proceed:

> aws cloudformation create-stack --stack-name MyNetworkStack --template-body file://my-network-template.yaml
 
An error occurred (ValidationError) when calling the CreateStack operation: Invalid template property or properties [Resuorces]

A semantic error, on the other hand, is not detected until the resource is created or updated. This happens when AWS CloudFormation attempts to invoke an underlying API on your behalf, but that API call returns an error. When this happens, AWS CloudFormation stops the creation or update process and (by default) attempts to roll back the stack to the previous state. For example, when you omit the DependsOn attribute for the route resource in the previous example, you can see the stack go through the UPDATE_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, and finally the UPDATE_ROLLBACK_COMPLETE states by using either the AWS Management Console (as shown in Figure 10.2) or by calling the DescribeStacks API from the command line.

> aws cloudformation update-stack --stack-name MyNetworkStack --template-body file://bad-network-template.yaml
{
    "StackId": "arn:aws:cloudformation:us-west- 2:1234567890:stack/MyNetworkStack/48b718d0-8b6b-11e7-a582-503f20f2ade6"
}
 
> aws cloudformation describe-stacks --stack-name MyNetworkStack
{
    "Stacks": [
        {
            lines omitted for brevity

            "StackStatusReason": "The following resource(s) failed to create: [MyDefaultRoute, MyGatewayAttachment, MySubnet2cRouteTableAssociation]. ",
            "StackStatus": "UPDATE_ROLLBACK_IN_PROGRESS",
            "DisableRollback": false,
        }
    ]
}

To find the reason for the failure, you would look at the stack events. This would typically be done by going to the AWS CloudFormation console, selecting the stack, and clicking on the events tab to view the errors, as shown in Figure 10.3.

For programmatic use, the DescribeStackEvents API can also be used.

Template Parameters

The template that you have designed hard codes all of the values for your resources. To be able to reuse the template, you will want to customize resource properties such as the subnet Availability Zone and CIDR ranges. While you could edit the template for each stack that you create, this kind of customization makes it difficult to roll out new changes from a centralized team because each customized template must have the patches applied manually.

Template parameters allow users to specify values at stack creation or update time. They are provided in a Parameters block.

Parameters:
  VPCCIDR:
    Type: String
    Default: "10.17.0.0/16"
    Description: The CIDR range to assign to the VPC.
    AllowedPattern: "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/[0-9]{1,2}"
    ConstraintDescription: An IPv4 block in CIDR notation is required, e.g. 10.17.0.0/16
  SubnetAZ:
    Type: AWS::EC2::AvailabilityZone::Name
    Default: us-west-2c
    Description: The availability zone to assign to the subnet.
  SubnetCIDR:
    Type: String
    Default: "10.17.0.0/16"
    Description: The CIDR range to assign to the subnet.
    AllowedPattern: "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/[0-9]{1,2}"
    ConstraintDescription: An IPv4 block in CIDR notation is required, e.g. 10.17.1.0/24

Each parameter must specify a type. Basic types are String, Number (which can be an integer or floating-point number), CommaDelimitedList (a list of strings separated by commas), and List<Number> (a list of numbers separated by commas).

In addition, there are AWS-specific parameter types, such as the AWS::EC2::AvailabilityZone::Name for the SubnetAZ parameter. In the AWS CloudFormation user interface, these are rendered as drop-down menus to make it easier to specify valid inputs, as shown in Figure 10.4. A full list of the valid AWS-specific types can be found in the AWS CloudFormation User Guide.

The Description field is optional but highly recommended; it is displayed alongside the parameter to help guide the user. Default is also optional; having reasonable default values assists new users who are trying your template for the first time.

There are additional fields that allow you to verify user input before changes to the stack are made. The previous example uses AllowedPattern, which specifies a regular expression against which the user input is matched. ConstraintDescription is text that is displayed to the user if the verification fails. Other verification fields are MinLength and MaxLength for strings and MinValue and MaxValue for numbers. Without these validations, the user must wait for a stack creation or update operation to fail and attempt to find the issue in the events.

To use the parameters user interface for the first time, upload the template from the AWS CloudFormation console.

Using parameters within the template uses the Ref intrinsic function introduced earlier. Using parameters for the VPC and subnet resource properties would look like the following listing:

MyVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VPCCIDR
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyVPC
        - Key: Environment
          Value: Testing
  MySubnet2c:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Ref SubnetAZ
      CidrBlock: !Ref SubnetCIDR
      Tags:
        - Key: Name
          Value: MySubnet2c
      VpcId: !Ref MyVPC

It is also possible to specify parameters through the API. This is useful for programmatically specifying the parameters. When updating a stack, you have the option of specifying a new value or instructing AWS CloudFormation to use the previous value. For example, the following would update the Availability Zone and CIDR range of the subnet while keeping the VPC CIDR range intact:

> aws cloudformation update-stack --stack-name MyNetworkStack --use-previous-template --parameters ParameterKey=VPCCIDR,UsePreviousValue=true  ParameterKey=SubnetCIDR,ParameterValue=10.17.2.0/24  ParameterKey=SubnetAZ,ParameterValue=us-west-2b

{
    "StackId": "arn:aws:cloudformation:us-west- 2:123456789012:stack/MyNetworkStack/48b718d0-8b6b-11e7-a582-503f20f2ade6"
}

Verifying Changes with Change Sets

Manually calculating the changes being made when a stack is updated can be difficult. For example, VPC and subnet CIDR ranges are immutable. What might seem like a small change, such as narrowing a CIDR range from /24 to /26, would actually require building a new subnet, re-creating any Amazon EC2 instances in the new subnet, terminating the instances from the old subnet, and deleting the old subnet—a potentially disruptive change. The AWS CloudFormation User Guide lists the impact of each parameter change, but it can be easy to miss a detail in a larger template.

Change sets allow you to visualize and approve a proposed change to a stack before it is updated. From the AWS CloudFormation console, select the stack and then click on Actions and Create Change Set for Current Stack, as shown in Figure 10.5.

After specifying a change set name and giving a new value for the subnet CIDR range, you can examine the changes, as shown in Figure 10.6. Note that the Replacement value for the subnet resource is True, indicating that a new subnet will be created and the old one deleted.

To execute the change set, you can click on the Execute button in the upper-right corner. On the other hand, if you elect not to accept the changes, you can delete the change set by clicking on Other Actions and then Delete.

When updating a stack using the AWS CloudFormation user interface, the review screen automatically creates and displays a change set for the update operation.

Retaining Resources

When you delete a stack, AWS CloudFormation deletes all associated resources by default. You might prefer to retain the resources instead, however. For example, you might instruct a new user to create a stack from a template for a new account to ensure that all networking resources are created. After this is done, the user might want to remove the stack so that it is no longer visible in the AWS Management Console. You have to be able to remove the stack but keep the resulting network configuration.

In the template, you can specify which resources to retain by adding a DeletionPolicy attribute with a value of Retain. This must be applied to each resource. For example, the following would retain the VPC created by the template.

Resources:
  MyVPC:
    Type: AWS::EC2::VPC
    DeletionPolicy: Retain
    Properties:
      CidrBlock: !Ref VPCCIDR
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: MyVPC
        - Key: Environment
          Value: Testing

Configuring Non-AWS Resources

AWS CloudFormation supports almost all AWS Cloud services available. You may also want to configure non-AWS resources when creating or updating a stack.

For example, consider the case of setting up a VPC with only a private subnet connected back to an on-premises network through a VPN, as shown in Figure 10.7. You can configure the VPN gateway, customer gateway, and VPN connection directly through AWS CloudFormation. The on-premises router will require custom configuration that AWS CloudFormation cannot perform natively, however.

Custom resources allow us to bridge this gap. Within a template, a custom resource is a resource of type AWS::CloudFormation::CustomResource or any type in the form Custom::ResourceType. When a custom resource is created, updated, or deleted, AWS CloudFormation sends a notification to an Amazon Simple Notification Service (Amazon SNS) topic or invokes an AWS Lambda function. Within the event, AWS CloudFormation provides the following properties:

• LogicalResourceId: The name of the resource in the stack (for example, MyVPC for the VPC in the previous example).
• OldResourceProperties: The previous user-specified properties during an update operation.
• PhysicalResourceId: The physical resource ID returned by an earlier invocation of your function. This is provided for update and delete requests only.
• RequestId: A unique ID for the request.
• RequestType: Create, Update, or Delete.
• ResourceType: The type specified for the resource.
• ResourceProperties: The user-specified properties for the resource.
• ResponseURL: A pre-signed Amazon S3 URL to which your code must write output.
• StackId: The AWS CloudFormation stack Amazon Resource Name (ARN).

AWS CloudFormation then waits for your code to PUT output to the pre-signed Amazon S3 URL. The body of this object is a JSON mapping object with the following attributes:

• LogicalResourceId: The LogicalResourceId from the request event.
• Status: Whether the custom resource operation was successful (SUCCESS) or if it failed (FAILED)
• Reason: If the request failed, this describes the reason for the failure.
• PhysicalResourceId: If the request was successful, this is an identifier specific to your function.
• RequestId: The RequestId from the request event.
• StackId: The StackId from the request event.
• Data: If the request was successful, this property contains a mapping of arbitrary key-value attributes that can be retrieved using the GetAtt intrinsic function.

To configure a VPN for a router, for example, you could use an AWS Lambda function written in Python that uses the Paramiko library http://www.paramiko.org/ to use Secure Shell (SSH) to access the router and execute router commands.

Security Best Practices

In the previous examples, we have not specified what permissions AWS CloudFormation should have when performing operations. In this case, AWS CloudFormation uses a temporary session created from your credentials when you create or update the stack. If you have administrative permissions, for example, then AWS CloudFormation will also have administrative permissions when executing a stack operation.

A better practice is to limit the permissions to the least privilege necessary for stack management. To do this, you can create an AWS Identity and Access Management (IAM) service role that specifies the calls that AWS CloudFormation can make. This role remains associated with the stack for all future operations. This can be used to avoid granting excessive permissions to end users. For example, a user may not have the expertise required to set up a VPN for a VPC; having them attempt this directly may cause disruption to your organization’s network operations. Instead, you can allow them to use and update an AWS CloudFormation stack from a template using the service role.

If a template includes certain IAM resources, AWS CloudFormation requires you to acknowledge this when creating or updating the stack. You do this by passing the capabilities flag CAPABILITIES_IAM or CAPABILITIES_NAMED_IAM (if the resources have custom names).

To prevent a stack from accidental deletion, stack termination protection can be enabled. To prevent specific resources within a stack from accidentally being replaced, modified, or deleted, you can designate them as protected resources by attaching a stack policy to the stack. For example, the following stack policy prevents the VPC and subnet from the earlier example from being replaced.

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Principal": "*",
            "Resource": "*"
        },
        {
            "Effect": "Deny",
            "Action": "Update:Replace",
            "Principal": "*",
            "Resource": ["LogicalResourceId/MyVPC",
                         "LogicalResourceId/MySubnet"]
        }
    ]
}

For the exam, you should be familiar with AWS CloudFormation concepts, including templates, stacks, native AWS resources types related to networking, custom resources, and change sets.

Configuration Management

Using a local file system to store templates might work for a small organization with a single administrator, but it quickly stops scaling beyond that point. A version control system for checking in the resulting templates is a must. Modern systems such as Git (https://git-scm.com/) are robust yet lightweight enough for even a single administrator to use.

If your organization already has a version control system in place, this can be used to store your AWS CloudFormation templates. Otherwise, you can use AWS CodeCommit to create a Git repository for your template. The AWS CodeCommit User Guide at http://docs.aws.amazon.com/codecommit/latest/userguide/welcome.html has tutorials at http://docs.aws.amazon.com/codecommit/latest/userguide/ getting-started-topnode.html, which guide you through creating a repository and using basic Git commands to interact with it.

Branches allow you and your collaborators to organize your code to avoid conflicts. For most infrastructure template projects, there will just be a single default branch named master by convention. If you were testing something risky such as upgrading an existing VPC to IPv6, you might create a separate ipv6 branch, switch your development there, and create test stacks from it. When you are confident that the changes are correct, you can merge the ipv6 branch back into the master branch.

Continuous Delivery

Keeping AWS CloudFormation templates and test and production stacks in sync is a tedious task if done manually. Many organizations require human review and approval before changes are applied to a production environment, but manually computing these changes is an error-prone process. AWS CodePipeline provides a way to automate the synchronization and change computation steps.

Approvals

Before deploying to production, your organization may require human review of the necessary changes. Typically, you will want to be able to see the actual changes that will be deployed before making an approval decision.

To do this, you typically separate the deployment into three separate actions. The first action is a deployment action that computes an AWS CloudFormation change set but does not execute it. The second action is an approval action, which sends a notification using Amazon SNS, and then waits for an approval signal. The third action is another deployment action that executes the AWS CloudFormation change set created by the first action.

Monitoring Network Health Metrics

Amazon CloudWatch allows you to collect metrics and log files, set alarms, and invoke actions in response to these alarms. Many of the metrics of interest are collected automatically. For example, you can view the state of VPN tunnels and connections by going to the Amazon CloudWatch console, clicking on Metrics, and browsing through the VPN service options. In Figure 10.9, we’ve plotted the state of a VPN connection and the two tunnels. After bringing the connection up, we briefly interrupted one of the tunnels on the connection and then restored it.

When graphing a metric, you can view different statistics, including minimum, maximum, average, and various percentile values. The sum statistic is all of the values submitted over a time period added together. This is typically used for calculating the total volume of a metric—for example, the number of bytes over a VPN tunnel.

You can also create custom metrics using either the AWS CLI or a Software Development Kit (SDK) for a programming language. Metrics belong to a namespace, which can be used to collect related metrics. Metrics are further subdivided by dimensions: up to 10 key/value pairs that contain additional information to identify the metric uniquely. For example, if you wanted to record the packet loss to a given host every 60 seconds, you might run this loop on a Linux host:

#!/bin/sh
remote_ip=192.0.2.17
ping_count=5
 
while true; do
  packet_loss=$(ping -c $ping_count $remote_ip | grep ’packet loss’ | 
                sed -e ’s/^.*received, //’ -e ’s/% packet loss.*//’)
  aws cloudwatch put-metric-data --namespace NetOps 
    --metric-name PacketLoss --unit Percent --value "$packet_loss" 
    --dimensions RemoteIp="$remote_ip";
  sleep 60;
done;

From the Amazon CloudWatch console, you can then view graphs for these metrics. For example, Figure 10.10 shows the plot of packet loss to three hosts.

If you have a networking appliance deployed to Amazon EC2, it may offer integration with Amazon CloudWatch.

To create a unified view of your network health metrics, you can create a dashboard. From a metric graph, selection Actions, and then click on Add to Dashboard. You can add multiple graphs to a dashboard in line, stacked area, or number formats. Figure 10.11 shows a network health dashboard containing metrics about your VPN connection and the Amazon EC2 instances it serves within your VPC.

Creating Alarms for Unusual Events

While dashboards provide you with a view into the health of your network, you don’t want to rely on someone monitoring the dashboard to maintain high availability. Amazon CloudWatch metrics can be used to create alarms that send alerts to an Amazon SNS topic. A topic can send notifications to subscribed email addresses, mobile phones (via SMS or mobile push), HTTP endpoints, AWS Lambda functions, and Amazon Simple Queue Service (Amazon SQS) queues.

To create an alarm, select the alarm bell icon next to a metric graph. You will see a wizard like the one shown in Figure 10.12. In this example, we’ve created a packet loss alarm to one of the endpoints and subscribed the PacketLoss topic to it.

The format of the alarm will vary in a way that is appropriate for the receiver. For example, Figure 10.13 shows the same alarm received over SMS and email for the packet loss metric.

Collecting Text Logs

Having a central repository such as Amazon CloudWatch Logs for collecting human-readable logs generated by applications and networking devices helps correlate events and can help reduce the time to recovery if a system failure occurs. Each log event is a record of some activity. Sequences of log events from the same source are stored in a log stream. Each log stream belongs to a log group that defines retention, monitoring, and access control policies.

For example, when you enable flow logs for a VPC, you specify a log group to which to send events. Within the log group, a log stream is created for each elastic network interface within the VPC. Here is a sample of a flow logs log stream.

2 123456789012 eni-a1b2c3d4 192.0.2.139 172.31.0.5 43160 1433 6 1 40 1504769338 1504769393 REJECT OK
2 123456789012 eni-a1b2c3d4 172.31.0.5 192.0.2.30 123 123 17 1 76 1504769338 1504769393 ACCEPT OK
2 123456789012 eni-a1b2c3d4 192.0.2.109 172.31.0.5 42481 1433 6 1 40 1504769399 1504769453 REJECT OK
2 123456789012 eni-a1b2c3d4 192.0.2.229 172.31.0.5 6000 1433 6 1 40 1504769399 1504769453 REJECT OK
2 123456789012 eni-a1b2c3d4 172.31.0.5 192.0.2.139 0 0 1 1 32 1504769399 1504769453 ACCEPT OK
2 123456789012 eni-a1b2c3d4 192.0.2.139 172.31.0.5 0 0 1 1 32 1504769399 1504769453 ACCEPT OK

The AWS CLI can be used to send log events to a log stream directly. For example, if you have a routing appliance running in Amazon EC2, you could use a script to write your own flow logs using a command such as this:

> aws logs create-log-stream --log-group-name FlowLogs 
  --log-stream-name router-1

> aws logs put-log-events --log-group-name FlowLogs 
  --log-stream-name router-1 
  --log-events timestamp=1483228800000,message="2 123456789012 router-1 192.0.2.254 172.31.0.10 44178 80 6 1 1718 1483228800 1483228860 REJECT OK"

{
    "nextSequenceToken":  "49576537118627494925011401022786533057489097985223586498"
}

It’s more likely that you will want to send an existing log file on the host directly to Amazon CloudWatch Logs, however. The CloudWatch Logs agent provides this functionality. On an Amazon EC2 instance running Amazon Linux, you can install this directly from the package repository.

> sudo yum install -y awslogs
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package awslogs.noarch 0:1.1.2-1.10.amzn1 will be installed
--> Finished Dependency Resolution ...

Installed:
  awslogs.noarch 0:1.1.2-1.10.amzn1
 
Complete!
> # Configure files to send to Amazon CloudWatch Logs
> sudo nano /etc/awslogs/awslogs.conf
> sudo service awslogs start
Starting awslogs:                                          [  OK  ] > sudo chkconfig awslogs on

The default configuration file sends the /var/log/messages file to a log group also named /var/log/messages and uses the Amazon EC2 instance ID as the log stream name.

Converting Logs to Metrics

The text logs captured by Amazon CloudWatch Logs can be filtered and sent to another service for processing or storage. Services capable of receiving these logs are Amazon CloudWatch metrics, AWS Lambda, Amazon Kinesis Streams, and Amazon Elasticsearch.

For example, the following metric filter could be used to match all REJECT entries from a flow log.

[version, account_id, eni_id, src_ip, dst_ip, src_port, dst_port, protocol, packets, bytes, start_time, end_time, action="REJECT", status]

Next, you can send this to Amazon CloudWatch as a metric. The fields that make sense would be either packets or bytes. Specify $bytes as the metric value when creating the metric filter.

When graphing the values, it’s important to keep in mind how the values are being sent to Amazon CloudWatch as data points. For example, in the following abbreviated log stream (with byte values highlighted), there are six individual entries:

2 ... eni-a1b2c3d4 192.0.2.139 172.31.0.5 43160 443 6 2 80 ... REJECT OK
2 ... eni-a1b2c3d4 192.0.2.220 172.31.0.5 44792 443 6 1 40 ... REJECT OK
2 ... eni-a1b2c3d4 192.0.2.109 172.31.0.5 51482 443 6 1 40 ... REJECT OK
2 ... eni-a1b2c3d4 192.0.2.229 172.31.0.5 64809 443 6 1 40 ... REJECT OK
2 ... eni-a1b2c3d4 192.0.2.174 172.31.0.5 37829 443 6 1 40 ... REJECT OK
2 ... eni-a1b2c3d4 192.0.2.139 172.31.0.5 43161 443 6 2 80 ... REJECT OK

The total number of bytes transmitted in this window is 320. A common mistake is to try to graph the maximum value here (80), which just happens to be a host that attempted connections quickly enough to have two packets captured in a single entry. Instead, you need to graph the sum of the data points.

For the exam, you should know how to configure Amazon CloudWatch Logs filters and Amazon CloudWatch metrics. Knowledge of other subscribing services is helpful but not required.