Instance Types

EC2 instances come in a range of sizes, referred to as instance types, to suit various use cases. Instance types differ wildly in the amount of resources allocated to them and, correspondingly, their pricing. The M6g.medium instance type has 4.0 GB of memory and 1 virtual CPU core, whereas its significantly bigger brother C5.18xlarge has 144 GB of memory and 72 virtual CPU cores. Each virtual CPU is a hyperthread of an Intel Xeon core in the C5 instance class, or a dedicated core of an Amazon Graviton2 ARM processor in the case of the M6g class. AMD chips have also recently returned to the forefront of EC2 offerings, with instance types like the very cost-effective t3a.micro (1 GB or memory, 2 vCPUs).

For most of the examples in the book, we will use the t3.micro instance type, among the smaller and one of the cheapest instance types suitable for any operating system choice, making it ideally suited to our demonstrations.

Figure 1-1. Using EC2instances.info to compare the type options available in class m6g

In production, picking the right instance type for each component in your stack is important to minimize costs and maximize performance, and benchmarking can be the key when optimizing this decision.

Processing Power

EC2, along with the rest of AWS, is built using increasingly customized hardware running Amazon’s software to provide the services and APIs. Because Amazon adds this hardware incrementally, several hardware generations are in service at any one time.

Amazon provides a rather vast selection of instance types, the current generation of which is described at the EC2 Instance Types page. The previously mentioned t3a.micro instance type therefore refers to a third generation (3) general-purpose burstable performance instance (t), backed by and AMD processor (a). An immediate update of already running applications is generally not required as older generations remain available for provisioning, with their original functionality intact. We advise operators to adopt instances of the latest generation whenever designing a new or updated application, so as to benefit from the increased capabilities of newer hosting hardware. Amazon’s relentless drive to reduce costs is also often reflected in the improved pricing offered for newer instance classes.

Few EC2 instance types have ever been discontinued in almost 15 years, including cc1, cg1, and hi1. This record is made possible by market forces: as newer instance types become available, their significantly better price/performance ratio induces a user migration away from the previous generation. A reduced demand base in turn allows Amazon to continue to supply those deprecated instance types without having to add capacity with old hardware that may be unavailable. Eventually, AWS reaches out to the few remaining users of the type to help them migrate to a newer instance type-no official deprecation announcements are made.

AWS machine images may make use of either of the two virtualization types originally supported by the Xen hypervisor: paravirtualized or hardware virtual machine (HVM). Recent instance classes use Amazon’s own Nitro hypervisor under the hood, a lightweight, speed-optimized variant of KVM which uses the HVM-style format in combination with NVMe and SR-IOV drivers. It is not necessary to be conversant in the finer differences of virtualization technology¹ to make effective use of AWS, but the two approaches present boot-time differences to the guest OS environment. A given Linux machine image will only support booting one virtualization type as a result, a requirement easily met by filtering any image search with the appropriate virtualization type.

Amazon recommends using HVM virtualization on current-generation AMIs. Where that approach is not suitable, it becomes necessary to determine what virtualization type is supported by the older generation of a specific instance type. This is quickly accomplished by launching a test HVM instance from the AWS CLI and watching for a helpful error message. The AWS documentation also provides insight into what virtualization type is supported by what older instance type.

Different combinations of CPU, memory, network bandwidth, and even custom hardware differentiate AWS instance types. There are forty-two instance type classes in the current generation at the time of writing, including general purpose (the m4, m5, m5a, m5n, m5zn, m6g, a1, mac), burstable performance (t2, t3, t3a, t4g), compute optimized (c4, c5, c5a, c5n, c6), memory optimized (the r5, r6, x2), storage optimized (i3 for random I/O performance, d3 for sequential I/O, h1 for aggregate throughput or d2 for cost), and a range of accelerated types (p2 with GPUs, p3, p4, and inf1 with dedicated machine learning tensor cores, g3 and g4 with advanced graphics, and f1 with programmable FPGA cores). These in turn include a multitude of types with resource allotments of increasing size, bringing the total number of choices we can select from to above three hundred. This wide range includes a number of bare-metal options (the .metal subtype), hosting workloads as varied as macOS (mac1.metal) and SAP HANA (u-24tb1.metal with 24 TB of memory, the instance type with most RAM yet).

With such dizzing variety, the operator configuring a new workload can find solace in limits: unless a very specific need (like hardware acceleration) is specified, one will do well to remember that very high CPU or greedy memory requirements characterize legacy workloads. Modern workload design scales out over multiple nodes, not up within a single instance’s specs.

Taking a scientific approach to benchmarking is the only way to really be sure you are using the right instance type. AWS makes it really simple to run the very same workload configuration with a succession of different instance types, considerably simplifying this task. The most common approach in the AWS user community is to start with an instance type considered high-CPU for the workload under consideration. While running top, drive the CPU to 100% using your application’s load generator of choice. Now examine memory use: if you observe the instance running out of memory before the CPU is at full throttle, switch to a higher-memory instance type. Continue this process until you achieve a reasonable balance.

Alongside fixed-performance instance classes, including the M5, C5, and R5 types, EC2 offers burstable performance instances like the t2 and t3 classes. Burstable performance instances generally receive a CPU performance baseline but can “burst” above this limit for a time. Bursting is governed by CPU credits that are accumulated when the instance runs without its full allotment of CPU. A CPU credit represents use of a full CPU core for one minute.

A practical example will illustrate the accounting mechanism EC2 employs: the t3.micro instance type allocates two virtual CPUs to your cloud instance, with twelve CPU credits earned each hour, representing a 20% share of a virtual CPU core. Let’s assume our workload is a web server, often idling while waiting for requests. If the CPU load falls below the allocated 20% baseline, CPU credits not spent are added to that instance’s credit up to a maximum of 288, an instance-type specific limit representing 24 hours of CPU credits. If a recently-launched instance does not yet have a credit balance to spend (or has run out of credits), in unlimited mode it will accrue an additional flat-rate charge per CPU hour-this is the t3 class’s default behavior.

Burstable performance is particularly useful for workloads that do not consistently use their full share of the CPU, but benefit from having access to additional, fast CPUs when the occasion arises—applications include small databases, web servers, and development systems.

Specific instance types may provide the latest advanced features found in Intel hardware, including on-chip support for AES encryption and the Advanced Vector Extensions instruction set. The P3 instance type is currently the most prominent example of enhanced compute support, featuring 2496 NVIDIA K80 cores and 12 GB of GPU memory. Advanced compute options are rapidly evolving; their most recent iteration is documented in the instance types page, which we recommend you review often.

EC2 instances can be purchased in three ways. Allocated by the hour and requiring no upfront commitment, on-demand instances are the default and are used exclusively throughout this book. Reserved instances represent a prepaid commitment on the part of a customer, which is usually rewarded by AWS with very steep discounts, up to 75% of on-demand pricing. Spot instance pricing requires no upfront commitment, and their pricing and availability fluctuates according to the supply and demand of EC2 compute capacity. The customer may define a maximum hourly price not to be exceeded, and EC2 will automatically shut those instances down if their spot pricing tops the set threshold.

Storage

There are two options when it comes to virtual disk storage for your instances: instance storage (also known as ephemeral storage) and Elastic Block Store (or EBS). Both are simply block storage devices that can be attached to instances. Once attached, they can be formatted with your operating system’s tools and will act like a standard disk. AWS storage comes in two flavors: magnetic disks and solid-state drives (SSDs). SSDs provide higher read and write performance when compared to magnetic disks, but the cost is slightly higher.

There are some key differences between instance storage and EBS. Instance storage is directly attached to the physical host that runs your instance, whereas EBS is attached over the network. This has implications in terms of disk latency and throughput, so we recommend performing another series of benchmarks to see which is best if your application is sensitive to latency or I/O jitter.

I/O speeds are not the only difference—EBS has features that make it preferable to instance storage in nearly all usage scenarios. One of the most useful is the ability to create a snapshot from an EBS. A snapshot is a copy of an EBS volume at a particular point in time. Once you have created a snapshot, you can then create additional EBS volumes that will be identical copies of the source snapshot. You could, for example, create a snapshot containing your database backups. Every time a new instance is launched, it will have a copy of the data ready for use. EBS snapshots form the backbone of many AWS backup strategies.

When an instance is terminated, any data stored on instance storage volumes is lost permanently. EBS volumes can persist after the instance has been terminated. Given all of the additional features, using EBS volumes is clearly preferable except in a few cases, such as when you need fast temporary storage for data that can be safely discarded.

Multiple volumes (of either type) can be attached to an instance, leading to pretty flexible storage configurations. The Block Device Mapping facility allows multiple volumes to be associated with an instance at boot time. It is even possible to attach multiple volumes to an instance and build a software RAID array on them—an advantage of volumes appearing as block storage devices to the operating system.

In June 2012, AWS began offering SSDs as a higher-performance alternative to magnetic storage, and over time introduced multiple options with different performance levels and cost. Some instance types now include an SSD-backed instance store to deliver very-high random I/O performance, with types I2 and R3 being the first to support TRIM extensions. Instance types themselves have evolved to include high-I/O instances (type I3), aimed at delivering high IOPS from up to 8 local SSD drives, while dense storage instances (type D3) offer the lowest price per-disk throughput in EC2 and balance cost and performance, using 24 local magnetic drives.

EBS Magnetic and SSD volumes are currently limited to 16 TB in size, limits easily exceeded by dense storage (d3) instances, which can boot with 48 TB of local disk storage. Whereas EBS volumes can be provisioned at any time and in arbitrary configurations, the number and size of available instance store volumes varies with instance type, and can only be attached to an instance at boot time. In addition, EBS volumes can be dynamically resized, which is also used to redefine their performance at runtime.

Figure 1-2. ncdu can tally a 2 GB solid-state EBS filesystem in under one second

EBS SSD options include a number of performance flavors. General-purpose SSD volumes (gp2) are provisioned with 3 IOPS per GB, with burst performance reaching 3,000 IOPS for extended periods, while gp3 volumes can provision IOPS independent of capacity. Provisioned IOPS SSD volumes allow the user to define the desired level of performance, up to 64,000 IOPS and 1000 MB/s of throughput. A less costly option is offered by the EBS-optimized instances , which include dedicated EBS bandwidth between 500 and 4,000 Mbps depending on the specific instance type. At the time of writing M6g, M5, M4, C6g, C5, C4, R6g, P3, P2, G3, and D2 instance types are launched as EBS-optimized by default, with no additional charges. EBS-optimized instances use a tuned configuration stack requiring corresponding support on the machine image’s part for optimal performance (see “Finding Ubuntu Images” for details on locating optimized images).

Long-term storage options are best supported by the S3 service, but a block storage option is available through Cold HDD EBS volumes. Backed by magnetic drives, Cold HDD volumes offer the lowest cost per GB of all EBS volume types, and still provide enough performance to support a full-volume scan at burst speeds. EBS also supports native at-rest encryption that is transparently available to EC2 instances and requires very little effort on the administrator’s part to deploy and maintain. EBS encryption has no IOPS performance impact and shows very limited impact on latency, making it a general-purpose architectural option even when high security is not strictly required.

AWS storage options also include EFS, a scalable, cloud-native implementation of NFS. While not suitable as a boot device, EFS filesystems can be mounted and shared by multiple instances, even across regions, unlike filesystems built on a block EBS device. Just as is the case for some provisioned IOPS volumes, throughput and IOPS of EFS shares scale with the size of a filesystem and can provide burst performance to accomodate the quirky performance needs of file workloads-EFS can support performance over 10 GB/sec and peaks at 500000 IOPS.

Shared persistent storage across cloud-native workloads should rely on an object store-Amazon S3 in the case of AWS. Period. Shared filesystems are the telltale sign of legacy applications migrated into a cloud datacenter, and that is the bread-and-butter use case for EFS: the lift-and-shift transfer of an older application designed to share data over NFS. EFS can help you shoehorn a legacy workload into EC2, but that a cloud workload does not make. If you are to carry out such a migration, our advice is to make sure the effort required does not exceed that of redesigning the application with a modern architecture.

Networking

At its simplest, networking in AWS is straightforward—launching an instance with the default networking configuration will give you an instance with a public IP address. Many applications will require nothing more complicated than enabling SSH or HTTP access. At the other end of the scale, Amazon offers more-advanced solutions that can, for example, give you a secure VPN connection from your datacenter to a Virtual Private Cloud (VPC) within EC2.

At a minimum, an AWS instance has one network device attached. The maximum number of network devices that can be attached depends on the instance type. Running ip addr show on your instance will show that it has a private IP address in the default 172.31.0.0/16 range. Every instance has a private IP and may have a public IP; this can be configured at launch time or later, with the association of an Elastic-IP address. Elastic IP addresses remain associated with your account, and can be carried between instances.

Amazon Virtual Private Cloud enables you to provision EC2 instances in a virtual network of your own design. A VPC is a network dedicated to your account, isolated from other networks in AWS, and completely under your control. You can create subnets and gateways, configure routing, select IP address ranges, and define its security perimeter—a series of complex tasks that are bypassed by the existence of the default VPC. The default VPC includes a default subnet in each availability zone, along with routing rules, a DHCP setup, and an internet gateway (Figure 1-3). The default VPC enables new accounts to immediately start launching instances without having to first master advanced VPC configuration, but its security configuration will not allow instances to accept connections from the internet until we expressly give our permission, by assigning our own security group settings.

Figure 1-3. The default VPC in the newest rendition of the Console UI

The default security group allows all outbound traffic from instances to reach the internet, and also permits instances in the same security group to receive inbound traffic from one another, but not from the outside world. Instances launched in the default VPC receive both a public and a private IP address. Behind the scenes, AWS will also create two DNS entries for convenience.

For example, if an instance has a private IP of 172.31.16.166 and a public IP of 54.152.163.171, their respective DNS entries will be ip-172-31-16-166.ec2.internal and ec2-54-152-163-171.compute-1.amazonaws.com. These DNS entries are known as the private hostname and public hostname.

It is interesting to note that Amazon operates a split-view DNS system, which means it is able to provide different responses depending on the source of the request. If you query the public DNS name from outside EC2 (not from an EC2 instance), you will receive the public IP in response. However, if you query the public DNS name from an EC2 instance in the same region, the response will contain the private IP:

# From an EC2 instance
$ dig ec2-54-152-163-171.compute-1.amazonaws.com +short
172.31.16.166
# From Digital Ocean
$ dig ec2-54-152-163-171.compute-1.amazonaws.com +short
54.152.163.171

The purpose of this is to ensure that traffic does not leave the internal EC2 network needlessly. This is important as AWS has a highly granular pricing structure when it comes to networking, and Amazon makes a distinction between traffic destined for the public internet and traffic that will remain on the internal EC2 network. The full breakdown of costs is available on the EC2 Pricing page.

If two instances in the same availability zone communicate using their private IPs, the data transfer is free of charge. However, using their public IPs will incur internet transfer charges on both sides of the connection. Although both instances are in EC2, using the public IPs means the traffic will need to leave the internal EC2 network, which will result in higher data transfer costs.

By using the private IP of your instances when possible, you can reduce your data transfer costs. AWS makes this easy with their split-horizon DNS system: as long as you always reference the public hostname of the instance (rather than the public IP), AWS will pick the cheapest option.

Most of the early examples in the book use a single interface with default DNS arrangements, but we will look at more sophisticated topologies in later chapters.

$ curl -O https://ip-ranges.amazonaws.com/ip-ranges.json &> /dev/null
# Find the IP address ranges served by us-east-1
$ jq '.prefixes[] | select(.region=="us-east-1")' < ip-ranges.json
{
  "ip_prefix": "3.2.0.0/24",
  "region": "us-east-1",
  "service": "AMAZON",
  "network_border_group": "us-east-1-iah-1"
}
{
  "ip_prefix": "15.230.137.0/24",
  "region": "us-east-1",
[ output truncated ]
# Find the IP address ranges used by EC2
$ jq -r '.prefixes[] | select(.service=="EC2") | .ip_prefix' < ip-ranges.json
3.5.140.0/22
35.180.0.0/16
3.2.0.0/24
[ output truncated ]

Launching from the Management Console

Most people take their first steps with EC2 via the Management Console, which is the public face of EC2. Our first journey through the Launch Instance Wizard will introduce a number of new concepts, so we will go through each page in the wizard and take a moment to look at each of these in turn. Although there are faster methods of launching an instance, the wizard is certainly the best way to familiarize yourself with related concepts.

Launching a new instance of an AMI

To launch a new instance, first log in to Amazon’s web console, open the EC2 section, and click Launch Instance. This shows the first in a series of pages that will let us configure the instance options. The first of these pages is shown in Figure 1-4.

Figure 1-4. AMI selection

As described earlier, Amazon Machine Images (AMIs) are used to launch instances that already have the required software packages installed, configured, and ready to run. Amazon provides AMIs for a variety of operating systems, and the Community and Marketplace AMIs provide additional choices. For example, Canonical provides officially supported AMIs for various versions of its Ubuntu operating system. Other open source and commercial operating systems are also available, both with and without support. The AWS Marketplace lets you use virtual appliances created by Amazon or third-party developers. These are Amazon Machine Images already configured to run a particular set of software; for example, many variations of AMIs running the popular WordPress blogging software exist. While some of these appliances are free to use (i.e., you only pay for the underlying AWS resources you use), others require you to pay an additional fee on top of the basic cost of the Amazon resources.

If this is your first time launching an instance, the My AMIs tab will be empty. Later in this chapter, we will create our own custom AMIs, which will subsequently become available in this tab. The Quick Start tab lists several popular AMIs that are available for public use.

Click the Select button next to the Amazon Linux 2 AMI, while leaving the architecture selector on its x86 default. This presents you with a selection of instance types to choose from (Figure 1-5).

Figure 1-5. Selecting the instance type

EC2 instances come in a range of shapes and sizes to suit many use cases. In addition to offering increasing amounts of memory and CPU power, instance types also offer differing ratios of memory to CPU. Different components in your infrastructure will vary in their resource requirements, so it can pay to benchmark each part of your application to see which instance type is best for your particular needs. The instance family selector at the top of the web page can help reduce clutter by narrowing down selection to a specific instance class. The Instance Type Explorer can provide some higher level insight to start your search from. You can also find useful community-developed resources to quickly compare instance types at EC2instances.info.

The t2.micro instance type is part of Amazon’s free usage tier in the us-east-1 region. New customers can use 750 instance-hours free of charge each month of the Linux and Windows instance types labeled “free tier elegible” in the wizard. This varies by region, as Amazon has been generously upgrading the free tier to the more powerful t3.micro. After exceeding these limits, normal on-demand prices will apply.

Select the checkbox next to t3.micro and click Review and Launch to move directly to the final screen, taking a shortcut from step 2 to step 7. Do feel free to use the slower t2.micro type if you have free credits to spend. Now you are presented with the review screen, which gives you a chance to confirm your options before launching the instance.

After reviewing the options, click Launch to spin up the instance. At the time of this writing, the wizard’s Quick Start process will automatically create a convenient launch-wizard-1 security group granting the instance SSH access from the internet at large. This is not the default security group previously discussed, and this helpfulness is not present when using the AWS CLI or API interfaces to create instances (Figure 1-6).

Figure 1-6. The Review screen (the prominent security warning is alerting you that SSH access has been opened with a default security group)

Key pairs

After clicking Launch, one last screen presents the available key pairs options (Figure 1-7).

Figure 1-7. Key pair selection

Key pairs provide secure access to your instances. To understand the benefits of key pairs, consider how we could securely give someone access to an AMI that anyone in the world can launch an instance of. Using default passwords would be a security risk, as it is almost certain some people would forget to change the default password at some point. Amazon has implemented SSH key pairs to help avoid this eventuality. Of course, it is possible to create an AMI that uses standard usernames and passwords, but this is not the default for AWS-supplied AMIs.

All AMIs have a default user: when an instance is booted, the public part of your chosen key pair is copied to that user’s SSH authorized keys file. This ensures that you can securely log in to the instance without a password. In fact, the only thing you need to know about the instance is the default username and its IP address or hostname.

This also means that only people with access to the private part of the key pair will be able to log in to the instance. Sharing your private keys is against security best practices, so to allow others access to the instance, you will need to create additional user accounts and configure them with passwords or (better yet) SSH authorized keys.

Note

The name of the default user varies between AMIs. For example, Amazon’s own AMIs use ec2-user, whereas Ubuntu’s official AMIs use ubuntu.

If you are unsure of the username, one trick you can use is to try to connect to the instance as root. The most friendly AMIs present an error message informing you that root login is disabled, and letting you know which username you should use to connect instead.

Changing the default user of an existing AMI is not recommended, but can be easily done. The details of how to accomplish this have been documented by Eric Hammond of Alestic. The following table enumerates default usernames for most popular Linux distributions:

Distribution	Default Username
Amazon Linux	ec2-user
Ubuntu	ubuntu
Debian	admin
RHEL	ec2-user (since 6.4), root (before 6.4)
CentOS	root
Fedora	ec2-user
SUSE	root
FreeBSD	ec2-user
BitNami	bitnami

You can create a new SSH key pair through the EC2 Key Pairs page in the AWS Management Console—note that key pairs are region-specific, and this URL refers to the US East 1 region. Keys you create in one EC2 region cannot be immediately used in another region, although you can, of course, upload the same key to each region instead of maintaining a specific key pair for each region. After creating a key, a .pem file will be automatically downloaded.

Alternatively, you can upload the public part of an existing SSH key pair to AWS. This can be of great help practically because it may eliminate the need to add the -i /path/to/keypair.pem option to each SSH command where multiple keys are in use (refer to ssh-agent’s man page if you need to manage multiple keys). It also means that the private part of the key pair remains entirely private—you never need to upload this to AWS, it is never transmitted over the internet, and Amazon does not need to generate it on your behalf, all of which have security implications.

Tip

Alestic offers a handy Bash script to import an existing public SSH key into each region.

Please note that EC2 does not accept the use of deprecated DSA keys. Imported keys need to be generated for the RSA algorithm.

From the Key Pairs screen in the launch wizard, you can select which key pair will be used to access the instance, or to launch the instance without any key pair. You can select from your existing key pairs or choose to create a new key pair. It is not possible to import a new key pair at this point—if you would like to use an existing SSH key that you have not yet uploaded to AWS, you will need to upload it first, just proceed following the instructions outlined on the EC2 Key Pairs page.

Once you have created a new key pair or imported an existing one, click “Choose from your existing Key Pairs,” select your key pair from the drop-down menu, and continue to the next screen. You have now completed the last step of the wizard—click Launch Instances to create the instance.

Tip

If you are a Windows user connecting with PuTTY, you will need to convert this to a PPK file using PuTTYgen before you can use it. To do this, launch PuTTYgen, select Conversions → Import Key, and follow the on-screen instructions to save a new key in the correct format. Once the key has been converted, it can be used with PuTTY and PuTTY Agent.

Figure 1-8. The expanded Launch Log details confirm a successful instance launch

Figure 1-9. The instance details view in the AWS Console

$ ec-2metadata --instance-id
instance-id: i-017d22a731fbcf6f9
$ ec2-metadata --ami-id
ami-id: ami-0be2609ba883822ec

#! /bin/bash
INSTANCE_ID=$(ec2-metadata --instance-id | cut -f2 -d' ')
AMI_ID=$(ec2-metadata --ami-id | cut -f2 -d' ')
echo "The instance $INSTANCE_ID was created from AMI $AMI_ID"

Terminating the instance

Once you have finished testing and exploring the instance, you can terminate it. In the Management Console, right-click the instance and select Terminate Instance.

Next, we will look at some of the other available methods of launching instances.

Tip

Since early 2013, Amazon has supplied operators with a mobile app interface to the AWS Management Console with versions supporting both iOS and Android devices. After multiple updates and enhancements, the app has become a reliable tool for administrators who need a quick look at the state of their AWS deployment while on the move.

The app’s functionality is not as comprehensive as the web console’s, but it showcases remarkable usability in its streamlined workflows (see Figure 1-10 for an example), and operators enjoy the quick access to select functionality it provides for over 15 services. Some users now even pull up their mobile phone to execute certain tasks rather than resorting to their trusted terminal!

Figure 1-10. Three side-by-side views of the AWS console mobile app

Launching with Command-Line Tools

If you followed the steps in the previous section, you probably noticed a few drawbacks to launching instances with the Management Console. The number of steps involved and the variety of available options engender complex documentation that takes a while to absorb. This is not meant as criticism of the Management Console—EC2 is a complex beast, thus any interface to it requires a certain level of complexity. Because AWS is a self-service system, it must support the use cases of many users, each with differing requirements and levels of familiarity with AWS itself. By necessity, the Management Console is equivalent to an enormous multipurpose device that can print, scan, fax, photocopy, shred, and collate.

This flexibility is great when it comes to discovering and learning the AWS ecosystem, but is less useful when you have a specific task on your to-do list that must be performed as quickly as possible. Interfaces for managing production systems should be streamlined for the task at hand, and not be conducive to making mistakes.

Documentation should also be easy to use, particularly in a crisis, and the Management Console does not lend itself well to this. Picture yourself in the midst of a downtime situation, where you need to quickly launch some instances, each with different AMIs and user data. Would you rather have to consult a 10-page document describing which options to choose in the Launch Instance Wizard, or copy and paste some commands into the terminal?

Fortunately, Amazon gives us precisely the tools required to do the latter. The EC2 command-line tools can be used to perform any action available from the Management Console, in a fashion that is much easier to document and much more amenable to automation.

If you have not already done so, you will need to set up the EC2 command-line tools according to the instructions in before continuing. Make sure you have set the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables or the equivalent values in the .aws/credentials file in your home directory.

# Describe all of your own images in the US East region
aws ec2 describe-images --owners self --region us-east-1

# Find Amazon-owned images for Windows Server 2019, 64-bit version
aws ec2 describe-images --owners amazon --filters Name=architecture,Values=x86_64 | grep Server-2019

# List the AMIs that have a specific set of key/value tags
aws ec2 describe-images --owners self --filters Name=tag:role,Values=webserver Name=tag:environment,Values=production

$ aws ec2 run-instances --image-id ami-0be3f0371736d5394 --region us-east-1 
--key federico --instance-type t2.micro --output text
740376006796	r-088ffb963f76af643
INSTANCES	0	x86_64	789c6f18-af67-435d-b4dd-75552b81a2f4	False	True	xen	ami-0be3f0371736d5394	i-0a71dbbba93ec6881	t2.micro	federico	2021-01-03T16:08:41.000Z	ip-172-31-36-115.ec2.internal	172.31.36.115		/dev/sda1	ebs	True		subnet-86a83ebb	hvm	vpc-934935f7
[ output truncated ]

ssh-import-id lp:f2

ssh-import-id gh:mikery gh:jbz

wget https://launchpad.net/~f2/+sshkeys -0 - >>
~/.ssh/authorized_keys && echo >> ~/.ssh/authorized_keys

$ aws ec2 describe-instance-status --instance-ids i-0a71dbbba93ec6881 
--region us-east-1 --output text
INSTANCESTATUSES	us-east-1e	i-0a71dbbba93ec6881
INSTANCESTATE	16	running
INSTANCESTATUS	ok
DETAILS	reachability	passed
SYSTEMSTATUS	ok
DETAILS	reachability	passed

$ aws ec2 describe-instances --instance-ids i-0a71dbbba93ec6881 
--region us-east-1 --output text
RESERVATIONS	740376006796	r-088ffb963f76af643
INSTANCES	0	x86_64	789c6f18-af67-435d-b4dd-75552b81a2f4	False	True	xen	ami-0be3f0371736d5394	i-0a71dbbba93ec6881	t2.micro	federico	2021-01-03T16:08:41.000Z	ip-172-31-36-115.ec2.internal	172.31.36.115	ec2-3-90-65-67.compute-1.amazonaws.com	3.90.65.67	/dev/sda1	ebs	True		subnet-86a83ebb	hvm	vpc-934935f7
BLOCKDEVICEMAPPINGS	/dev/sda1
EBS	2021-01-03T16:08:42.000Z	True	attached	vol-02497364a07d7b45f
[ output truncated ]
MONITORING	disabled
NETWORKINTERFACES		interface	06:89:16:4e:21:bb	eni-099ae3a3deede2346	740376006796	ip-172-31-36-115.ec2.internal	172.31.36.115	True	in-use	subnet-86a83ebb	vpc-934935f7
ASSOCIATION	amazon	ec2-3-90-65-67.compute-1.amazonaws.com	3.90.65.67
ATTACHMENT	2021-01-03T16:08:41.000Z	eni-attach-008a0c0b75b768929	True	0	0	attached
GROUPS	sg-384f3a41	default
PRIVATEIPADDRESSES	True	ip-172-31-36-115.ec2.internal	172.31.36.115
ASSOCIATION	amazon	ec2-3-90-65-67.compute-1.amazonaws.com	3.90.65.67
PLACEMENT	us-east-1e		default
SECURITYGROUPS	sg-384f3a41	default
STATE	16	running

$ aws ec2 terminate-instances --instance-ids i-0a71dbbba93ec6881 
--region us-east-1 --output text
TERMINATINGINSTANCES	i-0a71dbbba93ec6881
CURRENTSTATE	32	shutting-down
PREVIOUSSTATE	16	running
$ aws ec2 describe-instances --instance-ids i-0a71dbbba93ec6881 
--region us-east-1 --output text
RESERVATIONS	740376006796	r-088ffb963f76af643
INSTANCES	0	x86_64	789c6f18-af67-435d-b4dd-75552b81a2f4	False	True	xen	ami-0be3f0371736d5394	i-0a71dbbba93ec6881	t2.micro	federico	2021-01-03T16:08:41.000Z			/dev/sda1	ebs	User initiated (2021-01-03 17:09:18 GMT)	hvm
[ output truncated ]
STATE	48	terminated
STATEREASON	Client.UserInitiatedShutdown	Client.UserInitiatedShutdown: User initiated shutdown

$ ping -o 52.90.56.122; sleep 2; ssh [email protected]
PING 52.90.56.122 (52.90.56.122): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from 52.90.56.122: icmp_seq=3 ttl=48 time=40.492 ms
[ output truncated ]

Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1029-aws x86_64)

$ until ssh [email protected]; do sleep 1; done
ssh: connect to host 52.90.56.122 port 22: Connection refused
ssh: connect to host 52.90.56.122 port 22: Connection refused
[ output truncated ]

Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1029-aws x86_64)

Launching from Your Own Programs and Scripts

The command-line tools are useful from an automation perspective, as it is easy to call them from Bash or any other scripting language. While the output for some of the services can be rather complex, it is relatively straightforward to parse this output and perform dynamic actions based on the current state of your infrastructure. At a certain level of complexity, though, calling all of these external commands and parsing their output becomes time-consuming and error prone. At this point, it can be useful to move to a programming language with a client library to help you work with AWS directly.

Officially supported client libraries are available for many programming languages and platforms, including:

• Go

• Java

• PHP

• Python

• Ruby

• .NET

• iOS

• Android

The full set of AWS programming resources can be found at the AWS Sample Code site.

Most of the API coding examples in this book use the popular Python-based Boto library although other, equally capable libraries exist. Even if you are not a Python developer, the examples should be easy to transfer to your language of choice, because each library is calling the same underlying AWS API.

Regardless of your language choice, the high-level concepts for launching an instance remain the same: first, decide which attributes you will use for the instance, such as which AMI it will be created from, and then issue a call to the RunInstances method of the EC2 API.

When exploring a new API from Python, it can often be helpful to use the interactive interpreter. This lets you type in lines of Python code one at a time, instead of executing them all at once in a script. The benefit here is that you have a chance to explore the API and quickly get to grips with the various functions and objects that are available. We will use this method in the upcoming examples. If you prefer, you can also copy the example code to a file and run it all in one go with python filename.py.

If you do not already have the Boto library installed, you will need to install it with pip (pip3 install boto) before continuing with the examples. Once this is done, open the Python interactive interpreter by running python without any arguments:

$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

When you connect to an AWS service with Boto, Boto needs to know which credentials it should use to authenticate. You can explicitly pass the aws_access_key_id and aws_secret_access_key keyword arguments when calling connect_to_region, as shown here:

>>> AWS_ACCESS_KEY_ID = "your-access-key"
>>> AWS_SECRET_ACCESS_KEY = "your-secret-key"
>>> from boto.ec2 import connect_to_region
>>> ec2_conn = connect_to_region('us-east-1',
... aws_access_key_id=AWS_ACCESS_KEY_ID,
... aws_secret_access_key=AWS_SECRET_ACCESS_KEY)

Alternatively, if the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set, Boto will use these automatically:

$ export AWS_SECRET_ACCESS_KEY='your access key'
$ export AWS_ACCESS_KEY_ID='your secret key'
$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from boto.ec2 import connect_to_region
>>> ec2_conn = connect_to_region('us-east-1')

Boto will also automatically attempt to retrieve your credentials from the file ~/.aws/credentials if one is present, in which case exporting them to environment variables is not necessary.

Once you have connected to the EC2 API, you can issue a call to run_instances to launch a new instance. You will need two pieces of information before you can do this—the ID of the AMI you would like to launch, and the name of the SSH key pair you will use when connecting to the instance:

>>> ssh = ec2_conn.create_security_group('ssh', 'SSH access group')
>>> ssh
SecurityGroup:ssh
>>> ssh.authorize('tcp', 22, 22, '0.0.0.0/0')
True
>>> reservation = ec2_conn.run_instances('ami-0be3f0371736d5394', 
... instance_type='t3.micro', key_name='your-key-pair-name', 
... security_group_ids=['ssh'])
>>> instance = reservation.instances[0]

The call to run_instances does not, as might initially be suspected, return an object representing an instance. Because you can request more than one instance when calling the run_instances function, it returns a reservation, which is an object representing one or more instances. The reservation object lets you iterate over the instances. Here, we requested only one instance, so we simply took the first element of the list of instances in the reservation (in Python, that is done with reservation.instances[0]) to get our instance.

Now the instance is launching, and we have an instance (in the programming sense) of the instance (in the EC2 sense), so we can begin to query its attributes. Some of these are available immediately, whereas others do not get set until later in the launch process. For example, the DNS name is not available until the instance is nearly running. The instance will be in the pending state initially. We can check on the current state by calling the update() function:

>>> instance.state
'pending'
>>> instance.update()
'pending'
# After some time…
>>> instance.update()
'running'

Once the instance reaches the running state, we should be able to connect to it via SSH. But first we need to know its hostname or IP address, which are available as attributes on the instance object:

>>> instance.public_dns_name
'ec2-3-214-215-179.compute-1.amazonaws.com'
>>> instance.private_ip_address
'172.31.91.174'
>>> instance.id
'i-0bf338951ba9b23f8'

Terminating a running instance is just a matter of calling the terminate() function. Before we do that, let’s take a moment to look at how Boto can work with EC2 tags to help make administration easier. A tag is a key/value pair that you can assign to any number of instances to track arbitrary properties. The metadata stored in tags can be used as a simple but effective administration database for your EC2 resources. Setting a tag is simple:

>>> ec2_conn.create_tags([instance.id], {'environment': 'staging'})
True

Once an instance has been tagged, we can use the get_all_instances() method to find it again. get_all_instances() returns a list of reservations, each of which, in turn, contains a list of instances. These lists can be iterated over to perform an action on all instances that match a specific tag query. As an example, we will terminate any instances that have been tagged as being part of our staging environment:

>>> tagged_reservations = ec2_conn.get_all_instances(filters={
... 'tag:environment': 'staging'})
>>> tagged_reservations
[Reservation:r-066e2f09d3ea91640]
>>> tagged_reservations[0]
Reservation:r-066e2f09d3ea91640
>>> tagged_reservations[0].instances[0]
Instance:i-0bf338951ba9b23f8
>>> for res in tagged_reservations:
...     for inst in res.instances:
...         inst.terminate()
>>>

The previous example iterated over all the matching instances (only one, in this case) and terminated them. We can now check on the status of our instance and see that it is heading toward the terminated state:

>>> instance.update()
'shutting-down'
# After a moment or two…
>>> instance.update()
'terminated'

This example only scratches the surface of what Boto and other client libraries are capable of. The Boto documentation provides a more thorough introduction to other AWS services. Having the ability to dynamically control your infrastructure is one of the best features of AWS from a system administration perspective, and it gives you plenty of opportunities to automate recurring processes.

Creating the Stack

In this section, we will start with a basic stack template that simply launches an EC2 instance. Example 1-1 shows one of the simplest CloudFormation stacks.

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "A simple stack that launches an instance.",
  "Resources" : {
    "Ec2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "InstanceType": "t3.micro",
        "ImageId" : "ami-0be3f0371736d5394"
      }
    }
  }
}

aws cloudformation validate-template 
--template-body file://MyStack.json

The Resources section is an object that can contain multiple children, although this example includes only one (EC2Instance). The EC2Instance object has attributes that correspond to the values you can choose when launching an instance through the Management Console or command-line tools.

CloudFormation stacks can be managed through the Management Console, with the command-line tools, or with scripts leveraging client-side libraries such as Boto.

One advantage of using the Management Console is that a list of events is displayed in the bottom pane of the interface. With liberal use of the refresh button, this will let you know what is happening when your stack is in the creating or updating stages. Any problems encountered while creating or updating resources will also be displayed here, which makes it a good place to start when debugging CloudFormation problems. These events can also be read by using the command-line tools, but the Management Console output is a much more friendly human interface.

It is not possible to simply paste the stack template file contents into the Management Console. Rather, you must create a local text file and upload it to the Management Console when creating the stack. Alternatively, you can make the stack file accessible on a website and provide the URL instead. The same applies when using the command-line tools and API.

To see the example stack in action, copy the JSON shown in Example 1-1 into a text file. You may need to substitute the AMI (ami-0be3f0371736d5394) with the ID of an AMI in your chosen EC2 region (our preset value comes from the ever-popular default us-east-1). Use the command-line tools or Management Console to create the stack. Assuming you have stored your stack template in a file named example-stack.json, you can create the stack with this command:

 aws cloudformation create-stack --template-body file://example-stack.json 
--stack-name example-stack

If your JSON file is not correctly formed, you will see a helpful message letting you know the position of the invalid portion of the file. If CloudFormation accepted the request, it is now in the process of launching an EC2 instance of your chosen AMI. You can verify this with the aws cloudformation describe-stack-resources and aws cloudformation describe-stack-events commands:

$ aws cloudformation describe-stack-events --stack-name example-stack 
--output text
STACKEVENTS	6e21cdf0-4ead-11eb-9b7d-0a78092beb51	example-stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5		CREATE_COMPLETE		AWS::CloudFormation::Stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:54.883Z
STACKEVENTS	Ec2Instance-CREATE_COMPLETE-2021-01-04T16:53:53.311Z	Ec2Instance	i-0863b9fb0712d6437	{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro"}	CREATE_COMPLETE		AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:53.311Z
STACKEVENTS	Ec2Instance-CREATE_IN_PROGRESS-2021-01-04T16:53:37.683Z	Ec2Instance	i-0863b9fb0712d6437	{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro"}	CREATE_IN_PROGRESS	Resource creation Initiated	AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:37.683Z
STACKEVENTS	Ec2Instance-CREATE_IN_PROGRESS-2021-01-04T16:53:35.910Z	Ec2Instance		{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro"}	CREATE_IN_PROGRESS		AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:35.910Z
STACKEVENTS	609db8b0-4ead-11eb-a444-12cd418968c5	example-stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5		CREATE_IN_PROGRESS	User Initiated	AWS::CloudFormation::Stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:32.249Z

$ aws cloudformation describe-stack-resources --stack-name example-stack 
--output text
STACKRESOURCES	Ec2Instance	i-0863b9fb0712d6437	CREATE_COMPLETE	AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T16:53:53.311Z
DRIFTINFORMATION	NOT_CHECKED

aws cloudformation describe-stack-events prints a list of events associated with the stack, in reverse chronological order. Following the chain of events, we can see that AWS first created the stack, and then spent less than 20 seconds launching the instance, before declaring the stack creation complete. The second command shows us the ID of the newly launched instance: i-0863b9fb0712d6437.

Updating the Stack

Updating a running stack is an equally straightforward task. But before doing this, a brief digression into the way AWS handles resource updates is called for.

Some attributes of AWS resources cannot be modified once the instance has been created. Say, for example, you launch an EC2 instance with some user data. You then realize that the user data was incorrect, so you would like to change it. Although the Management Console provides the option to View/Change User Data, the instance must be in the stopped state before the user data can be modified. This means you must stop the instance, wait for it to enter the stopped state, modify the user data, and then start the instance again.

This behavior has an interesting implication for CloudFormation. Using the previous example, imagine you have a CloudFormation stack containing an EC2 instance that has some user data. You want to modify the user data, so you update the stack template file and run the aws cloudformation update-stack command. Because CloudFormation is unable to modify the user data on a running instance, it must instead either reload or replace the instance, depending on whether this is an EBS-backed or instance store−backed instance.

If this instance was your production web server, you would have had some unhappy users during this period. Therefore, making changes to your production stacks requires some planning to ensure that you won’t accidentally take down your application. We won’t list all of the safe and unsafe types of updates here, simply because there are so many permutations to consider that it would take an additional book to include them all. The simplest thing is to try the operation you want to automate by using the Management Console or command-line tools to find out whether they require stopping the instance.

We already know that user data cannot be changed without causing the instance to be stopped, because any attempt to change the user data of a running instance in the Management Console will fail. Conversely, we know that instance tags can be modified on a running instance via the Management Console; therefore, updating instance tags with CloudFormation does not require instance replacement.

Changing some other attributes, such as the AMI used for an instance, will also require the instance to be replaced. CloudFormation will launch a new instance using the update AMI ID and then terminate the old instance. Obviously, this is not something you want to do on production resources without taking care to ensure that service is not disrupted while resources are being replaced. Mitigating these effects is discussed later, when we look at Auto Scaling and launch configurations.

If in doubt, test with an example stack first. CloudFormation lets you provision your infrastructure incredibly efficiently—but it also lets you make big mistakes with equal efficiency. With great power (which automation offers) comes great responsibility.

To see the update process in action, we will first update the instance to include some tags. Update the example-stack.json file so that it includes the following line in bold—note the addition of the comma to the end of the first line:

…
        "InstanceType": "t3.micro",
        "Tags": [ {"Key": "foo", "Value": "bar"}], 
        "ImageId": "ami-0be3f0371736d5394"
      }
…

Now we can update the running stack with aws cloudformation update-stack and watch the results of the update process with aws cloudformation describe-stack-events:

$ aws cloudformation update-stack --template-body file://example-stack.json 
--stack-name example-stack --output text
arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5

$ aws cloudformation describe-stack-events --stack-name example-stack 
--output text
STACKEVENTS	4496d650-4ec2-11eb-9a26-12324404f95d	example-stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5		UPDATE_COMPLETE		AWS::CloudFormation::Stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:23:04.620Z
STACKEVENTS	44350740-4ec2-11eb-ad89-0e22fadd816f	example-stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5		UPDATE_COMPLETE_CLEANUP_IN_PROGRESS		AWS::CloudFormation::Stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:23:03.977Z
STACKEVENTS	Ec2Instance-UPDATE_COMPLETE-2021-01-04T19:23:01.921Z	Ec2Instance	i-0863b9fb0712d6437	{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro","Tags":[{"Value":"bar","Key":"foo"}]}	UPDATE_COMPLETE		AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:23:01.921Z
STACKEVENTS	Ec2Instance-UPDATE_IN_PROGRESS-2021-01-04T19:22:46.344Z	Ec2Instance	i-0863b9fb0712d6437	{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro","Tags":[{"Value":"bar","Key":"foo"}]}	UPDATE_IN_PROGRESS	AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:22:46.344Z
STACKEVENTS	3708d600-4ec2-11eb-8902-0e39579e7d83	example-stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5		UPDATE_IN_PROGRESS	User Initiated	AWS::CloudFormation::Stack	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:22:41.882Z

 $ aws cloudformation describe-stack-resources --stack-name example-stack 
 --output text
STACKRESOURCES	Ec2Instance	i-0863b9fb0712d6437	UPDATE_COMPLETE	AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T19:23:01.921Z
DRIFTINFORMATION	NOT_CHECKED

Finally, the aws ec2 describe-tags command will show that the instance is now tagged with foo=bar:

$ aws ec2 describe-tags --filters Name=resource-type,Values=instance 
Name=resource-id,Values=i-0863b9fb0712d6437 --output text
TAGS	aws:cloudformation:logical-id	i-0863b9fb0712d6437	instance	Ec2Instance
TAGS	aws:cloudformation:stack-id	i-0863b9fb0712d6437	instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5
TAGS	aws:cloudformation:stack-name	i-0863b9fb0712d6437	instance	example-stack
TAGS	foo	i-0863b9fb0712d6437	instance	bar

Notice the additional tags in the aws:cloudformation namespace. When provisioning resources that support tagging, CloudFormation will automatically apply tags to the resource. These tags let you keep track of that stack, which “owns” each resource, and make it easy to find CloudFormation-managed resources in the Management Console.

Looking Before You Leap

When you have more than one person working on a stack template, it can be easy to find yourself in a situation where your local copy of the stack template does not match the template used by the running stack.

Imagine that two people are both making changes to a stack template stored in a Git repository. If one person makes a change and updates the stack without committing that change to Git, the next person to make an update will be working with an out-of-date stack template. The next update will then revert the previous changes, which, as previously discussed, could have negative consequences. This is a typical synchronization problem whenever you have two independent activities that could be happening concurrently: in this case, updating Git and updating the actual AWS stack.

Happily, Amazon has provided a tool that, in combination with a couple of shell commands, will let you make certain that your local copy of the stack does indeed match the running version. Use the aws cloudformation get-template command to get a JSON file describing the running template, clean the output with sed and head, and finally use diff to compare the local and remote versions. If we did this before updating the example stack to include tags, we would have obtained the following results:

$ aws cloudformation get-template --stack-name example-stack 
| jq .TemplateBody > example-stack.running 
$ diff <(jq '.' example-stack.running) <(jq '.' example-stack.json)
8a9,14
>         "Tags": [
>           {
>             "Key": "foo",
>             "Value": "bar"
>           }
>         ],

These commands could be wrapped in a simple script to save typing. Changes to production CloudFormation stacks should always be preceded by a check like this, especially if working in a team. This check should be incorporated into the script used for updating the stack: if it happens automatically, there is no chance of forgetting it.

$ jd example-stack.running example-stack.json
@ ["Resources","Ec2Instance","Properties","Tags"]
+ [{"Key":"foo","Value":"bar"}]

Deleting the Stack

Deleting a running stack will, by default, result in the termination of its associated resources. This is quite frequently the desired behavior, so it makes for a sensible default, but at times, you would like the resources to live on after the stack itself has been terminated. This is done by setting the DeletionPolicy attribute on the resource. This attribute has a default value of Delete.

All resource types also support the Retain value. Using this means that the resource will not be automatically deleted when the stack is deleted. For production resources, this can be an added safety net to ensure that you don’t accidentally terminate the wrong instance. The downside is that, once you have deleted the stack, you will need to manually hunt down the retained resources if you want to delete them at a later date.

The final option for the DeletionPolicy attribute is Snapshot, which is applicable only to the subset of resources that support snapshots. Currently, these include the Relational Database Service (RDS) instances, Amazon Redshift data warehousing clusters, ElastiCache Memcached clusters, AWS Neptune graph database instances, and EC2’s EBS volumes. With this value, a snapshot of the database or volume will be taken when the stack is terminated.

Remember that some resources will be automatically tagged with the name of the CloudFormation stack to which they belong. This can save some time when searching for instances that were created with the Retain deletion policy.

Deleting a stack is done with the aws cloudformation delete-stack command. Again, you can view the changes made to the stack with aws cloudformation describe-stack-events:

$ aws cloudformation delete-stack --stack-name example-stack

$ aws cloudformation describe-stack-events --stack-name example-stack --output text
STACKEVENTS	Ec2Instance-DELETE_IN_PROGRESS-2021-01-04T20:27:40.827Z	Ec2Instance	i-0863b9fb0712d6437	{"ImageId":"ami-0be3f0371736d5394","InstanceType":"t3.micro","Tags":[{"Value":"bar","Key":"foo"}]}	DELETE_IN_PROGRESS	AWS::EC2::Instance	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/609b47b0-4ead-11eb-a444-12cd418968c5	example-stack	2021-01-04T20:27:40.827Z
[ output truncated ]

Events are available only until the stack has been deleted. You will be able to see the stack while it is in the DELETE_IN_PROGRESS state, but once it has been fully deleted, aws cloudformation describe-stack-events will fail.

aws cloudformation update-termination-protection 
--stack-name example-stack --enable-termination-protection