AMI Types

In the early days of EC2, the only available AMI type was what is now known as an instance store−backed AMI. As the Elastic Block Store service was introduced and evolved, an additional type of AMI was created: the EBS-backed AMI. The key architectural difference between the two is in where the disk image that contains the root volume is stored.

For EBS-backed AMIs, the disk image is simply an EBS snapshot. When launching a new instance from such an image, a volume is created using this snapshot, and this new volume is used as the root device on the instance.

Instance store−backed AMIs are created from template disk images stored in S3, which means the disk image must be copied from S3 each time an instance is launched, introducing a startup delay over EBS-backed instances. Because the image must be downloaded from S3 each time, the root volume size is also limited to 10 GB, whereas EBS-backed instances have their root volumes limited to a more generous 16 TB. The choice of storage technology used for the AMI also affects instance boot time, with EBS-backed AMIs starting in under one minute whereas instance-store start-up times can reach as high as five minutes.

In practice, an EBS-backed AMI is nearly always the best option. This type of AMI can be temporarily stopped and restarted without losing any data, whereas instance store−backed AMIs can only be terminated, at which point all data stored on the volume is lost.

It is easy to identify what type of storage is in use by a given AMI. In the AWS Console, this attribute is found in the Root Device Type field in the Details tab of an AMI. The command line approach is equally straightforward:

 $ aws ec2 describe-images --image-ids ami-0be3f0371736d5394 | 
jq '.. | .RootDeviceType?'
"ebs"

Building Your Own AMI

AMI builds should be automated as soon as possible, if they take place with any kind of regularity. It is tedious work and involves a lot of waiting around. Automating the process means you’ll probably update AMIs more frequently, reducing a barrier to pushing out new features and software upgrades. Imagine you learn of a critical security flaw in your web server software that must be updated immediately. Having a procedure in place to create new AMIs and push them into production will help you respond to such scenarios rapidly and without wasting lots of time.

To demonstrate the procedures for creating an AMI and some of the useful features that AMIs provide, let’s create an AMI using the command-line tools. This AMI will run an Nginx web server that displays a simple welcome page. We will look at a method of automating this procedure later in the book, in Chapter 10.

Begin by selecting an AMI to use as a base. We will be using our usual Ubuntu 20.04 image with the ID ami-0be3f0371736d5394. Launch an instance of this AMI with aws ec2 run-instances, remembering to specify a valid key pair name and security group granting access to SSH, then use aws ec2 describe-instances to find out the public DNS name for the instance:

$ # if you have not created a security group for SSH access yet,
$ # you need to do that first:

$ aws ec2 create-security-group --group-name ssh --description "SSH Access"
{
    "GroupId": "sg-07ec8513a0bf9acf2"
}

$ aws ec2 authorize-security-group-ingress --group-name ssh --protocol tcp 
--port 22 --cidr 0.0.0.0/0

$ aws ec2 describe-security-groups --group-names ssh --output text
SECURITYGROUPS	SSH Access	sg-07ec8513a0bf9acf2	ssh	740376006796	vpc-934935f7
IPPERMISSIONS	22	tcp	22
IPRANGES	0.0.0.0/0
IPPERMISSIONSEGRESS	-1
IPRANGES	0.0.0.0/0

$ aws ec2 run-instances --image-id ami-0be3f0371736d5394 --region us-east-1 
--key federico  --security-groups ssh --instance-type t3.micro
740376006796	r-070f3b8d826755da5
INSTANCES	0	x86_64	5b656d7d-6324-466a-a6b5-5e581fcc4201	False	True	xen	ami-0be3f0371736d5394	i-0d3f9965712b45500	t3.micro	federico	2021-01-09T16:09:06.000Z	ip-172-31-89-214.ec2.internal	172.31.89.214		/dev/sda1	ebs	True		subnet-c2b704ce	hvm	vpc-934935f7
[ output truncated ]

$ aws ec2 describe-instances --instance-ids i-0d3f9965712b45500 
--region us-east-1 --output text
RESERVATIONS	740376006796	r-070f3b8d826755da5
INSTANCES	0	x86_64	5b656d7d-6324-466a-a6b5-5e581fcc4201	False	True	xen	ami-0be3f0371736d5394	i-0d3f9965712b45500	t3.micro	federico	2021-01-09T16:09:06.000Z	ip-172-31-89-214.ec2.internal	172.31.89.214	ec2-34-236-36-177.compute-1.amazonaws.com	34.236.36.177	/dev/sda1ebs	True		subnet-c2b704ce	hvm	vpc-934935f7
[ output truncated ]

Once the instance has launched, we need to log in via SSH to install Nginx. If you are not using Ubuntu, the installation instructions will differ slightly. On Ubuntu, update the package repositories and install Nginx as follows:

$ ssh ubuntu@ec2-34-236-36-177.compute-1.amazonaws.com
The authenticity of host 'ec2-34-236-36-177.compute-1.amazonaws.com (34.236.36.177)' can't be established.
ECDSA key fingerprint is SHA256:ERzqPqdJzxJwYYqmCF0uYof4y9K4NDTo1pBFkCE7exQ.
Are you sure you want to continue connecting (yes/no)? yes

$ sudo apt update
[ output truncated ]

$ sudo apt install nginx-full --assume-yes
[ output truncated ]

By default, Nginx is installed with a welcome page stored at /usr/share/nginx/html/index.html. If you like, you can modify this file to contain some custom content.

Once the instance is configured, we need to create a matching AMI using aws ec2 create-image. This command will automatically create an AMI from a running instance. Doing so requires that the instance be stopped and restarted, so your SSH session will be terminated when you run this command. In the background, a snapshot of the EBS volumes used for your instance will be made. This snapshot will be used when launching new instances through a newfangled AMI ID. Because it can take some time before snapshots are ready to use, your new AMI will remain in the pending state for a while after aws ec2 create-image completes. The image cannot be used until it enters the available state. You can check on the status in the Management Console or with the aws ec2 describe-images command:

$ aws ec2 create-image --instance-id i-0d3f9965712b45500 --region us-east-1 
--name test-image --output text
ami-0bfd16f7927a830a5

$ aws ec2 describe-images --region us-east-1 --image-ids ami-0bfd16f7927a830a5 
--output text
IMAGES	x86_64	2021-01-10T02:31:57.000Z	True	xen	ami-0bfd16f7927a830a5	740376006796/test-image	machine	test-image	740376006796	Linux/UNIX	False	/dev/sda1	ebs	simple	pending	RunInstances	hvm
BLOCKDEVICEMAPPINGS	/dev/sda1
EBS	True	False	snap-0e8fda8383ec4176a	8	gp2
BLOCKDEVICEMAPPINGS	/dev/sdb	ephemeral0
BLOCKDEVICEMAPPINGS	/dev/sdc	ephemeral1

When your new image is ready, it can be launched by any of the means described previously. Launch a new instance based on this image and get the public DNS name with aws ec2 describe-instances. Connect via SSH, then confirm that Nginx has started automatically:

$ service nginx status
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-01-10 04:16:04 UTC; 2min 39s ago
       Docs: man:nginx(8)
   Main PID: 645 (nginx)
      Tasks: 3 (limit: 1134)
     Memory: 11.8M
     CGroup: /system.slice/nginx.service
             ├─645 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─646 nginx: worker process
             └─647 nginx: worker process

[ output truncated ]

Although we have configured Nginx and have a running web server, you can’t access the Nginx welcome page just yet. If you try to visit the instance’s public DNS name in your web browser, the request will eventually time out. This is because EC2 instances are, by default, protected by a firewall that allows only connections from instances in the same security group—incoming HTTP connections have to be explicitly enabled with the same processes we used to allow inbound SSH connections. These firewalls, known as security groups, are discussed in the next chapter.

Remember that both this instance and the original instance from which we created the image are still running. You might want to terminate those before moving on to the next section. The two-line script can be used to terminate all running EC2 instances in your account to clean the slate after running a few experiments.

Tagging your images is a good way to keep track of them. This can be done with the aws ec2 create-tags command. By using backticks to capture the output of shell commands, you can quickly add useful information, such as who created the AMI, as well as static information like the role:

$ aws ec2 create-tags --resources ami-0bfd16f7927a830a5 
--tags Key=role,Value=webserver Key=created-by,Value=`whoami` 
Key=stage,Value=production

$ aws ec2 describe-tags --output text | grep ami-0bfd16f7927a830a5
TAGS	created-by	ami-0bfd16f7927a830a5	image	federico
TAGS	role	ami-0bfd16f7927a830a5	image	webserver
TAGS	stage	ami-0bfd16f7927a830a5	image	production

Deregistering AMIs

Once an AMI is no longer required, it should be deregistered, which means it will no longer be available to use for launching new instances. Although they are not particularly expensive, it is good practice to regularly remove old AMIs because they clutter up the interface and as they accumulate they will contribute to a gradual increase of your AWS costs.

aws ec2 describe-snapshots --owner-ids self 
--output text

You must also delete the snapshot used to create the root volume. This will not happen automatically.

AWS used to allow you to delete the snapshot before deregistering the AMI. Doing so meant you would have an AMI that looks as though it is available and ready for use, but will, in fact, fail when you tried to launch an instance. This has since changed, in one of countless examples of AWS transparently improving infrastructure behind the scenes. Deleting the snapshot backing an AMI now errors out with a helpful error message:

$ aws ec2 delete-snapshot --region us-east-1 --snapshot-id snap-0e8fda8383ec4176a

An error occurred (InvalidSnapshot.InUse) when calling the DeleteSnapshot operation: The snapshot snap-0e8fda8383ec4176a is currently in use by ami-0bfd16f7927a830a5

However, operators can still get themselves in trouble carelessly decommissioning AMIs. If a deregistered AMI is referenced in Auto Scaling groups, it might be some time before you notice the problem. The only option in that case is to quickly create a new AMI and update the Auto Scaling group.

You can check to see whether a particular AMI is in use by running instances with the aws ec2 describe-instances command. For example:

$ aws ec2 describe-instances --output text --filters 
Name=image-id,Values=ami-0bfd16f7927a830a5
RESERVATIONS	740376006796	043234062703	r-04284239167699bbd
INSTANCES	0	x86_64	examp-Ec2In-1RM2I5Y54AWJ1	False	True	xen	ami-0bfd16f7927a830a5	i-0015bd2909173d777	t3.micro	2021-01-11T04:05:23.000Z	ip-172-31-88-197.ec2.internal	172.31.88.197	ec2-3-239-84-160.compute-1.amazonaws.com	3.239.84.160	/dev/sda1	ebs	True		subnet-c2b704ce	hvm	vpc-934935f7
[ output truncated ]
TAGS	aws:cloudformation:stack-id	arn:aws:cloudformation:us-east-1:740376006796:stack/example-stack/37295ab0-53c2-11eb-b1cc-0a23b021aa67
TAGS	aws:cloudformation:stack-name	example-stack
TAGS	aws:cloudformation:logical-id	Ec2Instance

This works for individual instances. For instances that were launched as part of an Auto Scaling group, we can instead use the aws autoscaling describe-launch-configurations command. Unfortunately, this command does not accept a filter argument, so it cannot be used in quite the same way. As a workaround, you can grep the output of aws autoscaling describe-launch-configs for the AMI ID.

Performing these checks before deleting AMIs en masse can save you from a rather irritating cleanup exercise.

Once you are sure the AMI is safe to deregister, you can do so with aws ec2 deregister-image:

$ aws ec2 deregister-image --image-id ami-0bfd16f7927a830a5 --region us-east-1 

Remember to delete the snapshot that was used as the root volume of the AMI. You can find it through the aws ec2 describe-snapshots command. When AWS creates a new snapshot, it uses the description field to store the ID of the AMI it was created for, as well as the instance and volume IDs referencing the resources it was created from. Therefore, we can use the AMI ID as a filter in our search, returning the ID of the snapshot we want to delete:

$ aws ec2 describe-snapshots --region us-east-1 --output text --filters 
Name=description,Values="Created by CreateImage*for ami-0bfd16f7927a830a5*"

SNAPSHOTS	Created by CreateImage(i-0d3f9965712b45500) for ami-0bfd16f7927a830a5 from vol-089c950bd54753f4e	False	740376006796	100%	snap-0e8fda8383ec4176a	2021-01-10T02:33:27.412Z	completed	vol-089c950bd54753f4e	8

$ aws ec2 delete-snapshot --snapshot-id snap-0e8fda8383ec4176a 
--region us-east-1

Filter string wildcards (stars) are a really handy feature, and are needed here to match those unknown volume and instance identifiers found in the description string.

Automatic snapshot management has long made everyone’s list of Amazon AWS missing features. A Google search can produce a long list of (mostly out of date) scripts using the EC2 API meant by different users to manage this deficiency. In 2019 Amazon finally introduced the eagerly awaited Data Lifecycle Manager service to address this gap, which is the object of the next section. Before we take the easy way out, let’s examine snapshot management as a great example of how API access and automation can combine to fill any kind of “missing feature” gap.

$ aws ec2 describe-snapshots --owner-ids self --output text 
--query 'Snapshots[?StartTime<=`2020-05-31`]'

Created by CreateImage(i-b0b59281) for ami-7c81c46b from vol-03d9b5d6	False740376006796	100%	snap-514c5be2	2016-09-25T23:22:52.000Z	completed	vol-03d9b5d6	8
Created by CreateImage(i-0bd62c3b) for ami-cc491edb from vol-44fce591	False740376006796	100%	snap-6d9afde5	2016-10-16T07:39:19.000Z	completed	vol-44fce591	8
TAGS	Name	Mezzanine test snap page 190 at exit, before db is deleted
Created by CreateImage(i-a6844bbe) for ami-6a5d367d from vol-73fe60d4	False740376006796	100%	snap-428f90da	2016-09-03T20:49:26.000Z	completed	vol-73fe60d4	8

$ aws ec2 describe-snapshots --owner-ids self 
--query 'Snapshots[*].{Label:Description,ID:SnapshotId,Time:StartTime}'

[
    {
        "Label": "Created by CreateImage(i-b0b59281) for ami-7c81c46b from vol-03d9b5d6",
        "ID": "snap-514c5be2",
        "Time": "2016-09-25T23:22:52.000Z"
    },
    {
        "Label": "Created by CreateImage(i-0bd62c3b) for ami-cc491edb from vol-44fce591",
        "ID": "snap-6d9afde5",
        "Time": "2016-10-16T07:39:19.000Z"
    },
    {
        "Label": "Created by CreateImage(i-a6844bbe) for ami-6a5d367d from vol-73fe60d4",
        "ID": "snap-428f90da",
        "Time": "2016-09-03T20:49:26.000Z"
    },
    {
        "Label": "Created by CreateImage(i-0d3f9965712b45500) for ami-0bfd16f7927a830a5 from vol-089c950bd54753f4e",
        "ID": "snap-0e8fda8383ec4176a",
        "Time": "2021-01-10T02:33:27.412Z"
    }
]
]

#! /bin/bash
REGION=us-east-1

echo "Clearing all EC2 snapshots older than one year from $REGION"

AGE_FILTER=``date +%Y-%m-%d --date '1 year ago'``
SNAPSHOTS=$(aws ec2 describe-snapshots --owner-ids self 
--query "Snapshots[?StartTime<=$AGE_FILTER].{ID:SnapshotId}" --output text)
for i in $SNAPSHOTS
do
 echo "deleting $i"
 aws ec2 delete-snapshot --region $REGION --snapshot-id $i
done

$ ./snapshot-annual-purge.sh
Clearing all EC2 snapshots older than one year from us-east-1
deleting snap-514c5be2
deleting snap-6d9afde5
deleting snap-428f90da

$ aws ec2 describe-images --filters Name=tag-key,Values="stage" 
Name=tag-value,Values="retired" --output text
IMAGES	x86_64	2017-05-29T04:16:43.000Z	xen	ami-838ac495	740376006796/test-image	machine	test-image	740376006796	False	/dev/sda1	ebs	simple	available	hvm
BLOCKDEVICEMAPPINGS	/dev/sda1
EBS	True	False	snap-1b793584	8	standard
BLOCKDEVICEMAPPINGS	/dev/sdb	ephemeral0
BLOCKDEVICEMAPPINGS	/dev/sdc	ephemeral1
TAGS	stage	retired

#!/usr/bin/env python

from boto.ec2 import connect_to_region

ec2_conn = connect_to_region('us-east-1')

print 'Deleting retired AMI images.
'

for image in ec2_conn.get_all_images(owners='self', filters={'tag:stage': 'retired'}):
     print ' Deleting image %s and associated snapshot' % (image.id)
     image.deregister(delete_snapshot=True)

The Data Lifecycle Manager

The natural destiny of the automation approach designed in the last section is turning it into a cloud-hosted service, ideally as an AWS Lambda function. Amazon did just that for us in 2019, introducing the Data Lifecycle Manager service to automate the management of EBS artifacts.

The Data Lifecycle Manager service provides operators with a mechanism to automatically manage snapshots, as well as AMIs backed by EBS. DLM works by defining policies for their automated creation or deletion, and even supports management actions on volume copies incoming from other AWS accounts.

Figure 2-1. Defining a lifecycle policy purging snapshots older than a year

The new service can be controlled from the AWS CLI through the aws dlm set of commands, but we find that initial policy creation is most easily carried out with the guided wizard in the AWS Console (Figure 2-1).

To define a snapshot creation cycle with cleanup logic matching our scripted automation example, choose EBS snapshots as the object of the policy. Select “instance” to limit its scope to instances the volumes are attached to (an alternative is to use volumes already pre-tagged by separate automation). We are targeting instances tagged with a stage value of production, and are not tagging the lifecycle policy itself at this time. DLM automatically applies the dlm:managed tag to label all entities it manages, and includes additional policy ID, schedule ID, and expiration time metadata by default in other tags. Further on in the wizard, toggle the option to copy tags from the source volume to maintain the consistency of stage labels found in this deployment.

The next section of the wizard requires the IAM role these operations are to execute under—we will discuss roles in the next chapter as we illustrate AWS security, but for now the default option shall suffice. At least one schedule needs to be attached to the policy, and up to four can be defined. In Figure 2-2 we define a policy taking snapshots every Thursday, to be retained for a year. A useful alternative is count-based retention, where a maximum total of snapshots is preserved and the oldest snapshots are automatically deleted once that threshold is exceeded.

Activate the policy by selecting the “enable policy” radio button and clicking on “create policy”. Snapshots of production EBS volumes will now be taken weekly on Thursday, and discarded automagically a year later. Automating these steps saves the operator from the need to track and perform manual cleanups, and rescues the CIO from incurring needless additional cost.

Figure 2-2. Defining a schedule for execution of the lifecycle policy

DLM is a great instrument to automate backups snapshots, ensuring the cleanup of older artifacts is also taken care of as part of the process. This new service includes advanced options to replicate snapshots to alternate regions, supporting disaster recovery architectures, and even supports copying EBS artifacts across accounts, providing a logical substitute to offsite backups.