13.1.1 Configuring Access Keys

With the right security policy, users can be in charge of their own access keys. A single access key comprises the access key ID and the access key secret. The ID does not need to be kept secret and remains accessible via the IAM user interface after generation. This allows, for example, disabling or deleting an access key by ID.

A user can configure up to two access keys. Having two keys allows for doing 0-downtime key rotations. The first step is to generate a new key. Then replace the old key everywhere. Afterward, disable the old key. Disabling the old key makes anything that tries to use it fail. If such a failure is detected, it is easy to re-enable the old key until the task using that key can be upgraded to the new key.

After a certain amount of time, when no failures have been observed, it should be safe to delete the old key.

Local security policies generally determine how often keys should be rotated, but this should usually be at least a yearly ritual. This should generally follow practices for other API secrets used in the organization.

Note that in AWS, different computation tasks can have their own IAM credentials.

For example, an EC2 machine can be assigned an IAM role. Other higher-level computation tasks can also be assigned a role. For example, an Elastic Container Service (ECS) task, which runs one or more containers, can be assigned an IAM role. Serverless Lambda functions, which run on infrastructure allocated on an as-needed basis, can also be assigned an IAM role.

The boto3 client automatically uses these credentials if running from such a task. This removes the need to explicitly manage credentials and is often a safer alternative.

13.1.2 Creating Short-Term Tokens

AWS supports short-term tokens (STS), which can be used for several things. They can be used to convert alternative authentication methods into tokens that can be used with any boto3-based program, for example, by putting them in an environment variable.

This can be used to improve auditing or to reduce the security impact of lost or stolen credentials.

With this logic, the web portal has no special access to the AWS account. After the user logs in, it uses the assertion to create a short-term token for operations.

On an account that has been configured with cross-account access, assume_token can return credentials for the granting account.

Even when using a single account, sometimes it is useful to create a short-term token. For example, this can be used to limit permissions. It is possible to create an STS with a limited security policy. Using limiting tokens in code that is more prone to vulnerabilities, such as direct user interactions, limits the attack surface.

13.2.1 Regions

EC2 machines run in regions. Regions usually have a human-friendly name (such as Oregon) and an identifier used for programs (such as us-west-2).

There are several regions in the United States, including North Virginia (us-east-1), Ohio (us-east-2), North California (us-west-1), and Oregon (us-west-2). There are also several regions in Europe, Asia Pacific, and more.

When you connect to AWS, you connect to the region you need to manipulate; boto3.client("ec2", region_name="us-west-2") returns a client that connects to the Oregon AWS data center.

It is possible to specify default regions in environment variables and configuration files, but it is often best to be explicit in code (or retrieve it from higher-level application configuration data).

EC2 machines also run in the availability zone. While regions are objective (every customer sees the region the same), availability zones are not: one customer’s us-west-2a might be another’s us-west-2c.

Amazon puts all EC2 machines into a virtual private cloud (VPC) network. For simple cases, an account has one VPC per region, and all EC2 machines belonging to that account are in that VPC.

A subnet is how a VPC intersects with an availability zone. All machines in a subnet belong to the same zone. A VPC can have one or more security groups. Security groups can set up various firewall rules about what network connections are allowed.

13.2.2 Amazon Machine Images

To start an EC2 machine, you need an operating system image. While it is possible to build custom Amazon Machine Images (AMIs), you can often use a ready-made one.

There are AMIs for all major Linux distributions. The AMI ID for the right distribution depends on the AWS region you want to run the machine. Once you have decided on the region and the distribution version, you need to find the AMI ID.

The CentOS wiki contains product codes for all relevant CentOS versions.

AMI IDs for Debian images can be found on the Debian wiki. The Ubuntu website has a tool to find the AMI IDs for various Ubuntu images based on region and version. Unfortunately, there is no centralized, automated registry. It is possible to search for AMIs with the UI, but this is risky. The best way to guarantee the authenticity of the AMI is to look at the creator's website.

13.2.3 SSH Keys

For ad hoc administration and troubleshooting, it is useful to be able to SSH into the EC2 machine. This might be for manual SSH, using Paramiko, Ansible, or bootstrapping Salt.

Best practices for building AMIs, which are followed by all major distributions for their default images, use cloud-init to initialize the machine. One of the things cloud-init does is allow a preconfigured user to log in via an SSH public key retrieved from the machine’s user data.

Public SSH keys are stored by region and account. There are two ways to add an SSH key; letting AWS generate a key-pair and retrieving the private key, or generating a key-pair ourselves and pushing the public key to AWS.

13.2.4 Bringing up Machines

The run_instances method on the EC2 client can start new instances. Choose IMAGE_ID based on the image you want to run by searching the console.

The API is a little counter-intuitive. In almost all cases, both MinCount and MaxCount need to be 1. For running several identical machines, it is much better to use an AutoScaling Group (ASG), which is beyond the scope of the current chapter. It is worth remembering that as AWS's first service, EC2 has the oldest API, with the least lessons learned on designing good cloud automation APIs.

While the API generally allows running more than one instance, this is not often done. The SecurityGroupIds imply which VPC the machine is in. When running a machine from the AWS console, a fairly liberal Security group is automatically created. Using this security group is a useful shortcut for debugging purposes, although it is better to create custom security groups.

The AMI chosen here is a CentOS AMI. While KeyName is not mandatory, it is highly recommended to create a key-pair, or import one, and use the name.

The InstanceType indicates the amounts of computation resources allocated to the instance. t2.micro is, as the name implies, a fairly minimal machine. It is useful mainly for prototyping but usually cannot support all but the most minimal production workloads.

13.2.5 Securely Logging In

When logging in via SSH, it is good to know beforehand what public key you expect. Otherwise, an intermediary can hijack the connection. Especially in cloud environments, the Trust On First Use (TOFU) approach is problematic. There are a lot of first uses whenever you create a new machine. Since VMs are best treated as disposable, the TOFU principle is of little help.

Look for the first line that starts with ssh-rsa. Now that you have the public key, there are several things you can do with it. If you just want to run the SSH command line, and the machine is not VPN-accessible-only, you want to store the public IP in known_hosts.

13.2.6 Building Images

Building your own images can be useful. One reason to do it is to accelerate start-up. Instead of booting up a vanilla Linux distribution and then installing needed packages, setting configuration, and so on, it is possible to do it once, store the AMI, and then launch instances from this AMI.

Another reason to do it is to have known upgrade times; running apt-get update && apt-get upgrade means getting the latest packages at the time of upgrade. Instead, doing this in an AMI build allows knowing all machines are running from the same AMI. Upgrades can be done by first replacing some machines with the new AMI, checking the status, and replacing the rest. This technique (used by Netflix, among others) is called an immutable image. While there are other approaches to immutability, this is one of the first successfully deployed in production.

One way to prepare machines is to use a configuration management system. Both Ansible and Salt have a local mode that runs commands locally instead of via a server/client connection.

This approach means a simple script running on a local machine or in a CI environment can generate an AMI from source code.

13.3.1 Managing Buckets

There are other options, but those are usually not needed. Some of those have to do with granting permissions on the bucket. In general, a better way to manage bucket permissions is how all permissions are managed by attaching policies to roles or IAM users.

The first two arguments are important. It is necessary to specify the bucket, and it is a good idea to make sure that responses are of known maximum size.

The Prefix parameter is useful, especially when using the S3 bucket to simulate a file system. For example, this is what S3 buckets that serve as websites usually look like. When exporting CloudWatch logs to S3, it is possible to specify a prefix exactly to simulate a file system. While internally, the bucket is still flat, you can use something like Prefix="2018/12/04/" to get only the logs from December 4, 2018.

When more objects qualify than MaxKeys, the response is truncated. The IsTruncated field in the response is True, and the NextMarker field is set. Sending another list_objects with the Marker set to the returned NextMarker retrieve the next MaxKeys objects. This allows pagination through consistent responses even in the face of mutating buckets, in the limited sense that you get at least all objects that were not mutated while paginating.

The value is a bytes object.

Especially for small to medium-sized objects, say up to several megabytes, this allows simple retrieval of all data.

Again, this works well for the case where the body all fits in memory.

As alluded to earlier, when uploading or downloading larger files (for example, videos or database dumps), you would like to be able to upload incrementally without keeping the whole file in memory at once.

Finally, it is often the case that you would like objects to be transferred directly out of S3 or into S3 without the data going through the custom code, but you do not want to allow unauthenticated access.

For example, a continuous integration job might upload its artifacts to S3. You would like to be able to download them through the CI web interface, but having the data pass through the CI server is unpleasant. This server now needs to handle potentially larger files where people would care about transfer speeds.

This URL can now be sent via email to people who need to view the recording, and they can download the video and watch it. In this case, you saved yourself any need to run a web server.

An even more interesting use case is allowing pre-signed uploads. This is especially interesting because uploading files sometimes requires subtle interplays between the web server and the web application server to allow large requests to be sent in.

This lets you upload the meeting recording locally, even if the meeting recording device does not have S3 access credentials. It is also possible to limit the maximum size of the files via generate_presigned_post to prevent potential harm from an unknown device uploading these files.

Note that pre-signed URLs can be used multiple times. Making a pre-signed URL only valid for a limited time can mitigate any risk of potentially mutating the object after uploading. For example, if the duration is one second, you can avoid checking the uploaded object until the second is done.