13

Working with Ephemeral Hosts

In this chapter, we will learn how to manage Windows Server as an ephemeral host; then, we will learn how EC2 Image Builder helps you build custom AMIs with all the necessary Windows Server patches, cached container images, and hardening.

The chapter will cover the following topics:

• The idea behind ephemeral hosts
• Why custom AMIs?
• Building a custom AMI pipeline

The idea behind ephemeral hosts

As a Windows sysadmin, never in my life would I imagine having an ephemeral (i.e., “lasting for a very short time,” according to the Oxford dictionary) or temporary Windows host in a production environment; the use case just isn’t there. Usually, Windows Servers are used to host long-running applications, where the hosts become aged after 4 or 5 years, or even more. Thousands of patches are applied, troubleshooting and hardening occur, and these sometimes help achieve years of uptime without a single reboot. This technique has been working for many years, but there is no place for this type of strategy in a Windows container world.

In the container world, containers are immutable, and hosts are ephemeral. That means the host doesn’t carry an application dependency or installation; it is just the container runtime and some additional agents for security if needed. If the server goes down, there is no dependency associated with the container and the application within. So, if there is no application installed, why should we care about troubleshooting or patching an existing host?

You shouldn’t be troubleshooting a running Windows container host; instead, just drain the containers running on it, and terminate the host. EC2 Auto Scaling will bring a new host based on the custom AMI you already set up, free of issues. The same happens for patching; if a patch is needed, get a new host based on an updated custom AMI and terminate the out-of-date host.

However, to achieve such beauty, you will need tools that automatically create custom AMIs and replace hosts as new patches become available. This is where EC2 Image Builder comes in handy.

Why custom AMIs

As already discussed in previous chapters, AWS provides optimized AMIs for each container orchestrator: the ECS-optimized Windows AMI and EKS-optimized Windows AMI. These are vanilla Windows installations with the required components to work with Amazon ECS or Amazon EKS clusters.

Every month, AWS launches a new AMI set that contains the latest Windows patches and components updates, such as the following:

• ECS-optimized Windows AMI:
  • Windows patches
  • ECS agent
  • Docker Community Edition (CE) updates
• EKS-optimized Windows AMI:
  • AWS IAM Authenticator
  • Windows patches
  • kubelet
  • kube-proxy
  • containerd
  • csi-proxy

Usually, in a production environment, you won’t rely on the AWS patch calendar if a strict security policy requires any new security patch to be applied within 24–48 hours of its release, and updating existing hosts becomes a challenge when the group of hosts is managed by an Auto Scaling group, meaning that new hosts will be launched and terminated based on resource utilization. For instance, let us assume you are about to update 10 hosts and in the next minute, it scales out to 20 more hosts, so now, how would you handle this volatile environment, where you may be patching a server that in 10 minutes might be flagged for termination due to the scaled-out activity?

You will need to have an AMI pipeline that automatically builds AMIs with new security patches, caches container image locally, installs agents, and updates the Launch Template attached to an existing Auto Scaling group, so that once the Auto Scaling API detects a new launch template version, it will automatically replace existing EC2 Windows instances with new updated ones.

In the next section, we will focus on how EC2 Image Builder helps you to build an automated pipeline that auto-replaces hosts based on the release of new patches and ensures that you have all your EC2 Windows instances up to date.

Building a custom AMI pipeline

EC2 Image Builder is a fully managed service that provides an automated pipeline to build and keep AMIs up to date and secure for you. As you may know, maintaining Windows Server images is very time-consuming, resource intensive, and error prone as since the old days we used to keep and maintain operational system images on VMware.

Important note

EC2 Image Builder is free, and you pay only for the underlying AWS resources used to create, store, and share the images.

EC2 Image Builder uses AWS Task Orchestrator and Executor (AWSTOE). An AWS document (https://docs.aws.amazon.com/pdfs/imagebuilder/latest/userguide/imagebuilder-ug.pdf) defines AWSTOE as a “standalone application that creates, validates, and runs commands within a component definition framework.” With AWSTOE, you can modify system configuration and orchestrate complex workflows without writing code. That means you can have a series of components written in YAML, which are instructions that you want to be executed on your operating system, and AWSTOE can simply run and validate against your operational system.

For Windows container hosts, independent of being ECS or EKS orchestrators, we want to leverage EC2 Image Builder to create a custom AMI, so we can have an automated AMI pipeline that can trigger EC2 Windows instances replacements in your cluster in an automated process:

Figure 13.1 – Custom AMI pipeline for ECS or EKS-optimized AMIs

Let’s explore each step of this pipeline:

1. An EC2 Image Builder component is created, in YAML style, that will execute steps 1 to 6 defined in the preceding figure, as follows:
   • Update Windows with the latest patches (component provided by AWS). This is a ready-to-use component provided by AWS that instructs the operational system to download and install the latest Windows Server patches available.
   • Pull the required container images, such as .NET Framework, sidecars, and applications (custom component).
   • Custom components that instruct Docker or containerd to pull the necessary container images.
   • Install ECS or EKS components (component provided by AWS). This ready-to-use component provided by AWS instructs the installation of the ECS agent or Kubernetes components such as kubelet, kubeproxy, and csi-proxy into your AMI.

The following example is a custom component that downloads .NET Framework applications from Microsoft MCR (public registry) and application container images from an Amazon ECR (private registry):

name: Images-Auto-cache
description: Pull the necessary container images
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: Dockerpull
        action: ExecutePowerShell
        inputs:
          commands:
            - Set-ExecutionPolicy Unrestricted -Force
            - (Get-ECRLoginCommand).Password | docker login --username AWS --password-stdin <ACCOUNT-ID>..dkr.ecr.us-east-1.amazonaws.com
            - docker pull <ACCOUNT-ID>.dkr.ecr.us-east-1.amazonaws.com/my-dotnet-application
            - docker pull mcr.microsoft.com/dotnet/framework/aspnet:4.8.1

2. In step 7, the pipeline executes the components on a temporary Amazon EC2 instance launched by EC2 Image Builder. Once it finishes, a Sysprep generalize process kicks in, so the image can be reused to launch new EC2 instances. The custom AMI is stored in your AMI library.
3. In step 8, EC2 Image Builder starts the process of distributing the AMI in selected regions. In my example, just us-east-1, and then enable EC2 Fast Launch.
4. Step 9 is optional, but it is the one that allows an automated EC2 Windows instance replacement. EC2 Image Builder can automatically create a new Launch Template version with the new AMI ID. If combined with an Auto Scaling Termination Policy, it can automatically replace EC2 instances that were launched by the oldest Launch Template.

Now that we have an idea of how an automated custom AMI pipeline works and why we need one, I want to call attention to step 8, which is enabling Fast Launch. If you go back to Chapter 6, Deploying a Fargate Windows-Based Task, in the Planning for serverless Windows containers section, I discussed a secret sauce that AWS uses on AWS Fargate called AWS Windows faster launching, which drastically reduces the several minutes implied by the Sysprep process. Well, this feature is also available for custom AMIs and it can be enabled via AWS CLI, EC2 console, or EC2 Image Builder.

To ensure that we remember what Sysprep is, I extracted this paragraph from Chapter 6:

Sysprep is a process on Windows that removes PC-specific information from the registry, such as the security identifier (SID). SID uniquely identifies any entity in a Windows-based network, including users, groups, computer accounts, etc. The process of removing these pieces of information is called generalizing, and it is used during Windows OS image creation.

When planning to deploy Windows Server at scale, companies use OS-prepared images, so they can quickly launch virtual machines in any environment, cloud, on-prem, VMware, Hyper-V, or elsewhere. When these OS-prepared images are launching the Windows virtual machine, Sysprep kicks in and starts the process of randomly created SIDs that were previously removed during the OS image.

Now, let’s compare the EC2 boot time between an EC2 Windows instance (ECS or EKS) launched with a custom AMI without Fast Launch and another one with Fast Launch enabled on the custom AMI. We will be comparing them from the time the EC2 API receives a call to launch a new EC2 Windows instance to the time the instance finishes the Sysprep process.

In the following figure, you can see "LaunchTime": 2022-09-27T02:30:42, which is the time the EC2 control plane received the RunInstance API call, and then in the EC2 instance system log, the Windows Sysprep process was completed 6 minutes after the API was invoked:

Figure 13.2 – The time frame from the EC2 Create API call until Windows Sysprep finishes without Fast Launch

The 6 minutes average is normal behavior on a Windows Sysprep process, but depending on the use case where scale-out needs to happen fast in order to meet demand, this can be a huge problem. Did you ever think about waiting 6 minutes for the EC2 Windows instance to become online, then 6 to 10 further minutes to join the ECS/EKS cluster and pull/extract container images? Unacceptable, right?

By enabling Fast Launch on the custom AMI, we avoid the Windows Sysprep process. You can see "LaunchTime": 2022-09-29T16:35:39, which is the time the EC2 control plane received the RunInstance API call, and then in the EC2 instance system log, the Windows Sysprep process was completed in 1 minute after the API was invoked. As a result, we were able to cut 5 minutes from the Windows boot time:

Figure 13.3 – The time frame from the EC2 Create API call until Windows Sysprep finishes with Fast Launch

In this section, we learned how valuable it is to have a custom AMI pipeline leveraging EC2 Image Builder features to auto-replace hosts in your cluster. I decided to spend little time on how to set up the pipeline because I already published a blog post on AWS channels that has a complete step-by-step guide on how to set the pipeline up. It is a live documentation that I keep updating as new features are launched.

You can find more information at the following link: https://aws.amazon.com/blogs/containers/speeding-up-windows-container-launch-times-with-ec2-image-builder-and-image-cache-strategy/.

Also, if you want to see a full demo and explanation on EC2 Fast Launch, I have recorded a video that is posted on YouTube at the AWS channel: https://www.youtube.com/watch?v=XjNOm04Lp7Y.

To wrap up, in this section, we learned how valuable a custom AMI can be by caching and installing all requirements directly into the AMI instead of running it on the bootstrap, and how Fast Launch can help speed up EC2 Windows boot time.

Summary

In this chapter, we started the fine-tuning. First, we learned why we should follow the strategy of managing EC2 Windows instances as ephemeral hosts on Amazon ECS and EKS clusters; then, we learned about custom AMIs and how crucial it is to have an automated custom AMI pipeline, and the benefits of using EC2 Image Builder as a build pipeline. Finally, we covered in great detail why you should use Fast Launch to speed up the EC2 Windows boot startup time.

In the next chapter, we will continue the fine-tuning and learn why it is essential to cache container images on the AMI, and how crucial this is for Windows containers.