7

Amazon EKS – Overview

In this chapter, we’ll learn about Amazon Elastic Kubernetes Service (EKS) and its relevant Windows components, such as node groups, worker nodes, and pods, and then dive deep into Windows networking on EKS. Finally, we’ll deploy an Amazon EKS cluster with a Linux and Windows node group using Terraform.

We are going to cover the following main topics:

• Amazon EKS – fundamentals
• Amazon VPC CNI for Windows
• Amazon EKS and Windows support

This chapter will give us the fundamentals for the upcoming Chapters 8, 9, and 10, which will dive deep into cluster interoperability and best practices for Windows node groups and pods.

Important note

Amazon EKS is a rapidly growing managed service with a fast-paced product roadmap that results in new features and functionality, which makes it impossible for published books to be up to date. Therefore, the information and solutions proposed in this book may change over time, and you should always check for the latest news on the official AWS roadmap web page (https://docs.aws.amazon.com/eks/latest/userguide/roadmap.html) .

Amazon EKS – fundamentals

Kubernetes has been proven to be the new standard to run containerized applications; it is cloud agnostic, robust, and widely used by companies of any size. All that comes at the expense of a learning curve, and it is crucial that companies, leaders, and managers invest in their DevOps teams with high-quality training and technical resources.

The Kubernetes ecosystem is big and complex, and my objective isn’t to teach you about Kubernetes; instead, this book intends to help you learn all the gotchas on how to successfully schedule and manage Windows containers in the form of pods on an Amazon EKS cluster. In order to accomplish that, we will be focusing on Windows container-related topics only, and leverage the beauty of using AWS managed services, such as Amazon EKS.

Amazon EKS is a managed Kubernetes orchestrator that allows customers to run containers as pods on top of node groups without needing to install and operate a Kubernetes control plane. You can choose to run Windows containers on Amazon EKS in three different forms:

• Amazon EKS in the cloud, where the control plane and data plane are deployed in the AWS cloud
• Amazon EKS on AWS Outposts, where the control plane can be deployed in the cloud or your data center, and the data plane is deployed in your data center
• Amazon EKS Anywhere, where both the control plane and data plane are deployed in your data center

Control plane

An Amazon EKS control plane consists of at least two API server instances and three etcd instances across three availability zones (AZs) within an AWS Region:

Figure 7.1 – Amazon EKS control plane representation

Customers interact with Amazon EKS through API servers, which are exposed by the cluster endpoint using standard tooling such as kubectl. By default, the API server endpoint is public to the internet. Access to the API server is secured using IAM and native Kubernetes role-based access control (RBAC). In addition, the public endpoint can be converted to private if needed.

Data plane

An Amazon EKS data plane consists of one or more Amazon EC2 nodes, which become part of a node group. Each node group must run the same EC2 instance type, AMI, and Amazon EKS node IAM role:

Figure 7.2 – Amazon EKS data plane representation

An Amazon EKS cluster can contain several node groups, for instance, a Linux node group and a Windows node group; as a result, you end up with a heterogenous Amazon EKS cluster.

In order to run Windows-based node groups, you must have at least one Linux-based node group responsible for running services such as CoreDNS, Cluster-Auto-Scaler, Metric Server, and so on. This is done because most Kubernetes services and add-ons were not developed to run on Windows containers, which makes sense as the Windows container footprint is much smaller than Linux.

It is common to see customers using a mix of managed nodes within Linux-based node groups and self-managed nodes with Windows-based node groups.

We have covered fundamental concepts for Amazon EKS, from the control plane to the data plane. Still, it is essential to highlight that as a managed service, you will be free of cluster control plane operations as it is part of the shared responsibility model. AWS is responsible for keeping the control plane up and running for you for just $0.10 per hour per cluster.

Amazon VPC CNI for Windows

Amazon EKS supports a native network plugin called Amazon VPC Container Network Interface (CNI) that has two main responsibilities:

• Creating Elastic Network Interfaces (ENIs) and attaching them to the Amazon EC2 nodes or using a combination of prefix-delegation per ENI
• Assigning a private IPv4 or IPv6 address from your VPC to each pod and service

By default, the number of IP addresses available to be assigned to pods is a calculation based on the number of IP addresses available on each ENI and the number of ENIs that can be attached to the Amazon EC2 instance. The following formula applies:

(Number of ENIs * IP addresses per interface) - 3 = Total IPv4 addresses available for pods

The minus three is a result of the following:

• One IP reserved for the Amazon EC2 node primary IP
• One IP reserved for the VPC CNI
• One IP reserved for kube-proxy

For instance, we can calculate that a Linux-based node running on an m5.large instance has a total of 27 IPs available to be assigned to pods. The following formula applies:

(3 ENI * 10 secondary IPs per ENI) - 3 = 27 IPv4 addresse available for pods

The number of primary ENIs and IPs per ENI can be found at the following link: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI.

However, the Amazon VPC CNI for Windows has some particularities that differ from the main formula and directly influence the Windows pod density per Windows-based nodes. In a nutshell, when this book was written, the Amazon VPC CNI for Windows only supported the primary ENI and its available IPs to be used with Kubernetes Pods. The following formula applies:

(Primary ENI * IP addresses per interface) - 3 = Total IP addresses available for Pods

The minus three is a result of the following:

• One IP reserved for the Amazon EC2 node primary IP
• One IP reserved for the VPC CNI
• One IP reserved for kube-proxy

For instance, we can calculate that a Windows-based node running on an m5.large instance has a total of 7 IPs available to be assigned to pods. The following formula applies:

(Primary ENI * 10 secondary IPs per ENI) - 3 = 7

As you can see, the way Amazon VPC CNI for Windows works reduces pod density per node compared to a Linux-based node. However, we also need to consider that usually, Windows pods are memory hungry, so the pod density per node based on available IPs won’t be a hard-stop adoption in most cases.

One point to highlight is that you aren’t limited to the VPC CNI; you also have great open source solutions such as Calico CNI, which supports Amazon EKS Windows nodes. However, understanding its limitation and the support you will get is essential for adopting a CNI not provided by AWS.

Amazon EKS and Windows support

When planning to have a heterogenous Amazon EKS cluster, it is imperative to understand what limitations are applied to Windows-based nodes and how that would affect your Amazon EKS cluster.

One of the most common mistakes I’ve seen is customers first adding Windows-based node groups in their Amazon EKS cluster for the next plan on what to do regarding logging, monitoring, security, patching, and EKS limitations. This mistake usually costs lots of money and hours for customers paying hourly rates to consult companies to implement these projects. All these should be planned ahead of time, identifying commercial or open source tools that will address your requirements.

I will discuss some of the most common solutions that don’t fit on Windows-based node groups right from the shelf, which will require additional development to get up and running:

• Karpenter: Not available for Windows-based nodes. If you need to use a tool that automatically adjusts the size of your Kubernetes cluster based on the following conditions:
  • Pods that failed to run in the cluster due to insufficient resources, such as IPv4 node exhaustion, as mentioned in the previous topic
  • Nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes

Then you will need to choose between keeping Karpenter to handle auto-scaling on Linux-based node groups and leveraging Kubernetes Cluster Autoscaler for Windows-based node groups. Or you can use Kubernetes Cluster Autoscaler across both node groups.

• Security groups for pods: Not available for Windows-based pods. So, if you are leveraging Amazon EC2 security groups to define rules that allow inbound and outbound network traffic to and from Linux pods, you won’t be able to achieve the same on Windows pods.

You will need to choose between managing firewall rules in two places or adopting a network policy solution that interoperates between Windows and Linux, such as the Calico network policy by Tigera.

• Custom networking: Not available for Windows-based nodes. Usually, customers use custom networking so that pods could use a different subnet or security group than the node’s primary network interface. If you are leveraging customer networking to schedule your pods on different subnets, you won’t be able to do the same with Windows-based pods, thereby having them all in the same subnet as the Windows-based nodes.
• IP prefixes (prefix-delegation): Not available for Windows-based nodes. As explained in the Amazon VPC CNI for Windows section, the number of IP addresses available to assign to pods is based on the number of IPs assigned to the primary ENI. It was introduced in Amazon VPC CNI version 1.9.0, the add-on that assigns /28 IPv4 address prefixes or /80 IPv6 on version 1.21. This is done by the CNI add-on assigning significantly more IP addresses to a network interface, thereby increasing the pod density per node.
• Amazon EKS pods on Fargate: Not available for Windows-based pods. So, no serverless Windows containers on Amazon EKS.
• Amazon EKS Windows managed node group: Not available for Windows-based nodes. Additionally, this item may change over time, as it is part of the Amazon EKS public roadmap.

These are the official not-supported features for Windows-based nodes, which can change over time. Still, you will also need to research additional plugins, sidecars, and open source solutions to use for observability and security and check whether these are compatible with Windows nodes and pods.

Personal thoughts

It may be intimidating to choose Kubernetes as a container orchestrator for Windows at first due to its complexity and the solution ecosystem that must orbit a cluster to operate your Kubernetes cluster on day two successfully. But by taking appropriate training and a dose of goodwill, you will get there.

I’ve been playing with Windows on Kubernetes since its release in 2018, and the lack of out-of-three drivers, security, logging, and monitoring open source solutions is a pain, and it persists until today. While we have a strong Windows Kubernetes community called SIG-Windows, where we have access to AWS, Microsoft, and Google engineers responsible for Windows integration in their managed Kubernetes offerings, it is still very small when compared with the Linux community. This adds lots of complexity when you need to solve a problem or request a new feature since Windows containers are second-class citizens on Kubernetes.

I just talked about successfully operating on day two, which means managing your cluster. Unfortunately, many open source components available for observability, security, monitoring, and logging aren’t available for Windows pods. As a result, you will need to rely on commercial third-party integrators if you don’t have the necessary skills to adapt the open source plugin to your needs.

I’m sorry if it sounds scary or puts you in doubt regarding whether Kubernetes is the right choice for Windows containers, but this is true. Many customers I have been working with successfully run Windows pods on Kubernetes – first, because they have the expertise to adapt open source plugins to their own needs; second, as they have money to buy third-party integration tools; and third, as they simply don’t need any complexity over Windows pods.

It isn’t solely about the orchestrator, but more importantly, the supportability you will have on tools that need to orbit your Kubernetes cluster.

Summary

In this chapter, we did a highlighted overview of the Amazon EKS control plane/data plane and how Amazon VPC CNI for Windows works; then, we went over a list of not-supported features and functionalities for Windows-based nodes and pods and how this can affect your cluster configuration. Finally, I shared some personal experiences about Kubernetes and Windows and what you should consider when choosing Amazon EKS over Amazon ECS as your Windows containers orchestrator.

In the next chapter, Preparing the Cluster for OS Interoperability, we will learn how to enable Windows support on an Amazon EKS cluster. In addition, we will learn how to avoid deployment disruption and scale in/out of Windows pods.