We'll start by deciding the name of our soon to be created cluster. We'll choose to call it devops23.k8s.local. The latter part of the name (.k8s.local) is mandatory if we do not have a DNS at hand. It's a naming convention kops uses to decide whether to create a gossip-based cluster or to rely on a publicly available domain. If this would be a "real" production cluster, you would probably have a DNS for it. However, since I cannot be sure whether you do have one for the exercises in this book, we'll play it safe, and proceed with the gossip mode.

We'll store the name into an environment variable so that it is easily accessible.

 1  export NAME=devops23.k8s.local

When we create the cluster, kops will store its state in a location we're about to configure. If you used Terraform, you'll notice that kops uses a very similar approach. It uses the state it generates when creating the cluster for all subsequent operations. If we want to change any aspect of a cluster, we'll have to change the desired state first, and then apply those changes to the cluster.

At the moment, when creating a cluster in AWS, the only option for storing the state are Amazon S3 (https://aws.amazon.com/s3/) buckets. We can expect availability of additional stores soon. For now, S3 is our only option.

The command that creates an S3 bucket in our region is as follows.

 1  export BUCKET_NAME=devops23-$(date +%s)
 2
 3  aws s3api create-bucket 
 4      --bucket $BUCKET_NAME 
 5      --create-bucket-configuration 
 6      LocationConstraint=$AWS_DEFAULT_REGION

We created a bucket with a unique name and the output is as follows.

{
    "Location": "http://devops23-1519993212.s3.amazonaws.com/"
}

For simplicity, we'll define the environment variable KOPS_STATE_STORE. Kops will use it to know where we store the state. Otherwise, we'd need to use --store argument with every kops command.

 1  export KOPS_STATE_STORE=s3://$BUCKET_NAME

There's only one thing missing before we create the cluster. We need to imnstall kops.

If you are a MacOS user, the easiest way to install kops is through Homebrew (https://brew.sh/).

 1  brew update && brew install kops

As an alternative, we can download a release from GitHub.

 1  curl -Lo kops
    https://github.com/kubernetes/kops/releases/download/$(curl -s
    https://api.github.com/repos/kubernetes/kops/releases/latest |
    grep tag_name | cut -d '"' -f 4)/kops-darwin-amd64
 2
 3  chmod +x ./kops
 4
 5  sudo mv ./kops /usr/local/bin/

If, on the other hand, you're a Linux user, the commands that will install kops are as follows.

 1  wget -O kops
    https://github.com/kubernetes/kops/releases/download/$(curl -s
    https://api.github.com/repos/kubernetes/kops/releases/latest |
    grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
 2
 3  chmod +x ./kops
 4
 5  sudo mv ./kops /usr/local/bin/

Finally, if you are a Windows user, you cannot install kops. At the time of this writing, its releases do not include Windows binaries. Don't worry. I am not giving up on you, dear Windows user. We'll manage to overcome the problem soon by exploiting Docker's ability to run any Linux application. The only requirement is that you have Docker For Windows (https://www.docker.com/products/docker-desktop) installed.

I already created a Docker image that contains kops and its dependencies. So, we'll create an alias kops that will create a container instead running a binary. The result will be the same.

The command that creates the kops alias is as follows. Execute it only if you are a Windows user.

 1  mkdir config
 2
 3  alias kops="docker run -it --rm 
 4      -v $PWD/devops23.pub:/devops23.pub 
 5      -v $PWD/config:/config 
 6      -e KUBECONFIG=/config/kubecfg.yaml 
 7      -e NAME=$NAME -e ZONES=$ZONES 
 8      -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID 
 9      -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY 
10      -e KOPS_STATE_STORE=$KOPS_STATE_STORE 
11      vfarcic/kops"

We won't go into details of all the arguments the docker run command uses. Their usage will become clear when we start using kops. Just remember that we are passing all the environment variables we might use as well as mounting the SSH key and the directory where kops will store kubectl configuration.

We are, finally, ready to create a cluster. But, before we do that, we'll spend a bit of time discussing the requirements we might have. After all, not all clusters are created equal, and the choices we are about to make might severely impact our ability to accomplish the goals we might have.

The first question we might ask ourselves is whether we want to have high-availability. It would be strange if anyone would answer no.

Who doesn't want to have a cluster that is (almost) always available? Instead, we'll ask ourselves what the things that might bring our cluster down are.

When a node is destroyed, Kubernetes will reschedule all the applications that were running inside it into the healthy nodes. All we have to do is to make sure that, later on, a new server is created and joined the cluster, so that its capacity is back to the desired values. We'll discuss later how are new nodes created as a reaction to failures of a server. For now, we'll assume that will happen somehow.

Still, there is a catch. Given that new nodes need to join the cluster, if the failed server was the only master, there is no cluster to join. All is lost. The part is where master servers are. They host the critical components without which Kubernetes cannot operate.

So, we need more than one master node. How about two? If one fails, we still have the other one. Still, that would not work.

Every piece of information that enters one of the master nodes is propagated to the others, and only after the majority agrees, that information is committed. If we lose majority (50%+1), masters cannot establish a quorum and cease to operate. If one out of two masters is down, we can get only half of the votes, and we would lose the ability to establish the quorum. Therefore, we need three masters or more. Odd numbers greater than one are "magic" numbers. Given that we won't create a big cluster, three should do.

With three masters, we are safe from a failure of any single one of them. Given that failed servers will be replaced with new ones, as long as only one master fails at the time, we should be fault tolerant and have high availability. The whole idea of having multiple masters does not mean much if an entire data center goes down.

Attempts to prevent a data center from failing are commendable. Still, no matter how well a data center is designed, there is always a scenario that might cause its disruption. So, we need more than one data center. Following the logic behind master nodes, we need at least three. But, as with almost anything else, we cannot have any three (or more) data centers. If they are too far apart, the latency between them might be too high. Since every piece of information is propagated to all the masters in a cluster, slow communication between data centers would severely impact the cluster as a whole.

All in all, we need three data centers that are close enough to provide low latency, and yet physically separated, so that failure of one does not impact the others.

Since we are about to create the cluster in AWS, we'll use availability zones (AZs) which are physically separated data centers with low latency.

There's more to high-availability to running multiple masters and spreading a cluster across multiple availability zones. We'll get back to this subject later. For now, we'll continue exploring the other decisions we have to make.

Which networking shall we use? We can choose between kubenet, CNI, classic, and external networking.

The classic Kubernetes native networking is deprecated in favor of kubenet, so we can discard it right away.

The external networking is used in some custom implementations and for particular use cases, so we'll discard that one as well.

That leaves us with kubenet and CNI.

Container Network Interface (CNI) allows us to plug in a third-party networking driver. Kops supports Calico, flannel, Canal (Flannel + Calico), kopeio-vxlan, kube-router, romana, weave, and amazon-vpc-routed-eni networks. Each of those networks comes with pros and cons and differs in its implementation and primary objectives. Choosing between them would require a detailed analysis of each. We'll leave a comparison of all those for some other time and place. Instead, we'll focus on kubenet.

Kubenet is kops' default networking solution. It is Kubernetes native networking, and it is considered battle tested and very reliable. However, it comes with a limitation. On AWS, routes for each node are configured in AWS VPC routing tables. Since those tables cannot have more than fifty entries, kubenet can be used in clusters with up to fifty nodes. If you're planning to have a cluster bigger than that, you'll have to switch to one of the previously mentioned CNIs.

The good news is that using any of the networking solutions is easy. All we have to do is specify the --networking argument followed with the name of the network. Given that we won't have the time and space to evaluate all the CNIs, we'll use kubenet as the networking solution for the cluster we're about to create. I encourage you to explore the other options on your own (or wait until I write a post or a new book).

Finally, we are left with only one more choice we need to make. What will be the size of our nodes? Since we won't run many applications, t2.small should be more than enough and will keep AWS costs to a minimum. t2.micro is too small, so we elected the second smallest among those AWS offers.

The command that creates a cluster using the specifications we discussed is as follows.

 1  kops create cluster 
 2      --name $NAME 
 3      --master-count 3 
 4      --node-count 1 
 5      --node-size t2.small 
 6      --master-size t2.small 
 7      --zones $ZONES 
 8      --master-zones $ZONES 
 9      --ssh-public-key devops23.pub 
10      --networking kubenet 
11      --authorization RBAC 
12      --yes

We specified that the cluster should have three masters and one worker node. Remember, we can always increase the number of workers, so there's no need to start with more than what we need at the moment.

The sizes of both worker nodes and masters are set to t2.small. Both types of nodes will be spread across the three availability zones we specified through the environment variable ZONES. Further on, we defined the public key and the type of networking.

We used --kubernetes-version to specify that we prefer to run version v1.8.4. Otherwise, we'd get a cluster with the latest version considered stable by kops. Even though running latest stable version is probably a good idea, we'll need to be a few versions behind to demonstrate some of the features kops has to offer.

By default, kops sets authorization to AlwaysAllow. Since this is a simulation of a production-ready cluster, we changed it to RBAC, which we already explored in one of the previous chapters.

The --yes argument specifies that the cluster should be created right away. Without it, kops would only update the state in the S3 bucket, and we'd need to execute kops apply to create the cluster. Such two-step approach is preferable, but I got impatient and would like to see the cluster in all its glory as soon as possible. 

The output of the command is as follows.

...
kops has set your kubectl context to devops23.k8s.local
    
Cluster is starting.  It should be ready in a few minutes.
    
Suggestions:
 * validate cluster: kops validate cluster
 * list nodes: kubectl get nodes --show-labels
 * ssh to the master: ssh -i ~/.ssh/id_rsa [email protected]
The admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
 * read about installing addons: https://github.com/kubernetes/kops/blob/master/docs/addons.md

We can see that the kubectl context was changed to point to the new cluster which is starting, and will be ready soon. Further down are a few suggestions of the next actions. We'll skip them, for now.

We'll use kops to retrieve the information about the newly created cluster.

 1  kops get cluster

The output is as follows.

NAME               CLOUD ZONES
devops23.k8s.local aws   us-east-2a,us-east-2b,us-east-2c
  

This information does not tell us anything new. We already knew the name of the cluster and the zones it runs in.

How about kubectl cluster-info?

 1  kubectl cluster-info

The output is as follows.

Kubernetes master is running at https://api-devops23-k8s-local-ivnbim-609446190.us-east-2.elb.amazonaws.com
KubeDNS is running at https://api-devops23-k8s-local-ivnbim-609446190.us-east-2.elb.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
    
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

We can see that the master is running as well as KubeDNS. The cluster is probably ready. If in your case KubeDNS did not appear in the output, you might need to wait for a few more minutes.

We can get more reliable information about the readiness of our new cluster through the kops validate command.

 1  kops validate cluster

The output is as follows.

Using cluster from kubectl context: devops23.k8s.local

Validating cluster devops23.k8s.local

INSTANCE GROUPS
NAME              ROLE   MACHINETYPE MIN MAX SUBNETS
master-us-east-2a Master t2.small    1   1   us-east-2a
master-us-east-2b Master t2.small    1   1   us-east-2b
master-us-east-2c Master t2.small    1   1   us-east-2c
nodes             Node   t2.small    1   1   us-east-2a,us-east-2b,us-east-2c

NODE STATUS
NAME                 ROLE   READY
ip-172-20-120-133... master True
ip-172-20-34-249...  master True
ip-172-20-65-28...   master True
ip-172-20-95-101...  node   True

Your cluster devops23.k8s.local is ready

That is useful. We can see that the cluster uses four instance groups or, to use AWS terms, four auto-scaling groups (ASGs). There's one for each master, and there's one for all the (worker) nodes.

The reason each master has a separate ASG lies in need to ensure that each is running in its own availability zone (AZ). That way we can guarantee that failure of the whole AZ will affect only one master. Nodes (workers), on the other hand, are not restricted to any specific AZ. AWS is free to schedule nodes in any AZ that is available.

We'll discuss ASGs in more detail later on.

Further down the output, we can see that there are four servers, three with masters, and one with worker node. All are ready.

Finally, we got the confirmation that our cluster devops23.k8s.local is ready.