Chapter 4: The Original GitOps – Continuous Deployment in Kubernetes

Understanding how the industry has arrived at the current state of continuous delivery and continuous deployment is essential in understanding the fundamentals of GitOps, what GitOps aims to solve, and why GitOps is important.

Even though GitOps is in its infancy, the practice has already evolved into a few different schools of thought. The first of these schools can be summed up as originalist, since it covers the practice expounded by the group that first coined the term GitOps.

In this chapter, we're going to cover the following main topics:  

• Original GitOps basics
• Kubernetes and operators
• Manifest explosion
• Benefits and drawbacks of originalist GitOps
• Common originalist GitOps tools

Original GitOps basics

The DevOps team was now in a mad scramble to get a GitOps process implemented and adopted. The development teams had continued to break down the old monolithic application into containers to deploy into a Kubernetes cluster, resulting in significant growth of deployable services across all clusters and teams. The haphazard service-oriented architecture style of deployments with tightly coupled version dependencies would soon result in a deployment process that is too heavy to support. This would require significant team growth and manual intervention just to achieve the business objectives.

By implementing Argo CD in the cluster, the team was able to quickly have deployments automatically sync with the cluster tied to the tool. Argo CD is a GitOps tool and closely aligns itself with the group that first coined the term GitOps in 2017. The original intent is to allow an in-cluster engine to execute the desired deployment, known as auto-syncing, with the Kubernetes cluster that the engine is tied to. The two main areas that the original GitOps method is most associated with are Kubernetes manifests, like Helm, and Infrastructure-as-Code files, like Terraform.

When there is a change in the code repository that houses either the Kubernetes manifest or the Terraform files, ArgoCD will auto-sync the changes with the cluster.

Since the team currently leverage Terraform and Kubernetes, the originalist style of GitOps would offer a quick solution for the DevOps team. The only thing that the team needed to better understand was how ArgoCD worked in Kubernetes in order to not be locked into that tool in the future.

Deadlines and compelling events always seem to loom overhead as teams attempt to accomplish a desired outcome. The need to make a deployment process scalable and reliable is difficult to achieve when under a time-crunch. This desired outcome is precisely many companies in the industryare wanting to move to a GitOps model of deployment. The native tracking, versioning, and visibility that comes with storing code files in a Git repository makes onboarding of users and services easy. However, the process of adopting originalist GitOps means that the execution engine must have already built and tested manifests.

Prior to GitOps being introduced a few years ago, Kubernetes deployments required advanced Kubernetes knowledge. Any solution that allowed all users to have easy access to interact with Kubernetes was sorely needed. The only way to make that happen was for a tool to be an intermediary between the user and the cluster. Most teams looking to solve the problem had to build out an internal platform. But even then, most of the Kubernetes interaction was still delegated to an administrator group.

What the originalist GitOps practice capitalized on was the native ability of Kubernetes to handle the deployment and configuration of desired resources. At first, the users would create a sprawling mess of Kubernetes flat files, which was eventually improved through templating engines. In essence, templating engines allow for manifest files to scale through variable overrides. Because Kubernetes allows the declarative files to dictate what is deployed, the originalist GitOps process only had to facilitate the execution of the deployment. This means that the GitOps tool wasn't technically doing the deployment but rather just executing the deployment command.

One major benefit with originalist GitOps tools is the ability for the tool to understand and enforce the desired end state of a Kubernetes workload based on what is in the Git repository. This process of synchronizing the workload in the cluster with the manifest in the Git repository is not a unique concept. Kubernetes leverages the same synchronization design between etcd, the replication controller, and the kubelet.

One of the major issues with any form of automation that is provided by a third-party tool is understanding how that tool works. When leveraging a tool like ArgoCD to handle any GitOps process inside of a Kubernetes cluster, it is important to understand what ArgoCD is doing in the cluster and how. Being locked into a tool is not something that any team wants and understanding the underlying behavior will help to avoid the lock-in dilemma.

Kubernetes and operators

One major issue with GitOps, as the DevOps team was experiencing, is that the platform that the application relies on is the main execution engine for the GitOps process. Although the execution in the underlying platform being inherent is helpful, it also means that the execution is like a black box where the underlying actions are relatively hidden.

To avoid this black box issue with the GitOps process, the DevOps team decided to dive deeper into what Argo CD is doing during an execution event. By documenting Argo CD's underlying tasks, the troubleshooting process will become much simpler. The first step to understand and document is how Kubernetes behaves when an execution command is triggered.

Up to this point, the teams are basically just running helm commands. The team will monitor the helm output for a success or failure message, and then moves on. What the DevOps team wanted to understand was how Kubernetes behaved when the command was run.

The deployment process for Kubernetes applications is significantly different than the quarterly release pipeline. In the quarterly releases, the teams had to be very specific as to what needs to be installed on which servers, along with any configuration requirements. But with Kubernetes, the entire deployment process is inherent to the platform, with a manifest declaring the desired end state.

A helm command would start the process by executing the desired templating command to bring the files together into a single large manifest. Then, the underlying engine in helm would execute a kubectl command to pass the manifest to Kubernetes. The kubectl command would pass the manifest from the YAML file to the etcd store via the kube-api-server. Etcd was then storing the desired end state of the application based on the manifest and other core system components would run certain actions based on the change in etcd. A controller would notice a change in etcd and move to steer the cluster to be in the desired state, which would involve the kube-scheduler, the kubelet, and other controllers. Once the resources were deployed onto the appropriate nodes, the response from the kube-api-server is a success message saying that the command was successful. When the resources were added to the appropriate nodes, the components on the nodes would then work to get the resources into the appropriate state by downloading the required containers and any relevant files. If there was an issue with the deployment, the users could execute a helm command to roll back the execution to a previous version and the previous process would execute again.

This underlying process remained entirely unchanged when using Argo CD. The helm process and kubectl process remained the same, as did the provision of the manifest. What the team was able to see from the Argo CD side is the use of a Kubernetes operator that would trigger and track the deployments instead.

A Kubernetes operator is essentially another controller that is programmed to execute a desired process, such as maintaining a customized resource according to that custom resource's definition requirements. The operator would live in a state of constant checking to ensure that the custom resource was always up to date, which is the main concept behind the Argo CD operator. Argo CD considers the custom resource as the actual resource to deploy and the definition is the manifest stored in the Git repository. When the custom resource is not in sync with the Git repository, the operator, similar to a controller, will work to steer the deployment in the right direction. This is accomplished by executing the appropriate helm or kubectl command on the underside.

Another benefit for the Argo CD operator is that it can also treat the cluster as a custom resource and leverage an Infrastructure-as-Code repository as the resource definition to stay consistent. By Argo CD maintaining both the Kubernetes resources and infrastructure resources through the operator configuration the DevOps team should only need to concern themselves with getting the Git repository into the desired state and setting up the connection between Argo CD and the Git repository. The main parts for the DevOps team to document are the rollback process and what the manifest administration requirement will look like.

Kubernetes seems simple from a cursory glance at the execution requirements, since the process typically leverages a simple command and some manifest files. However, when considering what the Kubernetes platform is made up of and how it operates, many would-be Kubernetes practitioners can become quickly overwhelmed.

Harkening back to the first few chapters, the foundational concepts of an application instance living on a server and needing to be updated have not changed in the world of containers and Kubernetes. The two major differences are the size of the application instances and the container orchestration process.

Before Kubernetes, teams were orchestrating which servers to update how to update and so on. Kubernetes does this same thing, but with less hands-on administration requirement from multiple teams. Kubernetes leverages two core concepts inside of a cluster that make automated orchestration a possibility: etcd and controllers.

etcd, which is the in-cluster data store, is intended to keep a consistent record of the state of the entire cluster. When the user requests information of Kubernetes, either a request to create resources in Kubernetes or to retrieve resource information from Kubernetes, etcd is the source that the request goes to. The initial step of the create command is to write to etcd, which then trickles down the execution process to create the resources in the cluster, and then etcd reports back the outcome of the execution. The kube-api-server is the communication gateway for all users that want to interact with etcd, but also for every component in the cluster to interact with etcd as well. This is especially true with the other orchestration automation component.

Every Kubernetes cluster has a set of system components that help with cluster maintenance and administration. etcd and kube-api-server are some of these core components, but the components that actually execute most of the work in the cluster are the controllers.

Each controller in Kubernetes manages one or more resource types in the cluster. The management process consists of the controller reading what etcd says about that resource type, reading what the cluster shows relating to the resource type, and then working to make the two stay in sync with each other. Controllers will accomplish their tasks either by executing kube-api-server requests, or taking effects directly on the cluster. The complexity of the inner workings of Kubernetes are often uncovered the hard way, which is to say that they are uncovered during troubleshooting an issue. Therefore, it is highly recommended that anyone pursuing originalist GitOps practices should have a thorough working knowledge of Kubernetes first.

A popular originalist GitOps tool in the market is Argo CD, which leverages the operator model inside of the Kubernetes cluster. The goal of Argo CD is to make sure that the current state of a designated resource set inside of the cluster matches the desired end state that is declaratively defined in the Kubernetes manifests in a Git repository. This operator model with Argo CD is exactly the same as the controller model, but there is one significant difference: A controller treats etcd as the source of truth, while Argo CD treats the Git repository as the source of truth. The underlying behavior of Argo CD in relation to the Kubernetes cluster is essential to understand, especially when considering the administrative requirement for Kubernetes manifests and failure strategies.

Manifest explosion

The DevOps team continued their exploration of the underlying operations and requirements for GitOps with Argo CD and Kubernetes. One of the parts they also wanted to understand was the process of onboarding and maintaining the manifests for Argo CD to work correctly. The goal is to make a repeatable, reliable, and scalable process for any current and future applications across the grow cluster size and count.

One of the first things they noticed about the manifest requirements is that every microservice required its own helm chart. Additionally, to add overrides files for different environments, the override files would have to reside in the same repository as the chart for Argo CD to leverage them. This Argo CD requirement meant that the DevOps team would have to maintain the core helm chart for every microservice and work through a pull request approval process for every chart or values file change on every microservice.

Because of this, the DevOps team would have to come up with a core chart and approved override files that could then be shared out to every application team. The team would also need to automate the creation and linking process of the charts, the repository, and Argo CD in the appropriate cluster. But this presented yet another issue for the team to consider: how would they support one application or microservice that needs to be deployed across different clusters?

The quarterly release process had some major issues with the onboarding process because every step had to be specifically defined and changed ahead of time. The DevOps team had to provide a configuration file for every group of servers that had to be updated and the script required constant maintenance. Although ArgoCD offered the ability to get away from the issue of script maintenance and administration, an enterprise-wide adoption would be very difficult to adopt.

The only possible solution to make the administration requirements lighter for the DevOps team would be to move the helm chart maintenance burden to the developers. It didn't matter who was responsible for maintaining the charts and files, the charts and files still need to be built and maintained. If the DevOps team did the building and maintaining processes, then they could enforce standardizations and easy compliance and auditing. But with developers are maintaining the charts, then different auditing steps would need to be in place.

There are some who would argue that the role of a Kubernetes administrator is not really a Kubernetes administrator, but more of a YAML engineer. This is because the vast majority of Kubernetes configurations are handled through YAML files. Therefore, being able to understand how a YAML file should be constructed, what templating is available, and other Kubernetes nuances is required.

Prior to the templating abilities that came about through go-templating and helm, Kubernetes administrators would maintain individual Kubernetes flat files. There were tools that allowed for the user to provide some inputs that the tool would convert into a Kubernetes manifest, but the maintenance requirements still existed. Storing the YAML files in a Git repository was nothing new when originalist GitOps was marketed. Rather, the ability to leverage the Git repository as the application source of truth was the breakthrough.

As mentioned before, Argo CD, and other originalist GitOps tools, are essentially Kubernetes controllers that work to steer the cluster state towards what the git repository would show. This operator model is an extension of the current controller-etcd relationship within the cluster. However, the manifest requirements that originalist GitOps requires can be very cumbersome.

Something to note about controllers in Kubernetes is that it is very difficult, if not impossible, to have a controller in one cluster monitor designated resources in another cluster as well. The closest that a controller can get to this type of functionality is to increase the underlying node pool and leverage namespaces for everything. The reason for this impossibility is because the controller's purpose is to compare the current resource state in the cluster with the desired resource state represented in the source of truth: etcd.

Originalist GitOps tools have the same issue that controllers do. It is very difficult for the originalist GitOps tool to accurately monitor and steer the resources in the cluster toward the desired state defined in the git repository. Additionally, since the originalist GitOps tool is treating the git repository as the source of truth for a set of resources in a cluster, to spread a single set of manifests across multiple resources and multiple clusters is impossible.

Any in-cluster originalist GitOps operators will require a combination of manifest files for each microservice and each cluster. Adding to this manifest requirement, any overrides or templating that a user would want to have requires heavy setup requirements and maintenance for the originalist GitOps tool and designated microservice. For example, if a team needed different values to be applied for every environment that the helm chart is deployed to, helm can leverage a command flag to do this. However, to tell the originalist GitOps tool to accomplish the same outcome would require the administrator to execute setup commands when adding the desired repository link. With that command, the administrator would have to add any specific variables to override, similar to helm set flags, and values files to override, similar to helm file flags. To make the process even more nuanced, if the chart needed to be deployed to multiple environments with a different values file depending on the environment, the GitOps administrator would have to set up a separate deployment for the same chart on every environment.

The requirement for the manifest hierarchy process in the originalist GitOps configuration results in a manifest explosion that must be tracked, maintained, administrated, audited, and scaled.

Benefits and drawbacks of originalist GitOps

As the DevOps team dove deeper into the world of originalist GitOps and tools such as Argo CD, they discovered some very promising benefits of adopting this type of process. They found that Argo CD allowed for the mandatory Kubernetes manifests to be the main driver for configuration and change to microservice resources in the cluster. And since the developers were already used to using git, the adoption process of originalist GitOps automation would not be too much of a burden for the developers to work through.

By treating the git repository and helm chart as the source of truth for the desired end state, the user should always be able to see what microservice configuration is in the cluster based on what the git repository shows. Also, since git has native versioning, a rollback to a previous version is as easy as reverting a version in git and letting Argo CD handle the deployment part.

One other benefit that the team found was that if a cluster went down, since the helm charts existed as the source of truth in the git repository, they could spin up another cluster and return it to the desired state in very little time. This shorter mean-time-to-restore for cluster recovery was a massive benefit for the company.

There were some drawbacks that the team found as well. They had a set of helm charts already created by the development teams, but the DevOps team didn't want to place the manifest creation and maintenance burden on the developers. Since Argo CD required that a chart existed in a single repo for every set of microservice resources that needed to be deployed to Kubernetes, there would be a manifest explosion.

Another drawback, which is significant for the engineering organization, is that the developers have a different cluster for every environment, meaning that there are four different clusters that need to be deployed to. Unfortunately, Argo CD does not easily allow for cross-cluster deployments of the same helm chart in sequential order. Adding to this issue, the company will require multiple production clusters, one for each major region of users across the globe. Argo CD does not have any way of being able to deploy the same chart across multiple clusters in different regions at the same time. The best option that the team could find was to leverage an Argo CD architecture known as an "app of apps," where a single git repository would store the files required to dictate applications and cluster, which requires manual executions only.

The last drawback, and probably the most important drawback the team found, was that the originalist GitOps process is only for deployments and specifically for the easiest part of the process. Every microservice manifest had to manually be created and tested with every values file combination to every environment and cluster in the architecture. Then, when the files were finalized and added to the git repository, Argo CD could be connected to execute the install command for the microservice. But all of the other required steps, such as approvals, tickets, testing, verification, and others, would have to be done outside of the GitOps process.

Argo CD is a great GitOps solution and can solve many problems for different companies and teams in the industry. But the DevOps team needed to see if maybe there was a better GitOps tool that was native to Kubernetes and could handle all of their delivery and deployment requirements.

Kubernetes is not the easiest platform to set up, maintain, administrate, or scale. The ability to dive deeper into the inner workings of a Kubernetes cluster is available but can quickly become overwhelming. In many cases, a Kubernetes administrator wants to set up a Kubernetes cluster in the default manner and keep it that way. Kubernetes practitioners that build out YAML files for microservices deployments would also like to keep their manifests as simple as possible. But these desires are not the norm for the industry today. The growing security concerns around network, secrets, access control, and cloud platforms require signficant customizations. Some microservices require autoscaling, others require node affinity, and then there are microservices that rely on the cluster defaults for these types of specifics.

There is a massive advantage when leveraging a GitOps tool to handle the deployments of microservices. Because of the requirements for the microservices to have some set of manifests to dictate which Kubernetes resources should be deployed set up for the microservice to work correctly, the GitOps tool can simply leverage that requirement. This ability of deployment automation by the GitOps tool makes for an easy user adoption process as well.

The main integration point that makes an automated GitOps process beneficial is the git repository. By treating the git repository as the source of truth for the desired state, the GitOps tool enables easy synchronization between the user and the cluster through automated events.

However, any process or tool change needs to have an evaluation process that is relative to the level of impact the change will have on the users, teams, or business. In the case of originalist GitOps, the deployment automation process and tool change would significantly impact the teams that are administrating the tool, as well as the production environment that is being deployed to.

When looking at a GitOps tool such as Argo CD, which has some major benefits for any and all Kubernetes practitioners, the fundamental question to answer is "what is the tool actually accomplishing?" The automation of the deployment, the automated synchronization of the git repository and the cluster, and even the ease of install and initial set up are all great benefits. And yet, the manifest sprawl requirement, the lack of cross-cluster deployment capabilities, and the limited functionality across the entire delivery process can be significant enough reasons to look for an alternative tool or process.

Common originalist GitOps tools

The DevOps team felt as though they were back to square one with their research and implementation. Argo CD would've been the perfect tool for them to use if they had the rest of the delivery process figured out and automated. What they needed to do now was evaluate other GitOps tools in the market to see if they could solve the delivery needs better than Argo CD was able to. After an extensive search for alternatives to Argo CD, they discovered many different tools that all marketed themselves as GitOps tools or Argo CD alternatives.

The first on the list was Flux, which came from a company called Weaveworks. Weaveworks was actually the company that coined the term GitOps a few years ago and they built a tool that is similar to what Argo CD does. And, considering the developers are leveraging helm charts, WeaveWorks has also built out a helm operator to work with their Flux GitOps tool as well. However, the manifest requirements and the lack of support for the rest of the delivery requirements are drawbacks that Flux and Argo CD have in common.

Another tool that the team looked at was GitKube, which is similar in the sense that it can automate deployments to Kubernetes based on a change to a git repository, but the tool is advertised for testing changes to a manifest in a development environment, rather than for production clusters.

Another GitOps tool that seems to offer more features and functionality is JenkinsX. The DevOps team was already very familiar with Jenkins for CI and found it difficult to not associate the difficulties to JenkinsX But the documentation for JenkinsX seemed to show that the requirements were lighter than Jenkins's.

The company wanted to leverage a different cluster for each environment and was considering a multi-cloud strategy at some point in the near future, which were architecture designs that JenkinsX seems to support. The team decided that they would begin a GitOps proof-of-concept process with JenkinsX to see if it could fulfill all of their requirements.

Finding the right tool for an immediate need is simple, especially in engineering teams. If a tasks takes too long, requires a manual interaction, or causes consistent problems, an engineer will look to automate it. This process of looking for an immediate solution to an immediate issue can be considered a falvor-of-the-month-type approach to problem solving. Rather than solving the cause of an issue, the flavor-of-the-month solution looks to solve the immediate pain. This decision making process always leads to technical debt down the road since it rearely solves the actual problem.

Most DevOps teams have experience with a tool that was implemented to solve a small or inconsequential issue. Every veteran engineer knows what it is like to join a new team and immediately be hamstrung by all of the rushed decisions that were made by previous team members. In fact, most engineers assume and expect that any new team or company they join will have some level of limitation or process debt that is left over from previously rushed decisions.

Argo CD has become a very popular originalist GitOps tool, in that it automates the synchronization of a git repository and a cluster. Weaveworks Flux is a similar tool that attempts to accomplish the same requirement. In fact, Weaveworks and Argo attempted at one point to build a hybrid solution called Argo Flux.

Considering what Argo CD and Flux are able to do and where they sit, both are great options for automating Kubernetes deployments.

If a company is leveraging multiple clusters, multiple clouds, or multiple regions, they might need to look into a tool that has explicit support for those architecture requirements. Although tools like Argo CD and Flux could be used across multiple clusters or regions, the functionality of the tools would be too disparate to be beneficial. A tool like JenkinsX has support for multiple clusters and clouds built in and is also built to solve originalist GitOps requirements.

Summary

Originalist GitOps is defined as a tool that automatically deploys Kubernetes resources to a Kubernetes cluster by synchronizing the cluster with a git repository. The analogy in this chapter showed what originalist GitOps tools are able to do, some requirements of those tools, benefits and drawbacks, and the different types of originalist GitOps tools used in the industry.

The next chapter will explore the purist GitOps school of thought, showing how it is different than originalist GitOps and what tools can be used to accomplish a purist GitOps practice.