The typical reason why most enterprises adopt the cloud is to accelerate application development. Applications are constantly evaluated and changed to add new features. Since everything is codified in the cloud, these new features need to be tested on the infrastructure of the target cloud. The final step in the life cycle of applications is the actual deployment of applications to the cloud and the handover to operations so that developers have their hands free to develop new features again, based on business requirements.
To speed up this process, organizations work in DevOps cycles, using release trains with continuous development and the possibility to test, debug, and deploy code multiple times per week, or even per day, so that these applications are constantly improved. Consistency is crucial: the source code needs to be under strict version control. That is what CI/CD pipelines are for: continuous integration and continuous delivery and deployment.
We will study the principles of DevOps, how CI/CD pipelines work with push and pull mechanisms, and how pipelines are designed so that they fit multi-cloud environments. Also, tooling for DevOps and CI/CD is discussed.
In this chapter, we're going to cover the following main topics:
Before we get into the principles of CI/CD and pipelines, we need to have a good understanding of DevOps. There are a lot of views on DevOps, but this book sticks to the definition and principles as defined by the DevOps Agile Skills Association (DASA). They define a DevOps framework based on six principles:
DevOps has been described in the literature as culture, a new way of working. It's a new way of thinking about developing and operating IT systems based on the idea of a feedback loop. Since cloud platforms are code-based, engineers can apply changes to systems relatively easily. Systems are code, and code can be changed, as long as changes are applied in a structured and highly controlled way. That's the purpose of CI/CD pipelines.
Continuous Integration (CI) is built on the principle of a shared repository, where code is frequently updated and shared across teams that work in the cloud environments. CI allows developers to work together on the same code at the same time. The changes in the code are directly integrated and ready to be fully tested in different test environments.
Continuous Delivery or Deployment (CD) focuses on the automated transfer of software to test environments. The ultimate goal of CD is to bring software to production in a fully automated way. Various tests are performed automatically. After deployment, developers immediately receive feedback on the functionality of the code.
CI/CD enables the DevOps cycle. Combined with CI/CD, all responsibilities, from planning to management, lie with the team, and changes can reach the customer much faster through an automated and robust development process. The following diagram shows the DevOps cycle with CI/CD:
In the next section, we will study how to get started with CI/CD.
CI/CD is widely adopted by enterprises, but a lot of projects fail. This section explains how enterprises can successfully implement CI/CD and how they can avoid pitfalls. The major takeaway should be that an implementation starts with consistency. That counts for cloud implementations as well as for CI/CD.
With CI, development teams can change code as often as they want, leading to the continuous improvement of systems. Enterprises will have multiple development teams, working in multi-cloud environments, which makes it necessary to have one way of working. Fully automated processes in CI/CD pipelines can help keep environments consistent. CI/CD and DevOps are, however, not about tools. They're about culture and sticking to processes.
To get to a successful implementation of DevOps, an organization is advised to follow these steps:
The DoD describes the conditions and the acceptance criteria a system must meet before it's deployed to production. The DoD is the standard of quality for the end product, the application, or IT system that needs to be delivered. In DevOps, teams work with user stories. An example of a user story is: as a responsible business owner for an online retail store, I want to have multiple payment methods so that more customers can buy our products online. This sets requirements for the development of applications and systems. The DoD is met when the user story is fulfilled, meaning that unit testing is done, the code has been reviewed, acceptance criteria are signed off, and all functional and technical tests have been passed.
The following diagram shows the concept of implementing a build and release pipeline with various test stages. The code is developed in the build pipeline and then sent to a release pipeline where the code is configured and released for production. During the release stages, the full build is tested in a test or Quality and Assurance Environment (Q&A). In Q&A, the build is accepted and released for deployment into production:
If all these steps are followed through, an organization can start working in DevOps teams using CI/CD. In the next sections, CI/CD is explained in more detail, starting with version control, and then discussing the functionality of commits, push and pull mechanisms in the pipeline.
By working from one code repository with different teams, version control becomes crucial in CI/CD. Git and Subversion are popular version control systems that enable teams to organize their files that form the source code, test scripts, deployment scripts, and configuration scripts used for applications and infrastructure components. Everything is code, which means that systems consist of a number of code packages: the code for the VM, code for how the VM should be configured based on policies, and the application code itself. A version control system also enables teams to retrieve the historical state of systems, in case a deployment fails or systems are compromised and need to be rebuilt.
Version control systems keep track of changes to files that are stored in the repository. In DevOps, these changes are commonly referred to as commits, something that we'll discuss further in the next section, Using push and pull principles in CI/CD, forking and merging code. A commit comprises the code change itself and metadata, holding information on who made the change and the rationale behind the code change. This ensures that code is kept consistent and with that, repeatable and predictable. It also means that teams are forced to document everything in the repository and bring it under version control.
This list contains many of the items that need to be under version control:
Once companies have implemented this, they need to maintain it. This is not a one-time exercise. Teams should confirm that version control is applied to application code, systems configuration, and automation scripts that are used in the CI/CD pipeline. Only if this is applied and used in a consistent way will enterprises be able to deploy new applications and systems rapidly, yet securely and reliably.
CI/CD pipelines work with branches, although other terms can be used for this. The master branch is sometimes referred to as a mainline or, when teams work in GCP, as a trunk. The most important principle to remember is that a development team has one master branch or mainline. Next, there are two ways of pushing new code to that master branch. These two methods are described in the following sections.
In this method, the developers work directly in the master code; they change small pieces of the code and merge these directly back into the master branch. Pushing code back to the master is called a commit. These commits are done several times per day, or at least as soon as possible. Working in this way ensures that releases can be done very frequently, as opposed to working in code forks that result in separate or feature branches, which are described in the second method. The following diagram shows the way of working with a direct push to a master branch:
The idea behind CI is that companies get rid of long, complex integrations. Developers work in small batches of the code that they frequently commit to the master. The big advantage is that developers immediately see whether the change is done correctly, with the possibility to revert the change without having a huge impact on the master as a whole. This is DevOps – the developers are responsible for the build, the commit, and the result: you break it, you fix it. Automated tests that are executed after the code commit are crucial to ensure that systems keep running without failures.
In this method, teams copy code from the master and create a separate or feature branch. This is also referred to as forking: developers create a feature branch by taking a copy from the source code, the master. They do their development on this forked code. In GCP, this is not trunk-based development, or better said: this is referred to as feature-driven development.
This method is often used for major developments, creating new features. Developers can work in isolation on the forked code, and when they're done, commit the code back to the master, where the new features or new builds are merged with the master. The downside is that this can lead to complex integrations. This can't be done on a frequent basis, but takes intensive testing before the merging takes place. The following diagram shows the way of working with feature branches:
In both methods, code gets pushed to the repository in, for example, GitHub. As soon as a developer has committed their code, they will do a pull request. This is the stage where the new, changed code is reviewed before the changes are actually merged into the master branch.
There are a few best practices to remember when working with CI/CD. One of the biggest pitfalls is that code reviews are too extensive, meaning that developers have to get approval from different stakeholders before they can push the code to production. This will cost a lot of time and effectively slow down the DevOps process. Companies that adopt DevOps should have two principles implemented:
If these two principles are followed, then extensive reviews and approval processes are not required – at least when developers work in small batches of the master. When creating new features and builds where developers work in a forked copy of the source or feature branch, a more intensive review process is advisable.
The development of code for applications can be cloud-agnostic, meaning that it doesn't matter to which cloud the code is pushed: the functionality of the code remains the same. However, a lot of developers will discover that it does matter and that's it not that simple to develop in a truly multi-cloud fashion. For the DevOps process itself, it probably doesn't matter on which platform the code lands, but it does matter as soon as teams push code into production. Then there are platform specifics that need to be taken into consideration.
In multi-cloud, developers also work from one repository, but during deployment, platform-specific configuration is added and tested. This is the staging phase. AWS, Azure, and GCP all have their specific provisioning features that need to be tested with the application code. In the staging phase, the code package is merged with infrastructure configuration for provisioning to a target cloud platform and is tested.
There are a few steps that developers have to take to make it successful. First of all, the DevOps way of working should be consistent, regardless of the platform where applications will eventually land. A company might want to run applications in Azure, AWS, GCP or even on-premises, but the way application code is written in DevOps cycles should be the same. Each of these platforms will have specific features to run the code, but that's a matter of configuration. Staging is meant to find out whether the whole package is ready for release to production.
In Chapter 8, Defining Automation Tools and Processes, we already learned that developers need to think in layers: they need to abstract the application layer from the resources in infrastructure and the configuration layer. That's the only way to get to a consistent way of application development with CI/CD. One other challenge that developers need to tackle in multi-cloud environments is the high rate of changes in the various cloud platforms. DevOps tools must be able to adapt to these changes in deployment, but without having to constantly change the application code itself.
Developers, however, need to have a good understanding of the target platforms and their specifics. It also makes sense to study the best practices in DevOps in these platforms and the recommended tooling: this will be discussed in more detail in the Exploring tooling for CI/CD section of this chapter about tooling for CI/CD. Most common tools are cloud-agnostic, meaning that they can work with different clouds, leveraging the native APIs.
So, the ground rules for the successful implementation of DevOps are as follows:
In terms of one framework, SAFe by Scaled Agile should be mentioned. SAFe stands for Scaled Agile Framework, and it's used by a lot of enterprises as the foundation of DevOps. One of the key assets of SAFe is the Agile Release Train.
The Agile Release Train is built around the principle of the standardization of application development and the release management of code. This is done by automating everything:
This supports the application life cycle management, the continuous improvement, and release of the application code. Details and courseware on SAFe can be found at https://www.scaledagileframework.com/DevOps/.
The tooling landscape for CI/CD and DevOps is massive and changes almost every month. A good overview is provided by Digital.ai at https://digital.ai/periodic-table-of-DevOps-tools; Digital.ai maintains and publishes the Periodic Table of DevOps. It's based on the format of the periodic table of elements, but contains an overview of various tools to execute DevOps in cloud environments. It also explains how to build a pipeline diagram using the tools that are selected from the periodic table.
There's no right or wrong answer in choosing the toolset, as long as it fits the need of the enterprise and people are trained in the usage of the tools. In this section, the native CI/CD tooling in the major clouds are discussed: Azure DevOps, AWS CodePipeline and CloudFormation, and Google Cloud Build.
The following screenshot shows the main menu of Azure DevOps with a project defined as Scrum that divides the work items in backlog items and sprints:
When a team starts in Azure DevOps, the first thing to do is to define a project and assign project members. Next, the project manager or product owner defines the development methodology. Depending on that choice, DevOps presents the possibilities to define work items, such as features, backlog items and tasks, and a board to plan the execution of these items. A work item or product can be a piece of code that may be deployed; in DevOps, this can be automated with Azure Pipelines, which after review actually deploys the code to the target cloud environment.
Developers can use Azure DevOps for AWS too, using the AWS Toolkit for Azure DevOps. The toolkit even accepts the use of AWS CloudFormation templates to provision and update AWS resources within Azure DevOps. In the Further reading section, a link to the documentation has been added.
The following screenshot shows the main menu of CodePipeline:
As shown in the screenshot, creating a pipeline from the CodePipeline main menu will start by pulling code from a repository that sits in CodeCommit. The pipeline itself is built in CodeBuild and deployed in CodeDeploy.
The following screenshot shows the menu of Cloud Build:
The main menu is very lean, as shown in the preceding screenshot. Only when a project is defined and started can developers start using Cloud Build to create the code repository. That service is available from the console, as shown in the following screenshot:
In this section, the native CI/CD toolsets of Azure, AWS, and GCP have been discussed. At the beginning of the section, it was mentioned that there are a lot of tools available to developers for developing, storing, testing, and deploying code. The main principles of CI/CD are the same, but there are differences in the way that these tools deploy code and especially how they test and validate code in an automated way. The tools discussed – Azure DevOps, AWS CodePipeline, and GCP Cloud Build – may cover a lot of functionality already, but typically, additional tools are required. In the Further reading section, some suggestions are provided for study material to help in choosing the right tools.
After completing this chapter, you should have a good understanding of the DevOps way of working and the use of CI/CD pipelines in cloud environments. Everything is code in cloud, from the application to the infrastructure and the configuration. Code needs to be stored in a central repository and brought under version control. That's where the CI/CD pipeline starts. Next, the DevOps team defines the phases of the pipeline, typically build, test, and deploy. Actions in these phases are automated as much as possible.
We discussed the push and pull principles in CI/CD pipelines using master and feature branches, describing the different methodologies to push and commit code to branches. If teams work consistently from one repository and with a unified way of working, they can deploy code to different clouds. Teams also need tooling: the last section provided an overview of the native CI/CD tooling in Azure, AWS, and GCP, working with Azure DevOps, AWS CodePipeline, and Cloud Build in GCP.
In the next chapter, the final concept for operations will be discussed: AIOps.
You can refer to the following links for more information on the topics covered in this chapter: