Chapter 18: Designing and Implementing CI/CD Pipelines

The typical reason why most enterprises adopt the cloud is to accelerate application development. Applications are constantly evaluated and changed to add new features. Since everything is codified in the cloud, these new features need to be tested on the infrastructure of the target cloud. The final step in the life cycle of applications is the actual deployment of applications to the cloud and the handover to operations so that developers have their hands free to develop new features again, based on business requirements.

To speed up this process, organizations work in DevOps cycles, using release trains with continuous development and the possibility to test, debug, and deploy code multiple times per week, or even per day, so that these applications are constantly improved. Consistency is crucial: the source code needs to be under strict version control. That is what CI/CD pipelines are for: continuous integration and continuous delivery and deployment.

We will study the principles of DevOps, how CI/CD pipelines work with push and pull mechanisms, and how pipelines are designed so that they fit multi-cloud environments. Also, tooling for DevOps and CI/CD is discussed.

In this chapter, we're going to cover the following main topics:

  • Understanding CI/CD and pipelines
  • Using push and pull principles in CI/CD
  • Designing the multi-cloud pipeline
  • Exploring tooling for CI/CD

Understanding CI/CD and pipelines

Before we get into the principles of CI/CD and pipelines, we need to have a good understanding of DevOps. There are a lot of views on DevOps, but this book sticks to the definition and principles as defined by the DevOps Agile Skills Association (DASA). They define a DevOps framework based on six principles:

  • Customer-centric action: Develop an application with the customer in mind – what do they need and what does the customer expect in terms of functionality? This is also the goal of another concept, domain-driven design, which contains good practices for designing.
  • Create with the end in mind: How will the application look when it's completely finished?
  • End-to-end responsibility: Teams need to be motivated and enabled to take responsibility from the start to the finish of the application life cycle. This results in mottos such as you build it, you run it and you break it, you fix it. One more to add is you destroy it, you rebuild it better.
  • Cross-functional autonomous teams: Teams need to be able and allowed to make decisions themselves in the development process.
  • Continuous improvement: This must the goal – to constantly improve the application.
  • Automate as much as possible: The only way to really gain speed in delivery and deployment is by automating as much as possible. Automation also limits the occurrence of failures, such as misconfigurations.

DevOps has been described in the literature as culture, a new way of working. It's a new way of thinking about developing and operating IT systems based on the idea of a feedback loop. Since cloud platforms are code-based, engineers can apply changes to systems relatively easily. Systems are code, and code can be changed, as long as changes are applied in a structured and highly controlled way. That's the purpose of CI/CD pipelines.

Continuous Integration (CI) is built on the principle of a shared repository, where code is frequently updated and shared across teams that work in the cloud environments. CI allows developers to work together on the same code at the same time. The changes in the code are directly integrated and ready to be fully tested in different test environments.

Continuous Delivery or Deployment (CD) focuses on the automated transfer of software to test environments. The ultimate goal of CD is to bring software to production in a fully automated way. Various tests are performed automatically. After deployment, developers immediately receive feedback on the functionality of the code.

CI/CD enables the DevOps cycle. Combined with CI/CD, all responsibilities, from planning to management, lie with the team, and changes can reach the customer much faster through an automated and robust development process. The following diagram shows the DevOps cycle with CI/CD:

Figure 18.1 – DevOps cycle with CI/CD

Figure 18.1 – DevOps cycle with CI/CD

In the next section, we will study how to get started with CI/CD.

Getting started with CI/CD

CI/CD is widely adopted by enterprises, but a lot of projects fail. This section explains how enterprises can successfully implement CI/CD and how they can avoid pitfalls. The major takeaway should be that an implementation starts with consistency. That counts for cloud implementations as well as for CI/CD.

With CI, development teams can change code as often as they want, leading to the continuous improvement of systems. Enterprises will have multiple development teams, working in multi-cloud environments, which makes it necessary to have one way of working. Fully automated processes in CI/CD pipelines can help keep environments consistent. CI/CD and DevOps are, however, not about tools. They're about culture and sticking to processes.

To get to a successful implementation of DevOps, an organization is advised to follow these steps:

  1. Implementing an effective CI/CD pipeline begins with all stakeholders implementing DevOps processes. One of the key principles in DevOps is autonomous teams that take end-to-end responsibility. It's imperative that the teams are given the authority to make decisions and act on them. Typically, DevOps teams are agile, working in short sprints of 2 to a maximum of 3 weeks. If that time is wasted on getting approval for every single detail in the development process, the team will never get to finish anything in time.
  2. Choose the CI/CD system. There are a lot of tools on the market that facilitate CI/CD. Jenkins is a popular one, but a lot of companies that work in Azure choose to work in Azure DevOps. Involve the people who have to work daily with the system and enable them to take a test drive. Then, make a decision and ensure all teams work in that system. Again, it's about consistency.
  3. It's advised to do a proof of concept. An important element of CI/CD is the automation of testing, so the first step is to create an automated process pipeline. Enterprises often already have quality and test plans, possibly laid down in a Generic Test Agreement (GTA). This describes what and how tests must be executed before systems are pushed to production. This is a good starting point, but in DevOps, organizations work with a Definition of Done (DoD).

    The DoD describes the conditions and the acceptance criteria a system must meet before it's deployed to production. The DoD is the standard of quality for the end product, the application, or IT system that needs to be delivered. In DevOps, teams work with user stories. An example of a user story is: as a responsible business owner for an online retail store, I want to have multiple payment methods so that more customers can buy our products online. This sets requirements for the development of applications and systems. The DoD is met when the user story is fulfilled, meaning that unit testing is done, the code has been reviewed, acceptance criteria are signed off, and all functional and technical tests have been passed.

    The following diagram shows the concept of implementing a build and release pipeline with various test stages. The code is developed in the build pipeline and then sent to a release pipeline where the code is configured and released for production. During the release stages, the full build is tested in a test or Quality and Assurance Environment (Q&A). In Q&A, the build is accepted and released for deployment into production:

    Figure 18.2 – Conceptual diagram of a build and release pipeline

    Figure 18.2 – Conceptual diagram of a build and release pipeline

  4. Automate as much as possible, as one of the principles of DevOps. This means that enterprises will have to adopt working in code, including Infrastructure as Code (IaC). In CI/CD, teams work from one repository, and this means that the application code and the infrastructure code is in the same repository, so that all teams can access it whenever they need to.

If all these steps are followed through, an organization can start working in DevOps teams using CI/CD. In the next sections, CI/CD is explained in more detail, starting with version control, and then discussing the functionality of commits, push and pull mechanisms in the pipeline.

Working under version control

By working from one code repository with different teams, version control becomes crucial in CI/CD. Git and Subversion are popular version control systems that enable teams to organize their files that form the source code, test scripts, deployment scripts, and configuration scripts used for applications and infrastructure components. Everything is code, which means that systems consist of a number of code packages: the code for the VM, code for how the VM should be configured based on policies, and the application code itself. A version control system also enables teams to retrieve the historical state of systems, in case a deployment fails or systems are compromised and need to be rebuilt.

Version control systems keep track of changes to files that are stored in the repository. In DevOps, these changes are commonly referred to as commits, something that we'll discuss further in the next section, Using push and pull principles in CI/CD, forking and merging code. A commit comprises the code change itself and metadata, holding information on who made the change and the rationale behind the code change. This ensures that code is kept consistent and with that, repeatable and predictable. It also means that teams are forced to document everything in the repository and bring it under version control.

This list contains many of the items that need to be under version control:

  • Application code
  • API scripts and references (what is the API for?)
  • Infrastructure components such as VMs, network devices, storage, images for operating systems, DNS files, and firewall configuration rules
  • Infrastructure configuration packages
  • Cloud configuration templates, such as AWS CloudFormation, Desired State Configuration (DSC) in Azure, and Terraform files
  • Code definitions for containers, such as Docker files
  • Container orchestration scripts, such as Kubernetes and Docker Swarm
  • Test scripts

Once companies have implemented this, they need to maintain it. This is not a one-time exercise. Teams should confirm that version control is applied to application code, systems configuration, and automation scripts that are used in the CI/CD pipeline. Only if this is applied and used in a consistent way will enterprises be able to deploy new applications and systems rapidly, yet securely and reliably.

Using push and pull principles in CI/CD

CI/CD pipelines work with branches, although other terms can be used for this. The master branch is sometimes referred to as a mainline or, when teams work in GCP, as a trunk. The most important principle to remember is that a development team has one master branch or mainline. Next, there are two ways of pushing new code to that master branch. These two methods are described in the following sections.

Pushing the code directly to the master

In this method, the developers work directly in the master code; they change small pieces of the code and merge these directly back into the master branch. Pushing code back to the master is called a commit. These commits are done several times per day, or at least as soon as possible. Working in this way ensures that releases can be done very frequently, as opposed to working in code forks that result in separate or feature branches, which are described in the second method. The following diagram shows the way of working with a direct push to a master branch:

Figure 18.3 – Developers merging code directly to the master branch

Figure 18.3 – Developers merging code directly to the master branch

The idea behind CI is that companies get rid of long, complex integrations. Developers work in small batches of the code that they frequently commit to the master. The big advantage is that developers immediately see whether the change is done correctly, with the possibility to revert the change without having a huge impact on the master as a whole. This is DevOps – the developers are responsible for the build, the commit, and the result: you break it, you fix it. Automated tests that are executed after the code commit are crucial to ensure that systems keep running without failures.

Pushing code to forks of the master

In this method, teams copy code from the master and create a separate or feature branch. This is also referred to as forking: developers create a feature branch by taking a copy from the source code, the master. They do their development on this forked code. In GCP, this is not trunk-based development, or better said: this is referred to as feature-driven development.

This method is often used for major developments, creating new features. Developers can work in isolation on the forked code, and when they're done, commit the code back to the master, where the new features or new builds are merged with the master. The downside is that this can lead to complex integrations. This can't be done on a frequent basis, but takes intensive testing before the merging takes place. The following diagram shows the way of working with feature branches:

Figure 18.4 – Developers working in a feature branch before merging to the master

Figure 18.4 – Developers working in a feature branch before merging to the master

In both methods, code gets pushed to the repository in, for example, GitHub. As soon as a developer has committed their code, they will do a pull request. This is the stage where the new, changed code is reviewed before the changes are actually merged into the master branch.

Best practices while working with CI/CD

There are a few best practices to remember when working with CI/CD. One of the biggest pitfalls is that code reviews are too extensive, meaning that developers have to get approval from different stakeholders before they can push the code to production. This will cost a lot of time and effectively slow down the DevOps process. Companies that adopt DevOps should have two principles implemented:

  • The four-eyes principle: Have code reviewed while programming. By working in developer pairs, where the second developer reviews the code of the first developer. This is also referred to as Extreme Programming. Peer review is another method: here, the author of the code reviews and at least one other developer, typically at the end of the development process.
  • Running automated test scripts is most important. These scripts must be executed before code is actually committed to the master branch to make sure that systems keep functioning after the code commit.

If these two principles are followed, then extensive reviews and approval processes are not required – at least when developers work in small batches of the master. When creating new features and builds where developers work in a forked copy of the source or feature branch, a more intensive review process is advisable.

Designing the multi-cloud pipeline

The development of code for applications can be cloud-agnostic, meaning that it doesn't matter to which cloud the code is pushed: the functionality of the code remains the same. However, a lot of developers will discover that it does matter and that's it not that simple to develop in a truly multi-cloud fashion. For the DevOps process itself, it probably doesn't matter on which platform the code lands, but it does matter as soon as teams push code into production. Then there are platform specifics that need to be taken into consideration.

In multi-cloud, developers also work from one repository, but during deployment, platform-specific configuration is added and tested. This is the staging phase. AWS, Azure, and GCP all have their specific provisioning features that need to be tested with the application code. In the staging phase, the code package is merged with infrastructure configuration for provisioning to a target cloud platform and is tested.

There are a few steps that developers have to take to make it successful. First of all, the DevOps way of working should be consistent, regardless of the platform where applications will eventually land. A company might want to run applications in Azure, AWS, GCP or even on-premises, but the way application code is written in DevOps cycles should be the same. Each of these platforms will have specific features to run the code, but that's a matter of configuration. Staging is meant to find out whether the whole package is ready for release to production.

In Chapter 8, Defining Automation Tools and Processes, we already learned that developers need to think in layers: they need to abstract the application layer from the resources in infrastructure and the configuration layer. That's the only way to get to a consistent way of application development with CI/CD. One other challenge that developers need to tackle in multi-cloud environments is the high rate of changes in the various cloud platforms. DevOps tools must be able to adapt to these changes in deployment, but without having to constantly change the application code itself.

Developers, however, need to have a good understanding of the target platforms and their specifics. It also makes sense to study the best practices in DevOps in these platforms and the recommended tooling: this will be discussed in more detail in the Exploring tooling for CI/CD section of this chapter about tooling for CI/CD. Most common tools are cloud-agnostic, meaning that they can work with different clouds, leveraging the native APIs.

So, the ground rules for the successful implementation of DevOps are as follows:

  • One repository
  • One framework or way of working
  • A unified toolset that can target multi-cloud environments

In terms of one framework, SAFe by Scaled Agile should be mentioned. SAFe stands for Scaled Agile Framework, and it's used by a lot of enterprises as the foundation of DevOps. One of the key assets of SAFe is the Agile Release Train.

The Agile Release Train is built around the principle of the standardization of application development and the release management of code. This is done by automating everything:

  • Build by automating the process of compiling the code
  • Automated test of units, acceptance, performance, and load
  • Continuous integration, automated by running integration tests and releasing units that can be deployed to production
  • Continuous deployment, by automated deployments to different environments of the version-controlled code
  • Additional automation tools for configuration, provisioning, security by design, code review, audit trail, logging, and management

This supports the application life cycle management, the continuous improvement, and release of the application code. Details and courseware on SAFe can be found at

Exploring tooling for CI/CD

The tooling landscape for CI/CD and DevOps is massive and changes almost every month. A good overview is provided by at; maintains and publishes the Periodic Table of DevOps. It's based on the format of the periodic table of elements, but contains an overview of various tools to execute DevOps in cloud environments. It also explains how to build a pipeline diagram using the tools that are selected from the periodic table.

There's no right or wrong answer in choosing the toolset, as long as it fits the need of the enterprise and people are trained in the usage of the tools. In this section, the native CI/CD tooling in the major clouds are discussed: Azure DevOps, AWS CodePipeline and CloudFormation, and Google Cloud Build.

Working with Azure DevOps

Azure DevOps enables teams to build and deploy applications; it caters for the full development cycle. Azure DevOps contains the following:

  • Boards: This is the planning tool in Azure DevOps and supports scheduling with Kanban and Scrum. Kanban works with cards, moving tasks through stages, while Scrum works with short sprints to accomplish tasks.
  • Repos: This is the repository in Azure DevOps for version control, based on Git or Team Foundation Version Control (TFVC). The term Team Foundation still refers back to the original name of Azure DevOps: Visual Studio Team Foundation Server (TFS).
  • Pipelines: This is the CI/CD functionality in Azure DevOps, which supports the build and releases of code to cloud environments. It integrates with Repos and can execute scheduled tasks from Boards.
  • Test Plans: This allows teams to configure test scripts, manually and automated.
  • Artifacts: This feature allows developers to share code packages for various sources and integrate these into pipelines or other CI/CD tooling. Artifacts supports Maven, Node Packet Manager (NPM, for Node.js and JSON), and NuGet packages.

The following screenshot shows the main menu of Azure DevOps with a project defined as Scrum that divides the work items in backlog items and sprints:

Figure 18.5 – Main menu of Azure DevOps

Figure 18.5 – Main menu of Azure DevOps

When a team starts in Azure DevOps, the first thing to do is to define a project and assign project members. Next, the project manager or product owner defines the development methodology. Depending on that choice, DevOps presents the possibilities to define work items, such as features, backlog items and tasks, and a board to plan the execution of these items. A work item or product can be a piece of code that may be deployed; in DevOps, this can be automated with Azure Pipelines, which after review actually deploys the code to the target cloud environment.

Developers can use Azure DevOps for AWS too, using the AWS Toolkit for Azure DevOps. The toolkit even accepts the use of AWS CloudFormation templates to provision and update AWS resources within Azure DevOps. In the Further reading section, a link to the documentation has been added.

Working with AWS CodePipeline

AWS CodePipeline is the CI/CD tool for AWS and offers development teams a tool to deploy applications and infrastructure resources. CodePipeline provides the following:

  • Workflow modeling: You could see this as the planning tool in CodePipeline. Workflow modeling defines the different stages for the release of code: build, test, and deploy. Teams can create tasks that need to be executed in the different stages.
  • Integrations: As any CI/CD tool, CodePipeline works with version control for the source code. With Integrations, developers can use various sources, such as GitHub, but also the native AWS service CodeCommit (the default in the main menu of CodePipeline), Amazon Elastic Container Registry (ECR), and Amazon S3. Provisioning and update code is done with AWS CloudFormation. AWS Integrations can do a lot more, such as continuous delivery to serverless applications with Serverless Application Model (SAM) and automating triggers with AWS Lambda functions to test whether application code has been deployed successfully.
  • Plugins: It looks like AWS mainly uses its own tools, but developers absolutely have freedom of tools. AWS Plugins allow the use of GitHub for version control and Jenkins for deployment, for example.

The following screenshot shows the main menu of CodePipeline:

Figure 18.6 – The main menu of AWS CodePipeline

Figure 18.6 – The main menu of AWS CodePipeline

As shown in the screenshot, creating a pipeline from the CodePipeline main menu will start by pulling code from a repository that sits in CodeCommit. The pipeline itself is built in CodeBuild and deployed in CodeDeploy.

Be aware that Artifacts is not the same as it is in Azure. In AWS, Artifacts uses a S3 artifacts bucket where CodePipeline stores the files to execute actions in the pipeline.

Working with Google Cloud Build

The CI/CD tool in GCP is Cloud Build. The main functions in Cloud Build are as follows:

  • Cloud Source Repositories: These are private Git repositories that are hosted on GCP. This is where the pipeline workflow starts: developers can store, access, and pull code from this repository using Cloud Build and Cloud Pub/Sub. Creating a repository can be done through the GCP UI portal or Google Cloud Shell with the gcloud source repos create command. After the creation of the repository, developers can start pushing code to it with the git add, git commit, and git push commands from the gcloud console.
  • Artifact Registry: This is basically the same service as Artifacts in Azure DevOps. It allows the creation and management of repositories that hold Maven and NPM packages. In GCP, Artifact Registry is also used to create repositories for Docker container images.
  • Cloud Build: This is the engine of the CI/CD functionality in GCP. In Cloud Build, developers define the pipelines. It imports source code from Cloud Source Repositories, but can also pull code from other sources, such as GitHub. Cloud Build tests and deploys the code to the targeted GCP infrastructure. Cloud Build integrates with a lot of different solutions – for example, with Jenkins and the open source tool Spinnaker for automated testing and continuous delivery. These solutions can also be used to work with the Google Kubernetes Engine (GKE) to enable CI/CD on container platforms running Kubernetes.

The following screenshot shows the menu of Cloud Build:

Figure 18.7 – Introduction screen to start with Cloud Build in the GCP console

Figure 18.7 – Introduction screen to start with Cloud Build in the GCP console

The main menu is very lean, as shown in the preceding screenshot. Only when a project is defined and started can developers start using Cloud Build to create the code repository. That service is available from the console, as shown in the following screenshot:

Figure 18.8 – Starting a repository in GCP's Cloud Source Repositories

Figure 18.8 – Starting a repository in GCP's Cloud Source Repositories

In this section, the native CI/CD toolsets of Azure, AWS, and GCP have been discussed. At the beginning of the section, it was mentioned that there are a lot of tools available to developers for developing, storing, testing, and deploying code. The main principles of CI/CD are the same, but there are differences in the way that these tools deploy code and especially how they test and validate code in an automated way. The tools discussed – Azure DevOps, AWS CodePipeline, and GCP Cloud Build – may cover a lot of functionality already, but typically, additional tools are required. In the Further reading section, some suggestions are provided for study material to help in choosing the right tools.


After completing this chapter, you should have a good understanding of the DevOps way of working and the use of CI/CD pipelines in cloud environments. Everything is code in cloud, from the application to the infrastructure and the configuration. Code needs to be stored in a central repository and brought under version control. That's where the CI/CD pipeline starts. Next, the DevOps team defines the phases of the pipeline, typically build, test, and deploy. Actions in these phases are automated as much as possible.

We discussed the push and pull principles in CI/CD pipelines using master and feature branches, describing the different methodologies to push and commit code to branches. If teams work consistently from one repository and with a unified way of working, they can deploy code to different clouds. Teams also need tooling: the last section provided an overview of the native CI/CD tooling in Azure, AWS, and GCP, working with Azure DevOps, AWS CodePipeline, and Cloud Build in GCP.

In the next chapter, the final concept for operations will be discussed: AIOps.


  1. Systems must meet the acceptance criteria before it's signed off as ready. In DevOps, a specific term is used for this sign off – what is that term?
  2. What is a commonly used framework for working in DevOps structures?
  3. Azure DevOps has a specific tool for agile planning, such as Kanban or Scrum. What is this tool called?
  4. Rate the following statement true or false: Artifact Registry in Google Cloud Build also supports Docker images.

Further reading

You can refer to the following links for more information on the topics covered in this chapter:

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.