Appendix A. CD systems

In this appendix

  • reference list of CD system features discussed in this book
  • overview of common CD systems and features they provide

In this appendix, we’ll be taking a look back at all the CD system features that have been discussed in this book, and looking at which features are available across several common CD systems.

This book has used GitHub Actions to demonstrate many CD system features. However, when choosing which CD system is best for your needs, it is important to consider and weigh all of your options (including building your own CD system if needed—see chapter 11). We’ll be looking at several CD systems:

  • Argo Workflows

  • CircleCI

  • GitHub Actions

  • Google Cloud Build

  • Jenkins Pipeline

  • Tekton

This list is not exhaustive. Use the reference list of CD system features in the “Feature list” section when evaluating CD systems not covered here.

CD System features by chapter

In the 13 chapters of this book, you’ve seen a whirlwind of features that CD systems can support to make it easy to define powerful, reusable tasks and pipelines. CD system features are highlighted in these chapters:

  • Chapter 2—Gives an overview of the basic elements of a CD pipeline, and defines the terminology used throughout this book to refer to pipeline elements (including events, triggers, webhooks, tasks, and pipelines)

  • Chapter 3—Teaches about config as code and how important it is to treat CD configuration as code by storing it in version control

  • Chapter 7—Shows all the places that bugs can sneak into our code and demonstrates the usefulness of periodic triggering

  • Chapter 9—Shows the importance of building software artifacts safely and discusses the features we need to make that happen

  • Chapter 12—Focuses the config-as-code lens on scripts in particular, reinforcing that all CD configuration should be treated as code, and that CD pipelines and tasks need to be reusable

  • Chapter 13—Shows all of the pipeline-level features we need in order to be able to build effective pipelines, and why we need the features

When evaluating CD systems, these are features to keep an eye out for. If they aren’t present, it doesn’t mean you can’t use the system; it just means there might be more work for you to do to get the same functionality.

Feature list

This appendix will examine the following features across common CD systems:

Triggering features (see chapters 2 and 7)

Event-based triggering—Pipelines will need to be triggered and executed in response to various events—for example, when pull requests are opened or merged.

Periodic triggering—Regular scheduled execution of pipelines can help reveal issues such as flaky tests and can be used to support release strategies such as nightly releasing.

Safe and reliable build-process features (see chapters 3 and 9)

Config as code (aka build as code)—Storing your CD pipelines in version control is crucial; the same best practices you apply to your business logic should be applied to all the data that makes up your software.

Run as a service—Without a consistent service executing your CD pipelines, it is impossible to ensure that your software is built consistently, and auditing is nearly impossible.

Ephemeral environments—Starting from a clean environment every time you build an artifact ensures that you get the same results every time. This is often accomplished using one-time-use VMs and containers for execution.

Units of execution (see chapter 2)

Pipelines—Graph-based orchestrations of tasks with control-flow features specific to CD use cases.

Tasks—Units of logic that are ideally highly cohesive, loosely coupled, well factored, and reusable.

Task and pipeline features (see chapters 12 and 13)

Inputs and outputs—To reuse tasks and pipelines, it must be possible to provide them with inputs that support customizing their behavior. Outputs make it possible to plug tasks and pipelines together so their behavior can vary based on the behavior of other tasks and pipelines.

Conditional execution—Being able to control at runtime which parts of a pipeline execute makes it possible to write very flexible pipelines that can be more easily reused.

Finally behavior—CD pipelines often contain behavior that must happen, even when other parts of the pipeline fail. Cleaning up environments and sending notifications such as instant messages are examples.

Parallel execution—Running tasks in a pipeline that have no dependencies on one another can reduce overall pipeline execution time.

Matrix-based execution—An extended version of parallel execution, this allows tasks to be automatically run multiple times in parallel for combinations of input, supporting complex testing use cases such as sharding.

Pipelines using pipelines—In addition to making tasks reusable, it is often useful to define reusable pipelines that combine tasks, linking their inputs and outputs, so these pipelines can be reused in multiple scenarios.

Argo Workflows

Argo Workflows (https://argoproj.github.io/workflows/) is one of several projects under the Argo banner. Argo Workflows is an open source Kubernetes-native workflow engine that was donated to the Cloud Native Computing Foundation (CNCF) by Intuit (and originally created by the company Applatix). It supports CD use cases but was created to solve workflow automation use cases more broadly as well.

Triggering features

Triggering is supported through the companion project Argo Events, which supports multiple event sources (including version control systems such as GitHub, and cloud integrations).

Safe and reliable build-process features

You must host and run Argo yourself; it can be used to provide a CD service within an organization.

Containers are the basic unit of execution, providing ephemeral environments.

Units of execution

Workflows (corresponding to pipelines in this book) execute Workflow templates.

Workflow templates (reusable workflows) contain steps and/or directed acyclic graphs.

Task and pipeline features

Workflow templates are reusable, and they take input parameters and produce results via artifacts. Steps can declare inputs and outputs (which can be parameters or artifacts).

Conditional execution is supported with a when syntax.

Matrix-style execution is supported via the loops feature.

Pipelines-using-pipelines behavior can be achieved by workflow templates calling other workflow templates.

CircleCI

CircleCI (https://circleci.com/docs/2.0/concepts/) is a CD system provided by the company of the same name, built around the idea of using the repo as the source of truth for CD configuration (config as code).

Triggering features

Integrates directly with version control systems and can trigger execution on events from those systems, including GitHub and Bitbucket, and workflows can be scheduled to execute periodically.

Safe and reliable build-process features

CircleCI was built around the idea of config as code and expects to find CD configuration for your project in a folder called .circleci in the root of your repo.

It can be used via the public hosted service or can be run as a server in a self-hosted mode.

Build environments (called executors) are ephemeral and can be backed by VMs or containers; it is possible to reuse VMs across jobs if desired (which would mean the execution environment is intentionally not ephemeral); containers are never reused.

Units of execution

Pipelines (corresponding to pipelines in this book) contain workflows as well as triggering information.

Workflows (also roughly corresponding to pipelines in this book) orchestrate jobs.

Jobs (corresponding to tasks in this book) contain sequential steps.

Commands can be reused within steps in jobs.

Orbs (also roughly corresponding to tasks in this book but also going a bit beyond it) define reusable jobs, commands, and executors.

Task and pipeline features

Parameters can be declared by jobs, commands, and executors. Using outputs between jobs is accomplished by persisting data to workspaces or by using the BASH_ENV environment variable.

Steps within a job can be executed conditionally using the when and unless keywords.

Finally behavior at the step level is supported by specifying the condition always.

Jobs within a workflow can run concurrently or sequentially. To run jobs sequentially, use the requires keyword to declare dependencies between jobs. Steps in a job can be made to run in parallel with the parallelism keyword.

The matrix keyword allows for jobs to be executed multiple times, once for each unique combination of the matrixed values.

GitHub Actions

GitHub Actions (https://docs.github.com/en/actions) is a CD system that is built into GitHub (see appendix B for more on version control systems). GitHub Actions has been used to demonstrate many of the scenarios in this book because you can easily create your own GitHub repositories and configure your own GitHub Actions to try them out, with no cost to you. It supports the features in this book as part of its extensive functionality.

Triggering features

Triggering on many different activity types for each repository, from scheduled cron events, to pull request updates, to interaction with GitHub issues.

Safe and reliable build-process features

It not only supports using a config-as code approach, but that is the only way to set up GitHub workflows. The definitions of these workflows live in the repository that triggers them, and it is possible to refer to workflows and actions that live in other repositories.

When using public GitHub, the CD system is hosted and run by GitHub itself. If you use GitHub Enterprise in self-hosted mode, you are responsible for configuring and running the platform that executes the actions.

Each GitHub Actions job is run in an ephemeral environment that is created to run the job and is torn down afterward. The job can be run as an entire VM, or a container within a VM.

Units of execution

Workflows (corresponding to pipelines in this book), orchestrate jobs.

Jobs (corresponding to tasks in this book) contain sequentially executed steps.

Actions (also corresponding to tasks) are reusable jobs.

Task and pipeline features

Jobs within workflows can declare and emit outputs that can be used by other jobs within the same workflow as inputs. Reusable workflows can define inputs and outputs.

Jobs within GitHub workflows can use the if statement to define conditions under which they will execute.

Finally behavior is also supported using this syntax, by specifying if always() as a condition for a job to indicate it should always run.

By default, all jobs in a workflow will execute in parallel, unless the needs syntax is used to indicate that one job should run after another job.

The matrix keyword allows for jobs to be executed multiple times, once for each unique combination of the matrixed values.

Workflows can be defined as reusable workflows that can be used by other workflows.

Google Cloud Build

Google Cloud Build (GCB; https://cloud.google.com/build) is the CD platform provided by Google as part of Google’s cloud offering. Initially created as a way to build container images, it quickly expanded into being a generalized tool for CD.

Triggering features

Triggered execution is supported based on events from integrated version control systems including GitHub, GitLab, and Bitbucket, as well as webhook triggering via GCP’s Pub/Sub and scheduled periodic triggering.

Safe and reliable build-process features

GCB triggering can be configured to read the build definition from the triggering repository, supporting config as code.

GCB is provided as a service hosted and run by Google as part of Google Cloud Platform (GCP).

Steps in GCB builds are executed as ephemeral containers that are run on VMs.

Units of execution

Builds are made up of steps. Builds correspond to the tasks concept in this book, and somewhat to the pipelines concept as well (since steps can be executed as graphs, with steps being executed sequentially or in parallel).

Each step is executed as a container.

Task and pipeline features

Builds can use inputs provided at runtime via the user-defined substitutions feature.

By default, steps in a build execute sequentially; the waitFor keyword can be used to explicitly declare the order in which the steps are meant to complete and can be used to create graphs where some steps are executed in parallel.

Jenkins Pipeline

Jenkins (https://www.jenkins.io/doc/book/pipeline/) is one of the most well known and probably the most ubiquitous CD systems. Jenkins is open source and via the huge ecosystem of plugins, can be made to do pretty much anything. In this section, we’ll assume Jenkins is being used with the Jenkins Pipeline suite of plugins, which focus on supporting CD pipelines in Jenkins.

Triggering features

Jenkins Pipeline can be configured to trigger on a variety of events, including the pollSCM trigger, which can be used to poll various version control systems (such as GitHub), and cron triggers, which allow you to schedule execution.

Safe and reliable build-process features

Jenkins pipelines are defined in Jenkinsfiles, which can be stored in version control and read directly by Jenkins, enabling config as code for the pipeline definitions.

You must host and run Jenkins yourself. It can be used to provide a CD service within an organization.

Various kinds of execution environments are supported, including containers that can be used as ephemeral environments.

Units of execution

Pipelines (corresponding to pipelines in this book) are made of stages.

Stages (corresponding to tasks in this book), are made of steps.

Task and pipeline features

Pipelines can use the keyword parameters to declare inputs they require.

The when directive can be used to execute stages conditionally.

Finally behavior is supported by the post section of the pipeline, which indicates steps to run after the completion of the rest of the stages (even if those stages fail).

Stages in a pipeline can be made to run in parallel by declaring them within nested parallel blocks.

The matrix keyword allows for stages to be executed multiple times, once for each unique combination of the declared axes.

Pipelines can use other pipelines that are declared in Jenkins shared libraries.

Pipelines can be either declarative or completely scripted using the programming language Groovy. Scripted pipelines have complete flexibility in supported conditional execution and finally behavior (via try/catch).

Tekton

Tekton (https://tekton.dev/ and https://github.com/tektoncd) is an open source Kubernetes-based CD system that was donated to the Continuous Delivery Foundation (CDF) by Google. Its mission is to define a conforming standard that can be supported by many CD systems. The features described in this book are primarily supported by the Tekton Pipelines and Tekton Triggers projects. As the newest of the CD systems listed in this appendix, some of these features are brand-new or still in the proposal stage.

Tekton is the CD system I co-created!

Triggering features

Triggering on any arbitrary event is enabled via the Tekton Triggers project, which supports any source of events and has built-in support for GitHub, GitLab, and Bitbucket triggering features.

Safe and reliable build-process features

It can be run as a service inside of a Kubernetes cluster, and used directly or used as a platform on which to build another CD service.

The basic unit of execution in Tekton is a container, so execution environments are completely ephemeral.

Units of execution

Pipelines (corresponding to pipelines in this book) orchestrate tasks.

Tasks (corresponding to tasks in this book) sequentially execute steps.

Each step is executed as a container.

Task and pipeline features

Pipelines and tasks can both define inputs (parameters) and outputs (results).

Conditional execution of tasks within a pipeline is supported via the when syntax.

Finally behavior in pipelines is supported by a finally section in the pipeline that defines tasks that must always execute.

By default, any tasks in a pipeline will execute in parallel, unless a dependency has been expressed between them. Dependencies are expressed by a task declaring that it needs a result from another task (and so it must run after that task) or by using the runAfter keyword to express ordering.

The matrix keyword within a pipeline can be used to execute a task for all unique combinations of the specified array values in the matrix declaration.

At the time of writing, work is in progress to support these features.

Out-of-the-box support for config as code by referencing pipelines and tasks in version control (and other locations, such as in OCI registries) directly.

Pipelines reusing other pipelines: in the same way that pipelines orchestrate tasks, they could also refer to other pipelines.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.138.134.107