Version Control

Continuous Integration requires that all code that’s needed to build the software be stored in a Version Control system.

A version control system must provide, at minimum, the following features:

1. Store current (latest) revisions of files and folders in any arbitrary structure and depth.

2. Storing these files and folders under a unified “repository” and not merely as disparate elements.

3. Store old (historic) revisions of files and folders — including those that have been subsequently deleted, renamed, moved, or otherwise modified.

4. Discrete and chronological versioning of all revisions — up to an including the current revision — of all these files and folders, so that it’s easy and unambiguous to track the history of any one file over time.

5. Ability to push (commit) changes to the code repository in a deterministic manner. (That is: a push should be accepted or rejected based on clear rules.)

6. Ability to query the code repository to detect any fresh changes.

7. Provide a command-line-interface (CLI) for the “push code”, “pull code”, and “query changes” features.

In addition, the following features are highly desirable:

8. Store multiple independent branches in the code repository, where branches can be created (forked), deleted, and rejoined (merged) with other branches.

9. Ability to resolve conflicts (which happen when two or more incompatible changes are made to the same file/folder).

10. Provide all these features in the CLI, without the need to resort to a graphical user interface (GUI). This facilitates automation.

11. Support or provide a GUI for users who need/prefer it. This facilitates widespread adoption.

In common practice, each member of a development team regularly commits their code to one (or more) shared code repositories. Each person may commit code several times during a typical workday, leading to dozens (or even scores) of CI builds running daily.

A version control system like Git provides all the features listed above and many others. Git can be used as a distributed version control system with no centralized repository — individual developers sharing code with each other in a peer-to-peer fashion. Git’s “patch” feature can be used to share changes with other team members using any existing, out-of-band mechanism, e.g. shared network folders, or even e-mail.

For the purpose of enabling CI, it’s much more common to have a centralized Git server to which all developers connect. This centralized Git server contains the definitive and canonical code repositories. All other team mambers are expected to push code to and pull code from this centralized Git server.

With a centralized Git server, it’s fairly common (bordering on universal) to use a PAAS provider ² instead of installing and maintaining a Git server of one’s own. With the ready availability of PAAS Git providers, including several that offer a generous “zero-price” tier — such as GitHub, GitLab, and Bitbucket — makes this an irresistible option.

In this book, we’ll use Github as our version control system.

Build Server and Agent

To automatically run the build, we need a computer to run the build. In reality, there are multiple processes that need to run:

1. A Build Server process to regularly monitor the version control system and detect any changes

2. A Build Agent process to run a build whenever there are changes

   1. There may be multiple Build Agent processes to either run builds concurrently, or to run them on different operating systems, or to build them with different sets of dependencies

Typically, the Build Server conscripts one Build Agent for each build that needs to run. The Build Agents are independent of (and therefore, unaware of the existence of) other Build Agents.

In this book, we’ll use the Build Server and Build Agents provided by GitHub Actions. We will use declarative programming to indicate which Build Agents we need and what should be installed on it. This declarative style is common in CI/CD systems — like GitHub Actions — which provide cloud-based Build Agents.

If the Build Agents are independent of each other, how do they share artifacts? This is where an Artifact Repository comes in.

Artifact Repository

To share build artifacts between Build Agents, an Artifact Repository is used. In principle, an Artifact Repository is a shared file system that each Build Agent can access. Advanced features provided by an Artifact Repository may include versioning of each build artifact, seamless back-up of artifacts for recoverability, and fine-grained read/write privileges (i.e. allowing specific Build Agents or other processes read-only or read-write access, as needed).

The Artifact Repository is similar to the Version Control system insomuch as both are used to store and version files and folders. They two could even share the same underlying implementation. The key difference is in what they are used to store. The Version Control system is used to store source files that are authored and managed directly by the developers crafting the software. The Artifact Repository, in contrast, stores files generated by the act of building the software. Many of these files are binaries — executable programs, libraries, and data files. However, other generated files are not binary: API and code documentation, test results, and even source files generated during the build process. Regardless of whether the files are binary (i.e. not meant for human eyes) or human-readable, storing them in the Artifact Repository ensures they are kept separate from the source files — the fountainhead of the software system.

We do not need an Artifact Repository in this chapter, because we do not have any artifacts to share between different builds. However, GitHub does provide a mechanism to store build artifacts using the same code repository that stores the source code.

Deployment Environment

After a successful CI build has run in a Build Agent, the build artifacts thus generated need to be deployed into a Deployment Environment. This allows these artifacts to be tested (mostly automated tests but also people) and released to end users (mostly people but also automated systems).

The deployment of build artifacts to one or more Deployment Environment is a key step in achieving Continuous Deployment and Continuous Delivery.

In this chapter, we’ll focus on the first phase: Continuous Integration. Continuous Deployment and Continuous Delivery — deploying the packaged software in an environment and ensuring that it is delivered to the end users — are out of scope of this book.

Create Github account

To create a CI pipeline using Github actions, we need a Github account. If you already have a Github account, great! You may skip this section.

If you don’t have a Github account, create one by visiting https://github.com. You do not need to pay anything for a free account, which is enough for our needs. (A free Github account is sufficient for many individual developers, as it allows unlimited public and private repositories and 2000 Github actions minutes per month.

All you need to create a Github account is a valid e-mail address. It’s strongly recommended that you set up two-factor authentication, which can be done in a variety of ways. See Github’s documentation for more details.

Verify Github account

Make sure you can log into your Github account. If you decide to use the SSH protocol to interact with Github, you’ll need to generate and add an SSH key. If you have other projects on Github and you regularly push and pull code to them, you probably don’t need to do much by way of verifying your Github account.

If you haven’t used your Github account in a while, you may want to fork a repo to verify your account is in pristine working condition. Go to https://github.com/saleem/tdd-book-code and use the “Fork” option to fork the repo. See Figure 13-2.

Figure 13-2. Fork a repository, like the one containing the code for this book, to verify that your Github account is working as expected.

Of course, you’ll use your hand-written (and highly cherished) code for the rest of this chapter — not the pre-fabricated and intellectually unsatisfying code you forked from this book’s Github site! The purpose of forking is to verify that your Github account is working correctly.

Push code repository to Github

Up until the end of the preceding chapter, we regularly committed our code to our local Git repository. Now is the time when wen push our code repository to Github. The conceptual difference between these two actions is shown in Figure 13-3.

Figure 13-3. Difference between committing code to a local repository and pushing code to a remote repository.

We first create a project in Github which will house all the code in our local code repo. To do this, we click the “New Repo” button. This starts a short (two screen long) workflow to create a new repository.

Figure 13-4 shows the first screen. This is where we enter the repository name. We use tdd-book-code for the name, which is the same name as our TDD Project Root folder. This makes things easier for us to remember. We also have the choice to make the repository private, which means non one else can see it; or leave it public, which means we can collaborate on it with others. Do not select any of the options under the “Initialize this repository with” heading. We already have a repository with several files in it.

Figure 13-4. First step of creating a new repository on Github.

The second screen shows the quick setup guide. We will use the instructions under the section …or push an existing repository from the command line, as shown in Figure 13-5.

Figure 13-5. Second step of creating a new repository on Github.

The command-line instructions in this section are already configured for your Github user name — you can simply copy and paste the three lines of code verbatim. Leave the browser screen as it’s shown in Figure 13-5 and use the commands in a shell window to push your code to Github.

Figure 13-6 shows the results of the three commands for my Github repo. Notice that the first two commands silently succeed. The third command — git push -u origin main — produces some output on the screen.

Figure 13-6. Pushing code from our local git repository to the Github repository.

After you’ve successfully pushed your code to Github, simply refresh the browser window that previously showed the commands (shown in Figure 13-5). The contents of that browser page should change, showing you the code you just pushed, as seen in Figure 13-7.

Figure 13-7. Code after it’s been pushed to the Github repository.

Voila: our code is ready to be spruced up with the awesomeness that’s continuous integration!

Prepare for CI build scripts

Our code has different folders with source code for the three languages. Here’s the complete folder structure under the TDD PROJECT ROOT folder.

tdd-book-code
├── go
│   ├── go.mod
│   ├── money_test.go
│   └── stocks
│       ├── bank.go
│       ├── money.go
│       └── portfolio.go
├── js
│   ├── bank.js
│   ├── money.js
│   ├── portfolio.js
│   └── test_money.js
└── py
    ├── bank.py
    ├── money.py
    ├── portfolio.py
    └── test_money.py

Our CI build scripts will be in a new folder, actually, a new subfolder in a new folder. It needs to be named .github/workflows. Pay attention to the . in the front! It’s imperative that this folder be created exactly as named.

To create this folder all at once, type the following command in the shell, from the TDD PROJECT ROOT

mkdir -p .github/workflows

This will create both the .github folder and the workflows folder underneath it, all at once.

Our CI scripts will be in YAML format. Our YAML scripts for Go, Javascript, and Python will follow a similar structure, which is shown in the following code fragment.

name: Name of script 
on:
  push:
    branches: [ main ] 
jobs:
  build:
    name: Build 
    strategy:
      matrix: 
...
        platform: [ubuntu-latest, macos-latest, windows-latest] 
    runs-on: ${{ matrix.platform }} 
    steps: 
    - name: Set up language-specific environment 
...
    - name: Check out code
      uses: actions/checkout@v2 
    - name: Test 
      run:...
      shell: bash 

A meaningful name for the entire script

The script runs on each push to the main branch

There is only one job in each script, named “Build”

Uses a matrix build strategy, allowing us to build on multiple OSes and language versions

We signal our intent to use the “latest” versions of Ubuntu, macOS, and Windows OSes in the matrix.platform variable

The previously defined matrix.platform variable is used here to run the build

There will be exactly three steps in our Build job for each CI script

First step: language-specific environment configuration will be done here

Second step: this is how we’ll check out code, regardless of language

Third step: language-specific commands will run the tests here

We specify that the bash shell should be used for the preceding commands in the third step

The script structure is dense but packs a lot of punch! Let’s analyze its various components.

The first line is the name of the script. The property name is used in many places in this script. Names can be anything we want; therefore, it’s best to name it something that will describe the purpose of the script well. We’ll name each script after the language for which it is intended.

Next we describe when the script should run. The on: {push: {branches: [ main ]}} section dictates that the script should run on every push to the main branch.³

Next we define our jobs section. There is exactly one job in each script: build. We chose “Build” as the name of this job. We choose a “matrix strategy” for our builds. A matrix strategy is a powerful feature provided by GitHub actions: it allows us to run the same build on multiple operating systems, language compilers, etc. This is immensely helpful to ensure that our code builds and runs on a variety of environments, not just the one we are currently using. If you have ever heard any rendition of the “it works on my machine” joke, you know how important this feature is!

Our matrix comprises two dimensions: operating systems and language compilers. We’ll choose the three popular families of operating systems for each language, viz Ubuntu, macOS, and Windows. The compiler dimension will vary for each language. The runs-on property ensures that our build will run on each of these three operating systems.

Figure 13-8 shows the general formulation of the matrix.

Figure 13-8. General Build Strategy Matrix.

The last section lists the steps in our Build process. Each CI script will have three steps, the first and last of which are language specific.

1. The first step will set up the build environment needed by that language.

2. The second step, checking out the code from the GitHub repository, is identical for all three build scripts. This step uses the checkout action provided by GitHub Actions.

3. The last step runs the tests for the specific language. This step will look familiar to us: it will include commands to run the tests for each language that we have used throughout this book.

With this overview of the structure of CI build script and YAML behind us, let’s get to the business of writing the specific build scripts for each of our three languages.