Chapter 18
Working with A Build Server
Many projects use a build server such as Hudson, Jenkin or Cruise Control to carry out regular automated builds with unit tests and integration tests. Git can naturally serve as a source for software to be built. It is interesting when information about successful builds also flow into the Git repository. If a repository knows which version was last successfully built and tested, the developer can see with a simple diff command how its local development differs from this version. Moreover, it will be possible to construct a branch with a history of successfully tested versions. This is very useful if you later want to fix a tricky error that was not caught by the tests.
This workflow shows how to integrate Git such that
- the current version will be built and tested regularly,
- the developer at any time will be able to compare their workspace with the last successfully tested version, and
- a history will be built on successfully tested versions to assist in troubleshooting.
Overview
The brain of this workflow is the build server (see Figure 18.1). The build server is configured such that it brings the latest commits from the master branch of the central repository in the build repository regularly. In the build server, the sources are built and tested.
There will be two branches in the build repository. The last-build branch simply refers to the version in the master branch that was last built successfully. The build-history branch contains all successfully built versions of the software.
The last-build branch is transferred to the build server after each successful build in the central repository. With a diff command against the last-build branch, every developer can always show which changes have not yet been tested successfully.
In the event of a hard to find error, the developer can get the build-history branch from the build server to his/her local repository and check in which successfully built version the error first appeared.
Requirements
Central repository: The project uses a central repository that defines the current status of the software.
Continuous integration: The development for the next release is regularly integrated into a common branch (the master branch). This integration will not happen only at the end of development, but immediately after a change is completed.
Build server: The project uses a build server to do automated builds and tests.
Test suite: There must be a suite of unit tests and or integration tests that can be started with a script.
Workflow “Working with A Build Server”
A build server builds and test the current version of the software regularly. As a result, we get a history of successfully built versions. In addition, the last successful build is marked in the central repository.
Figure 18.1: Workflow overview
Process and Implementation
Preparing the build server
A separate repository must be set up for the build server. The easiest way to create a build repository is by using the well known clone command. Using clone, however, would clone the entire central repository with all the branches. As a matter of fact, for the build server we only need the master branch and perform an init and a fetch commands on it.
Step 1: Create an empty build repository
First, create a new empty repository.
> mkdir buildrepo
> cd buildrepo
> git init
Step 2: Get the master branch from the central repository
With the remote add command we link the build repository with the central repository. To prevent fetching all the branches with the fetch or pull command, we specify a branch explicitly with the -t parameter.
> git remote add -t master origin <central repo>
The parameters used in the command are
-t master: Only the master branch is automatically transferred at a future fetch, pull and push command.
origin: The name of the newly added remote. We chose the same remote name on purpose so that the clone command would also use this name.
<centrl repo>: The URL to the central repository.
So far, no commits have been transferred from the central repository to the build repository. So, first we will do a fetch to transfer the data.
> git fetch
Step 3: Create a build-history branch
As the last step, we need to create a new branch called build-history that will contain a history of successful builds. It is recommended that the branch start at the first commit of the origin/master branch.
If we were to start the build-history branch at the current commit of the origin/master branch, existing commits on the origin/master branch would be included in the build history.
Unfortunately, there is no simple Git command to find the first commit of a branch. The best thing we can do is output the log and look for the last entry.
> git log --oneline --first-parent origin/master| tail -1
3a05e26 init
The parameters used in the command are as follows.
--oneline: Prints only one line of the commit log.
--first-parent: Prints only the first parents of the commits. This will lead to less processing and thus a faster result.
origin/master: The first commit should come from the master branch on the central repository.
| tail -1: Prints only the last line of the log output.
After the first commit is found, the build-history branch can be created and activated. We can also specify the start commit with the checkout command.
> git checkout -b build-history 3a05e26
The -b option creates a new branch and activates it in the workspace.
Git on the Build Server
The following steps describe how to work with Git on the build server. Typically they would be implemented in a script and integrated into the respective build server infrastructure.
The build server always works on the build-history branch.
Step 1: Get the changes from the central repository
The latest commits are fetched from the master branch on the central repository.
> git fetch
Step 2: Check if a build is required
You need to check if there are fresh commits by comparing the current build-history branch with the origin/master branch using the diff command. If no difference is detected, the build must not even be started and the process should be terminated.
> git diff --shortstat --exit-code origin/master
1 files changed, 68 insertions(+), 144 deletions(-)
The parameters used in the command are
--shortstat: With this option, not all changes will be displayed in detail. Instead, only a brief statistics on the number of changed files will be displayed.
--exit-code: This option ensures that the command returns 1 if there are differences (otherwise 0). This way, you can evaluate the result easily.
origin/master: The difference is to be checked against the master branch on the central repository.
Step 3: Clean up the workspace
If a previous build has failed, the local workspace will contain the merge results of this broken build. You should reset the workspace just to be safe.
> git reset --hard HEAD
You can delete all unversioned files with the clean command.
> git clean --force
The --force option forces the execution of the clean command.
Step 4: Bring changes to the local build-history branch
The new commits on the master branch must be included in the build-history branch. Since there is no development on the build-history branch, a merge command would always cause a fast forward merge, i.e., the build-history branch would simply refer to the current commit on the master branch.
Since we want to use the first-parent history on the build-history branch to retrieve the successful builds, we need to run merge with the --no-ff option.
> git merge --no-ff --no-commit origin/master
The parameters used in the merge command are
--no-ff: Do no allow fast-forward merges.
--no-commit: With this option, the merge is indeed performed on the workspace, but initially it is not committed. Only after successful building and running of the tests, a commit will be created.
Step 5: Do a build
Now the build server is used to build the current workspace and run tests. Git is not involved in this case. In the event of an error, the workflow is aborted and the build server informs the developers by email.
Step 6: Do a commit
If the build is successful, the prepared commit will be executed. The commit message should contain the build number assigned by the build server.
> git commit -m "build <build-nummer>"
If the build or tests failed, we can simply cancel at this point. In the next cycle, the workspace will be reset again (see Step 3).
Step 7: Mark the last successful build
The last-build branch on the central repository should always point to the last successfully built commit on the origin/master branch.
For this purpose, a local last-build branch is first created in the build repository or set to the correct commit. It is a little tricky to set this branch to the correct commit on the origin/master branch instead of on the merge commit on the current build-history branch.
You can see the various parents of a merge commit in Git marked by the ^- notation: ^1 stands for the parent commit of the current branch, and ^2 for the parent commit of the added branch.
Take a look at commit Z In Figure 18.2. Its first parent is commit Y from the build-history branch, and its second parent is commit D from the origin/master branch.
Figure 18.2: The build-history branch and the master branch
The following branch command creates a new last-build branch or modifies the existing branch so that it points to the commit from the origin/master branch:
> git branch --force last-build HEAD^2
The --force ensures that the branch command always creates a new last-build branch, even if one already exists.
The parameter HEADˆ2 references the built commit on the origin/master branch.
After the branch points locally to the right commit, it must still be transmitted to the central repository. For this, use the push command.
> git push --force origin last-build:last-build
The parameters are as follows
--force: This option ensures that the local last-build branch is always displaced in the central repository with the new commit, even if the commit is not a successor of the last Last-Build commit.
origin: This parameter references the central repository.
last-build:last-build: This parameter indicates that the local last-build branch is transferred to the central last-build branch.
Comparing the Local Developer Version with the Last Successful Build
Every time there is a content error in a developer repository after a merge with the central repository, it is useful to compare the changes with respect to the last successful build.
The following section describes how you can compare your local version status with the last successfully built version on the build server.
Step 1: Check the central commit
First you should check if there are commits from other developers that are not yet successfully built in the central repository, because the error could come from someone else.
For this purpose, you can use the log command to determine whether there are commits in the central origin/master branch that are not yet included in the origin/last-build branch. In the example in Figure 18.3, you would find commit C.
Figure 18.3: Using the last-build branch
> git log origin/last-build..origin/master
You can use the diff command to compare the changes.
> git diff origin/last-build origin/master
Step 2: Review the local commits
Next, you can check the changes in your own version compared to the last successful build.
> git diff origin/last-build
In the example in Figure 18.3, this would compare the contents of commit F with the contents of commit B. All changes in commits C, D, E, and F will be shown.
Troubleshooting with the Build History
In most cases, it is easy to find the location in the code that is causing an error. Often it is enough to read the description of the error in order to know at what point something must have gone wrong. Occasionally, however, errors creep in, which are difficult to find. Then it can be very helpful to know exactly when the error first appeared in the software. Since you can restore older versions of the software quickly with Git, it is possible to systematically isolate the error. You can start with an old version, in which the error is not occurring. Then you can get a later version and check whether the error occurs there. By repeating this in the version history you can find the commit in which the error first occurs. With any luck, the associated diff is small, and you have the error nailed very precisely.
However, the process can be quite a chore. Git therefore offers the bisect command that can help with this. The command finds the faulty commit by getting the “middle” commit enabled and tested in the affected area, and then proceed with the commit to the left or right of the last commit.
This method works most efficiently when it only has to consider versions that were previously built and tested successfully at least once. Otherwise, you would waste a lot of time with old versions that cannot be built due to serious errors.
Here, our build history comes into play, as it contains only commits that have been built successfully, and in which all unit tests have been successfully executed.
The build history, however, is not included in the local repository, nor is it present in the central repository of this branch. Using the fetch command, it is possible to import commits from other repositories.
Step 1: Link the build repository
To access the build repository, we create a remote named “build” in the developer repository.
> git remote add build <build-repo-url>
Step 2: Transfer the build history
Fetch the build history to the local developer repository using the fetch command.
> git fetch build
Step 3: Create a local branch
It is useful to create a local branch that is based on the build-history branch.
> git checkout -b history build/build-history
Step 4: Do a bisect
Define a suitable initial “good” commit and start debugging with the bisect command.
> git bisect start HEAD <good-commit>
Git will now select a “middle” commit for testing. We perform our test and depending on the results define the current commit as “good” or “bad.”
> git bisect good
or
> git bisect bad
Figure 18.4: Troubleshooting with bisect
Since Git looks at merge commits at both parents, it may happen that the chosen commit does not come from our build-history branch. Figure 18.4 shows a typical commit history. If the algorithm that Git employs checks the commits between commit X and commit Z, then commits B to D can be selected from the master branch.
Step 5: interpret the results
After a successful bisection process the faulty commit is found in the build history. Behind every commit in the build history, however, several commits on the master branch can pop (see commit Y in Figure 18.4). To print the log messages of the possible faulty commits, a little acrobatics needs to be done again.
The second parent (^2) of each build commit always corresponds to a commit on the master branch. The first parent (^1) of a build commit corresponds to the previous build commit.
The following log command determines, on the basis of a “faulty” build commit, all commits on the master branch that could have led to the error:
> git log <bad-commit>^1^2..<bad-commit>^2
If commit Y in Figure 18.4 has been identified as defective, then the log command would show commits B and C as a possible cause of the error.
Step 6: Clean up
After the bisect command has been completed, all unnecessary branches, commits, and remotes can be removed from the developers repository:
> git bisect reset
> git checkout master
> git branch -D build-history
> git remote rm build
Why Not the Alternatives?
Why No Tags?
An alternative implementation of the build history would be to store successful builds rather than merge commits in a separate branch as tags. This would even make the process simpler as you do not have to prepare and perform merge commits. The following are points to consider:
- It would create a lot of tags, and the non-build tags would be hard to find.
- The more efficient troubleshooting with bisection only on commits that can be built and successfully tested would not be possible.
- The logical order of the builds (tags) would implicitly manifested only by the build number.
Also for the last-build branch we could use a tag instead of the branch. This would require the tag to be deleted and recreated after each successful build. This works in a local repository, but once a tag has been transferred to the central repository, there will be problems.
While it is still possible to delete and recreate a tag centrally, all clones will not get the updated tag by calling the pull command as it would ignore any existing tags.
Why not Put the Build History in the Central Repository?
In the implementation described, the build history will not be published in the central repository, but will only be visible in the build repository.
Why will it make no sense to store the build history in the central repository?
The main reason is that otherwise the normal commit history would be “defaced” through the many merge commits. Every successful build enforces a new merge commit, based on the origin/master branch and build-history branch. In the normal viewing of a project history, it is irrelevant which version was built successfully. The merge commits would leave the log output appear cluttered.
The worst thing of all is if there are multiple build servers instead of one. For example because both the master branch and the codefreeze branch would be built, there would be more build commits in the central repository.
The beauty of the lattice approach is that it is always possible to bring together the commits in the build history and the normal project commits in a repository. For this purpose, a fetch, both from the central repository and the build repository, is performed.