CI builds start by retrieving a fresh copy of the latest source code from the SCM repository. Scripted builds used for other purposes may build only those modules that have changed, but CI builds always start from scratch. Rebuilding everything eliminates the possibility of dependency changes sneaking in undetected.

Having used the word “always,” we now have to point out the exceptions to the rule. When using CI in large, complex systems, you may have to break the build into smaller pieces to reduce the amount of time required. Breaking down the build based on module relationships and dependencies ensures that your team builds and tests all code affected by a change.

For example, consider a financial system that contains modules for managing user account information and tracking the stock market, as in Figure 5-1. The two modules interact with a database that drives the system’s user interface. A system this small probably doesn’t warrant breaking the build into pieces, but it is useful for demonstrating how to apply CI to large, complex systems effectively.

Figure 5-1. Part of a financial system

To break the build into smaller pieces, start by looking for independent modules. The user management and market tracking modules are completely decoupled, so they are candidates for separate builds. Changes to the user management module don’t affect the market tracking module. Likewise, changes to the market tracking module don’t affect the user management module. The decoupled modules may be built separately; however, both modules may indirectly affect the user interface via the database, so you should build (and test) the user interface module as part of both builds, as in Figure 5-2.

Figure 5-2. Separate builds for decoupled modules

CI builds are end-to-end processes. End-to-end builds start from source code and include all the actions necessary to create the final software product. The definitions of “all the actions necessary” and “final software product” are arbitrary, and different development teams may apply different meanings to each. For example, the simplest end-to-end build consists of nothing more than compiling source code into an executable file, as shown in Figure 5-3.

Figure 5-3. Simple end-to-end build

While a simple end-to-end build implements a repeatable process for creating executables, it doesn’t incorporate all the steps typically performed prior to releasing a software product. CI builds are often more complex and may include:

Figure 5-4 shows a typical end-to-end build used in a CI process, including automatic deployment of the product. Be aware that automatic deployment is a tricky proposition. Updating a product while it is being tested or used can cause confusion and can result in decreased productivity, lost data, and a lack of confidence in the product.

Builds resulting in deployable software packages are sometimes referred to as push button builds because a single command creates the entire product from scratch. Builds that are started automatically at preset times are called scheduled builds.

Figure 5-4. Typical end-to-end build

CI builds employ automated reporting mechanisms to ensure that developers, managers, and possibly even customers always know the state of the build. Report content depends on the complexity of the build; simple builds may report compiler errors, whereas more complex builds may include error messages, unit and integration test results, code coverage statistics, adherence to standards, and statistics generated by performance testing.

An important part of the process is getting reports to developers quickly. Broken builds disrupt the normal flow of development because developers working on other parts of the system can’t use the latest version of source code from SCM. The sooner the developers who are responsible for the bad code know the build is broken, the sooner they can fix it and restore the flow.

Results can be reported in a number of different ways. Some of the more common methods are:

Using a dedicated build server prevents interference from other activities. Other activities may slow down or even halt a build due to limited resources or access issues.

The primary function of a build server is to build and test the project. Build servers doing double duty, such as hosting the test database for a web application or hosting the web server itself, will require more time to build the software. Small projects with few developers may be able to utilize a less powerful machine or allow the machine to perform other tasks without sacrificing build speed. However, large projects with many developers and frequent code check-ins will benefit from having a powerful, dedicated build server.

Dedicated build servers allow you to build and test the product in a well-known environment. Users performing other tasks on a shared build server may load different versions of libraries and supporting software or make changes to the runtime or test environment resulting in build failures.

Build servers should not be used for prototyping, experimenting with new technologies, or evaluating new software packages. Each of these activities can introduce new versions of libraries and make system-level configuration changes that are incompatible with the build environment.

One drawback of using a dedicated build server is the expense of obtaining an additional computer, one that will not be used by developers for writing code. In addition, you may require a fairly powerful machine to deal with large, complex builds.

Setting up a dedicated build server requires an initial investment, but the payoff is well worth the expense. One of the main drivers behind CI is immediate feedback, so the more powerful the machine, the better. Faster builds mean faster feedback to developers. If at all possible, consider using a powerful, dedicated build server to implement CI.

Because CI is based on rebuilding the system whenever the codebase changes, it requires the capability to detect such changes automatically. CI servers typically include the capability to interface with a wide variety of SCM systems, allowing them to monitor the repository and take action when the codebase changes, or to defer action when no changes are detected.

Once a CI server has detected a change in the codebase, it starts an automated build process by invoking a build script. Typically, CI servers incorporate a mechanism to ensure that all pending changes to the SCM repository are completed before invoking the build script. Once it has started the build script, the CI server monitors the process and waits for it to finish.

The final responsibility of a CI server is to report the build results to the development team and anyone else who needs to know the build status. CI servers commonly provide error notification via email and web-based access to the latest build results.

A strict interpretation of CI does not include scheduled builds, but such builds can be useful in managing large, complex products. Using a combination of builds triggered by code changes and scheduled builds, developers can create a system that performs fast-running unit and integration tests for all code changes and longer tests in the background. Most CI servers allow you to schedule builds for a particular time of day—a nightly build, for example—or on a periodic basis.

An effective CI process requires more than a collection of tools and scripts; it requires acceptance by the development team members. CI works by integrating small pieces of functionality at regular intervals. Developers must adhere to a few rules to make the CI process effective:

An effective CI implementation requires users to check changes into SCM frequently. Frequent check-ins tend to limit both the size and scope of changes, which in turn limit the possible sources of error when a test fails. CI helps speed debugging by limiting the source of a test failure to recent changes.

Feedback for changes is not limited to test failures; it includes successful integrations as well. CI provides both positive and negative feedback about recent changes, allowing developers to see the effects of those changes quickly. The negative or positive quality of those effects can then drive additional changes or a determination that no more work is required.

Traditional software development life cycles include an integration phase sandwiched between the end of development and the release of the product. CI minimizes—and sometimes eliminates completely—the integration phase. By spreading the integration effort out over the entire development cycle, CI allows developers to integrate smaller changes. Integrating smaller changes frequently doesn’t simply spread the same effort out over a longer period; it actually reduces the integration effort. Because the changes are small and limited to specific sections of the source code, isolation of errors is quicker and debugging is easier.

An undetected defect in one module can propagate through the software system via dependencies with other modules. When the defect is finally discovered, the fix affects not only the defective module, but also all modules dependent on it. First, the defect propagates through the system, and then the fix propagates through the system.

Continuous integration minimizes the propagation of defects through the system by detecting the problem early (via thorough automated unit and integration testing), before it has a chance to propagate.

A significant part of any development effort involves modifying existing code to account for changing requirements and refactoring modules to improve the overall software implementation. Developers can modify and refactor code with confidence because CI is providing a safety net. The unit and integration tests that run as part of the build give immediate feedback to developers, who can roll back changes if something causes the build to break.

Because CI builds the project whenever the code changes, it keeps the latest build in sync with the code repository. A CI implementation that reads from an SCM repository and deploys the build to a test environment ensures that developers, testers, and users are all working with the latest version of software.

Because CI reports the state of every build, it provides up-to-date visibility into the current state of the project. The snapshot may include:

As discussed earlier in the section Dedicated Build Servers,” CI works best with a powerful computer dedicated to performing builds and all the associated tasks. Although CI can be implemented without a dedicated build server, the benefits generally outweigh the costs of extra hardware, even when the machine is a very powerful (i.e., expensive) one. Even so, there are at least two ways to get started with CI on something less than the biggest, fastest machine on the market: piggyback on an existing server or use a less powerful machine that no one wants.

Piggybacking on an existing server or shared computer, such as a test machine or an underutilized web server, is a quick way to implement CI. Such machines are often already running the typical deployment environment, so you only have to load the CI-related software. When following this approach, the more stable the environment, the better. A server whose configuration is changed often will lead to inconsistent build results.

Implementing CI on a less powerful machine—perhaps one left over from the last time the development machines were upgraded—provides the benefits of a dedicated machine at the cost of fast builds. Whether a less powerful machine is adequate for a particular project will depend on a number of factors, such as the number of developers and the frequency of check-ins (both directly affect the frequency of builds), the size of the build, and the number and type of automated tests. Less powerful machines may be enough for small development efforts. Larger, more complex projects can use this approach to demonstrate the benefits of CI in an effort to lobby management for the funds to obtain a more powerful machine.

Both of these approaches can provide a starting point for implementing CI, but there is no substitute for a powerful, dedicated build server. In the end, the justification is simply that the time saved during integration far outweighs the upfront costs of adequate hardware.

Though it is possible to implement CI from scratch with custom scripts and applications, the easiest approach is to use a third-party CI server application, many of which are available as open source software. Implementing CI with open source packages can reduce the initial cost to almost nothing, although there are maintenance and training costs to consider. Other software packages required for implementing CI—such as an SCM system, a build scripting language, and a unit test framework—are also available as open source software.

As with most software, there is a learning curve associated with CI software; however, CI software packages are usually independent of the project software, so experience gained on one project transfers easily to new projects. In addition, you can implement CI in a stepwise fashion, starting simply and enhancing the process as developers learn more about the package.

In addition to the initial setup tasks, CI requires ongoing maintenance of the build scripts, although the amount of ongoing maintenance is fairly small. This type of maintenance typically involves creating build scripts for new modules and adding tasks to the build process.

Creating the build script for a new module can be time-consuming, since you must take care to ensure that all dependencies are accounted for and all required libraries are available. As noted earlier, one way to simplify the task is to use an IDE to create a build script that can be invoked by the CI process.

Spending time upfront to organize the project structure (in conjunction with the SCM repository structure) and create extensible build scripts often reduces work necessary for both the addition of tasks and the addition of modules. You can add tasks to the build script such that they apply to all modules or a specific subset of them. And good build scripts will automatically incorporate repository changes, so adding and deleting modules is as simple as adding or removing the module from the SCM repository.

Continuous integration is most effective when used with a robust set of unit tests; after all, without testing, CI degrades to simply compilation of the source code. Legacy code often lacks unit tests (at least automated unit tests), so retrofitting CI onto an existing project can be a daunting task.

The best approach to implementing CI in a legacy system is to begin by creating build scripts for the existing modules, including any unit tests that exist. Unit tests are then added to legacy modules as the modules are modified or refactored, expanding unit test coverage to the entire legacy codebase over time. New modules are required to have unit tests before being allowed into the build. Chapter 4 covers the implementation of unit tests for legacy code in detail.