Chapter 8. Continuous Integration
Continuous integration (CI) is an essential practice that involves putting together code that has been created by different developers and ascertaining if the code components can compile and run together successfully. CI requires a robust automated testing framework, which we will discuss further in Chapter 20, and also provides the basis for ensuring code quality through both static and instrumented code scanning. Continuous integration often involves merging together code that has been written by different developers and is essential for code quality. The fundamental value of this practice is that integrating a small amount of code as early as possible can avoid much bigger issues later. It would be difficult to imagine an effective ALM that does not embrace integrating code frequently, although I will discuss a couple of situations where it is difficult or even impossible to achieve. When code cannot be integrated early and often, there is increased risk that must be identified and addressed. It is also important to understand that continuous integration relies upon many other practices, including continuous testing, effective build engineering, static and instrumented code analysis, and continuous deployment, discussed in Chapter 9.
8.1 Goals of Continuous Integration
The goal of continuous integration is to support parallel development while avoiding costly mistakes that often result in code defects. CI also provides better quality by identifying any potential collisions early in the process with merging code from two or more developers. Integrating code early and often makes it much easier to identify any potential problems during the merge process and, more importantly, remediate the code to include desired changes without defects.
8.2 Why Is Continuous Integration Important?
Continuous integration is important because it reduces risk by identifying merge conflicts right away, while the changes are still fresh in everyone’s mind. Without CI, developers would have a much more difficult time identifying potential code-merge problems and take much more time to fix them once identified.
However, the general rule is, whenever possible, to integrate early and often. When this is not feasible, then late-binding integration should be identified as a risk that must be addressed. We will discuss strategies for dealing with this situation. It is tempting to call this approach “continuous everything,” alluding to the fact that so many practices depend upon a well-designed continuous integration platform.
8.3 Where Do I Start?
Getting started with continuous integration requires that you have your build procedures completely automated, a task generally accomplished using a build scripting tool such as Apache Ant, Maven, or Make. You also need a continuous integration server such as Jenkins to start the build and record the results. Unit tests and code scanning should be part of the build process. Equally important is an automated procedure to deploy the code to a test environment and then start automated testing once the code has been deployed. We discuss build engineering in Chapter 6.
8.4 Principles in Continuous Integration
The basic principle behind continuous integration is that it is much easier to find and fix problems when code is integrated early and often. But much more can be achieved in continuous integration. Continuous integration provides a great opportunity to improve code quality while we have access to the code and we can so easily build not only the production baseline of the code, but also any variants to facilitate testing and quality assurance. This is analogous to a great automobile mechanic who, regardless of any specific complaints or repairs, also routinely checks one’s oil, tires, and a few other important dependencies every time the car is brought in. CI servers could be viewed as the software equivalent of putting your car up on the lift in the mechanic’s shop. So our principles should include taking a very broad view of code quality throughout the continuous integration process.
8.5 Challenges of Integration
The basic challenge of integration is that code developed on separate variants—often branches—tends to diverge. This means that two developers might be writing code that will have to run together using code baselines that are different from each other. The solution is to integrate code early and often, as in continuous integration, and also ensure that developers are coding from the same baseline. However, there are also times when it is necessary to keep code separate because the decision to include features has not yet been made. This is sometimes called “cherry picking,” although a more descriptive term is late-binding integration. Integrating code late in the process is challenging because it is much more difficult to find and fix problems in code that was written days or even weeks ago.
The more basic challenge comes when trying to get developers together to look at code conflicts discovered during the merge process. Obviously, this gets challenging when one developer commits his code on a Monday and then takes the rest of the week off to go camping, leaving his colleagues to try and deal with the merge difficulties. There are strategies for dealing with this situation, but the first approach is simply to commit changes to the repository on a frequent basis.
8.6 Commit Frequently
Committing code to the version control system frequently is an essential best practice. In continuous integration, it is common to trigger a build with each commit. However, when large teams are working together, this approach can be less than completely effective because the sheer volume of triggered builds can result in too much activity that is a burden for developers to monitor throughout the day. For example, with a team of ten developers committing code several times a day, you can end up with 40 or 50 builds to monitor. Many teams find that a nightly build is sufficient, especially in the beginning of the process.
However, when you get closer to release time, it is often common to have the CI server build on each commit so that problems are identified early enough without affecting the release date. This is a practical example of where you may need to have a little more muscle in your process as you get closer to the release date.
8.7 Rebase and Build Before Commit
Rebasing means that you update your local workspace with changes before you commit your latest changes back to the repository. In some version control systems, this means that you should pull the latest changes made by other developers before you commit your latest changes to the repository. This is important because while you were working in your own private sandbox, other people may have committed their changes to the version control system. If you commit your own changes without rebasing first, you may introduce changes that do not compile against the baseline, thereby affecting all of the other developers. To avoid this risk, you should proactively pull the latest changes to your sandbox, verify that your code builds, and then commit your changes, which I often refer to as publishing your changes back to the team. This procedure is known as rebasing and is closely related to another practice known as a preflight build, where you use the production build farm to test your build and ensure that it will not fail during the official build—this is often performed on a nightly basis. If you don’t rebase and problems result, you will likely get blamed for breaking the build for everyone else. There are other merge challenges as well.
8.8 Merge Nightmares
We have seen several commonly used version control systems that did not handle merges well. We have also seen code that was so complicated it was very challenging to address the problem with merge conflicts. The more code involved, the longer the time required to resolve any merge conflicts found. Sometimes the changes made are complicated, and perhaps they cannot actually exist together in the same codebase. We have seen situations where developers reviewed their changes together only to realize that they had each adversely impacted the other’s work. In these challenging situations, sometimes all they can do is drop the changes and start over again, this time with a deeper understanding of how their code is structured and interdependent. Complexity can be a factor, but size is often the more common challenge. In general, integrating smaller units is often much easier to accomplish than trying to manage a monolithic set of code changes, which may or may not be able to work together. Simplicity in this case comes directly from smaller units of integration.
8.9 Smaller Units of Integration
There is no doubt that dealing with a merge conflict of only five lines is much easier than a merge conflict with a hundred lines. Integrating smaller units is much simpler and often shows the value of the continuous integration process. The earlier that developers get feedback that they are touching the same code, the sooner they can collaborate and address any potential design and coding issues. The frequency of the integrations is essential for success.
8.10 Frequent Integration Is Better
Performing continuous integration more frequently is better for several reasons. The first, as we have discussed, is that dealing with a smaller set of code changes significantly reduces complexity. But there is much more to this than just the number of lines of code (LOC). By making continuous integration a daily activity, you introduce a cadence that can help you with testing, code scanning, and deployment. DevOps puts a strong value on feedback loops, and CI gives immediate feedback when your developers use this methodology on a daily basis. Your team also gets used to this practice and actually becomes quite good at using CI for daily benefit. For one thing, it is simply easier to find and deal with any issues found.
8.10.1 Easier to Find Issues
When your CI server is used daily, you get immediate feedback on whether the build succeeded or failed. But this is only the beginning. If you are practicing CI correctly, then you will also run code scans and unit tests, deploy the code to a test environment, and then run automated tests in a comprehensive practice that is becoming known as continuous testing, discussed further in Chapter 20. Finding issues is essential, but even more important is that it is much easier to fix problems once they are found.
8.10.2 Easier to Fix Problems
When you find a problem immediately, it is almost always easier to fix. When teams do not fully utilize continuous integration, they often do not find problems until later in the process, and consequently it can be much more difficult to understand and resolve these issues. We have seen many teams implement continuous integration but fail to realize its full benefits, usually because they fail to react quickly to broken builds.
8.10.3 Fix Broken Builds
When the CI server reports a broken build, it is essential that the problem be addressed in a timely manner, even if this requires that the team stops what they are doing and examines the problem. Ignoring broken builds is a symptom of much bigger problems and needs to be addressed as soon as possible. We have seen the same issues with ignoring failed unit tests, errors in the code scans, and other types of automated tests. The continuous integration process will only succeed if these issues are quickly recognized and addressed.
8.11 Code Reviews
Often overlooked, code review is a key area that is implemented within continuous integration. Many products perform code scans, reporting potential issues related to performance, failing to meet coding standards including security, and more. When the CI server reports that these problems exist, this is an excellent opportunity to take a DevOps approach and assemble a cross-functional team to review the code and recommend adjustments. We like to use the feedback from the CI process to foster better communication via working sessions that address any issues that are discovered. One common limitation that often undermines successful continuous integration is a lack of sufficient servers dedicated to the automated application build, package, and deployment.
8.12 Establishing a Build Farm
Continuous integration relies heavily upon having fast and reliable build servers. We usually refer to these machines as being a build farm. Ideally, you should have a build server and build agents to help manage the automation of the application build, package, and deployment. The build farm should be controlled and managed by a centralized build engineering team. Organizations that follow the itSMF ITIL v3 framework refer to this group as release and deployment management (RDM). The build farm is a key resource and should be protected from unauthorized changes. We do like to provide the ability for developers to run unofficial “production-like” builds, discussed in Section 8.13, on the build farm, but they should never be allowed to make changes to the build machines without working with the RDM team, whose members are responsible for ensuring that the build farm is working.
When resources are stretched thin, sometimes it is advisable to utilize virtualization and cloud-based resources.
8.12.1 Virtualization and Cloud Computing
We see many more companies embracing virtualization and cloud-based computing strategies. This has grown out of developers needing the speed and agility of fast builds and test environments to facilitate continuous integration and especially continuous testing. Some companies are embracing on premises (ON-PREM) hypervisors, which are used to provide on-demand virtual machines (VMs). With this strategy, developers can request a large virtual machine for a few hours to build, package, and deploy code in a half hour, when it used to take half a day. This approach can be combined with automated unit, functional, application programming interface (API), service virtualization, and performance testing, each of which will be discussed further in Chapter 20, “QA and Testing in the ALM.” We will also discuss cloud-based computing strategies, including related DevOps strategies, in Chapter 17.
Do be aware that public vendor-provided, cloud-based resources come with significant risk. Not only have some cloud-based providers suffered from reliability issues, but information security continues to be a major concern. We do not see large companies such as banks and trading firms being very comfortable with using cloud-based public vendors for machines that contain customer information, but we do see many developers grabbing a couple of VMs for early development work, including support for continuous integration and related testing.
8.13 Preflight Builds
Preflight builds enable the developer to run the build privately on his or her machine before turning the code over to the build engineering team to verify that the code will compile on the build platform. Preflight builds save a lot of time by identifying anomalies without the volleyball game of tossing the build over the net to operations—only to have them toss the build back when it fails. While getting operations involved early in the build is a good example of the DevOps best practice of “left-shift,” preflight builds are an example of what we have come to consider as “right-shift,” where operations provides resources to developers early in the lifecycle so that the quality of the work can be verified, thus saving everyone time and effort.
8.14 Establishing the Build and Deploy Framework
Developers have a tough job. Understanding user requirements, designing scalable architectures, and implementing comprehensive solutions has always been challenging work. It is common, almost expected even, that the work to implement a comprehensive solution will take longer than expected. Add to this the fact that requirements are often a “moving target” that are not even fully understood when first specified. It is no wonder that technology professionals are often scrambling to get all of the technology pieces working together—not to mention actually writing the complex code itself that we have come to expect from enterprisewide solutions. Very often, creating a comprehensive build and deployment framework just doesn’t rise to the top of the priority stack, and the result is builds are often not well understood and code that may build for one person may not even compile successfully for someone else—unfortunately, often the build engineer. There are many reasons for this problem.
Too often, developers do not fully understand their own build dependencies, especially when they are using powerful development frameworks that may abstract and effectively hide these complexities. Unfortunately, the result is that code may build successfully on one machine and actually yield different results on another machine. Consequently, it is essential to establish a consistent build framework, including all compile and runtime dependencies. In Java, this may mean that you need to establish the classpath and ensure that each developer uses the same environment settings. On the .Net framework, you may need to establish similar compile and runtime dependencies. We have seen many strong teams really struggle with this challenge. Some developers ask more senior colleagues to help them set up their development environments and then have no idea how to deal with these configuration details themselves. It is essential to establish a comprehensive build framework and then manage and communicate the changes that will surely be required over time.
We like to see the continuous integration framework be the reference architecture for all official application builds by ensuring that all builds are performed via automated scripts that establish and specify all dependencies from librarian search paths to compiler switches and other compile and runtime dependencies. This required focus on reliability should ensure that continuous integration provides traceability and transparency.
8.15 Establishing Traceability
Continuous integration helps provide much-needed traceability to the software development process by ensuring that there is a single source of information on important baseline builds, which sometimes become milestone releases. The agile ALM is often a long journey, spanning months or even years from project inception to delivery. The CI server, integrated with the version control system, becomes a fundamental resource for providing traceability throughout this journey. In practice, the CI server shows the work evolving from each of the teams, as well as the fundamental effort to actually integrate everyone’s work in a timely manner. Sometimes the sheer volume of information can be daunting for even the most clever technology professionals.
The agile ALM inherently includes many complexities, and developers understandably struggle with compartmentalizing all of the information that they must process during the software and systems development effort. We find that information overload is a common source of mistakes, defects, and ultimately rework, among other forms of waste. Establishing timely and easy access to traceability can be a huge benefit by efficiently providing stakeholders with essential information that one might normally expect to be tracked manually. The continuous integration server actually becomes the central source of information about successful milestone releases or, as we will soon explain, interesting builds. Traceability, and for that matter, the entire continuous integration process, helps ensure effective communication, which is essential for productivity and quality.
Managing information overload is essential for success in the agile ALM. Traceability helps a great deal in addressing this problem. It is critical to ensure that there is effective communication throughout the entire continuous integration process.
8.16 Better Communication
Communication is often overlooked in the technology development effort. The continuous integration process helps facilitate communication in many important ways. CI can be an effective information radiator helping to keep everyone advised of the status (including success and failure) of application build, package, and deployments. There is nowhere that this is more important than in the integration process itself.
CI provides a central point for the work of more than one developer to be integrated together, a juncture that often requires particularly good collaboration and communication between the developers who wrote the code. We see this as an area where many teams are really challenged. Sometimes it is because of the often-distributed working environments that many of us are operating within today. We may have one or more resources in the Northeast in the United States working closely with other resources based in Europe or India. Fortunately, we have good technologies that allow us to share a screen and collaborate on our work easily. But, we still see language and cultural barriers often resulting in situations where collaboration and communication can be challenging at best and sometimes almost impossible.
Communication and traceability are two very important areas for the agile ALM. But ultimately, the CI process must discover problems immediately and ensure that all relevant stakeholders are aware and able to respond. When something is broken, we need to immediately identify the right resource to address the problem in a timely manner. This process has become known as fingering and blaming the person who introduced the change that caused the problem, most often a “broken” build.
8.17 Finger and Blame
The terminology may seem a bit harsh, but identifying exactly who introduced a change that resulted in the CI server reporting a failed build, package, or deployment (including tests) is essential for the team to stay productive. It is especially true that the earlier we identify the person who broke the build—sometimes because he or she forgot to rebase or take advantage of the preflight build—the better. The “finger and blame” step of continuous integration refers to the important step of identifying who introduced the problem and who is presumably the right person to fix the issue. When teams choose to build and deploy on every commit of the code to the version control system, then fixing anomalies is much easier because the amount of code to review is relatively small compared with teams that have multiple developers committing code, but only performing integration builds on a weekly basis. Unfortunately, building on each commit can be impractical when there are ten or more developers committing code several times a day. Sometimes, the pragmatic choice is just to run a single integration build once each night.
8.18 Is the Nightly Build Enough?
We have known teams that relied upon an integration build executed each night long before it was popular to talk about continuous integration or even agile development. Sometimes, the nightly build is the best choice compared with trying to manage dozens of builds per day when teams decide to build, package, and deploy on each commit of changes to the version control system. There are several pragmatic reasons for this choice. First, setup for the build and resources required may be such that it is only practical to run one build per night. This works well when each member of the team ensures that their changes will integrate successfully by running a preflight build and, of course, rebasing their own sandbox before committing their changes to the enterprise version control system. The problem with the nightly build approach is that the more people who have committed changes to the system, the more difficult it can be to track down the cause of any integration problems. This may be acceptable early in the software and systems lifecycle, but may not be tolerable at all the week before your system is scheduled to be released.
We find that teams often need a little more structure in their processes when they are nearing the release date—especially if delays could cause the team to miss an important deliverable. We find that urgency is sometimes related to external dependencies such as the deadline for filing a tax return.
Process in continuous integration is essential. Choosing and successfully implementing the right tools is also absolutely essential.
8.19 Selecting the Right Tools
It would be impossible to implement continuous integration without automation. Selecting the right tools can save a lot of time and make it much easier to successfully implement continuous integration. For years, many of us scripted these solutions, which truly became a labor of love. Sometimes, our home-grown build automation tools were more complicated than the applications we were building. The good news is that there are now many quality tools in this space, including popular open-source solutions and full-featured commercial solutions. The first step is picking the right continuous integration server.
8.19.1 Selecting the Right CI Server
Continuous integration servers come in all sizes and shapes. When selecting your CI server, you first need to establish your requirements for evaluation. We find that most teams focus on the features available, but may not adequately consider ease of use and time to administer and support. More basic continuous integration servers typically work from one configuration file, which can be difficult to administer. We ran into this situation when we first started working with CruiseControl. It was relatively easy to set up and get our first build to work, but then we discovered that supporting builds from multiple teams was difficult because we could introduce a change for one development team that might unintentionally affect another group. We found some commercial solutions that offered convenient menu-driven interfaces, making it easy to allow each team to configure and support their own builds with minimal risk of impacting other teams.
We have also seen organizations where each group simply set up and administered their own independent continuous integration server, which was limited to the development environment. As CI servers became easier to manage and support, this has turned out to be a perfectly viable approach. Some CI servers also integrate more easily with products in the software delivery lifecycle such as deployment frameworks. We have also found that reporting can be a differentiator between CI solutions. Successful continuous integration servers end up building many components. This has led to reuseability being another consideration in continuous integration. The demand for shared prebuilt components leads us to consider the requirements for shared repositories.
8.19.2 Selecting the Shared Repository
We first encountered shared repositories when supporting robust build frameworks, including Maven. As previously described, many companies find that open-source solutions give them all the functionality that they need, whereas others find that they really need the power, flexibility, and support of a commercial product solution. We believe that it is essential to once again consider your requirements and then conduct a bake-off between the leading vendors. All shared repos are not the same, and some have more advanced features, such as better reporting, which typically run counter to the time and effort to administer and ease of use. Choosing the right CI server and shared repository depends largely upon the requirements for your team and whether the solution will be for the specific needs of one group or be implemented as an enterprise solution.
8.20 Enterprise Continuous Integration
Enterprise continuous integration solutions need to consider what may be a variety of disparate requirements from each of the teams in the organization. We have worked with large financial services organizations where one team may be comprised of high-performance math wizards, known as quants, who come up with complex formulas for trading. Other teams may be handling mission-critical back-office batch processing that must be extremely reliable. Meeting the needs of different teams across a large enterprise can feel a lot like selling the same solution across different companies. Our focus is to share a minimum set of best practices to help teams implement continuous integration while allowing them the technical creativity to take things to the next level if so desired. The important thing is to have a consistent approach to training and support.
8.21 Training and Support
After 30 years in the trenches of IT, we are adamant that training is the hill to die on. We view training as being the most important investment that an organization can make to boost its success in implementing CI. Even if you must use a less-than-optimal continuous integration server, proper training can make the difference between success and failure. We provide a lot of training online, at conferences, and in corporate training rooms. Training team members to use each specific tool in an effective way is very important. In a corporate environment, we find that there is an even more essential goal of teaching one specific, consistent way to use the tool. Vendors often provide training in what their tool can do. Our training is often focused on what your team should do. We believe that enterprise deployment of tools and processes benefits from this approach. Picking the right approach to training and support is essential. It is equally important to realize that the tools implemented to support continuous integration must be deployed and tested successfully as well.
8.22 Deploy and Test
Implementing even the most basic continuous integration system can involve some complexity. It is very important to manage the deployment process of the tools, including planning for testing and ongoing support. We find that many teams fail to realize the importance of creating a repeatable process to install and configure the tools. We like to treat the implementation of continuous integration servers and shared repositories as we would any other systems development effort. This means that we start by understanding our dependencies, and then we create a checklist, and even scripts, to automate the installation and configuration of our tools. Having a repeatable process for installation and configuring our tools means that we can recover in the event of a significant outage by simply reinstalling our software and recovering any necessary data from the version control system or perhaps backups. We recognize that most folks don’t take the installation of tools this seriously—we do, and we have repeatedly seen that this is the best approach for enterprise tools implementation. Getting things right the first time can be challenging, and thus we also take an iterative approach to both tools implementation and the processes they support.
8.23 Tuning the Process
You will not be able to implement a comprehensive continuous integration process perfectly the first time that you try. Tuning the process is a journey that is best approached in an agile iterative way.
Sometimes processes are too informal to really help avoid mistakes. In this case, the steps of the process should be reviewed and updated. The only thing worse than a process that is missing a few steps, is a heavy process that involves too many time-consuming checkpoints. Lean processes are designed to incorporate just enough ceremony to be effective.
8.23.1 Getting Lean
Saying that a process is Lean indicates that it does not contain extra steps, which waste time and introduce unnecessary bureaucracy. Getting Lean requires its own journey. We typically find that processes early in the agile ALM can be relatively informal. But as the project progresses and deadlines approach, you will need more structure to avoid costly mistakes. Obviously, it is essential to ensure that you have enough “ceremony” in place to avoid errors, which are often prevented by well-designed and right-sized processes. For continuous integration, we often find that there are just too many builds being executed, and developers simply start ignoring the complaints of the CI server. Sure, we always insist that there be a designated resource responsible for dealing with broken builds, but in practice, too many builds often result in teams giving up entirely on continuous integration, which is obviously not the desired outcome. One approach of moderation is to identify “interesting” builds.
8.23.2 Interesting Builds
Builds that identify a milestone release are obviously of greater interest than builds that are mid-sprint. We never want to ignore any broken build, but in practice, it is often necessary to set priorities. This is where tagging a milestone build as being “interesting” helps identify specific builds that should be monitored more carefully. Getting continuous integration implemented successfully has many benefits. One of the most essential is that CI establishes the processes and technical procedures that will help lead to automated deployment.
8.24 CI Leads to Continuous Deployment
In Chapter 9, we will focus on continuous deployment, which although considerably more complicated than continuous integration, nonetheless benefits from many of the same principles and lessons learned. What you should understand in this section is that effective continuous integration establishes the required automated procedures that can then be used by the deployment framework to support continuous deployment.
8.25 Conclusion
Continuous integration is a well-established industry best practice that helps to deliver quality code and improve overall programmer productivity. Choosing the right tools and establishing right-sized processes are fundamental if you are going to succeed with CI. We emphasize that continuous integration is a journey that should be implemented using agile principles. Choosing the right tools is essential, as is providing training, administration, and ongoing support.