The idea behind the daily-build-and-smoke-test process is simple: build the product and test it every day. The following sections describe some of the ins and outs of this simple idea.

Build daily. The most fundamental part of the daily build is to build the product every day. As Jim McCarthy says, treat the daily build as the heartbeat of the project (McCarthy 1995c). If the heart isn't beating, the project is dead.

A little less metaphorically, some describe the daily build as the synch pulse of a project (Cusumano and Selby 1995). Different developers' code can get a little bit out of synch between synch pulses, but with each synch pulse, all the code has to come back into alignment. When you insist on keeping the pulses close together, you prevent some developers from getting out of synch entirely.

Use an automated build tool (such as make) to reduce the repetitive work associated with building daily.

Check for broken builds. In order for the daily-build process to be effective, the software that's built has to work. If the software isn't usable, the build is considered to be broken, and fixing the build becomes top priority.

Each project sets its own standard for what constitutes "breaking the build." The standard needs to set a quality level that's strict enough to keep showstopper defects out of the daily build but lenient enough to disregard trivial defects (because undue attention to trivial defects can paralyze progress).

At a minimum, a good build should:

Some projects set stricter standards and include compiler and linker errors and warnings in their definitions of good builds. The litmus test in any case is whether the build is stable enough to be released to testing. If it's stable enough to test, it isn't broken.

Smoke test daily. The smoke test should exercise the entire system from end to end. It does not have to be an exhaustive test, but it should be capable of detecting major problems. By definition, if the build passes the smoke test, that means that it is stable enough to be tested and is a good build.

Without the smoke test, the daily build is meaningless. The smoke test indicates whether the build worked. If you treat the build itself as the heartbeat of the project, the smoke test is the stethoscope that allows you to determine whether the heart is beating. It is the first line of testing, the first line of defense against program deterioration. Without the smoke test, the daily build is just a way of verifying that you have a clean compile.

The smoke test must evolve as the system evolves. At first, the smoke test will probably be something simple, such as testing whether the system can say, "Hello World." As the system develops, the smoke test will become more thorough. The first test might take a matter of seconds to run. As the system grows, the smoke test can grow to thirty minutes, an hour, or more.

The build group should make maintaining the smoke test its top priority. The quality of their work on the smoke test should factor into their performance evaluations. The build group will be under pressure from the developers not to make the smoke test too "picky," so its members should be chosen from the ranks of quality assurance rather than from the ranks of developers. Regardless of everyone's best intentions, putting a developer in charge of the smoke test is too much like putting a fox in charge of the chicken coop. The group's members should report to the QA lead rather than to the development lead.

This is one of the areas in which designing your system for testing comes in handy. The smoke test should be automated, and sometimes it's easier to automate if you have built-in hooks for testing.

Establish a build group. On most projects, tending the daily build becomes a big enough task that it needs to be an explicit part of someone's job. On large projects, it can become a full-time job for more than one person. For example, on Windows NT 3.0, there were four full-time people in the build group (Zachary 1994). It also takes effort to keep the smoke test up to date as the product grows and evolves.

On smaller projects you might not need to devote a whole person to the daily build. In that case, put someone from quality assurance in charge of it.

Add code to the build only when it makes sense to do so… Although daily builds require that you build the system daily, they do not require that developers add every new line of code they write to the system every day. Individual developers usually don't write code fast enough to add meaningful increments to the system every day. They work on a chunk of code and then integrate it when they have a collection of code in a consistent state. Developers should maintain private versions of the source files they're working on. Once they have created a complete set of modifications, they make a private build of the system using their modifications, test it, and check it in.

…but don't wait too long to add code to the build. Although few developers will check in code every day, beware of developers who check in code infrequently. It's possible for a developer to become so embroiled in a set of revisions that every file in the system seems to be involved. That undermines the value of the daily build. The rest of the team will continue to realize the benefit of incremental integration but that particular developer will not. If a developer goes more than a few days without checking in a set of changes, consider that developer's work to be at risk.

Require developers to smoke test their code before adding it to the system. Developers need to test their own code before they add it to the build. A developer can do this by creating a private build of the system on a personal machine, which the developer then tests individually. Or the developer can release a private build to a "testing buddy," a tester who focuses on that developer's code. The goal in either case is to be sure that the new code passes the smoke test before it's allowed to influence other parts of the system.

Create a holding area for code that's to be added to the build. Part of the success of the daily-build process depends on knowing which builds are good and which are not. In testing their own code, developers need to be able to rely on a known good system.

Most groups solve this problem by creating a holding area for code that developers think is ready to be added to the build. New code goes into the holding area, the new build is built, and if the build is acceptable, the new code is migrated into the master sources.

On small- and medium-sized projects, a version-control system can serve this function. Developers check new code into the version-control system. Developers who want to use a known good build simply set a date flag in their version-control–options file that tells the system to retrieve files based on the date of the last-known good build.

On large projects or projects that use unsophisticated version-control software, the holding-area function has to be handled manually. The author of a set of new code sends email to the build group to tell them where to find the new files to be checked in. Or the group establishes a "check-in" area on a file server where developers put new versions of their source files. The build group then assumes responsibility for checking new code into version control after they have verified that the new code doesn't break the build.

Create a penalty for breaking the build. Most groups that use daily builds create a penalty for breaking the build. If the build is broken too often, it's hard for developers to take the job of not breaking the build seriously. Breaking the build should be the exception, not the rule. Make it clear from the beginning that keeping the build healthy is the project's top priority. Refuse to take a broken build casually. Insist that developers who have broken the build stop their other work until they've fixed the build.

A fun penalty can help to emphasize the priority of keeping the build healthy. Some groups give out suckers to developers who break the build. The sucker who broke the build has to tape a sucker to his office door until he fixes the build. Other projects make the person who broke the build responsible for running the build until someone else breaks it. On one project I worked on, the person who broke the build had to wear a hat made out of an umbrella until he fixed the build. Other groups have guilty developers wear goat horns or contribute $5 to a morale fund.

Some projects establish a penalty with more bite. Microsoft developers on high-profile projects such as NT, Windows 95, and Excel have taken to wearing beepers in the late stages of their projects. If they break the build, they are called in to fix the build regardless of the time of day or night. The success of this practice depends on the build group being able to determine precisely who broke the build. If they aren't careful, a developer who's mistakenly called to fix the build at 3:00 in the morning will quickly become an enemy of the daily-build process.

Release builds in the morning. Some groups have found that they prefer to build overnight, smoke test in the early morning, and release new builds in the morning rather than the afternoon. There are several advantages to smoke testing and releasing builds in the morning.

First, if you release a build in the morning, testers can test with a fresh build that day. If you generally release builds in the afternoon, testers feel compelled to launch their automated tests before they leave for the day. When the build is delayed, which it often is, the testers have to stay late to launch their tests. Because it's not their fault that they have to stay late, the build process becomes demoralizing.

When you complete the build in the morning, you have more reliable access to developers when there are problems with the build. During the day, developers are down the hall. During the evening, developers can be anywhere. Even when developers are given beepers, they're not always easy to locate.

It might be more macho to start smoke testing at the end of the day and call people in the middle of the night when you find problems, but it's harder on the team, it wastes time, and in the end you lose more than you gain.

Build and smoke even under pressure. When schedule pressure becomes intense, the work required to maintain the daily build can seem like extravagant overhead. The opposite is true. Under pressure, developers lose some of their discipline. They feel pressure to take design and implementation shortcuts that they would not take under less stressful circumstances. They review and unit test their own code less carefully than usual. The code tends toward a state of entropy more quickly than it would during less stressful times.

Against this backdrop, the daily-build-and-smoke-test process enforces discipline and keeps pressure-cooker projects on track. The code still tends toward a state of entropy, but you bring that tendency to heel every day. If you build daily and the build is broken, identifying the developer responsible and insisting on an immediate code fix is a manageable task. Bringing the code back from its state of entropy is a relatively small undertaking.

If you wait two days—until twice as many defects are inserted into the code—then you have to deal both with twice as many defects and with multiplicative interaction effects of those defects. It will take more than twice as much effort to diagnose and correct them. The longer you wait between builds, the harder it is to bring the build back into line.