The Bad Test Death Spiral, aka the SCUMmy Cycle

Sometimes teams embark on TDD and experience some good results for a while. Then things start sliding toward oblivion, speeding up quickly and ultimately ending up with the decision to abandon TDD. What causes this “bad test death spiral,” and how do you prevent it?

The problem isn’t unique to TDD. There’s also the “bad agile death spiral” in which the short iterations of pseudo-Agile appear to produce good results for a while. A year or eighteen months in, however, the team is flabbergasted at the mess on their hands. The net result is that Agile gets abandoned and blamed for the waste.

Ben Rady and Rod Coffin described the SCUMmy cycle in their Agile2009 conference presentation, “Continuous Testing Evolved.”^[34] Their acronym SCUM describes the kinds of bad tests that generate the degradation: Slow, Confusing, Unreliable, Missing. Here’s one possible path along the downward spiral (I’ve seen this path firsthand a few times in shops):

1. The team writes mostly integration-style tests. These tests tightly couple to volatile or slow dependencies, such as the database or other external APIs. While this results in slower tests, initially it doesn’t add up to much real pain, because the team is still able to run a few hundred tests in a minute or two. (Think “frog in slowly boiling water.”)

2. The growing body of tests begins to pass the pain threshold. Now there are enough tests to take several minutes to run.

3. Developers run the test suite less frequently or run subsets of the tests. The team also discovers that the tests are more difficult. They are typically longer, with more required setup, and thus require increased effort to understand and analyze when there’s a problem. Other problems creep in, with tests failing intermittently because of volatile conditions outside the control of the unit tests. Developers begin to discover that the tests are often false alarms, indicating a problem not with the production system but instead with the test design.

4. Developers delete tests. The knee-jerk reaction to difficult tests is to start disabling or even deleting them. Developers find that deleting is much easier than spending an hour fixing a poorly designed test.

5. Defects begin to increase. The tests that remain likely don’t cover enough detailed bits of logic and now offer only little value in preventing defects. (They never offered much in the way of documentation value either.)

6. The team, or management, questions the value of TDD. The team tries to plod on, but it’s fairly obvious that the investment was a waste.

7. The team abandons TDD. Management makes a note of the highly visible fiasco.

What’s a team to do? If the path to TDD is doomed, why even start?

Ideally, you’ve learned in this book that TDD requires you to constrain the scope of each test so that it tests a small piece of isolated logic. Systems built in this manner are unlikely to experience the bad test death spiral. Still, a quality system doesn’t magically emerge from doing TDD. You and your team must actively seek to squash poor design in both the tests and the code. Doing that also requires the team to understand what good tests and code look like.

The following progression counters each step in the spiral. If you’re actively monitoring what’s going on, chances are you can prevent the spiral from progressing further downward.

1. The team writes mostly integration-style tests. Learn how to write unit tests. Reread this book, attend training, hire a coach, hold dojos, review more, read more, and so on. You’ll also want to increase your knowledge of good design and code constructs.

2. The growing body of tests begins to pass the pain threshold. Split into slow and fast suites. Establish what it means for a test to be fast (5ms or less on a dev machine?). Require the team to acknowledge when they are adding another test to the slow suite. Learn what it takes to restructure a test (and the corresponding code) so that it’s fast. Establish a habit of incrementally but regularly trying to transform a slow test into a fast test.

3. Developers run the test suite less frequently or run subsets of the tests. Fail the fast suite execution when a test exceeds the slow threshold. (I succeeded in modifying Google Test to this end for a recent customer.) This reinforces the importance of fast tests to the development team.

4. Developers delete tests. Monitor coverage. While establishing a coverage metric goal is of questionable value (see ​Using the Code Coverage Metric Effectively​), you do want to prefer increasing or at least stable coverage numbers. Unfortunately, replacing a bad test with equivalent coverage by good unit tests can require a bit of effort. But until the proper habits are in place, you’re better off incrementally taking the time to do the right thing than giving up.

5. Defects begin to increase. Always write a test first in response to a defect. Defects are opportunities to recognize the deficiencies in TDD practice. For each defect, insist that an appropriate failing unit test gets written before the problem is fixed.

6. The team, or management, questions the value of TDD. Commit to quality. Insist that the practices of TDD, as well as other practices such as acceptance testing, refactoring, and pairing, are primarily about delivering quality software in the long haul and are not just about minimizing defects. Ensure that it’s clear how and why these practices relate to quality, and ensure that the team employs the practices in a way that helps realize the goal of quality. Lack of concern over quality will devastate your system—the “bad quality death spiral” is even more unforgiving.

7. The team abandons TDD. Don’t wait ’til it’s too late! Management rarely tolerates second attempts at what they think is the same path to failure.

As with anything, you can practice TDD inappropriately and fail. But it’s possible to succeed and to succeed wildly; otherwise, I wouldn’t have bothered writing this book. Blaming the technique for insistence on doing it poorly isn’t cool.