Chapter 23. Using Tests for Other Things
“By fighting, you never get enough, but by yielding, you get more than you expected.”
Lawrence G. Lovasik
Acceptance tests define the functionality of a program. But you can use them for more than just that—measuring doneness, estimating, and breaking down a story.
Uses of Acceptance Tests
Acceptance tests are a communication mechanism between the members of the triad. They clarify the customer requirements and are a specification of how the system works. But you can also employ them for other purposes. They are a measure of how complete an implementation is; a means of estimating the effort to implement a story, and a method for story breakdown.
Degree of Doneness
If there are multiple acceptance tests for a business rule or a story, the ratio of successful tests to the total number of tests can provide a rough guide to how much of the story has been implemented. For example, if you have ten acceptance tests and three are passing, the story is “about” 30% complete.
It is possible that the “worst” test case was saved for last. So the effort to implement that story represents more than its fair share of the total effort. That is why this is only a guide to doneness, rather than an exact measurement.
Make all the tests pass for a story before moving to another story. Otherwise, you may wind up with lots of stories that are not done.
Estimation Aid
An estimate may be required for implementing a story in many environments. The number and complexity of the acceptance tests can be a rough guide to the effort required to implement a story. The tests for a new story can be compared to the tests run against completed stories. You can develop your own heuristics as to how the number and complexity influence the effort. Large custom setups (givens) and numerous state changes (thens) usually imply a much larger effort than tests with small setups and few state changes.
Breaking Down Stories
You may need to break down a story into smaller stories for the purposes of fitting stories into iterations. Mike Cohn [Cohn01] offers some ways to break down a story. For example, you can start with a basic user interface and then add more bells and whistles, such as images. You can implement something manually, such as calling users about overdue rentals, and later automate it with robot calling. You can do something simple, such as a single check-out per CD, and then have a story for checking out multiple CDs at once.
In addition, you can use acceptance tests as a breakdown mechanism. The test for each scenario of a use case can become a separate story. A complex business rule can become a separate story. The test for a complicated business rule calculation may be expressed in a table with lots of rows. You can break the calculation into separate stories by assigning a set of rows to each story.
With the address example in Chapter 17, “Decouple with Interfaces,” the tests themselves suggest a way to break down the story. The tests for a U.S. address are separate from the tests for a Canadian address. Therefore, U.S. address verification can be a different story than Canadian address verification.
Developer Stories
Another reason for breaking down a story is so that multiple teams can help implement it. Chapter 16, “Developer Acceptance Tests,” showed how Debbie made up acceptance tests for user interface components and functional modules that are to be created by other developers. If a story has to be broken down, you should create acceptance tests for each of the substories. The acceptance tests help decouple the stories by clarifying the responsibilities of each of the stories. They provide doneness criteria for each story. It is far more effective for distributed teams to work on decoupled stories than to work on tasks for the same story [Eckstein01].
Tests as a Bug Report
For nontrivial bugs, you can write a test as documentation of the bug. The definition of a nontrivial bug is something more than a misspelled word, a bad color, or an unaligned dialog box. For each bug, the discoverer should create an acceptance test that shows that the desired behavior has not been achieved. With the bug, the acceptance test fails. When it passes, the bug has been fixed.1
For example, suppose that you were coding the discount example in Chapter 4, “An Introductory Acceptance Test,” and you did not have an acceptance test. If a bug was reported for a Good customer, you would create an acceptance test. To ensure that the bug fix does not affect other behavior, include related test cases that are currently passing. For example, here are tests around other values for a Good customer. You might also have tests for the Excellent customer as well, depending on the situation - the cost involved if the bug fix might alter that Customer Rating as well.
Root Cause Analysis
If a bug like the preceding appears, you have an opportunity to do root-cause analysis.2 What you are looking for is why the bug appeared. Is it something in the process itself, or was it a random event? Was the case in production not covered by a test case, and if so, why not? Were the data values not expected? For example, suppose a data value was supposed to be between 1 and 100, but the value in production was found to be 101. In that case, you create an acceptance test that demonstrates the correct behavior—limiting the value to 100. Because an acceptance test represents a requirement, the need for this acceptance test being created after implementation may represent a missed requirement An acceptance test that has the wrong values is a misinterpreted requirement.
The missed or misinterpreted requirement may be traced to a random event, such as, “We had to get this done in four hours before release.” Alternatively, it may be traced to a common cause, such as, “The customer never collaborates with us before we start implementing.”3
If you find yourself getting buried in analysis, try something you think might prevent the problem from reoccurring and see what happens. If it doesn’t work, try something else.
Production Bugs
One of the most important measures for a team process is the number of bugs that have escaped to production. You should examine the root cause or causes of each escaped bug so you can discover how to prevent more of them from escaping in the future.
Regression Testing
The primary purpose of acceptance tests is to translate the customer requirements into code. If you have acceptance tests for all requirements, you can use the set of acceptance tests as a regression test suite. Unless the requirement associated with an acceptance test changes, all acceptance tests should pass. If a change is made to the implementation to accommodate a new requirement, all previous acceptance tests should still pass. If previous tests break when a new requirement is introduced, you may have issues in the design of your code.
If the acceptance tests are automated, run them as often as possible. This provides immediate feedback that a change in the application has caused some previously implemented functionality to break.
Summary
• You can use acceptance tests as any of the following:
• A rough guide to story completeness
• A rough way to estimate relative story effort
• A way to break up stories
• Distributed teams that break up a story should have acceptance tests for their part of the story.
• Examine the root cause of why an acceptance test was missed or incorrect. If possible, change the process to eliminate the cause.