Aside from making the testing process repeatable, you need to measure the project so that you can tell for sure whether changes improve or degrade it. Here are a few kinds of data you can collect to measure your project:
Administrative description of the defect (the date reported, the person who reported it, a title or description, the build number, the date fixed)
Full description of the problem
Steps to take to repeat the problem
Suggested workaround for the problem
Related defects
Severity of the problem—for example, fatal, bothersome, or cosmetic
Origin of the defect: requirements, design, coding, or testing
Subclassification of a coding defect: off-by-one, bad assignment, bad array index, bad routine call, and so on
Classes and routines changed by the fix
Number of lines of code affected by the defect
Hours to find the defect
Hours to fix the defect
Once you collect the data, you can crunch a few numbers to determine whether your project is getting sicker or healthier:
Number of defects in each class, sorted from worst class to best, possibly normalized by class size
Number of defects in each routine, sorted from worst routine to best, possibly normalized by routine size
Average number of testing hours per defect found
Average number of defects found per test case
Average number of programming hours per defect fixed
Percentage of code covered by test cases
Number of outstanding defects in each severity classification
In addition to project-level test records, you might find it useful to keep track of your personal test records. These records can include both a checklist of the errors you most commonly make as well as a record of the amount of time you spend writing code, testing code, and correcting errors.
Federal truth-in-advising statutes compel me to disclose that several other books cover testing in more depth than this chapter does. Books that are devoted to testing discuss system and black-box testing, which haven't been discussed in this chapter. They also go into more depth on developer topics. They discuss formal approaches such as causeeffect graphing and the ins and outs of establishing an independent test organization.
Kaner, Cem, Jack Falk, and Hung Q. Nguyen. Testing Computer Software, 2d ed. New York, NY: John Wiley & Sons, 1999. This is probably the best current book on software testing. It is most applicable to testing applications that will be distributed to a widespread customer base, such as high-volume websites and shrink-wrap applications, but it is also generally useful.
Kaner, Cem, James Bach, and Bret Pettichord. Lessons Learned in Software Testing. New York, NY: John Wiley & Sons, 2002. This book is a good supplement to Testing Computer Software, 2d ed. It's organized into 11 chapters that enumerate 250 lessons learned by the authors.
Tamre, Louise. Introducing Software Testing. Boston, MA: Addison-Wesley, 2002. This is an accessible testing book targeted at developers who need to understand testing. Belying the title, the book goes into some depth on testing details that are useful even to experienced testers.
Whittaker, James A. How to Break Software: A Practical Guide to Testing. Boston, MA: Addison-Wesley, 2002. This book lists 23 attacks testers can use to make software fail and presents examples for each attack using popular software packages. You can use this book as a primary source of information about testing or, because its approach is distinctive, you can use it to supplement other testing books.
Whittaker, James A. "What Is Software Testing? And Why Is It So Hard?" IEEE Software, January 2000, pp. 70–79. This article is a good introduction to software testing issues and explains some of the challenges associated with effectively testing software.
Myers, Glenford J. The Art of Software Testing. New York, NY: John Wiley, 1979. This is the classic book on software testing and is still in print (though quite expensive). The contents of the book are straightforward: A Self-Assessment Test; The Psychology and Economics of Program Testing; Program Inspections, Walkthroughs, and Reviews; Test-Case Design; Class Testing; Higher-Order Testing; Debugging; Test Tools and Other Techniques. It's short (177 pages) and readable. The quiz at the beginning gets you started thinking like a tester and demonstrates how many ways a piece of code can be broken.
Bentley, Jon. "A Small Matter of Programming" in Programming Pearls, 2d ed. Boston, MA: Addison-Wesley, 2000. This essay includes several good examples of test scaffolding.
Mackinnon, Tim, Steve Freeman, and Philip Craig. "Endo-Testing: Unit Testing with Mock Objects," eXtreme Programming and Flexible Processes Software Engineering - XP2000" Conference, 2000. This is the original paper to discuss the use of mock objects to support developer testing.
Thomas, Dave and Andy Hunt. "Mock Objects," IEEE Software, May/June 2002. This is a highly readable introduction to using mock objects to support developer testing.
http://www.junit.org. This site provides support for developers using JUnit. Similar resources are provided at cppunit.sourceforge.net and nunit.sourceforge.net.
Beck, Kent. Test-Driven Development: By Example. Boston, MA: Addison-Wesley, 2003. Beck describes the ins and outs of "test-driven development," a development approach that's characterized by writing test cases first and then writing the code to satisfy the test cases. Despite Beck's sometimes-evangelical tone, the advice is sound, and the book is short and to the point. The book has an extensive running example with real code.