8. Reporting Bugs

Everyone

“A group of programmers were presenting a report to the Emperor. “What was the greatest achievement of the year?” the Emperor asked. The programmers spoke among themselves and then replied, “We fixed 50% more bugs this year than we fixed last year.” The Emperor looked on them in confusion. It was clear he did not know what a “bug” was. After conferring in low undertones with his chief minister, he turned to the programmers, his face red with anger. “You are guilty of poor quality control. Next year there will be no bugs!” he demanded. And sure enough, when the programmers presented their report to the Emperor the next year, there was no mention of bugs.”¹

—Geoffrey James, The Zen of Programming

Figure 8.1 Lifecycle of a (real) bug. Real bugs have a well-defined lifecycle, according to their species. Their metamorphosis is a good metaphor for software bugs, which often start with a simple observation of a probable source of customer dissatisfaction (the egg) and proceed to a well-defined report of the observed symptoms, steps to reproduce, and a technical investigation (the larva), to a fix for the problem (the chrysalis), and finally to a working build with the fix verified (the adult). Like real bugs, not all software bugs survive to adulthood. Of course, unlike real bugs, software bugs can retreat back to their larval state when the fixes are unsuccessful. Those are the reactivations.

In a value-up paradigm, quality is everyone’s business all the time. Prevention of errors is one of the most important aspects to every activity. So why do I have a separate chapter on bugs—isn’t that antiquated thinking? In most organizational cultures I have seen, bugs take on such an important life that they should be addressed head on. And despite the best-known lifecycle and development practices, bugs still occur, although their nature changes. Hence this chapter. MSF for Agile Software Development defines this principle:

Quality is everyone’s job every day. Quality requires both bug prevention and solution verification. Utilize practices such as code analysis and peer reviews to prevent bugs as well as maximize the testing to find bugs. All roles are responsible for prevention and verification of bugs.²

Consistent with MSF, I call bugs “bugs” and not “defects.” “Defect” often has connotations of blame and nonconformance to spec, both of which are red herrings. For value-up purposes, anything that diminishes perceived value is a bug and should be considered for triage and potential change. For the same reason, I do not distinguish here between bugs and change requests, although in certain processes, they would be treated differently. Take them all to triage the same way.

A Cautionary Tale

Let’s revisit the practices of two testers on our project, Caliban and Ariel. At the end of the project, Caliban had reported 100 bugs; Ariel had reported 74. (Assume that they were all equal priority and severity.) Prospero, who is their manager, must pick one of the two testers for a key new project. Whom should Prospero pick? Based on this information alone, he’d probably pick Caliban.

Before choosing, however, Prospero took a look at the state of their bugs at the end of the project (see Table 8.1).

Table 8.1 Breakdown of Reasons Given for Closure of the Bugs

Prospero noted that 80% of the bug fixes customers received were the result of Ariel’s work, and only 20% were the result of Caliban’s efforts. As a result, he picked Ariel.

The point of this exercise is to raise the significance of the bug’s lifecycle, not just its initial state as an ovum on a leaf. A lot can happen between the time when you report a bug and the time (if ever) when the customer receives software with the fix. It’s important that you create that ovum with the right material to mature on the right path.

A (Software) Bug’s Life

Not unlike biological bugs, software bugs go through a very precise set of states (see Figure 8.2). In VSTS, the default states and transitions that are allowed depend on the process template that you choose for the project. The selection and timing of permissible states, the groups who authorize advancing a bug to the next state, and the allowable reasons for the transition are governed by a set of rules in the chosen process. If you have the permission, you can customize the rules further to match your team’s workflow.

Figure 8.2 This state chart is the lifecycle of a software bug in MSF for CMMI Process Improvement.

Anyone can propose a bug, that is, request that it be queued for fixing.

A project manager can activate the bug and assign it to a developer.

After investigating and fixing the code, the developer marks the bug as resolved with the next check-in (see Figure 8.3). The developer is not allowed to close the bug directly.

Figure 8.3 In VSTS, merely by checking off the bug in the check-in dialog, the developer associates the changeset being delivered with the bug being fixed. The association is permanent and is used to populate the build report and the metrics warehouse.

The check-in process automatically resolves the bug when the corresponding source code is checked-in.

A tester can see the resolved bug in the corresponding query. Only the tester closes the bug after verifying that it passes the tests to prove the fix. (Of course, if the test fails, then the tester would reactivate the bug.)

In practice, workflows can be simpler or more complex. MSF Agile, for example, does not require a proposed state before activation, does not enforce different permissions, and instead relies on trust. On the other hand, some processes, especially in late iterations, do not allow check-in until the code has been reviewed. Work items can enforce this rule through another state or a separate field, and check-in policy can enforce the workflow.

Whatever they are, these rules are the tangible form of the process for your current project.

Bug Reporting Is Like Journalism

Caliban has an anecdote about his favorite bug. It goes something like this:

I found this bug in the lab months ago and reported it. No one paid attention; so they triaged it out and marked it deferred. Last week our biggest customer found it too and all of a sudden it’s a crisis. Now we need to ship a service release, which means that I’ll be in the lab all weekend checking the fix. Why couldn’t they have paid attention when I reported it the first time?

Why indeed? Very often the answer is that the bug report was not clear enough to communicate the impact to the triage committee, and the reported bug was closed. Prospero, Caliban’s manager, has a different view of Caliban’s story: “Why can’t Caliban make his bug reports clear enough for the team to act on them?” (And, as before, Prospero did not choose Caliban for his next project.)

The lesson from Caliban’s experience is well summarized in this quote from Kaner, Bach, and Pettichord:

Your advocacy drives the repair of the bugs you report.

Any bug report that you write is an advocacy document that calls for the repair of the bug.

Some bugs will never be fixed. Your responsibility is not to make sure that all the bugs are fixed. Your responsibility is to report bugs accurately and in a way that allows the reader to understand the full impact of the problem.

How well you research and write the report will often have a substantial effect on the probability that the bug will be fixed.³

Writing up a bug well is like writing newspaper (or web page) copy. It is journalism. The title is like the headline, and you should expect most readers never to look “below the fold.” The journalist’s six basic questions—what, where, how, who, why, and when—need to be covered in a summary sentence or two.

Know Your Audience

The first audience for your bug reports is the triage committee. (Triage is discussed in Chapter 4, “Project Management.”) They will see a query that looks something like Figure 8.4.

Figure 8.4 Keep in mind what your readers will see during triage. It’s not all the detail, just what fits on the query results list. You should write bugs to be viewed in this format.

The triage committee includes the busiest people on the project. It is common for them to spend less than one minute reviewing each bug. (An important reason for daily triage is to keep the length of the list down so that individual work items get the attention they deserve.) In that minute, the fate of your bug report will be decided, as was the fate of Caliban’s bugs. If Caliban had spent a little more time on the quality of his bug reports, the value of his work (and his performance rating) might have been considerably higher.

Learn from History

Reading bug reports, especially after they have been triaged, is a good way to learn about your project and your audience. Auditing triage meetings is another—you learn what is important to stakeholders in practice. Here are some tips on using your bug database:

• Execute the repro steps on interesting bugs. It’s a good way to learn the software under test in its more interesting cases (and a good way to learn how to write clear steps).

• Read the commentary to see how other team members answer bug reports to learn both what they think is important and how to improve your reporting style.

• Study closed bug reports to see which bugs have and have not been fixed, based on the way they were reported. If you want yours fixed, report them like the ones that historically do get fixed.

• If you’re working on an existing solution, read tech support logs for errors to check and stories to support your bugs.

Subjective Data

The most important question for the readers of your report is: What’s the impact on the user? Most of the time, the answer requires some thought. Ask yourself questions like the following:

• What will the customer say to Product Support when she sees this?

• What will the customer expect from experience with other software (including competitors’ products)?

• What will sales reps or trainers say if they hit this in a demo?

• What scenarios will be affected by this bug?

• What is the downstream impact of this bug?

• How many users will care and how often?

• Is this an isolated occurrence?

• Is there a security risk here?

• Are other qualities of service affected?

• Write this story first. How you answer these questions will influence both how much time you should spend researching the bug and what data you should collect with the report.

Titles Matter

As you can see in Figure 8.4, the title is often the only information that the reader sees. If the title is unclear, when truncated to fit the column width, the rest of the information may never be read.

Descriptions Stick

The persistent next level of information about the bug is the description. History changes can get very long, but the description should be the succinct detail about the bug.

One Report Per Bug

If you have two bugs, then write two reports. Keeping each observed bug in its own report will make all the reports easier to understand.

Objective Data

After you have written the subjective data about the bug, it is time to gather objective data.

Figure 8.6 The bug work item type, here from MSF for CMMI Process Improvement, captures the information that all its users need to query, track, and analyze the bug.

Quality of Service

Be clear about the type of issue that you are reporting—what QoS is affected.

Related Feature or Code

If you can relate the bug to a specified feature that you are testing or to specific areas of the code under test, then do so.

Test to Reproduce

If you have a test to reproduce the bug, identify it.

Attachments

What data can you attach to the bug report to demonstrate the conditions in which the bug occurs? All these can be useful, where applicable:

• Screenshots

• Data files

• Configuration files

• Trace files

• Server logs

• Dr. Watson logs

Reproduction Steps

How do you reproduce the bug in the minimum number of steps with the least setup? Eighty percent of the developer’s effort in working on a bug is often in the reproduction—how well can you streamline that process? Anything unnecessary is clearly wasteful. Make the repro steps accurate and clear. Check them before submitting the bug.

Preconditions

Is special setup needed to reproduce the error, such as loading a particular database? If the problem can be demonstrated with production data or customer data, it’s particularly compelling.

Assessment Data

The assessment will change a lot through the bug’s life as people refine the diagnosis and observation, debug the source, try and test fixes, study related bugs, and so on. It is your job to keep the bug history accurate and thorough as you learn more about the bug and its conditions.

Some of the assessment you can do on the initial report, and more of it will evolve with the bug history and conversation.

Is It a Duplicate?

Spend a little bit of time looking through the database to determine whether this is more information about an existing bug or something new, but don’t go crazy. It’s easy enough for someone to mark your bug as a duplicate later. Obviously, if you have discovered more about an existing bug, update the existing one with your findings.

How General Is It?

Uncorner your corner cases.⁵ Frequently, you will discover bugs using extreme values, unusual configurations, or long entry sequences. Essential to understanding the importance of the bug is to know how specific or general the conditions must be. Here are some tips:

• Vary the data. If you discover the bug with a particular set of input values, try different ones. Especially try different equivalence classes. How dependent is the bug’s appearance on specific values or classes?

• Vary the configurations. If you find the bug in a particular configuration of hardware and software components, try it on a very different set of machines.

• Vary the flows. If the bug appeared under a particular sequence of actions (especially if it is a long one), see whether there are other (preferably shorter) paths that lead to the same result.

• Look for suspect interactions. If something seems to be a strange side effect, but you’re not sure, keep poking. For example, after you use one program, another might suddenly seem slower. Perhaps they rely on some common component, data, or OS service.

• Are there more? Hypothesize where similar bugs may be lurking. Look for them.

How Severe Is It?

When you first find a bug, there is a great temptation to be satisfied and to report it. In fact, you might be at the tip of a much worse problem. Note the initial symptom but also keep using the software under test. You are likely to discover poorly handled error paths and exceptions, unforeseen interactions, and other problems that would baffle a user (and tech support).

Plan

At initial reporting, the primary planning tools you have are priority and ownership. Depending on your process, these may be settable only by the triage committee, or in less formal organizations, the submitter may fill these in directly. Priority should be handled in the same way as it is for project management.

Iteration

Think carefully about the planned iteration. Often in early iterations, bugs are filed appropriately against functionality that won’t work until later. A purpose of triage is to remove this noise from developers’ immediate queues.

Summary

This chapter focused on bug reporting. Obviously, as all previous chapters discussed, it is far better to prevent bugs from occurring with good requirements, architecture, and development practices than to discover them and incur the rework necessary to fix them. Nonetheless, they happen, and their details should be captured by everyone who finds them.

When you write up a bug, be conscious of your audience. Most of your readers will only see the title, perhaps in truncated form. Work hard to make the title summarize the impact and importance of the bug clearly; otherwise, the rest of the information might be overlooked.

A useful mnemonic for completing the rest of the report is SOAP (subjective-objective-assessment-plan), borrowed from the world of hospitals. In particular, the medical analogy can help you think hard about the assessment and plan, and it can help move you away from thinking of bugs as a numbers game.

Endnotes