Most projects have more bug reports that can be fixed given limited resources and time. To characterize how the quality of bug reports increases the chances of bugs getting fixed, we sampled 150,000 bugs from Apache, Eclipse, and Mozilla (50,000 per project). These bugs had various resolutions, such as FIXED, DUPLICATE, MOVED, WONTFIX, and WORKSFORME. We divided the bug reports into two groups, successful and not successful, and then used statistical tests such as Chi-Square and Kruskal-Wallis (p<.05) to check for relationships between the success of bug reports and the presence of information items (code samples, stack traces, patches, screenshots). Our comparisons were:
- Resolution of bug reports
We compared bug reports resolved as FIXED against bug reports with other resolutions, such as WONTFIX and WORKSFORME. We treated duplicate bug reports as a separate group because for some the master report is fixed, whereas for others the master report is not fixed.
- Lifetime of bug reports
We compared bug reports with a short lifetime against bug reports with a long lifetime. A fast turnaround for bug reports is desirable, especially for users, but also for developers.
We also checked for the influence of readability on the success of bug reports. To measure the readability of bug reports, we used the Style tool, which “analyses the surface characteristics of the writing style of a document” [Cherry and Vesterman 1981]. The readability of a text is measured by the number of syllables per word and the length of sentences. Readability measures are used by Amazon.com to inform customers about the difficulty of books and by the U.S. Navy to ensure readability of technical documents. For our experiments we used the following seven readability measures: Kincaid, Automated Readability Index (ARI), Coleman-Liau, Flesh, Fog, Lix, and SMOG Grade.
Our findings from this experiment are:
Bug reports with stack traces get fixed sooner. (Apache, Eclipse, and Mozilla)
Bug reports that are easier to read get fixed sooner. (Apache, Eclipse, and Mozilla)
Including code samples in a bug report increases the chances of it getting fixed. (Mozilla)
Independently from us, Hooimeijer and Weimer observed for Firefox that bug reports with attachments get fixed later, and bug reports with many comments get fixed sooner [Hooimeijer and Weimer 2007]. They also confirmed our results that easy-to-read reports are fixed faster. Panjer observed for Eclipse that comment count and activity as well as severity have the most effect on bugs’ lifetimes [Panjer 2007]. Schröter et al. validated our finding that bugs with stack traces get fixed sooner and further emphasized the importance of adding stack traces to bug reports [Schröter et al. 2010]. A study conducted by Guo et al. at Microsoft for Windows Vista and Windows 7 [Guo et al. 2010] found that number of reassignments, organizational and geographical distribution, and reputation of bug opener influence the chances of bug reports getting fixed.
The findings in this section show that well-written bug reports help in comprehending the problem better, consequently increasing the likelihood of the bug getting fixed in a shorter time.