Through an online survey, we asked over 150 developers from three large and successful open source projects—Apache, Eclipse, and Mozilla—what information in bug reports they consider valuable and helpful when resolving bugs. We also contacted bug reporters from the same projects to find out what information they provide and what information is most difficult to provide.
Our online survey is presented in Table 24-1. The survey sent out to developers comprised four questions in two categories:
- Contents of bug reports
(D1) Which items have developers previously used when fixing bugs?
(D2) Which three items helped the most?
Insight into this issue can help develop guides or tools for reporters to provide information in bug reports that focuses on the details that are most important to developers. We provided the developers with 16 items to choose from, some selected from the Mozilla bug-writing guidelines[32] and others found as standard fields in the Bugzilla database. Developers were free to check as many items as they liked for the first question (D1), but at most three for the second question (D2), thus indicating the importance of the choices in their perspectives.
- Problems with bug reports
(D3) Which problems have developers encountered when fixing bugs?
(D4) Which three problems caused the most delay in fixing bugs?
Our motivation for these questions was to find the prominent obstacles faced by developers that can be tackled in the future by more cautious, and perhaps even automated, bug reporting. Typical problems include reporters accidentally providing incorrect information, such as the wrong operating system.
Other problems in bug reports include poor use of language (ambiguity), bug duplicates, and incomplete information. Spam recently has become a problem, especially for the TRAC issue tracking system. We decided not to include the problem of incorrect assignments to developers, because bug reporters have little influence on the triaging of bugs. In total, we provided 21 problems that developers could select. Again, they were free to check as many items for the third question (D3), but at most three for the fourth question (D4). For a complete list of items, see Table 24-1.
We asked bug reporters the following three questions, divided into two categories (again, see Table 24-1):
- Contents of bug reports
(R1) Which items have reporters previously provided?
(R2) Which three items were most difficult to provide?
We offered the same 16 items to reporters that we offered to developers. This allowed us to check whether the information provided by reporters is in line with what developers frequently use or consider to be important (by comparing the responses for R1 with D1 and D2). The second question helped us to identify items that are difficult to collect and for which better tools might support reporters in this task. Reporters were free to check as many items as they wanted for the first question (R1), but at most three for the second question (R2).
- Contents considered to be relevant
(R3) Which three items do reporters consider to be most relevant for developers?
Again we listed the same items to see how much reporters agree with developers (comparing R3 with D2). For this question (R3), reporters were free to check at most three items, but could choose any item, regardless of whether they selected it for question R1.
Additionally, we asked both developers and reporters about their thoughts and experiences with respect to bug reports (D5 and R4).
Table 24-1. The questionnaire presented to Apache, Eclipse, and Mozilla developers (Dx) and reporters (Rx)
| Contents of bug reports. | D1: Which of the following items have you previously used when fixing bugs? | |||
| D2: Which three items helped you the most? | ||||
| R1: Which of the following items have you previously provided when reporting bugs? | ||||
| R2: Which three items were the most difficult to provide? | ||||
| R3: In your opinion, which three items are most relevant for developers when fixing bugs? | ||||
| ▢ product | ▢ hardware | ▢ observed behaviour | ▢ screenshots | |
| ▢ component | ▢ operating system | ▢ expected behaviour | ▢ code examples | |
| ▢ version | ▢ summary | ▢ steps to reproduce | ▢ error reports | |
| ▢ severity | ▢ build information | ▢ stack traces | ▢ testcases | |
| Problems with bug reports. | D3: Which of the following problems have you encountered when fixing bugs? | |||
| D4: Which three problems caused you most delay in fixing bugs? | ||||
| You were given: | There were errors in: | The reporter used: | Others: | |
| ▢ product name | ▢ code examples | ▢ bad grammar | ▢ duplicates | |
| ▢ component name | ▢ steps to reproduce | ▢ unstructured text | ▢ spam | |
| ▢ version number | ▢ test cases | ▢ prose text | ▢ incomplete information | |
| ▢ hardware | ▢ stack traces | ▢ too long text | ▢ viruses/worms | |
| ▢ operating system | ▢ non-technical language | |||
| ▢ observed behaviour | ▢ no spell check | |||
| ▢ expected behaviour | ||||
| Comments. | D5/R4: Please feel free to share any interesting thoughts or experiences. | |||
[32] The Mozilla bug-writing guidelines describe principles of effective bug reporting (e.g., be precise and clear, one bug per report) and list information that is essential to every bug report, such as steps to reproduce it, actual results, and expected results [Goldberg 2010].