Prologue: We Have Solid Empirical Evidence!

In recent years, empirical research in software engineering has gained a lot of maturity. This is in part thanks to the availability of a large amount of data from different kinds of software repositories, as well as of techniques to mine them, eg, for tracing data from heterogeneous sources, summarizing changes, or mining unstructured and semistructured data sources. Noticeably, the analysis of data from software repositories has also gained a lot of attention in industry, where developers rely on software data to make important decisions. The roles, competencies, and working styles of data scientists in software development has been recently investigated by Kim et al. [1].

Suppose we are software engineering researchers and we have developed a new recommendation tool that identifies certain kinds of code’s bad smells. We want to convince practitioners to adopt our tool. Also, suppose that, by using data from software repositories, and by using appropriate statistical methods, we are able to show that our new code smell detector achieves 90% precision and 60% recall, and that classes affected by such smells tend to be significantly more defect-prone than others. Hopefully, such findings can be useful to provide practitioners with convincing arguments about the performance of our novel tool and its potential usefulness. Is this sufficient to convince them to adopt the tool? Does it show that detecting code’s bad smells is relevant and useful? What are we missing?

Correlation is Not Causation and, Even If We Can Claim Causation…

No matter how we carefully conduct our (quantitative) empirical study, it would not tell us the whole story. Let us assume that, by doing a quantitative study, we know that our tool is very precise and has a good recall, better than an alternative tool. Let us also assume that we conduct a controlled experiment to study the relation between smells and code defect-proneness. In a controlled experiment, we observe one or more dependent variables (in our case defect-proneness) while varying the level of some independent variables (eg, separating classes containing bad smells from others) and controlling others (eg, determining whether the class size could influence our results). Results of the experiments show that classes containing code smells identified by our tools are more fault-prone than others. However, this does not tell us the whole story.

Specifically, we still do not know why our recommender worked better (or worse) than the baseline, how developers have used it during their tasks, and whether there are clear directions for improvement. More than that, there are problems in claiming the causation of our findings. Imagine our tool is able to highlight potentially defect-prone classes based on the presence of some code smells, and it seems to exhibit a very good precision and recall. Is the presence of code smell really what causes defects? Or, maybe, defective code is subject to more patches and then becomes “smelly”? Or, perhaps a specific module of our projects is just too complicated to become smelly and defect-prone? While in some cases, a purely quantitative analysis may have results sufficient to determine the direction of a causality relation (eg, smells are always introduced in the code before defects occurs, therefore we could claim that smells may cause defects and not vice versa, ie, defect fixes make source code smelly). In some other cases this is not possible, either because it is not possible to determine a temporal precedence between two phenomena, or because causation depends on factors we are ignoring.

Other than that, quantitative findings might tell us how good our approach is, but tells us little on how to improve it. Why is my smell detector producing so many false positives, especially when applied on certain (sub)systems? And, why does it fail to detect some true smells?

In some other circumstances, it can happen that a quantitative study does not show the effect of a factor we are not observing. A typical example is a study by Bird et al. [2] on the effect of developers’ geographical distribution. Quantitative results showed a limited effect of geographical distribution, though developers suggest this is because of countermeasures they have taken (not observable in the study).

What are the recipes to properly complement the gained quantitative evidence? In short, you should try to complement quantitative findings with qualitative ones. While, as exemplified herein, quantitative studies will provide you with empirical evidence in terms of numerical relations between variables, qualitative studies will help you to interpret and explain a phenomenon by observing it in its context. This is often achieved through the analysis of people observations, interviews, and documents [3].

Collect Your Data: People and Artifacts

To complement our quantitative findings with qualitative insights, we can rely on at least two precious sources of information: (i) artifacts contained in the software repositories and (ii) the projects’ contributors.

Build a Theory Upon Your Data

Once data has been collected, the purpose of qualitative analysis is to provide an interpretation to it and, ideally, build a theory. To this aim, techniques inspired to grounded theory [6], such as open coding, are particularly suitable.

Open coding starts with the identification of keywords/regularities in the observed artifacts, and in the consequent creation of categories, that should be internally cohesive and consistent, and overall inclusive of the investigated reality. The open coding process should involve multiple and independent coders to limit subjectiveness, eg, one person performs the coding, the other replicates it, and then coding reliability is assessed through measures such as the Krippendorff’s Alpha [7]. Multiple iterations of the process aim to merge, split, or remove categories. Such categories, their relationships, and the concepts abstracted from them allow you to answer questions such as:

• What do developers discuss before fixing a bug? How do discussions converge to a solution?

• What kinds of bugs occur in files affected by code smells?

• What are the typical problems developers encountered when using our new recommender system? What are the main suggestions for improvement?

Conclusion: The Truth is Out There!

Studying a software process/product, or evaluating a novel approach/tool requires multiple, and different kinds of, evaluations. On one hand, quantitative studies have different scales, different levels of control, and different risks, eg, surveys, case studies, and controlled experiments, as well as their replication to increase statistical power and external validity of results. On the other hand, combining such quantitative studies with qualitative ones will help provide convincing arguments of when and why a prediction model or a recommender system works (or does not work).

In summary, information is there, but you need to look at it from different perspectives: quantitative (numbers) and qualitative (concepts).

Suggested Readings

Seaman [8] has discussed the application of qualitative methods in software engineering, and specifically about observing/interviewing developers, and complementing this with coding and theory building. Quinn Patton [5] provides a thorough overview of qualitative methods—mainly in the context of social science application. Last, but not least, a “must read” is the seminal work on grounded theory research [6].

Some chapters of this book specifically focus on certain qualitative analysis techniques. In their chapter “Avoiding Survey Design Traps: A Successful Process for Sailing Past the Temptations of Persephone’s Sirens” Barik and Murphy-Hill provide hints on how to conduct survey research. In his chapter “Card-sorting: From Text To Themes” Zimmermann illustrates the card-sorting method that can be used to analyze open-text survey responses. Menzies’s chapter “Correlation is not Causation” explains, through simple examples, the difference between claiming correlation and causation.