Bikeshedding in Software Engineering

A group of scientists sit down on a nice sunny day to examine a large tabular data set they have experimentally gathered. The first order of business is to figure out what program they should use for analysis; should they use Excel? OpenOffice? R? An online collaborative spreadsheet? Resolving the matter to everyone’s satisfaction takes 1 h. Next they tackle the column names: should they be all caps? Can they contain spaces? Should they be short, or should they be descriptive? Agreeing to a naming scheme takes another 3 h.

Throughout this time, the data is just sitting there. Row upon row. Waiting to give insight. But the volume is overwhelming; it is more comfortable to tackle issues that are easy to reason about and have less technical baggage. This phenomenon is known as “bikeshedding,” or Parkinson’s law of triviality. Rather than exploring the data itself, the meeting gets derailed by administrative minutiae. This phenomena is also related to “analysis paralysis,” whereby transitioning from talking about something to actually doing it is overwhelming and inhibits progress.

Summarizing Temporal Data

As in the preceding bikeshedding example, it is easy to be overwhelmed with the complexity of large datasets. Faced with a daunting data set (once of course it is appropriately formatted and named), we often want to build some form of summarization. Given the tabular form of the data, it is often tempting to count events or to build distributions because these are easy to think about; however, in data sets that are temporal in nature it is possible to find interesting relationships between data points by noting how they are related through time. One possible quick summarization approach that can often reveal interesting insights that are difficult to spot with a more tabular approach are state-transition diagrams.

To create a state-transition diagram, first identify the “states” that are possible (for instance, for a test execution you could have a pass, error, or fail; for a code review you could have the review request, a positive review, or a negative review). Next, iterate through the data and figure out at each point in time what state is appropriate in each instance in time. Finally, draw a state-transition diagram where each node is a state and edges represent transitions between the states. Annotate each edge with a count representing the number of times it occurred in the data. Two important things are often visible in these diagrams. First, the most dominant transitions are instantly visible and can help to guide future investigations. Second, unexpected edges are often present.

While any expected edge confirms your understanding of the data, unexpected edges provide insight into how your model of the data could be incorrect. These unexpected transitions are therefore a great place to start a qualitative investigation of the individual data points they represent and provide a concrete way to move from a summarization to an in-depth examination of specific rows in your data.

A concrete example is shown in the following figure. In this work, we compared code review practices between Webkit and Firefox. The possible states a code review can be in are shown in the rectangles. The shaded rectangles represent terminal states for a code review. The only distinction between states that are terminal and those that are transitory is that reviews do not transition out of terminal states; a patch that is updated and resubmitted for a new review is considered as if it were new. We next examined how the code reviews move between the states; the annotations on the edges describe these. A request review is annotated with an r?, while a positive review is r + and a negative review is r −. The Firefox project also uses a super-review for some patches, which accounts for the sr + and sr − edges. The percentage labels correspond to number of patches flowing across each state transition divided by the total number of total starting edges (review requests, in this case). For example, 7.9% of the 34,749 patches in Webkit (or 2745) were resubmitted for additional review, even though they had been accepted.

By examining the summarization of the tens of thousands of data points provided by the state-transition diagram, we can see some huge differences between the two projects. The most notable of these is the edge between submitted and resubmitted for Webkit; while at first we thought this was an error in our data, it gave us a concrete question to approach the Webkit developers about and greatly contributed to our understanding of the Webkit code review process (it turns out Webkit runs automated analyses on patches; if an analysis turns up a problem, the developer will “obsolete” their own patch and resubmit a new one).

Recommendations

(1) Data can be overwhelming; while tabular summarization is often tempting, this can abstract away important relationships between data points.

(2) The flows between implicit states in the data are often hard to identify and can provide tremendous insight into what the data represents.

(3) Starting a qualitative investigation from events captured by unexpected transitions often highlights conceptual misunderstandings which can be used to refine and improve the initial model or suggest future research ideas.