As Simple as Possible… But not Simpler!

The first reasonable approach for unstructured software data summarization is to tap into the methods devised by the community of Information Retrieval, whose target is exactly to retrieve information from natural language documents, and to combine them with some basic visualization techniques.

So, let us say we want to summarize the email information produced in just a single discussion thread on the mailing list of an open-source software project. A common solution would be to collect all the terms present in those emails and display them in a tag cloud [1] where the most frequently occurring terms are represented in larger fonts and the less frequently occurring ones are represented in smaller fonts. The more visible a term is in the tag cloud, the greater should be its semantic value. Terms are defined as single words, divided by space or punctuation, or some special characters.

Let us take the content of the following email in Fig. 2, adapted from a real email sent to the mailing list of an open-source system (ArgoUML).

If we had to summarize the content, we could say that there are three people involved and the content refers to a bug affecting OS X, which is solved by patching the class “Explorer.” How good would a tag cloud represent this threaded email? If we create a tag cloud directly from it, we obtain the following (see Fig. 3).

The tag cloud already gives us some interesting information, but it also contains a lot of noise, which reduces the visibility of what really matters. Apparently, taking “off-the-shelf” methods from information retrieval and data visualization is quick and simple, but it is probably a “too simple” solution, not good enough to solve our problem.

You Need Structure!

Why did the preceding method not work well? Because most off-the-shelf techniques are prepared to work on a very specific kind of data. If we took newspaper articles and followed the same approach, we would probably have obtained more interesting results. But documents generated by software engineers are substantially different from those generated by writing professionals, such as newspaper journalists. Developers might use jargon more often and have a more terse language, and there is a lot of implicit knowledge not expressed. Then, emails contain text that is not relevant for analyzing their content, such as authors' signatures. Finally, and perhaps most important, software engineers often mix languages: natural language is mixed with source code, stack traces, and patches.

If we really want to extract useful information, which can be summarized, from unstructured software data, we first have to find the latent structure it has and exploit it correctly.

How can we give structure to a wall of text, such as an email? We first can realize that there is a simple underlying structure we can and should take advantage of, and build on top of it. We should take into account that threaded emails already give us information about how many people are involved in the discussion (the starting “>”s tell us about the indentation level). Then we can exploit a structure that is already there: emails are divided by authors in lines; our previous research showed that this is a very good starting point in finding the structure of a message [2]. Considering these two aspects, we obtain the following version of the same email (see Fig. 4), which visually already gives us much more information (this is what every good email client would do).

Once we have this basic structure, the next step is realizing that, in most of the cases, the different languages used in emails are used in different lines. In this way, we simplified our problem into a classification problem: how can we assign each line to the language it belongs to? From our email, we would like to have a categorization that looks like the following (see Fig. 5).

Is it possible to automatically tag each line with the language it is written in? Yes! Researchers developed a number of methods to classify text in different categories. A classical example is the case of classifying a whole email into “spam” or legitimate. In the case of development emails, we developed simple methods to recognize lines of code from other text [2] and more complex ones to recognize more complex languages, such as those used in signatures, from natural language content [3]. As an example, here we describe how the lines of Java source code can be recognized in the content of an email, with a very simple, yet effective approach, and see what its impact is on a final tag cloud summary.

If we consider lines 17–20 and 25–28, we note a peculiarity present in many programming languages (eg, Java, C, C#, Perl): the developer must end each statement with a semicolon or a curly bracket (mainly used to open or close a block). Based on this intuition, a simple approach that verifies whether the last character of a line is a semicolon or a curly bracket might be a good way to detect source code. Plus, we can write a simple regular expression to recognize the lines of a stack trace (9–11), as they have a very clear structure, without any nested blocks.

We tested this approach on thousands of emails from five Java source systems and we found that it is working well in practice and can be used as a basis for further data analysis on emails [2].

Now, if we take our initial email, we remove the signatures (by for example, eliminating everything at the end after the dashes) and apply the source code detection approach, we can generate a new tag cloud, as the one depicted below (see Fig. 6). Now we see that more important terms (such as “Explorer” and “NullPointerException”) started to emerge, thus creating a summary that gives a much better idea of the content.

With more sophisticated approaches [3], we can even parse the different parts of the content of a development email and remove more noise, thus making the most important content emerge and creating better summaries.