Calculating the Bug Count

Next we will calculate the total bug count by date. This will show how many new bugs are opened by day.

To do this, we pass bugs$Date into the table() function, which builds a data structure of counts of each date in the bugs data frame:

totalBugsByDate <- table(bugs$Date)

So the structure of totalBugsByDate looks like the following:

> totalBugsByDate
  
2013-01-04 2013-01-08 2013-01-09 2013-01-10 2013-01-14
 2 4 5 3 1

Let’s plot this data out to get an idea of how many bugs are opened each day:

plot(totalBugsByDate, type="l", main="New Bugs by Date", col="red", ylab="Bugs")

This code creates the chart shown in Figure 6-1.

Now that we have a count of how many bugs are generated each day, we can get a cumulative sum by using the cumsum() function. It takes the new bugs opened each day and creates a running sum of them, updating the total each day. It allows us to generate a trend line for the cumulative count of bugs over time:

> runningTotalBugs <- cumsum(totalBugsByDate)
>
> runningTotalBugs
01-04-2013 01-08-2013 01-09-2013 01-10-2013 01-14-2013 01-16-2013 2 6 11 14 15 17

This is exactly what we need to now plot out the way the bug backlog grows or shrinks each day. To do that, let’s pass runningTotalBugs to the plot() function. We set the type to "l" to signify that we are creating a line chart and then name the chart Cumulative Defects Over Time. In the plot() function, we also turn the axes off so that we can draw custom axes for this chart. We will want to draw custom axes so that we can specify the dates as the x-axis labels.

To draw custom axes, we use the axis() function. The first parameter in the axis() function is a number that tells R where to draw the axis:

• 1 corresponds to the x-axis at the bottom of the chart
• 2 to the left of the chart
• 3 to the top of the chart
• 4 to the right of the chart

plot(runningTotalBugs, type="l", xlab="", ylab="", pch=15, lty=1, col="red",
main="Cumulative Defects Over Time", axes=FALSE)
axis(1, at=1: length(runningTotalBugs), lab= row.names(totalBugsByDate))
axis(2, las=1, at=10*0:max(runningTotalBugs))

This code creates the time series chart shown in Figure 6-2.

This shows the progressively increasing bug backlog, by date.

The complete R code so far is as follows:

bugExport <- "/Applications/MAMP/htdocs/allbugs.csv"
bugs <- read.table(bugExport, header=TRUE, sep=",")
bugs <- bugs[order(as.Date(bugs$Date,"%m-%d-%Y")),]
  
sprintDates <- "/Applications/MAMP/htdocs/iterationdates.csv"
sprintInfo <- read.table(sprintDates, header=TRUE, sep=",")
#output ordered list to new flat file
write.table(bugs, col.names=TRUE, row.names=FALSE, file="/Applications/MAMP/htdocs/allbugsOrdered.csv", quote = FALSE, sep = ",")
totalBugsByDate <- table(factor(bugs$Date))
plot(totalBugsByDate, type="l", main="New Bugs by Date", col="red")
runningTotalBugs <- cumsum(totalBugsByDate)
plot(runningTotalBugs, type="l", xlab="", ylab="", pch=15, lty=1, col="red", main="Cumulative Defects Over Time", axes=FALSE)
axis(1, at=1: length(runningTotalBugs), lab= row.names(totalBugsByDate))
axis(2, las=1, at=10*0:max(runningTotalBugs))
  
write.table(totalBugsByDate, col.names=TRUE, row.names=FALSE, file="/Applications/MAMP/htdocs/runningTotalBugs.csv", quote = FALSE, sep = ",")

Let’s take a look at the criticality of the bugs, which shows not just when the bugs are opened but also when the most severe (or non-severe) bugs are being opened.

Examining the Severity of the Bugs

Remember that when we exported the bug data we included the Severity field, which indicates the level of criticality of each bug. Each team and organization might have their own classification of severity, but generally they include these:

• Blockers are bugs so severe that they prevent the launch of a body of work. They generally have broken functionality or are missing sections of a widely used feature. They can also be discrepancies with contractually or legally binding features such as closed captioning or digital rights protection.
• Criticals are bugs that are severe but not so damaging that they gate a release. They can have broken functionality of less-used features. The scope of accessibility, or how widely used a feature is, is usually a determining factor between making a bug a blocker or a critical.
• Minors are bugs with very minimal if any impact and might not even be noticeable to an end user.

To break out the bugs by severity, we simply call the table() function, just as we did to break out bugs out by date, but this time add in the Severity column as well:

bugsBySeverity <- table(factor(bugs$Date),bugs$Severity)

This code creates a data structure that looks like so:

 Blocker Critical Minor
 01-04-2013 0 0 2
 01-08-2013 1 0 3
 01-09-2013 3 0 2
 01-10-2013 1 0 2
 01-14-2013 0 0 1
 01-16-2013 1 0 1
 01-22-2013 1 0 1

We can then plot this data object. The way we do this is to use the plot() function to create a chart for one of the columns and then use the lines() function to draw lines on the chart for the remaining columns:

plot(bugsBySeverity[,3], type="l", xlab="", ylab="", pch=15, lty=1, col="orange", main="New Bugs by Severity and Date", axes=FALSE)
lines(bugsBySeverity[,1], type="l", col="red", lty=1)
lines(bugsBySeverity[,2], type="l", col="yellow", lty=1)
axis(1, at=1: length(runningTotalBugs), lab= row.names(totalBugsByDate))
axis(2, las=1, at=0:max(bugsBySeverity[,3]))
legend("topleft", inset=.01, title="Legend", colnames(bugsBySeverity), lty=c(1,1,1), col= c("red", "yellow", "orange"))

This code produces the chart shown in Figure 6-3.

This is great, but what if we want to see the cumulative bugs by severity? We can simply use the preceding R code, but instead of plotting out the columns, we can plot out the cumulative sum of each column:

plot(cumsum(bugsBySeverity[,3]), type="l", xlab="", ylab="", pch=15, lty=1, col="orange", main="Running Total of Bugs by Severity", axes=FALSE)
lines(cumsum(bugsBySeverity[,1]), type="l", col="red", lty=1)
lines(cumsum(bugsBySeverity[,2]), type="l", col="yellow", lty=1)
axis(1, at=1: length(runningTotalBugs), lab= row.names(totalBugsByDate))
axis(2, las=1, at=0:max(cumsum(bugsBySeverity[,3])))
legend("topleft", inset=.01, title="Legend", colnames(bugsBySeverity), lty=c(1,1,1), col= c("red", "yellow", "orange"))

This code produces the chart shown in Figure 6-4.

We posted all the R code discussed so far on RPubs at http://rpubs.com/tomjbarker/4169.

Reading in the Data

We add in some code to read in the data (the allbugsOrdered.csv file that was output from R earlier). Recall that this file contains the entire bug data ordered by date.

We use the d3.csv() function to read this file:

• The first parameter is the path to the file.
• The second parameter is the function to execute once the data is read in. It is in this anonymous function that we add most of the functionality, or at least the functionality that is dependent on having data to process.

The anonymous function accepts two parameters

• The first catches any errors that may occur
• The second is the contents of the file being read in

In the function, we first loop through the contents of the data and use the date formatter to convert all the values in the Date column to a native JavaScript Date object:

d3.csv("allbugsOrdered.csv", function(error, data) {
        data.forEach(function(d) {
        d.Date = parseDate(d.Date);
});
});

If we were to console.log() the data, it would be an array of objects that look like Figure 6-5.

Within the anonymous function but after the loop, we use the d3.nest() function to create a variable that holds the bug data grouped by date. We name this variable nested_data:

nested_data = d3.nest()
.key(function(d) { return d.Date; })
.entries(data);

The nested_data variable is now a tree structure—specifically a list that is indexed by date, and each index has a list of bugs. If we were to console.log() nested_data, it would be an array of objects that look like Figure 6-6.

Drawing on the Page

We are ready to start drawing to the page. So let’s step out of the callback function and go to the root of the script tag and write out the SVG tag to the page by using the margins, width, and height that defined previously:

var svg = d3.select("body").append("svg")
 .attr("width", width + margin.left + margin.right)
 .attr("height", height + margin.top + margin.bottom)
        .append("g")
 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

This is the container in which we draw the axes and the trend lines.

Still at the root level, we add a D3 scale object for both the x and y axes, using the width variable for the x-axis range and the height variable for the y-axis range. We add the x and y axes at the root level, passing in their respective scale objects, and orienting them at the bottom and left:

var xScale = d3.time.scale()
 .range([0, width]);
var yScale= d3.scale.linear()
 .range([height, 0]);
var xAxis = d3.svg.axis()
 .scale(xScale)
 .orient("bottom");
var yAxis = d3.svg.axis()
 .scale(yScale)
 .orient("left");

But they still aren’t showing on the page. We need to return to the anonymous function that we created in the d3.csv() call and add the nested_data list that we created as the domain data for the newly created scales:

xScale.domain(d3.extent(nested_data, function(d) { return new Date(d.key); }));
yScale.domain(d3.extent(nested_data, function(d) { return d.values.length; }));

From here, we need to generate the axes. We do this by adding and selecting an SVG g element, used for generic grouping, and adding this selection to the xAxis() and yAxis() D3 functions. This also goes in the anonymous callback function that gets invoked when the data is loaded.

We also need to transform the x-axis by adding the height of the chart so that it is drawn at the bottom of the graph.

svg.append("g")
 .attr("transform", "translate(0," + height + ")")
 .call(xAxis);
  
svg.append("g")
 .call(yAxis)

This creates the start of the chart with meaningful axes shown in Figure 6-7.

The trend line needs to be added. Back at the root level, let’s create a variable named line to be an SVG line. Assume for a minute that we have already set the data property for the line. We haven’t yet, but we will in a minute. For the x value of the line, we will have a function that returns the date filtered through the xScale scale object. For the y value of the line, we will create a function that returns the bug count values run through the yScale scale object.

var line = d3.svg.line()
 .x(function(d) { return xScale(new Date(d.key)); })
 .y(function(d) { return yScale(d.values.length); });

Next, we return to the anonymous function that processes the data. Right below the added axes, we will append an SVG path. We set the nested_data variable as the datum for the path and the newly created line object as the d attribute. For reference, the d attribute is where we specify path descriptions. See here for documentation around the d attribute: https://developer.mozilla.org/en-US/docs/SVG/Attribute/d.

svg.append("path")
 .datum(nested_data)
 .attr("d", line);

We can now start to see something in a browser. The code so far should look like so:

<!DOCTYPE html>
<head>
<meta charset="utf-8">
</head>
<body>
        <script src="d3.v3.js"></script>
<script>
var margin = {top: 20, right: 20, bottom: 30, left: 50},
 width = 960 - margin.left - margin.right,
 height = 500 - margin.top - margin.bottom;
  
var parseDate = d3.time.format("%m-%d-%Y").parse;
var xScale = d3.time.scale()
 .range([0, width]);
  
var yScale = d3.scale.linear()
 .range([height, 0]);
  
var xAxis = d3.svg.axis()
         .scale(xScale)
         .orient("bottom");
  
var yAxis = d3.svg.axis()
         .scale(yScale)
         .orient("left");
  
var line = d3.svg.line()
 .x(function(d) { return xScale(new Date(d.key)); })
        .y(function(d) { return yScale(d.values.length); });
  
var svg = d3.select("body").append("svg")
 .attr("width", width + margin.left + margin.right)
 .attr("height", height + margin.top + margin.bottom)
        .append("g")
 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
  
d3.csv("allbugsOrdered.csv", function(error, data) {
        data.forEach(function(d) {
                d.Date = parseDate(d.Date);
        });
  
 nested_data = d3.nest()
                .key(function(d) { return d.Date; })
                .entries(data);
  
         xScale.domain(d3.extent(nested_data, function(d) { return new Date(d.key); }));
         yScale.domain(d3.extent(nested_data, function(d) { return d.values.length; }));
         svg.append("g")
         .attr("transform", "translate(0," + height + ")")
         .call(xAxis);
                svg.append("g")
                 .call(yAxis);
                svg.append("path")
                 .datum(nested_data)
                 .attr("d", line);
  
});
  
</script>
</body>
</html>

This code produces the graphic shown in Figure 6-8.

But this isn’t quite right. The shading of the path is based on the browser’s best guess of intent, shading what it perceives to be the closed areas. Let’s use CSS to explicitly turn off shading and instead set the color and width of the path line:

<style>
.trendLine {
 fill: none;
 stroke: #CC0000;
 stroke-width: 1.5px;
}
</style>

We created a style rule for any element on the page with the class trendLine. Let’s next add the class to the SVG path in the same block of code in which we create the path:

Svg.append("path")
 .datum(nested_data)
 .attr("d", line)
 .attr("class", "trendLine");

This code produces the chart shown in Figure 6-9.

Looking much better! There are some minor things we should change, such as adding text labels to the y-axis and trimming the width of the axis lines to make them neater:

.axis path{
 fill: none;
 stroke: #000;
 shape-rendering: crispEdges;
}

This will give us tighter-looking axes. We just need to apply the style to the axes when we create them:

svg.append("g")
 .attr("transform", "translate(0," + height + ")")
 .call(xAxis)
 .attr("class", "axis");
svg.append("g")
 .call(yAxis)
 .attr("class", "axis");

The results can be seen in Figure 6-10.

This is great so far, but it shows no real benefit from doing this in R. In fact, we wrote quite a bit of additional code just to get parity and didn’t even do any data cleaning that we did in R.

The real benefit of using D3 is adding interactivity.

Adding Interactivity

Say we have this time series of new bugs, and we were curious what the bugs were in that large spike in mid-February. By taking advantage of the fact that we are working in HTML and JavaScript, we can extend this functionality by adding in a tooltip box that lists the bugs for each date.

To do this, we first should create obvious areas in which users can mouse over, such as red circles at each data point or discrete date. To do that, we simply need to create SVG circles right below where we added in the path, in the anonymous function that is fired when the external data is read in. We set the nested_data variable as the data attribute of the circles, make them red with a radius of 3.5, and set their x and y attributes to be tied to the date and bug totals, respectively:

svg.selectAll("circle")
.data(nested_data)
.enter().append("circle")
        .attr("r", 3.5)
        .attr("fill", "red")
        .attr("cx", function(d) { return xScale(new Date(d.key)); })
        .attr("cy", function(d) { return yScale(d.values.length);})

This code updates the existing time series so it looks like Figure 6-11. These red circles are now areas of focus in which users can mouse over and see additional information.

Let’s next code up a div to act as the tooltip that we will show with relevant bug data. To do this we will create a new div, right below where we created the line variable at the root of the script tag. We do this in D3 once again by selecting the body tag and appending a div to it, giving it a class and id of tooltip – both so that we can have the tooltip style apply to it (which we will create in just a minute) and so we can interact with it by ID later on in the chapter. We will have it be hidden by default. We will store a reference to this div in a variable that we will call tooltip:

var tooltip = d3.select("body")
 .append("div")
 .attr("class", "tooltip")
 .attr("id", "tooltip")
 .style("position", "absolute")
 .style("z-index", "10")
 .style("visibility", "hidden");

We next need to style this div using CSS. We adjust the opacity to be only 75 percent visible, so that when the tooltip shows up over a trend line we can see the trend line behind it. We align the text, set the font size, make the div have a white background, and give it rounded corners:

.tooltip{
        opacity: .75;
        text-align:center;
        font-size:12px;
        width:100px;
        padding:5px;
        border:1px solid #a8b6ba;
        background-color:#fff;
        margin-bottom:5px;
        border-radius: 19px;
        -moz-border-radius: 19px;
        -webkit-border-radius: 19px;
}

We next have to add a mouseover event handler to the circles to populate the tooltip with information and unhide the tooltip. To do this, we return to the block of code in which we created the circles and add in a mousemove event handler that fires off an anonymous function.

Inside the anonymous function, we overwrite the innerHTML of the tooltip to display the date of the current red circle and how many bugs are associated with that date. We then loop through that list of bugs and write out the ID of each bug.

svg.selectAll("circle")
 .data(nested_data)
 .enter().append("circle")
 .attr("r", 3.5)
 .attr("fill", "red")
 .attr("cx", function(d) { return xScale(new Date(d.key)); })
 .attr("cy", function(d) { return yScale(d.values.length);})
 .on("mouseover", function(d){
 document.getElementById("tooltip").innerHTML = d.key + " " + d.values.length + " bugs<br/>";
 for(x=0;x<d.values.length;x++){
 document.getElementById("tooltip").innerHTML += d.values[x].ID + "<br/>";
 }
 tooltip.style("visibility", "visible");
 })

If we want to take this even further, we can create links for each bug ID that link back to the bug tracking software; list descriptions of each bug; and if the bug tracking software has an API to interface with, we can even have form fields that could let us update bug information right from this tooltip. Only our imagination and the tools available to us limit the possibilities of how far we can extend this concept.

Finally, we add a mousemove event handler to the red circles so that we can reposition the tooltip contextually whenever the users mouse over a red circle. To do this, we use the d3.mouse object to get the current mouse coordinates. We use these coordinates to simply reposition the tooltip with CSS. So we don’t cover the red circle with the tooltip, we offset the top property by 25 pixels and the left property by 75 pixels.

svg.selectAll("circle")
 .data(nested_data)
 .enter().append("circle")
 .attr("r", 3.5)
 .attr("fill", "red")
 .attr("cx", function(d) { return xScale(new Date(d.key)); })
 .attr("cy", function(d) { return yScale(d.values.length);})
 .on("mouseover", function(d){
 document.getElementById("tooltip").innerHTML = d.key + " " + d.values.length + " bugs<br/>";
 for(x=0;x<d.values.length;x++){
 document.getElementById("tooltip").innerHTML += d.values[x].ID + "<br/>";
 }
 tooltip.style("visibility", "visible");
 })
 .on("mousemove", function(){
 return tooltip.style("top", (d3.mouse(this)[1] + 25)+"px").style("left", (d3.mouse(this)[0] + 70)+"px");
 });

A tooltip should display when the mouse hovers over one of the red circles (see Figure 6-12).

The complete source code should now look like this:

<!DOCTYPE html>
<html>
<meta charset="utf-8">
<head>
<style>
body {
 font: 15px sans-serif;
}
.trendLine {
 fill: none;
 stroke: #CC0000;
 stroke-width: 1.5px;
}
.axis path{
 fill: none;
 stroke: #000;
 shape-rendering: crispEdges;
}
.tooltip{
        opacity: .75;
        text-align:center;
        font-size:12px;
        width:100px;
        padding:5px;
        border:1px solid #a8b6ba;
        background-color:#fff;
        margin-bottom:5px;
        border-radius: 19px;
        -moz-border-radius: 19px;
        -webkit-border-radius: 19px;
}
</style>
</head>
<body>
        <script src="d3.v3.js"></script>
<script>
var margin = {top: 20, right: 20, bottom: 30, left: 50},
 width = 960 - margin.left - margin.right,
 height = 500 - margin.top - margin.bottom;
  
var parseDate = d3.time.format("%m-%d-%Y").parse;
var xScale = d3.time.scale()
 .range([0, width]);
var yScale = d3.scale.linear()
 .range([height, 0]);
var xAxis = d3.svg.axis()
         .scale(xScale)
         .orient("bottom");
var yAxis = d3.svg.axis()
         .scale(yScale)
         .orient("left");
var line = d3.svg.line()
 .x(function(d) { return xScale(new Date(d.key)); })
 .y(function(d) { return yScale(d.values.length); });
var tooltip = d3.select("body")
 .append("div")
 .attr("class", "tooltip")
 .attr("id", "tooltip")
 .style("position", "absolute")
 .style("z-index", "10")
 .style("visibility", "hidden");
  
var svg = d3.select("body").append("svg")
 .attr("width", width + margin.left + margin.right)
 .attr("height", height + margin.top + margin.bottom)
        .append("g")
 .attr("transform", "translate(" + margin.left + "," + margin.top + ")");
  
d3.csv("allbugsOrdered.csv", function(error, data) {
        data.forEach(function(d) {
                d.Date = parseDate(d.Date);
        });
  
 nested_data = d3.nest()
                .key(function(d) { return d.Date; })
                .entries(data);
  
         xScale.domain(d3.extent(nested_data, function(d) { return new Date(d.key); }));
         yScale.domain(d3.extent(nested_data, function(d) { return d.values.length; }));
         svg.append("g")
         .attr("transform", "translate(0," + height + ")")
         .call(xAxis)
         .attr("class", "axis");
  
                svg.append("g")
                 .call(yAxis)
         .attr("class", "axis");
                svg.append("path")
                 .datum(nested_data)
                 .attr("d", line)
         .attr("class", "trendLine");
                svg.selectAll("circle")
                 .data(nested_data)
                 .enter().append("circle")
                 .attr("r", 3.5)
                 .attr("fill", "red")
                 .attr("cx", function(d) { return xScale(new Date(d.key)); })
                 .attr("cy", function(d) { return yScale(d.values.length);})
                         .on("mouseover", function(d){
                                 document.getElementById("tooltip").innerHTML = d.key + " " + d.values.length + " bugs<br/>";
                                 for(x=0;x<d.values.length;x++){
                                         document.getElementById("tooltip").innerHTML += d.values[x].ID + "<br/>";
                               }
                                tooltip.style("visibility", "visible");
                })
                        .on("mousemove", function(){
                                return tooltip.style("top", (d3.mouse(this)[1] + 25)+"px").style("left", (d3.mouse(this)[0] + 70)+"px");
 });
});
</script>
</body>
</html>