A key performance indicator (KPI) is a performance measurement used in most organizations to evaluate an organization’s success or the success of a particular activity within the organization. Often, KPIs are used to measure the effects of a change project—for instance, implementing a good DevOps process—or to evaluate the progress of a development project.
You can use the score from a DevOps online assessment as a KPI and compare the assessment scores before and after the implementation of a DevOps process improvement. In this way, you get an indication of whether you have improved as a result of implementing a new process.
During projects, you should also be able to use the reports from your DevOps toolset to determine whether you’re constantly improving your work. Continuous improvement, in my opinion, is something to strive for. When it comes to project management, you can, for instance, look at the team’s velocity (how fast the team is able to work) and determine whether it’s increasing or decreasing. By using reports and metrics from your DevOps tools, you can choose the KPIs you want and learn how to evaluate them.
Project management
Architecture, analysis, and design
Developer practices
Software testing
Release management
Keep one thing in mind while you read about some of the reports. Reports are not available in Azure DevOps—only in TFS 2018. For Azure DevOps, I encourage you to use charts instead to get the information you need. Charts are described later in this chapter.
Some reports require that the team project collection that contains your team project was provisioned with SQL Server Reporting Services. The report is not available if Reports does not appear when you open Team Explorer and expand your team project node.
Metrics for Project Management
To get good metrics about the status of your projects, it’s important to measure your progress. You can do this in several ways. If you’re using Agile as a methodology, many of these metrics and reports should be familiar. To others, they may be new. Keep in mind that not all of these reports are available in Azure DevOps, but only in TFS on-prem.
Agile Metrics
Backlog overview
Sprint burndown
Velocity report
Release burndown
Remaining work
Unplanned work
Velocity (how much work a team can take on in a sprint) is important, especially for a product owner planning how much work can be accomplished in coming sprints. Velocity is usually a measure of the effect per story point that the team can accomplish.
Metrics for Architecture, Analysis, and Design
Lines of code: Presents an approximate number based on Intermediate Language (IL) code. A high count may indicate that a type or method is doing too much work and should be split up. This may also be a warning that code will be hard to maintain.
Class coupling: Measures the coupling to unique classes through parameters, local variables, return types, method calls, generic or template instantiations, base classes, interface implementations, fields defined on external types, and attribute decoration. Strive for low coupling. High coupling indicates a design that is difficult to reuse and maintain because of its many interdependencies on other types.
Depth of inheritance : Indicates the number of class definitions that extend to the root of the class hierarchy. The deeper the hierarchy, the more difficult it may be to understand where particular methods and fields are defined and/or redefined.
Cyclomatic complexity : Is determined by calculating the number of different code paths in the flow of the program. It indicates the code’s complexity. A high complexity makes maintainability suffer, and it can also be hard to get good code coverage.
Maintainability index: Is an index value between 0 and 100 that represents the relative ease of maintaining the code. The higher the better; a rating of more than 60 is good. Less than that, maintainability suffers.
Circular references are nodes that have circular dependencies on one another.
Hubs are nodes that are in the top 25% of highly connected nodes.
Unreferenced nodes have no references from any other nodes.
Using these analyzers, you can determine whether you have loops or circular dependencies so that you can simplify them or break the cycles. You also can determine whether you have too many dependencies, which could be a sign that they’re performing too many functions. To make the code easier to maintain, test, change, and perhaps reuse, you need to look into whether you should refactor these code areas to make them more defined. You may also be able to find code that performs similar functionality and merge with it. If the code has no dependencies, you should reconsider keeping it.
Metrics for Developer Practices
Metrics for developer practices are KPIs that can help you understand if you’re working successfully to improve your code. These measures are useful from both the architectural and design viewpoints as well as from a developer viewpoint. Using them helps you improve how you design your application or system.
Code coverage
Code metrics
Compiler warnings
Code analysis warnings
Code Coverage
Code coverage shows you how much of the code has been covered by automated unit tests. You get the value as a percentage of the entire code base. The difficulty often is deciding what percentage is enough. Should you always strive for 100%? Or is 80% enough? This is something the team has to discuss with the product owner when using Scrum or a similar decision maker when using other processes. This value is input as the definition of done.
Code Metrics
You can look at several different code metrics. Lines of code, class coupling, depth of inheritance, cyclomatic complexity, and the maintainability index—as described in the earlier section “Metrics for Architecture, Analysis, and Design”—also apply to code metrics.
Compiler Warnings
Errors and warnings should be avoided in a project. Allowing more than zero errors or warnings tends to result in the team accepting lower quality in the code base, which over time causes the code to lose maintainability.
Track this metric to make sure the number of errors is zero. Ideally, this should be enforced by automatic build policies.
Code Analysis Warnings
Code analysis in development tools performs static analysis on code, which helps developers identify potential design, globalization, interoperability, performance, and security problems, to name a few. Much of this functionality is currently available only for .NET development; if you’re using Java, things may be different.
Code analysis tools provide warnings that indicate rule violations in managed code libraries. The warnings are organized into rule areas such as design, localization, performance, and security. Each warning signifies a violation of a code analysis rule.
Code analysis can be used to enforce company policies on the code developers write. Many DevOps tools offer good support for code analysis, and usually include a set of rules. Often, you can extend the functionality by writing your own rule set or by suppressing the rules you don’t want. Definitely discuss this analysis with your development team and the product owner, because the warnings have an impact on the effort required before the definition of done is fulfilled.
Metrics for Software Testing
Number of bugs per state: This metric tells you how many bugs are active, resolved, or closed; whether the number of active bugs is increasing and whether the number of resolved and closed bugs is constant. If the numbers stay constant, you need to look into how you perform your testing.
Number of bugs sent back from testers for more information (aka “reactivated bugs”): A large number of reactivated bus may indicate that communication between developers and testers must improve.
Code coverage: This metric shows how much of the code has been covered by automated unit tests. You get the value as a percentage of the entire code base.
Test run results: This metric indicates how your tests performing, and whether you have many failed tests. If you do, you need to look at what can be done to improve the tests.
Percentage of requirements covered by test cases: As the title implies, this metric indicates the percentage of requirements covered by test cases.
Percentage of requirements covered by testing: Do you actually verify the requirements with the test cases you have? If this figure is low and the figure for percentage of requirements covered by test cases is high, you may have an issue you need to deal with.
Example Reports
Bug status reports
Reactivations reports
Bug trend reports
Bug Status Report
Figure 7-8 shows the number of bugs over time. You can see how the numbers of active, closed, and resolved bugs change. In this case, the number of active bugs is decreasing and the number of closed and resolved bugs is increasing, leading to a point where the number of active bugs is zero.
Figure 7-9 shows a report that displays how many bugs are assigned to an individual user. You can also see the priority of each bug as well as how many bugs have been resolved by the users.
Reactivations Report
Bug Trend Report
Metrics for Release Management
Number of software defects in production [the number of bugs or software defects of applications (versions) that are in production]
Percentage of successful software upgrades (excludes full installations)
Number of untested releases (not tested and signed off)
Number of urgent releases
Average costs of release, for which costs are most likely based on man-hours spent
Note ITIL is a set of practices for ITSM that focuses on aligning IT services with the needs of business. ITIL describes procedures, tasks, and checklists that aren’t organization specific but are used by organizations to establish a minimum level of competency. ITIL allows an organization to establish a baseline from which it can plan, implement, and measure. ITIL is used to demonstrate compliance and to measure improvement.
Example Reports
Build quality indicators
Build success over time
Build summary report
Active bugs: The number of active bugs that exist at the time of the build
Code churn: The number of lines of code that have been added, removed, and changed in the check-ins before the build
Code coverage: The percentage of code covered by tests
Inconclusive tests: The number of tests that didn’t succeed or were paused. If the build didn’t succeed, the tests are either not counted or counted as inconclusive.
Failed tests: The number of tests that failed during the build
Passed tests: The number of tests that passed during the build
These metrics are suggestions that you can use as a base for following up on progress and quality in your projects. Different DevOps tools offer different possibilities for reporting and collecting information. Thus, it’s important that you think through what you want for your organization when choosing an DevOps platform.
Using Charts to Monitor Metrics
With Azure DevOps, you can add charts for work item queries to the dashboard from the widget catalog. These charts are configurable. You can choose a query, a chart type, and visualization options while staying in the dashboard context. Chart types include pie, bar, column, stacked bar, line, area, and stacked area. You can also display data in a pivot table. And, you can still add charts from the Charts tab in the work hub and configure them just like you’ve always done.
Summary
Metrics and KPIs are valuable for any organization if it wants to evaluate progress and quality. This chapter showed some examples of metrics used with Agile projects, but they are not limited to these projects. Metrics help you run your projects more efficiently, with greater application quality as an end result.
Keep in mind that different organizations find different metrics valuable. This chapter gave examples of metrics that are commonly used, but there may be others that are better suited for you.
The next chapter walks you through a scenario in which you implement Agile processes on different levels in a software development project.