Metrics and ALM Assessment for Software Testing
Software testing is an important area. Testing should be a constant part of any development and not only a phase at the end of the project. Microsoft’s online ALM assessment can help us to greatly improve our testing practices. There are also good metrics we can use during our projects to make sure we have high-quality testing in place.
Metrics
There are a number of metrics to can use as KPIs for software testing.
- Number of bugs per state. How many bugs do we have that are active, resolved, or closed? Are the number of active bugs increasing and number of resolved and closed bugs constant? Then we need to look into how we perform our testing.
- Number of bugs sent back from testers for more information. A large number might indicate that communication between developers and testers must improve.
- Code coverage. This shows us how much of the code has been covered by our automated unit tests. We get the value as a percentage of the whole code.
- Tests run results. How are our tests performing? Do we have many failed tests? If so, what can we do to improve this?
- Percent Requirements covered by test cases. Do we write test cases for all our requirements? If not, what is the reason?
- Percent Requirements covered by testing. Do we actually run the tests we have test cases for? If this figure is low and the figure for Percent Requirements covered by test cases is high we might have an issue we need to deal with.
Standard Reports
The metrics we get in our report concerning testing can be really helpful in our projects. The reports described here are only available in the two MSF process templates and not in the Scrum template. Let’s take a brief look at the reports:
- Bug status (Agile, CMMI)
- Reactivations (Agile, CMMI)
- Bug trend (Agile, CMMI)
- Test Case Readiness (all)
- Test Plan Progress (all)
Bug Status Report
The first covered is the Bug status report. This report gives us information about the cumulative bug count based on bug state, priority, who it is assigned to, and of course, the bug severity. We see the number of bugs and the number of resolved bugs (see Figure 24-1).
Figure 24-1. Bug status report
Reactivations Report
The Reactivations report (see Figure 24-2) is used to see how many bugs have been resolved or closed too early. If the bug needs to be opened again it is called a reactivation. A high number indicates that the developers can improve their bug fixing process and not close or resolve the bugs unless they really are ready to be closed.
Figure 24-2. Reactivations report
Bug Trend Report
Next is the Bug trend report (see Figure 24-3). This report helps us track the rate at which our team is finding, resolving, and closing bugs.
Figure 24-3. Bug trend report
Test Case Readiness Report
A test case can have two states; Design or Ready. When the test case is created the state is set to Design and after the team has reviewed and approved the test case, it is set to Ready. The Test Case Readiness report (see Figure 24-4) provides an area graph that shows how many test cases are in the Design or Ready state over a time period that we specify. By reviewing this data, we can easily determine how quickly our team is designing test cases and making them ready for testing. The value of this report is perhaps questionable. What does it really say to us? When would we need this information? Decide for yourself if this is something you need in your projects.
Figure 24-4. Test case readiness report
Test Plan Progress Report
After the team creates test plans and starts to run tests by using Microsoft Test Manager, we can use the Test Plan Progress report (see Figure 24-5) to track the team’s progress in testing the product. We will get information about the number of test cases that passed, how many failed, and the number of inconclusive, blocked, or never run tests among other things. This information can be good input to a project manager in conjunction with the code coverage metrics described in Chapter 14. We can also see this report from inside Test Manager.
Figure 24-5. Test plan progress report
Note The reports in this section are mapped to the particular process template they support. If you find a report you like, it is possible to customize it to work with another template. In Chapter 32 we describe how to customize an existing report to do this and other things.
Custom Reporting
The reporting capabilities in TFS give us access to most of the information we manage in our ALM process. In the previous section we have seen how standard reports give us metrics for our project at a general level. By customizing, extending, and creating new reports we can really get the intelligence to know what works well in our projects and what does not.
Note In Chapter 32 we will look at the details of reporting in TFS, including how to create custom reports based on the data models described here.
Data Warehouse Model
Test Result Tables
To query about data for test results, test case association, and test outcome we can use the test result tables’ bugs (see Figure 24-6).
Figure 24-6. Test results data model
Test Run Coverage Tables
The test run coverage tables (see Figure 24-7) lets us report on code coverage for a test run.
Figure 24-7. Test run coverage data model
Also see the Work Item Test Results tables in Chapter 9, as well as the Build Details tables in Chapter 29 for more warehouse data useful for analyzing the testing process.
Assessment
To help us evaluate an organization’s maturity in different ALM areas Microsoft has developed their ALM assessments (see Chapter 4). Based on the score of the assessment we rreceived, a maturity level for a specific area that we can use for evaluating which direction to take our ALM efforts.
So based on the score, we can help the organization reach the maturity level they need for these areas. Table 24-1 list questions that can be used as a basis for an ALM assessment in the software testing area.
Table 24-1. ALM Assessment Questions
| Area | Sample Question | Discussion |
|---|---|---|
| Test Resource Management | Is there a dedicated test lead in place? | |
| Are there appropriate tools available to perform automated testing? | TFS will help with this. | |
| Is there a management system in place to track work items, defects, and change requests? | Here we have another good opportunity to show the benefits of TFS. | |
| Test Planning | Is a test plan or strategy in place before starting testing? | This is something for the project manager or product owner to consider. We can always benefit from having and executing this plan. |
| Is the test team represented at the design review stage? | The test team should always be involved in our opinions. | |
| Do test plans consider integration testing with other systems and third-party products? | ||
| Test Management | Is test execution tracked against the test plan? | |
| Is the test plan followed? | ||
| Are the appropriate reporting processes in place? | TFS will definitely help us getting the metrics we need. | |
| Is the end-user or customer-acceptance criteria well defined and evaluated? | This should be included in the definition of done. | |
| Has testing the following non-functional requirements been taken into consideration where appropriate: performance, scalability, security, accessibility, regression, localization, and load/stress/soak? | This should be covered by the definition of done. | |
| Has code coverage been considered where appropriate? | Discuss with the customer what level of code coverage is necessary. TFS will help us implement the automated tests and give us statistics of code coverage. | |
| Test Types | Is User Acceptance Testing (UAT) used? | TFS includes good tools for handling UAT. The Test Manager is very helpful here. |
| Are automated UI tests performed? | Easy to implement using TFS. | |
| Are automated integration tests used? | Easy to implement using TFS. | |
| Any data generation tools? | Easy to implement using TFS. | |
| Any stress test tools? | Easy to implement using TFS. | |
| Any Performance Analysis tools? | Easy to implement using TFS. | |
| Database Testing | Do you have automated testing of your databases? | Don’t forget to include the database development in your testing. |
| Is there suitable test data to ensure application tests are valid? | ||
| Do you have a repeatable data set for testing? | We can use TFS and VS to accomplish this. This is a good developer and testing practice. |
Summary
In this chapter have seen how we can use TFS to retrieve information for KPI assessment and also how we can see the test status using standard reports from TFS.
We have also shown how many of the assessment questions from the Microsoft online assessment can help us plan for successful implementation of testing practices.
The next section of this book covers how we can use TFS to release our application.