122 ◾ Simple Statistical Methods for Software Engineering
Percentage of Bad Fix
is is a subprocess metric, governing the quality of defect resolution. Bad fixes con-
tains technical risk in testing, besides fuelling rework cost. In addition to poor resolu-
tions, fixing an existing bug can sometimes create additional bugs or expose other bugs.
Product Metrics
e ultimate purpose of testing is to evaluate and measure the quality of the soft-
ware under test (SUT), hence the importance of product metrics in testing.
Defect Counts
Defect Arrival Rate
e number of defects detected per day is the defect arrival rate. e cumulative
number of defects found can be plotted as a pattern.
Defect Closure Rate
e number of defects closed per day is the defect closure rate. e cumulative plot
of this number is the defect closure pattern. Figure 8.2 shows the closure pattern
together with the arrival pattern. e gap between the two patterns has a self-
evident meaning. e figure shows the nearly completed detection process while
closure seems to be ongoing.
Program testing can be used to show the presence of
bugs, but never their absence.
Dijkstra
Some organizations have achieved partial success in getting unit test defect
data. At the least, they collect defect count per module. e defect count is only
partial because not all defects are counted. Even partial defect data would help
in understanding defects in the product before even the formal test cycles start.
Software Test Metrics ◾ 123
Component Defect Count
Defects found in individual components can be separately analyzed. Figure 8.3
shows a bar graph of defects found in each component. is figure reveals defect
proneness of some components.
Component Defect Density
When defect count is normalized (divided) by size, we can calculate the metric
defects/function point (FP); we obtain fresh information about the quality levels
of each component. Now there is an objective basis to do an intercomparison of
quality of components. ese data can be obtained at the first round of tests and
updated during subsequent test iterations. e first round of results are a fairly good
indicator of the quality levels. is will serve as a guide in developing test plans for
the remaining rounds, and help capture more defects.
Cumulative defects
Testing time
Arrival
Closure
Figure 8.2 Defect arrival closure patterns.
1 2 3 4 5 6 7 8 9 10
Component number
Defect count
Figure 8.3 Defect per component.
124 ◾ Simple Statistical Methods for Software Engineering
Defect Classification
During defect capturing, we can register the different attributes of defects: defect
type, defect severity, phase injected, trigger, cause, and complexity. en we can
classify defects according to each of the attributes and derive broad perspectives
about defect population. Classification is also a measurement. e defect attri-
bute analysis report is a valuable input to the development team to support defect
prevention.
Testing Size: Test Case Point
e test case point (TCP) can be used as an estimator for the entire software devel-
opment project also.
Testing size can be measured in terms of TCP. Each test case is assigned a weight
based on test case complexity and test case type. Test case complexity depends on
test data, checkpoints, and precondition. e weights are as follows:
Checkpoint 1
Precondition 1, 3, 5
Test data 1, 3, 6
e rules for assigning weights are provided by Nguyen et al. [1]. Unadjusted
text case points are computed with the above weights. Adjustment is performed
according to the test case type. In a case study, the author reports, “e num-
ber of estimated TCPs was 8783 for the total 1631 test cases of the system, an
average of 5.4 TCPs per test case.” e total TCP represents testing size and
Box 8.4 SMoke TeST DefeCT DenSiTy
Smoke tests are very useful; they shorten the testing cycle. ey ensure that
the code is working and the build is stable. ey should be capable of expos-
ing major problems. It is instructive to calculate defect density for each com-
ponent with smoke test defects. Smoke test defect density is an indicator of
risks in the components and an early predictor of problems. e real power of
metrics is realized when metrics are used to predict. e smoke test is simple,
fast, and cheap, but smoke test defect density is priceless.
Smoke tests must be identified during requirements.
For example, safety requirements can be smoke tested very early in the
project.
e smoke test sets the minimum entry criteria for starting test execution.
Software Test Metrics ◾ 125
can be used to estimate effort and time required to complete testing. Also in an
experiment, Nguyen et al. noted that testers spent, on average, 3.21 minutes to
execute each TCP.
Patel et al. [2] have extended TCP calculation. ey have determined TCP sep-
arately for test case generation, manual execution, and automated execution. After
these calculations, they add all the TCPs to determine the total TCP for testing.
Risk Metric
Testing risk is measured using the FMEA table and expressed as risk priority num-
ber (RPN).
Jha and Kaner [3] have applied FMEA to detect risks and prioritize them for
mobile application.
Failure mode is the way a program could fail.
ey have prepared a catalogue of failure modes under different quality criteria.
e catalogue is pretty detailed and can be a useful reference to testers. ese fail-
ure modes catalogued by Jha and Kaner are domain specific; testers have to identify
their failure modes in the software testing, by using experience and knowledge.
Experienced testers know failure modes in software.
Having identified the failure modes, then they are prioritized using RPN. en
test cases are first developed to cover top 20% risks. Such inputs help testers to plan
testing using the paradigm “risk-based testing.” is enables testers to begin with
a small set, learn from the results, and progress. Kaner [4] advocate risk-based test
prioritization, test lobbying, and test design.
For every feature, we can measure RPN and prioritize features for testing. For
every product, we can measure RPN and prioritize areas for testing.
Allocate more resource and earlier testing for bigger numbered
(RPN) areas.
Cem Kaner
e purpose of risk measurement is to prioritize, to test less, to test sooner, and
to obtain more knowledge.
126 ◾ Simple Statistical Methods for Software Engineering
Predicting Quality
As testing progresses, the quality of SUT would be progressively measured. Halfway
through in testing, we have sufficient data to predict the quality level of the product
at the end of completing test execution. We should be in a position to predict resid-
ual defects when testing is performed and the application is released. On the basis
of this prediction, we can make a decision to stop testing once the targeted quality
level is achieved. If a quality level is defined as 99.999% correct code and if predic-
tion shows only 0.001% defects are undiscovered, we can release the application.
In this context, the purpose all test metrics is to make this prediction. Chapter 21
describes a method for predicting defects.
Metrics for Test Automation
Test automation requires special effort, almost a project on its own merit. Auto-
mation promises to rapidly find defects that manual testing cannot find. is
increases the quality of SUT.
Automation could be a huge effort, especially for large projects.
Windows NT 4 had 6 million lines of code, and 12 million lines of test code.
By using test automation metrics, this huge effort could be managed better and
more defects could be found in the SUT.
Return on Investment
Return on investment (ROI) of automation is a strategic measure that justifies auto-
mation. e simple formula for ROI is a direct ratio given as follows:
ROI
Cost saved by automation
Cost of automation
=
Cost savings from automation is usually distributed across several projects that
benefit from the automation scripts; calculating ROI from single project informa-
tion is not fair.
Percentage Automatable
Another lead metric in test automation is % Automatable. is metric must be
established upfront:
% Automatable
Number of test cases that can be
=
aautomated
Total number of test cases
× 100
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.