After completing this chapter, you will be able to:

• – understand the importance of measurement;
• – understand the measurement process according to the ISO 12207 standard;
• – understand the Practical Software and Systems Measurement method;
• – understand the ISO/IEC/IEEE 15939 measurement standard;
• – understand the measurement perspective according to the CMMI^® for Development;
• – understand the benefit of using surveys as a measurement tool;
• – understand how to implement a measurement program;
• – learn practical considerations for measurement;
• – understand the ISO/IEC 29110 measurement perspective;
• – learn the measurement requirements described in the IEEE 730 standard.

10.1 Introduction—the Importance of Measurement

Software measurement has been a research topic in software engineering for over 30 years [FEN 07]. Sadly, many measurement programs can hardly report the most basic software measurement such as: schedule, costs, size, and efforts. This means that little recent and factual information is available to project teams and their management [LAN 08].

We recall here the definition of software engineering. This definition puts emphasis on the importance of measuring software activities and products.

Today's organizations that develop software, either as standalone products or as components of systems, must continue to improve their performance and their software. Consequently, they must establish a performance target for their software development and maintenance processes. This would allow for better decision making and assessments of the rate of improvement compared with the needs of clients.

Victor Basili summarizes the many problems relative to measurement [BAS 10]. Organizations developing software experience many problems when trying to implement measurement programs. As an example, they often try to collect too much data where a good amount of it is not useful. They often do not implement a process to analyze data in a way that can help with strategic and tactical decision making. This situation leads to many problems such as a reduction of the benefits that can be achieved from a proper measurement program and disillusionment on the part of customers, management, and software developers. The inevitable result is the failure of the measurement initiative.

Watts S. Humphrey described the key roles related to software measurement, that is, understand and characterize, evaluate, control, predict, and improve [HUM 89]:

• – understand and characterize: measures allow us to learn about software processes, products, and services. Measures also:

• establish baselines, standards, and business and technical objectives;
• document the software process models used;
• set improvement objectives for software processes, products, and services;
• better estimate effort and schedule costs for a specific project;

• – evaluate: measures can be used to conduct cost/benefit analysis and determine if the objectives have been met;
• – control: measures can help with project control of resources, processes, products, and services by sounding alarms when control limits are surpassed, performance criteria are not met and standards are not followed;
• – predict: when software processes are stable and under control, measures can be used to predict budgets, schedules, resources needed, risks, and even quality issues;
• – improve: measures allow us to identify the root causes of defects and other inefficiencies where improvements can be proposed.

As an example, we illustrate the use of measurement to control the quality of a software project during development. Figure 10.1 presents the defect density of software components that have been inspected. The dotted line shows the level of quality required for a software component. For example, component number 10 should be inspected again after defects of the first inspection have been corrected.

Measurement, when available, allows the software project manager [SEI 10a]: to better plan and objectively assess the project state as well as the tasks that have been assigned to a supplier; to track the actual project performance against approved plans and objectives; to quickly identify process and product problems in order to act on them; and to collect baseline data useful for benchmarking future projects. We will also see that measures can be used to better estimate project schedules, requests for proposals, answers from suppliers during the selection process, supplier and competitors' offers and proposed project schedules.

A measurement program is also helpful with the improvement of the quality of software acquisition, development, maintenance, and infrastructure processes. The program must use a measurement repository where data are collected, analyzed, and measurements are reported and available to all stakeholders within the organization. This measurement repository should be designed to answer all types of questions, for decision making and performance indicators. It should also allow for the coherent measurement of the software processes improving quality and allowing for efficient defect removal.

Measurement has been at the origin of all science and is partly responsible for all scientific advancements. Measurement contributes to the maturing of a concept by quantifying it. Using measurement allows software processes to move from an artisanal state to a controlled and repeatable state. Software engineers should design and use sound measures to improve the maturity of the software processes.

Measuring allows us to understand the past, better inform current activities and try to predict the quality of developed products. This capacity will benefit projects where performance has not always been very precise historically with unclear schedules, budget overruns, and final products that include defects. In the past, measurement was considered as overhead to the project. Software development managers use measurement in their processes and set objectives. Measurement results are then used to take short term and active decisions during delivery and operation of systems. This helps identify and solve business-IT alignment issues or even with assigning work dynamically. For example, Google's rule of thumb is that a system reliability engineer must spend 50% of his time on software development/maintenance. To enforce this threshold, they measure where time is spent and use this measure to ensure that teams consistently follow the proposed ratio.

Figure 10.2 describes the software engineering management knowledge area of the SWEBOK. On the right hand side of the figure, four software engineering measurement sub-topics are presented: (1) how to establish and sustain measurement commitment; (2) how to plan the measurement process; (3) how to perform the measurement process; and (4) how to evaluate measurement.

10.2 Software Measurement According to ISO/IEC/IEEE 12207

Measurement is one of the many processes described in the ISO 12207 standard. Its purpose is to collect, analyze, and report objective data and information to support effective software management and demonstrate the quality of the products, services, and processes [ISO 17].

As a result of the successful implementation of the measurement process [ISO 17]:

1. information needs are identified;
2. an appropriate set of measures, based on the information needs, is identified or developed;
3. required data is collected, verified, and stored;
4. the data is analyzed and the results interpreted;
5. information items provide objective information that supports decisions.

The project shall implement the following activities and tasks in accordance with applicable organizational policies and procedures with respect to the measurement process [ISO 17]:

• – Prepare for measurement:

1. define the measurement strategy;
2. describe the characteristics of the organization that are relevant to measurement, such as business and technical objectives;
3. identify and prioritize the information needs.
4. select and specify measures that satisfy the information needs.
5. define data collection, analysis, access, and reporting procedures.
6. define criteria for evaluating the information items and the measurement process.
7. identify and plan for the necessary enabling systems or services to be used.

• – Perform measurement:

1. integrate manual or automated procedures for data generation, collection, analysis, and reporting into the relevant processes.
2. collect, store, and verify data.
3. analyze data and develop information items.
4. record results and inform the measurement users.

To obtain an overview of the measurement process, ISO 12207 refers the reader to the ISO 15939 standard that will be presented in a later section.

10.3 Measurement According to ISO 9001

ISO 9001 highlights the fact that a quality system needs a measurement component to be efficient. In addition to the typical process components, Figure 10.3 describes where performance measurement applies.

Clause 7.1.5 of ISO 9001 entitled “Monitoring and measuring resources” describes some measurement obligations [ISO 15]: “The organization shall determine and provide the resources needed to ensure valid and reliable results when monitoring or measuring is used to verify the conformity of products and services to requirements.”

Clause 9.1, entitled “Monitoring, measurement, analysis and evaluation,” states that [ISO 15]: “The organization shall determine:

• – what needs to be monitored and measured;
• – the methods for monitoring, measurement, analysis, and evaluation needed to ensure valid results;
• – when the monitoring and measuring shall be performed;
• – when the results from monitoring and measurement shall be analyzed and evaluated.”

Lastly, clause 10.3 of the ISO 9001 standard entitled “Continual improvement” describes another necessity of the use of measurement [ISO 15]: “The organization shall continually improve the suitability, adequacy and effectiveness of the quality management system.”

10.4 The Practical Software and Systems Measurement Method

The PSM methods were developed for the American defense industry [JON 03]. It served as a major input to the ISO 15939 standard on systems and software engineering measurements. Given that standards do not usually explain how things should be done, the PSM is useful for its practical examples.

The objective of the PSM is to provide measurement guidelines to software project managers. In it, directives, examples, lessons learned and case studies are presented. It provides a measurement framework that is ready to be used by software project managers. It also explains how to define and design a software measurement program to support the information needs of customers when they acquire software and systems from external providers.

The PSM covers three perspectives: (1) the project manager so as to provide a good understanding of the measures and how to use them to manage their project; (2) the technical staff that conducts measurement during planning and execution phases; and (3) the management team so that they can understand the measurement requirements associated with software.

The nine principles of the PSM are (adapted from [PSM 00]):

1. Use issues and objectives to drive the measurement requirements.
2. Define and collect measures based on the technical and management processes.
3. Collect and analyze data at a level of detail sufficient to identify and isolate problems.
4. Implement an independent analysis capability.
5. Use a systematic analysis process to trace the measures to the decisions.
6. Interpret the measurement results in the context of other project information.
7. Integrate measurement into the project management process throughout the life cycle.
8. Use the measurement process as a basis for objective communications.
9. Focus initially on project-level analysis.

As shown in Figure 10.4, quantitative project management includes the following specialities: risk management, measurement, and financial performance management. The PSM concentrates primarily on the measurement process but also includes the interface to other specialties like risk management and financial performance management.

It has been observed that measurement is not as effective if it used in an independent and isolated process. Measurement can be effective in describing overall project challenges and also point to issues in interrelated systems. Measurement will be more effective when included in all aspects of project management. For example, it will be more effective when integrated with risk management and financial performance management [PSM 00]. Another chapter of this book discusses risk management in detail.

The PSM is composed of the following parts [PSM 00]:

• – Part 1, The Measurement Process, describes the measurement process at a summary level and provides an overview of measurement tailoring, application, implementation, and evaluation. Part 1 explains what is required to implement the measurement process for a project.
• – Part 2, Tailor Measures, describes how to identify project issues, select appropriate measures, and define a project measurement plan.
• – Part 3, Measurement Selection and Specification Tables, provides a series of tables that help the user select the measures that best address the project's issues. These tables support the detailed tailoring guidance of Part 2.
• – Part 4, Apply Measures, describes how to collect and process data, analyze the measurement results, and use the information to make informed project decisions.
• – Part 5, Measurement Analysis and Indicator Examples, provides examples of measurement indicators and associated interpretations.
• – Part 6, Implement Process, describes the tasks necessary to establish the measurement process within an organization.
• – Part 7, Evaluate Measurement, identifies assessment and improvement tasks for the measurement program as a whole.
• – Part 8, Measurement Case Studies, provides three different case studies that illustrate many of the key points made throughout the Guide. The case studies address the implementation of a measurement process on a DoD weapons system, an information system, and a government system in the operations and maintenance life cycle phase.
• – Part 9, Supplemental Information, contains a glossary, list of acronyms, bibliography, project description, comment form, and an index.
• – Part 10, Department of Defense Implementation Guide, this addendum provides information specific to implementing the PSM guidance on Department of Defense programs. It addresses implementation issues of particular concern to DoD acquisition organizations.

The PSM approach to software measurement, as illustrated in Figure 10.5, addresses the following four key measurement activities (adapted from [PSM 00]):

1. Tailor Measures

   The objective of this activity is to define the set of software and system measures that will provide the best understanding of challenges in the project at the lowest cost. A measurement plan documents the result of this first activity.

2. Apply Measures

   During this activity, measures are analyzed in order to provide the feedback necessary for effective decision making. Information on risks and financial performance can also be taken into account for decision making.

3. Implement Process

   This activity consists of three tasks:

   • – obtain organizational support: including the right to measure at all organizational levels;
   • – define responsibilities concerning measurement;
   • – provide resources, purchase required tools, and recruit personnel for this process.

4. Evaluate Measurement

   This activity includes four tasks:

   • – evaluate measures and indicators as well as their results;
   • – evaluate the measurement process according to three perspectives: (1) the quantitative performance evaluation of the measurement process; (2) a conformity assessment of the process executed versus the one that was planned; and (3) the measurement capability as compared to a standard recommendation;
   • – update the experience base with lessons learned;
   • – identify and implement improvements.

The key roles and responsibilities associated with measuring are (adapted from [PSM 00]):

• – executive manager: is typically a manager responsible for more than one project. This manager defines the expected high level performance and the business objectives. He ensures that individual projects align with the general measurement policy. He uses the measurement outputs to make decisions;
• – project or technical manager: this individual or group identifies the project challenges, reviews the measurement analysis, and acts on the information. In the case of the acquisition of complex software, the customer and the external provider will have a dedicated project manager that will use this information to make joint decisions.
• – measurement analyst: this role can be assigned to a person or a group. The responsibilities include the design of the measurement plan, data collection and analysis, as well as the presentation of results to all the stakeholders. Typically, in large and complex software acquisition projects, both the external provider and the customer have a measurement analyst assigned to the project;
• – project team: is the team responsible for the acquisition, development, and maintenance/operation of the software and systems. This team can include government or industry organizations as part of an Integrated Product Team (IPT). The project team collects measurement data periodically and uses it to orient engineering decisions.

The PSM defines seven categories of information that should be produced for software projects (adapted from [MCG 02]):

1. schedule: this measurement category aims at tracking the progress of the project at each step and milestone. A project that experiences delays will have a hard time meeting its delivery objectives. The project manager may have to make decisions such as reducing the functionality to be delivered or sacrificing its quality;
2. resources and costs: this measurement category evaluates the balance between the work to be done and the availability of human resources to do this work. A project that overruns its personnel budget will have difficulty completing the work unless some functionality is dropped or the quality reduced;
3. software size and stability: this measurement category addresses the stability of the progress made with respect to the delivery of functional and non-functional requirements. It uses the delivered and tested functional size to assess the delivery trend. The stability measurement considers functional change rates. Scope creep is characterized by a growing number of change requests being submitted. This situation will likely extend the schedule and increase the human resource costs;
4. product quality: another dimension of a software project that needs to be controlled is the product quality. This measurement category considers the current state of the defect removal trend for both functional and non-functional requirements. When a defective product is delivered to the customer acceptance testing step, it generates a large number of defect reports. Forcing delivery in this condition will drastically impact maintenance efforts;
5. process performance: this measurement category assesses the ability of the external providers to meet both the contract clauses as well as the requirements identified in the attachments of the contract. An external provider with weak control of his processes or experiencing weak productivity is an early sign of possible delivery problems;
6. effectiveness of technology used: this measurement category measures the effectiveness of the technology chosen to be used by the project to address the requirements. Relatively technical measures assess the software engineering techniques, like reuse, development methods and frameworks and software architectural concerns. It aims at discovering the use of risky technologies or those that have not been mastered;
7. user satisfaction: this last category assesses how the customers feel about the progress of the project and how it meets their requirements.

The next section presents the ISO/IEC/IEEE 15939 standard, which is the current international standard for software process measurement.

10.5 ISO/IEC/IEEE 15939 Standard

This section presents four key software measurement activities of ISO 15939 as well as examples of measures. ISO 15939 defines a software measurement process that applies to both the systems and software engineering disciplines for software suppliers and acquirers.

The ISO 15939 standard is aligned with the measurement requirements of ISO 9001. It elaborates the measurement process for software projects as described in ISO 15288 and ISO 12207.

In this standard, the measurement process is represented by a model that describes the activities and tasks to specify, implement, and interpret results. It does not describe how to perform these tasks nor give examples of measures. Its purpose is to describe the activities and tasks that are necessary to successfully identify, define, select, apply, and improve measurement within an overall project or organizational measurement structure. It also provides definitions for measurement terms commonly used within the system and software disciplines [ISO 17c].

As a result of the successful implementation of the measurement process, you can expect the following outcomes [ISO 17c]:

1. information needs are identified;
2. an appropriate set of measures, based on the information needs, is identified or developed;
3. required data is collected, verified, and stored;
4. the data is analyzed and the results interpreted;
5. information items provide objective information that supports decisions;
6. organizational commitment for measurement is sustained;
7. identified measurement activities are planned;
8. the measurement process and measures are evaluated;
9. improvements are communicated to the measurement process owner.

Note that the first five outcomes presented above are the same as those described in ISO 15288 and ISO 12207.

10.5.1 Measurement Process According to ISO 15939

A software measurement process should detail its activities and tasks to achieve its goal. Figure 10.6 presents the reference process model. It contains four activities, where each has a certain number of tasks [ISO 17c]:

1. Establish and sustain measurement commitment;
2. Prepare for measurement;
3. Perform measurement;
4. Evaluate measurement.

The model includes a feedback loop to the information technology life cycle processes and assumes that the organization has formalized them (i.e., technical and management processes). The activities are represented by an iterative cycle allowing for continuous feedback and improvement. This is an adaptation of the Plan-Do-Check-Act model widely used in process improvement.

The measurement repository described in Figure 10.6 collects data during a project iteration and stores historical data for all projects and software engineering processes.

The typical functional roles that are mentioned in the ISO 15939 standard are: stakeholder, sponsor, measurement user, measurement analyst, data provider, and measurement process owner.

10.5.2 Activities and Tasks of the Measurement Process

The measurement process is launched by the measurement requirements, also known as the technical and management information needs of the organization. The activities and tasks are described in Figure 10.7.

10.5.3 An Information Measurement Model of ISO 15939

Annex A of the ISO 15939 standard is only informative. It presents the model that links the information needs to the measures. This model shows what the measurement planner has to design during the planning, execution and evaluation stages. Three types of measures are presented: base measures, derived measures, and indicators. In this section, the measurement model is explained and followed by an example of its use.

Figure 10.8 presents the model and its components. Components are explained from the top to the bottom of the figure [ISO 17c]:

• – An entity is an object (e.g., a process, product, project, or resource) that is to be characterized by measuring its attributes. Typical engineering objects can be classified as products (e.g., design document, network, source code, and test case), processes (e.g., design process, testing process, and requirements analysis process), projects, and resources (e.g., the systems engineers, the software engineers, the programmers, and the testers). An entity may have one or more properties that are of interest to meet the information needs. In practice, an entity can be classified into more than one of the above categories.
• – An attribute is a property or characteristic of an entity that can be distinguished quantitatively or qualitatively by human or automated means. An entity may have many attributes, only some of which may be of interest for measurement. The first step in defining a specific instantiation of the measurement information model is to select the attributes that are most relevant to the measurement user's information needs. A given attribute may be incorporated in multiple measurement constructs supporting different information needs.
• – A measure is defined in terms of an attribute and the method for quantifying it. A measure is a variable to which a value is assigned. A base measure is functionally independent of other measures. A base measure captures information about a single attribute. Data collection involves assigning values to base measures. Specifying the expected range or type of values of a base measure helps to verify the quality of the data collected.
• – A measurement method is a logical sequence of operations, described generically, used in quantifying an attribute with respect to a specified scale. The operations may involve activities such as counting occurrences or observing the passage of time. The same measurement method may be applied to multiple attributes. However, each unique combination of an attribute and a method produces a different base measure. Some measurement methods may be implemented in multiple ways. A measurement procedure describes the specific implementation of a measurement method within a given organizational context:

• The type of measurement method depends on the nature of the operations used to quantify an attribute. Two types of method may be distinguished:
  • Subjective: quantification involving human judgment.
  • Objective: quantification based on numerical rules such as counting. These rules may be implemented by human or automated means.

• – A derived measure is a measure that is defined as a function of two or more values of base measures. Derived measures capture information about more than one attribute or the same attribute from multiple entities. Simple transformations of base measures (for example, taking the square root of a base measure) do not add information, thus do not produce derived measures. Normalization of data often involves converting base measures into derived measures that can be used to compare different entities.
• – A function is an algorithm or calculation performed to combine two or more base measures. The scale and unit of the derived measure depend on the scales and units of the base measures from which it is composed as well as how they are combined by the function.
• – An indicator is a measure that provides an estimate or evaluation of specified attributes derived from a model with respect to defined information needs. Indicators are the basis for analysis and decision making. These are what should be presented to measurement users. Measurement is always based on imperfect information, so quantifying the uncertainty, accuracy, or importance of indicators is an essential component of presenting the actual indicator value.
• – An information product is one or more indicators and their associated interpretations that address an information need; for example, a comparison of a measured defect rate to planned defect rate along with an assessment of whether or not the difference indicates a problem.

10.5.3.1 Example of a Measure

Figure 10.9 presents an example of a productivity measure originating from annex A of ISO 15939. The decision maker in this example needs to select a specific productivity level as the basis for project planning. The measurable concept is that productivity is related to effort expended and number of requirements implemented. Thus, effort and requirements are the measurable entities of concern.

This example assumes that the productivity can be estimated based on past performance. Thus, data for the base measures (numbered entries in the following table) need to be collected and the derived measure computed for each project in the data store.

The decision criteria are shown at the bottom of Figure 10.9. They are numeric boundaries or objectives used to assess the need for an action or for additional attention to productivity. Decision criteria help in the interpretation of a measure. They can be calculated or be based on the conceptual understanding of what is expected. Regardless of how the productivity number is arrived at, the uncertainty inherent in engineering means that there is a considerable probability that the estimated productivity will not be realized exactly. Estimating productivity based on historical data enables the computation of confidence limits that help to assess how close actual results are likely to come to the estimated value [ISO 17c].

Informative annex A of ISO 15939 also describes a “software artifact” quality measure followed by an example of project advancement measures. Informative annex B of the standard shows the mapping between work products and the measurement activities that have created the artifact. The measurement plan is the result of the execution of the planned activities and tasks. Informative annex F is an informative section listing typical information found in a measurement plan. Figure 10.10 presents examples.

Informative annex C of the standard describes criteria for selecting a measure, whereas informative annex D presents evaluation criteria for information products. Informative annex E presents evaluation criteria for the measurement process and informative annex G describes criteria for reporting information elements.

10.6 Measurement According to the CMMI Model

This section will explain some of the measurement practices proposed by the staged representation of the CMMI^®-DEV model. The practices appear in many process areas of the model: in the generic goals (GG), in the generic practices (GP), in the specific goals (SG), and in the specific practices (SP) of specific process areas like “project planning,” “project monitoring and control,” “organizational process definition,” and “quantitative project management.”

Measures collected by maturity level 1 organizations are often of poor reliability because at that maturity level, their processes are often chaotic and not documented. At maturity level 2, also referred to as “managed,” organizations have processes that are planned and executed. Therefore, at that level, it is possible to measure processes and software products. We recall here that one of the generic practices of maturity level 2 is “GP 2.8 Monitor and Control the Process,” and it refers to some process attributes such as the percentage of projects that use progress and performance measures and the number of outstanding open and closed corrective actions [SEI 10a].

Concerning software products developed by suppliers, the CMMI-DEV recommends that the acquirer needs to closely follow the project quality, schedule, and costs. Measurement and data analysis are key activities of project monitoring.

The ISO 15939 standard was used by the CMMI-DEV “measurement and analysis” process area. This allows both the systems engineering and software engineering communities to share the same measurement recommendations. The next text box describes the level 2 objectives and specific practices of the measurement and analysis process area.

This process area is used by many other process areas of the model. For example, for measuring project performance, the project monitoring and control process area should be consulted; for controlling software products, refer to the configuration management process area; for requirements traceability, the requirements management process area contains measurement guidelines; for organizational measurement, refer to the organizational process definition process area. To learn more about the appropriate use of statistical methods, the quantitative project management process area of CMMI-DEV provides more guidance.

10.7 Measurement in Very Small Entities

Worldwide, there are a large number of Very Small Entities (VSEs) that develop and maintain software. These are organizations, companies, departments, and projects involving up to 25 people. In an earlier chapter, the ISO 29110 was introduced. ISO 29110 proposes a four-stage roadmap referred to as a profiles. The profiles apply to VSEs in start-up mode; small projects that have a limited duration of six person-months; those that create only one product with only one team; VSEs that have more than a project with more than one team; and a VSE that wants to improve the management of its business and its competitiveness.

The activities of the ISO 29110 project management process that are related to measurement are: project planning activity, where size, effort, calendar, and resources are estimated and used in the preparation of the project plan, as well as in the project assessment and control activity, where progress is evaluated against the project plan.

The tasks of the software implementation process of ISO 29110 related to measurement are mainly those related to defects identified and corrected during reviews and testing.

10.8 The Survey as a Measurement Tool

Surveys are used by organizations to obtain an overview of complex questions, aid problem resolution, and support decision making. Surveys are tools that allow information to be collected quickly and anonymously. It can be done during meetings and, most often, using an internet survey tool that sends questionnaires or invites to participants to answer a survey by clicking on a web link.

Concerning SQA, surveys can be used to obtain service satisfaction information from individuals and organizations like developers, project managers, testers, configuration management, and sometimes suppliers. For example, a few weeks after the deployment of a new measurement program, surveys can be prepared by SQA to assess the level of satisfaction of customers of the organization concerning its products and services.

In this section, two case studies are presented: one survey conducted by the SEI concerning measurement and one survey conducted by the ISO working group for VSEs.

What is a survey? According to Kasunic, of the SEI [KAS 05], a survey is a collection of data and an analysis method where solicited individuals answer questions or comment on declarations previously elaborated.

The SEI developed a survey process with seven steps [KAS 05]:

1. identify the research objectives;
2. identify and characterize the target audience;
3. design the sampling plan;
4. design and write the questionnaire;
5. pilot test the questionnaire;
6. distribute the questionnaire;
7. analyze results and write a report.

According to Kasunic, a good survey has to be systematic, impartial, representative, quantitative, and repeatable. Although surveys show good results compared with other data collection techniques, they have limitations [KAS 05]:

• – To generalize for a population, a survey must follow strict procedures in defining which participants are studied and how they are selected.
• – Following the rules and implementing the survey with the rigor that is necessary can be expensive with respect to cost and time.
• – Survey data are usually superficial. It is not typically possible to go into any detail—that is, we are not capable of digging deeply into people's psyches looking for fundamental explanations of their unique understandings or behaviors.
• – Surveys can be obtrusive. People are fully aware that they are the subjects of a study. They often respond differently than they might if they were unaware of the researcher's interest in them.

The SEI conducted this survey to understand the state of the practice of software measurement. The following text box describes the results.

10.9 Implementing a Measurement Program

First of all, we would think that measurement is a quick and easy thing to do. But there are many obstacles to successfully implementing a measurement program:

• – we do not know why we are collecting measures;
• – we intend to collect too many measures;
• – some measures are not collected in the same manner in other projects;
• – there are no adequate tools to easily collect and analyze measurements;
• – measures are deployed without having been tested in pilot projects;
• – measurement adds to the current workload;
• – we think measures will not be used;
• – we believe measures will be used to assess our individual performance;
• – there is no commitment to measurement from the organization;
• – there is little support for the measurement program.

To overcome these potential obstacles, a seven-step implementation approach has been created, tested, and is recommended [DES 95]:

1. demonstrate the value and potential of the measurement program to upper management to gain their support;
2. implicate the delivery personnel early in the design of the program;
3. identify the key processes to be improved where most benefits would arise;
4. identify the measurement goals and objectives for these key processes;
5. design and publish the measurement program for comments;
6. identify the tools/processes to be used for measurement and test them;
7. launch a first pilot and then extend gradually.

10.9.5 Step 5: Design of the Measurement Program

This step consists in designing a measurement program that will allow management not only to see if the objectives have been reached, but also to understand why if they have not been reached. Figure 10.11 suggests the components of a measurement program: tools, standards, definitions, and a choice of measures. The implementation of this design will vary from one organization to another.

10.10 Practical Considerations

This section presents base measures recommended by the SEI. In every software project, management usually wants the same type of information (adapted from Carleton et al. (1992) [CAR 92]):

• – what is the size of the product to be developed?
• – do we have enough qualified/available personnel for the task ahead?
• – can we meet the schedule?
• – what level of quality is expected of this product?
• – where are we according to plans?
• – how are we doing on costs?

The base measures proposed by the SEI are: a size measure, an effort measure, a calendar measure, and a quality measure. For size, the SEI recommends the number of lines of source code for the following reasons (adapted from [PAR 92]):

• – it is easy, simply count the end of line markers;
• – counting methods do not greatly depend on the programming language;
• – it is easy to automate the counting of physical lines of code;
• – the majority of the data used to create cost estimation models like COCOMO [BOE 00] used lines of code.

With regards to the effort measure, the SEI recommends using the number of staff-hours. The organization should track both normal and overtime hours, whether they are paid or not. Ideally, hours dedicated on important activities like requirements, design, and testing should be calculated separately.

The SEI recommends also adopting structured methods to define two important aspects with respect to a measure for schedules: the dates and the exit criteria. It is recommended to compare dates (planned and actual) associated with project milestones, reviews, and audits.

Examples of calendar measures or schedule completion criteria are: milestones, end of phase reviews, audits, and deliverables approved by the client.

With regards to quality, the SEI recommends measuring problems and defects. It is recommended that the problems and software defects be used to help determine when products will be ready for customer delivery and to provide data for the improvement of both processes and products.

10.11 The Human Side of Measurement

The attitude of the people involved in measurement, that is to say the definition of measures, data collection, and analysis, is an important criterion to ensure that the program is useful to the organization. Measurement can affect the behavior of the individual observed. When something is measured, it is implied that it is important.

Every staff member wants to look good and therefore would like the measures to help him look good. When developing a measurement program, think of the behaviors we want to encourage and the behaviors that we do not want to encourage.

For example, if you measure software productivity in terms of lines of code per hour, developers will target that goal to meet the objective of productivity. They will perhaps focus on their own work to the detriment of the team and the project. Worse, they can find ways to program the same functionality with additional lines of code to impact the measure.

There is only one way to avoid this behavior and it is to focus measurement on processes and products instead. Here are some suggestions to promote the implementation of effective measures (adapted from [WES 05]):

• – once measures are collected, they should be used to make decisions. One sure way to undermine the measurement program is to accumulate them in a database and ignore them for decision making;
• – given that software development is an intellectual task, it is recommended to develop a set of measures to fully capture the complexity of the task. As a minimum, quality, productivity and project schedule should be measured;
• – to gain developer commitment to this program, they must have a sense of belonging (ownership). Participation in the definition, collection, and analysis of the measures will improve this sense of belonging. People working with a process on a daily basis have an intimate knowledge of this process. They can help suggest ways to better measure the process, ensure the accuracy of valid measurements and propose ways to interpret the results to maximize their utility;
• – provide regular feedback to the team about the data collected;
• – focus on the need to collect data (when team members see the data actually used, they are more likely to view the collection activity as important);
• – if team members are kept informed of how the measures are used, they are less likely to become suspicious of the measurement program;
• – benefit from the knowledge and experiences of team members by involving them in the analysis of data for process improvement efforts.

An anecdote about the invention of the inspection method by Fagan while he was working at IBM [BRO 02] was presented earlier in the chapter on reviews. In the following text box, we continue this story with a brief description of the difficulties he encountered.

10.12 Measurement and the IEEE 730 SQAP

Measurement, described in the IEEE 730 standard, is helpful in demonstrating that the software processes can and do produce software products that conform to requirements. This confirmation includes evaluating intermediate and final software products along with methods, practices, and workmanship. Evaluation further includes measurement and analysis of a software process as well as product problems and related causes and provides recommendations about ways to correct current problems. IEEE 730 also explains that the measurement activities and tasks can supply objective data to improve an organization's life cycle management process. Similarly, evaluating software products for compliance can identify improvement opportunities.

This standard provides a number of questions that the project team should use during planning and execution in order to ensure its conformity to the measurement requirements. For example, [IEE 14]:

• – Are requirements specific, measureable, attainable, realistic, and testable?
• – Have the information needs required to measure the effectiveness of technical and management processes been identified?
• – Has an appropriate set of measures, driven by the information needs, been identified and developed?
• – Have appropriate measurement activities been identified and planned?
• – Is the review process of the project measured and effective?
• – Have all corrective actions that were implemented proven to be effective as determined by effectiveness measures?
• – Have the measurement process and specific measures been evaluated?
• – Have improvements been communicated to the measurement process owner?

Of the 16 SQA activities recommended by this standard, activity 5.4.6 describes the measurement of software products while activity 5.5.5 describes the recommended process measurement of a software project.

10.13 Success Factors

Following are the factors that help or adversely affect software quality in an organization.