Suppose we want to express some quality attribute, say the complexity of a program text, in a single numeric value. Larger values are meant to denote more complex programs. If such a mapping C from programs to numbers can be found, we may next compare the values of C(P₁) and C(P₂) to decide whether program P₁ is more complex than program P_2. Since more complex programs are more difficult to comprehend and maintain, this type of information is very useful, e.g. for planning maintenance effort.

What then should this mapping be? Consider the program texts in Figure 6.1.

Most people will concur that text (a) looks less complex than text (b). Is this caused by:

Suppose we decide that the number of if statements is what counts. The result of the mapping is 0 for text (a) and 3 for text (b), and this agrees with our intuition. However, if we take the sum of the number of if statements, goto statements, and loops, the result also agrees with our intuition. Which of these mappings is the one sought for? Is either of them 'valid' to begin with? What does 'valid' mean in this context?

Figure 6.1. Two versions of a sort routine (a) structured and (b) unstructured

A number of relevant aspects of measurement, such as attributes, units and scale types can be introduced and related to one another using the measurement framework depicted in Figure 6.2. This framework also allows us to indicate how metrics can be used to describe and predict properties of products and processes, and how to validate these predictions.

Figure 6.2. A measurement framework (Source: B. Kitchenham, S. Pfleeger and N. Fenton, Towards a Framework for Software Measurement Validation, IEEE Transactions on Software Engineering 21:12, © 1995 IEEE. Reproduced with permission.)

The model in Figure 6.2 has seven constituents:

Measurement is a mapping from the empirical, 'real' world to the formal, relational world. A measure is the number or symbol assigned to an attribute of an entity by this mapping. The value assigned obviously has a certain unit, e.g. lines of code. The unit in turn belongs to a certain scale, such as the ratio scale for lines of code or the ordinal scale for the severity of a failure.

In mathematics, the term metric has a very specific meaning: it describes how far apart two points are. In our field, the term is often used in a somewhat sloppy way.

Sometimes it denotes a measure, sometimes the unit of a measure. We use the term to denote the combination of:

For each scale type, certain operations are allowed, while others are not. In particular, we cannot compute the average for an ordinal scale, but only its median (middle value). Suppose we classify a system as 'very complex', 'complex', 'average', 'simple' or 'very simple'. The assignment of numbers to these values is rather arbitrary. The only prerequisite is that a system that is classified as 'very complex' is assigned a larger value than a system classified as 'complex'. If we call this mapping W, the only requirement is: W(very complex) > W(complex) > ··· > W(very simple). Table 6.2 gives an example of two valid assignments of values to this attribute.

Suppose we have a system with three components, which are characterized as 'very complex', 'average' and 'simple', respectively. By assigning the values from the first row, the average would be 3, so the whole system would be classified as of average complexity. Using the values from the second row, the average would be 35, something between 'complex' and 'very complex'. The problem is caused by the fact that, with an ordinal scale, we do not know whether successive values are equidistant. When computing an average, we tacitly assume they are.

We often cannot measure the value of an attribute directly. For example, the speed of a car can be determined from the values of two other attributes: a distance and the time it takes the car to travel that distance. The speed is then measured indirectly, by taking the quotient of two direct measures. In this case, the attribute–relation model formalizes the relation between the distance traveled, time, and speed.

We may distinguish between internal and external attributes. Internal attributes of an entity can be measured purely in terms of that entity itself. Modularity, size, defects encountered, and cost are typical examples of internal attributes. External attributes of an entity are those which can be measured only with respect to how that entity relates to its environment. Maintainability and usability are examples of external attributes. Most quality factors we discuss in this chapter are external attributes. External attributes can be measured only indirectly, since they involve the measurement of other attributes.

Empirical relations between objects in the real world should be preserved in the numerical relation system that we use. If we observe that car A drives faster than car B, then we would rather like our function S, which maps the speeds observed to some number, to be such that S(A) > S(B). This is called the representation condition. If a measure satisfies the representation condition, it is said to be a valid measure.

The representation condition can sometimes be checked by a careful assessment of the attribute–relation model. For example, we earlier proposed to measure the complexity of a program text by counting the number of if statements. For this (indirect) measure to be valid we have to ascertain that:

Since neither of these statements is true in the real world, this complexity measure is not valid.

The validity of more complex indirect measures is usually ascertained through statistical means. Most of the cost estimation models discussed in Chapter 7, for example, are validated in this way.

Finally, the scale type of indirect measures merits some attention. If different measures are combined into a new measure, the scale type of the combined measure is the 'weakest' of the scale types of its constituents. Many cost-estimation formulae contain factors whose scale type is ordinal, such as the instability of the requirements or the experience of the design team. Strictly speaking, different values that result from applying such a cost-estimation formula should then be interpreted as indicating that certain projects require more effort than others. The intention though is to interpret them on a ratio scale, i.e. actual effort in man-months. From a measurement-theory point of view, this is not allowed.

Some of the first elaborate studies ((McCall et al., 1977) and (Boehm et al., 1978)) on the notion of 'software quality' appeared in the late 1970s. In these studies, a number of aspects of software systems are investigated that somehow relate to the notion of software quality. In the ensuing years, a large number of people have tried to tackle this problem. Many taxonomies of quality factors have been published. The fundamental problems have not been solved satisfactorily, though. The various factors that relate to software quality are hard to define. It is even harder to measure them quantitatively. On the other hand, real quality can often be identified surprisingly easily.

In the IEEE Glossary of Software Engineering Terminology, quality is defined as 'the degree to which a system, component, or process meets customer or user needs or expectations'. Applied to software, then, quality should be measured primarily against the degree to which user requirements are met: correctness, reliability, usability, and so on. Software lasts a long time and is adapted from time to time in order to accommodate changed circumstances. It is important to the user that this is possible within reasonable costs. The customer is therefore also interested in quality factors which relate to the structure of the system, such as maintainability, testability and portability, rather than its use.

We will start our discussion of quality attributes with McCall's taxonomy, which distinguishes between two levels of quality attributes. Higher-level quality attributes, known as quality factors, are external attributes and can, therefore, be measured only indirectly. McCall introduced a second level of quality attributes, termed quality criteria. Quality criteria can be measured either subjectively or objectively. By combining the ratings for the individual quality criteria that affect a given quality factor, we obtain a measure of the extent to which that quality factor is being satisfied. Users and managers tend to be interested in the higher-level, external, quality attributes.

For example, we cannot directly measure the reliability of a software system. We may however directly measure the number of defects encountered so far. This direct measure can be used to obtain insight into the reliability of the system. This involves a theory, which can be ascertained on good grounds, of how the number of defects relates to reliability. For most other aspects of quality though, the relation between the attributes that can be measured directly and the external attributes we are interested in is less obvious, to say the least.

Table 6.3 lists the quality factors and their definitions, as they are used in (McCall et al., 1977). The definition of software reliability in Table 6.3 is rather narrow. A more complete definition is contained in the IEEE Glossary of Software Engineering Terminology: 'The ability of a system or component to perform its required functions under stated conditions for a specified period of time.' It is often expressed as a probability.

The quality factors can be broadly categorized into three classes. The first class contains those factors that pertain to the use of the software after it has become operational. The second class pertains to the maintainability of the system. The third class contains factors that reflect the ease with which a transition to a new environment can be made. These three categories are depicted in Table 6.4.

In ISO standard 9126, a similar effort has been made to define a set of quality characteristics and sub-characteristics (see Table 6.5). Their definitions are given in Tables 6.6 and 6.7. Whereas the quality factors and criteria as defined in (McCall et al., 1977) are heavily interrelated, the ISO scheme is hierarchical: each sub-characteristic is related to exactly one characteristic.

The ISO quality characteristics strictly refer to a software product. Their definitions do not capture process quality issues. For example, security can be handled partly by provisions in the software and partly by proper procedures. Only the former is covered by the sub-characteristic 'security' of the ISO scheme. Furthermore, the sub-characteristics concern quality aspects that are visible to the user. Reusability, for example, is not included in the ISO scheme.

The ISO characteristics and sub-characteristics, together with an extensive set of measures, make up ISO's external and internal quality model. Internal quality refers to the product itself, ultimately the source code. External quality refers to the quality when the software is executed. For example, the average number of statements in a method is a measure of internal quality, while the number of defects encountered during testing is a measure of external quality.

Ultimately, the user is interested in the quality in use, defined in (ISO 9126, 2001) as 'the user's view of the quality of the software product when it is executed in a specific environment and a specific context of use'. It measures the extent to which users can achieve their goals, rather than mere properties of the software (see also Section 6.3). Quality in use is modeled in four characteristics: effectiveness, productivity, safety, and satisfaction. The definitions of these quality-in-use characteristics are given in Table 6.8.

Theoretically, internal quality, external quality and quality in use are linked: internal quality indicates external quality which, in turn, indicates quality in use. In general, meeting criteria at one level is not sufficient for meeting criteria at the next level. For instance, satisfaction is partly determined by internal and external quality measures, but also includes the user's attitude towards the product. The latter has to be measured separately. Note that internal quality and external quality can be measured directly. Quality in use can in general only be measured indirectly.

Quality factors are not independent. Some factors will impact one another positively, while others will do so negatively. For example, a reliable program is more likely to be correct and vice versa. Efficiency, on the other hand, will generally have a negative impact on most other quality factors. This means that we will have to make tradeoffs between quality factors. If there is a strong requirement for one factor, we may have to relax other factors. These tradeoffs must be resolved at an early stage. An important objective of the software architecture phase is to bring these quality factors to the forefront and make the tradeoffs explicit, so that the stakeholders know what they are in for (see Chapter 11). By doing so, we are better able to build in the desired qualities, rather than merely assessing them after the fact.

Users will judge the quality of a software system by the degree to which it helps them accomplish tasks and by the sheer joy they have in using it. The manager of those users is likely to judge the quality of the same system by its benefits. These benefits can be expressed in cost savings or in a better and faster service to clients.

To a tester, the prevailing quality dimensions will be the number of defects found and removed, the reliability measured, or the conformance to specifications. To the maintenance programmer, quality will be related to the system's complexity, its technical documentation, and so on.

These different viewpoints are all valid. They are also difficult to reconcile. Garvin (1984) distinguishes five definitions of software quality:

Transcendent quality concerns innate excellence. It is the type of quality assessment we usually apply to novels. We may consider Zen and the Art of Motorcycle Maintenance an excellent book; we may try to give words to our admiration but these words are usually inadequate. The practiced reader gradually develops a good feeling for this type of quality. Likewise, the software engineering expert may develop a good feeling for the transcendent qualities of software systems.

The user-based definition of quality concerns 'fitness for use' and relates to the degree in which a system addresses the user's needs. It is a subjective notion. Since different users may have different needs, they may assess a system's quality rather differently. The incidental user of a simple word-processing package may be quite happy with its functionality and possibilities while a computer scientist may be rather disappointed. The reverse situation may apply to a complex typesetting system such as LATEX.

In the product-based definition, quality relates to attributes of the software. Differences in quality are caused by differences in the values of those attributes. Most of the research into software quality concerns this type of quality. It also underlies the various taxonomies of quality attributes discussed in Section 6.2.

The manufacturing-based definition of quality concerns conformance to specifications. It is the type of quality definition used during system testing, whereas the user-based definition is prevalent during acceptance testing.

The value-based definition deals with costs and profits. It concerns balancing time and cost on the one hand, and profit on the other hand. We may distinguish various kinds of benefit, not all of which can be phrased easily in monetary terms (Simmons, 1996):

Software developers tend to concentrate on the product-based and manufacturing-based definitions of quality. The resulting quality requirements can be expressed in quantifiable terms, such as the number of defects found per man-month or the number of branching points per module. The quality attributes discussed in the previous section fall into these categories. Such quality requirements however can not be directly mapped onto the, rather subjective, quality viewpoints of the users, such as 'fitness for use'. Nevertheless, users and software developers have to come to an agreement on the quality requirements to be met.

One way to try to bridge this gap is to define a common language between users and software developers in which quality requirements can be expressed. This approach is taken in (Bass et al., 2003), where quality requirements are expressed in 'quality-attribute scenarios'. Figure 6.3 gives one example of how a quality attribute can be expressed in user terms. Quality-attribute scenarios have a role not only in specifying requirements, but also in testing whether these requirements are (going to be) met. For example, quality-attribute scenarios are heavily used in architecture assessments (see Chapter 11).

Quality is not only defined at the level of the whole system. In component-based development, quality is defined at the level of a component. For services, quality is defined at the level of an individual service. The environment of the component or service, i.e. some other component or service, will require certain qualities as well. So for components and services, there is a requires and a provides aspect to quality. Since a component or service generally does not know the context in which it is going to be embedded, it is difficult to decide on the 'right' quality level. One might then choose to offer different levels of quality. For example, a service that handles video streaming may be fast with low-quality images or slow with high-quality images. The user of that service then selects the appropriate quality of service (QoS) level. Quality aspects of components and services are touched upon in Chapters 18 and 19, respectively.

Developers tend to have a mechanistic, product-oriented view on quality, whereby quality is associated with features of the product. In this view, quality is defined by looking from the program to the user (user friendliness, acceptability, etc.). To assess the quality of systems used in organizations, we have to adopt a process-oriented view on quality as well, where quality is defined by looking from the user to the program. This leads to notions such as 'adequacy' and 'relevance'. For example, a helpdesk staffed with skilled people may contribute considerably to the quality of a system as perceived by its users, but this quality attribute generally does not follow from a product-based view of quality.

Figure 6.3. A quality-attribute scenario that can be used by both users and developers

A very eclectic view of quality is taken in Total Quality Management (TQM). In TQM, quality applies to every aspect of the organization and is pursued by every employee of that organization. TQM has three cornerstones:

TQM thus stresses improvement rather than conformance. CMM (see Section 6.6) builds on TQM and many of the requirements engineering techniques discussed in Chapter 9 owe a tribute to TQM as well.

The International Organization for Standardization (ISO) has developed ISO 9000, a series of standards for quality management systems (ISO 9000:2000, ISO 9001:2000, and ISO 9004:2000). ISO 9000 gives the fundamentals and vocabulary of the series of standards on quality systems. ISO 9001 integrates three earlier standards (labeled ISO 9001, ISO 9002 and ISO 9003) and specifies requirements for a quality system for any organization that needs to demonstrate its ability to deliver products that satisfy customer requirements. ISO 9004 contains guidelines for performance improvement. It is applicable after implementation of ISO 9001.

ISO 9001 is a generic standard that can be applied to any product. A useful complement for software is ISO/IEC 90003:2004, which contains guidelines for the application of ISO 9001 to computer software. It is a joint standard of ISO and the International Electrotechnical Commission (IEC). The scope of ISO/IEC 90003 is that it 'specifies requirements for a quality management system where an organization:

The standard is very comprehensive. It uses five perspectives from which to address the management of quality in software engineering:

Many organizations have tried to obtain ISO 9000 registration. The time and cost this takes depends on how much the current processes deviate from the ISO standards. If the current quality system is not already close to conforming, then ISO registration may take at least one year. ISO registration is granted when a third-party accredited body assesses the quality system and concludes that it does conform to the ISO standard. Reregistration is required every three years and surveillance audits are required every six months. ISO registration thus is a fairly drastic and costly affair, after which you certainly cannot sit back but have to keep the organization alert.

Since software development projects have some rather peculiar characteristics (frequent changes in requirements during the development process and the invisible nature of the product during its development), there is a need for quality assurance procedures which are tailored towards software development. This is the topic of Section 6.5.

The purpose of software quality assurance (SQA) is to make sure that work is done the way it is supposed to be done. More specifically, the goals of SQA (Humphrey, 1989) are:

Note that the SQA people themselves are not responsible for producing quality products. Their job is to review and audit, and to provide the project and management with the results of these reviews and audits.

There are potential conflicts of interest between the SQA organization and the development organization. The development organization may be facing deadlines and may want to ship a product, while the SQA people have revealed serious quality problems and wish to defer shipment. In such cases, the opinion of the SQA organization should prevail. For SQA to be effective, certain prerequisites must be fulfilled:

The review and audit activities and the standards and procedures that must be followed are described in a software quality assurance plan.

IEEE Standard 730 offers a framework for the contents of a Quality Assurance Plan for software development (IEEE730, 1998). Table 6.9 lists the contents of such a document. IEEE Standard 730 applies to the development and maintenance of critical software. For non-critical software, a subset of the requirements may be used.

IEEE Standard 983 (IEEE983, 1986) is a useful complement to Standard 730. IEEE Standard 983 offers further guidelines as to the contents of a quality assurance plan and its implementation, evaluation, and modification.

A software quality assurance plan describes how the quality of the software is to be assessed. Some quality factors can be determined objectively. Most factors at present can be determined only subjectively. Most often then, we will try to assess the quality by reading documents, by inspections, by walkthroughs, and by peer reviews. In a number of cases, we may profitably employ tools during quality assurance, in particular, for static and dynamic analysis of program code. The actual techniques to be applied here are discussed in Chapter 13.

Consider the following course of events in a hypothetical software development project. An organization is to develop a distributed library automation system. A centralized computer hosts both the software and the database. A number of local libraries are connected to the central machine through a Web-based interface. The organization has some experience with library automation, albeit only with stand-alone systems.

In the course of the project, a number of problems manifest themselves. At first they seem to be disconnected and they do not alarm management. It turns out that the requirements analysis has not been all that thorough. Local requirements turn out to differ on some minor points. Though the first such deviations can be handled quite easily, keeping track of all the change requests becomes a real problem after a while. When part of the system has been realized, the team starts to test the Web interface but it turns out to be too complex and time-consuming.

The project gets into a crisis eventually. Management has no proper means to handle the situation. It tries to cut back on both functionality and quality in a somewhat haphazard way. In the end, a rather unsatisfactory system is delivered two months late. During the subsequent maintenance phase, a number of problems are solved, but the system never becomes a real success.

Though the above description is hypothetical, it is not all that unrealistic. Many an organization has insufficient control over its software development process. If a project gets into trouble, it is usually discovered quite late and the organization can only react in a somewhat chaotic way. More often than not, speed is confused with progress.

An important step in trying to address these problems is to realize that the software development process can indeed be controlled, measured, and improved. In order to gauge the process of improving the software development process, Watts Humphrey developed a software maturity framework which has evolved into the Capability Maturity Model (CMM). This framework owes tribute to Total Quality Management (TQM), which in turn is based on principles of statistical quality control as formulated by Walter Shewart in the 1930s and further developed by W. Edwards Deming and Joseph Juran in the 1980s. Originally, there were separate CMM models for software engineering, systems engineering, and several others. These have now been integrated and carry the label CMMI.^[8] CMM and CMMI were developed at the Software Engineering Institute (SEI) of Carnegie Mellon University. The version described here is CMMI version 1.2 for software engineering and systems engineering (CMMI Product Team, 2006).

In CMM (and CMMI), the software process is characterized as one of five maturity levels, evolutionary levels toward achieving a mature software process. To achieve a certain maturity level, a number of process areas must be in place. These process areas indicate important issues that have to be addressed in order to reach that level. Taken together, the process areas of a level achieve the set of goals for that level. Figure 6.4 lists the maturity levels and associated process areas of CMMI.

CMMI's maturity levels can be characterized as follows:

In 1989, Humphrey investigated the state of software engineering practice with respect to the CMM (Humphrey et al., 1989). Although this study concerned the Department of Defense (DoD) software community, there is little reason to expect that the situation was much rosier in other environments. According to his findings, software engineering practice at that time was largely at the initial level. There were a few organizations operating at the repeatable level, and a few projects operating at the defined level. No organization or project operated at the managed or optimizing levels.

In the ensuing years, a lot has happened. Many organizations have initiated a software process improvement (SPI) program to achieve a higher maturity level. Most of these improvement programs concern a move to the repeatable or defined level. The number of organizations at these levels has significantly increased since 1989. There are still few organizations or projects at the managed or optimizing levels.

Reports from practical experience show that it takes about two years to move up a level. The cost ranges from $500 to $2000 per employee per year. The benefits, however, seem to easily outweigh the cost. Several companies have reported a return on investment of at least 5 to 1: every dollar invested in a process improvement program resulted in cost savings of at least five dollars.

In the preceding sections, we have discussed various ways to review the quality of a software product and the associated development process. The development organization itself should actively pursue the production of quality products, by setting quality goals, assessing its own performance and taking actions to improve the development process.

This requires an understanding of possible inadequacies in the development process and possible causes thereof. Such an understanding is to be obtained through the collection of data on both the process and the resulting products, and a proper interpretation of those numbers. It is rather easy to collect massive amounts of data and apply various kinds of curve-fitting techniques to them. In order to be able to properly interpret the trends observed, they should be backed by sound hypotheses.

An, admittedly ridiculous, example is given in Table 6.10. The numbers in this table indicate that black cows produce more milk than white cows. A rather naïve interpretation is that productivity can be improved significantly by repainting all the white cows.

Though the example itself is ridiculous, its counterpart in software engineering is not all that far-fetched. Many studies, for example, have tried to determine a relation between numbers indicating the complexity of software components and the quality of those components. Quite a few of those studies found a positive correlation between such complexity figures and, say, the number of defects found during testing. A straightforward interpretation of those findings then is to impose some upper bound on the complexity allowed for each component. However, there may be good reasons for certain components having a high complexity. For instance, Redmond and Ah-Chuen (1990) studied complexity metrics of a large number of modules from the MINIX operating system. Some of these, such as a module that handles escape character sequences from the keyboard, were considered justifiably complex. Experts judged a further decomposition of these modules not justified. Putting a simple upper bound on the allowable value of certain complexity metrics is too simplistic an approach.

An organization has to discover its opportunities for process improvements. The preferred way to do so is to follow a stepwise, evolutionary approach in which the following steps can be identified:

These steps are repeated, so that the effect of the actions is validated, and further hypotheses are formulated. By following this approach, the quest for quality will permeate your organization, which will subsequently reap the benefits.

One example of this approach is discussed in (Genuchten, 1991). He describes an empirical study of reasons for delay in software development. The study covered six development projects from one department. Attention was focused on the collection of data relating to time and effort, such as differences between a plan and reality. A one-page data collection form was used for this purpose (see Table 6.11).

Some 30 reasons for delay were identified. These were classified into six categories after a discussion with the project leaders, and finalized after a pilot study. The reasons for delay were found to be specific to the environment.

A total of 160 activities were studied from mid 1988 to mid 1989. About 50% of the activities overran their plan by more than 10%. Comparison of planned and actual figures showed that the relative differences increased towards the end of the projects. It was found that one prime reason for the difference between plan and reality was 'more time spent on other work than planned'. The results were interpreted during a meeting with the project leaders and the department manager. The discussion confirmed and quantified some existing impressions. For some, the discussion provided new information. It showed that maintenance actions constantly interrupted development work. The meeting included a discussion on possible actions for improvement. It was decided to schedule maintenance as far as possible in 'maintenance weeks' and include those in quarterly plans. Another analysis study was started to gain further insights into maintenance activities.

This study provides a number of useful insights, some of which reinforce statements made earlier:

In this chapter, we have paid ample attention to the notion of quality. Software quality does not come for free. It has to be actively pursued. The use of a well-defined model of the software development process and good analysis, design, and implementation techniques are a prerequisite. However, quality must also be controlled and managed. To be able to do so, it has to be defined rigorously. This is not without problems, as we saw in Sections 6.2 and 6.3. There exist numerous taxonomies of quality attributes. For each of these attributes, we need a precise definition, together with a metric that can be used to state quality goals, and to check that these quality goals are indeed being satisfied. Most quality attributes relate to aspects that are primarily of interest to the software developers. These engineer-oriented quality views are difficult to reconcile with the user-oriented 'fitness for use' aspects.

For most quality attributes, the relation between what is actually measured (module structure, defects encountered, etc.) and the attribute we are interested in is insufficiently supported by a sound hypothesis. For example, though programs with a large number of decision points are often complex, counterexamples exist which show that the number of decisions (essentially, McCabe's cyclomatic complexity) is not a good measure of program complexity. The issue of software metrics and the associated problems is further dealt with in Chapter 12.

Major standards for quality systems have been defined by ISO and IEEE. These standards give detailed guidelines as regards the management of quality. Quality assurance by itself does not guarantee quality products. It has to be supplemented by a quality program within the development organization. Section 6.7 advocates an evolutionary approach to establishing a quality program. Such an approach allows us to gradually build up expertise in the use of quantitative data to find opportunities for process improvements.

Finally, we sketched the software maturity framework developed by the Software Engineering Institute. This framework offers a means of assessing the state of software engineering practice, as well as a number of steps for improving the software development process. One of the major contributions of CMM and similar initiatives is their focus on continuous improvement. This line of thought has subsequently been successfully applied to other areas, resulting in, amongst others, a People-CMM, a Formal Specifications-CMM, and a Measurement-CMM.

Fenton and Pfleeger (1996) provide a very thorough overview of the field of software metrics. (Endres and Rombach, 2003) is a software engineering textbook focusing on empirical observations and measurements. The measurement framework discussed in Section 6.1 is based on (Kitchenham et al., 1995). Kaner and Bond (2004) also give a framework for evaluating metrics. (Kitchenham et al., 2007) discuss common errors made when applying software metrics. (Software, 1997b) and (JSS, 1995a) are special journal issues on software metrics. Many of the articles in these issues deal with the application of metrics in quality programs.

One of the first major publications on the topic of measurement programs is (Grady and Caswell, 1987). Success factors for measurement programs can be found in (Hall and Fenton, 1997) and (Gopal et al., 2002). Pfleeger (1995) elaborates on the relation between metrics programs and maturity levels. Niessink and van Vliet (1998b) give a CMM-like framework for the measurement capability of software organizations.

The best known taxonomies of software quality attributes are given in (McCall et al., 1977) and (Boehm et al., 1978). The ISO quality attributes are described in (ISO 9126, 2001) and (Côté et al., 2006). Critical discussions of these schemes are given in (Kitchenham and Pfleeger, 1996) and (Fenton and Pfleeger, 1996). (Suryn et al., 2004) give an overview of ISO/IEC 90003.

Garvin (1984) gives quality definitions. Different kinds of benefit in a value-based definition of quality are discussed in (Simmons, 1996). For an elaborate discussion of Total Quality Management, see (Bounds et al., 1994) and (Ishikawa, 1985).

The Capability Maturity Model (CMM) is based on the seminal work of (Humphrey, 1988) and (Humphrey, 1989). For a full description of the Capability Maturity Model Integrated (CMMI), see (CMMI Product Team, 2006). Practical experiences with software process improvement programs are discussed in (Wohlwend and Rosenbaum, 1994), (Diaz and Sligo, 1997), (Fitzgerald and O'Kane, 1999), (Conradi and Fuggetta, 2002) and (Dyb°a, 2005). Rainer and Hall (2003) discuss success factors and Baddoo and Hall (2003) discuss de-motivators for SPI. (Basili et al., 2002) discuss lessons learned from 25 years of process improvement at NASA. A survey of benefits and costs of software process improvement programs is given in (Herbsleb et al., 1997) and (Gartner, 2001). High-maturity, CMM level 5 organizations are discussed in (Software, 2000) and (Agrawal and Chari, 2007).

The Personal Software Process (PSP) is described in (Humphrey, 1996) and (Humphrey, 1997a). BOOTSTRAP is described in (Kuvaja et al., 1994) and SPICE in (El Emam et al., 1997).

Criticisms of CMM-like approaches are found in (Fayad, 1997) and (Fayad and Laitinen, 1997). El Emam and Madhavji (1995) discuss the reliability of process assessments.

Process improvement is the topic of several special journal issues; see (CACM, 1997) and (Software, 1994a). The journal Software Process: Improvement and Practice is wholly devoted to this topic.