Example isn't another way to teach.

It is the only way to teach.

—A. Einstein

The process of determining the extent to which a given object (e.g., an industrial product or a complex software system) satisfies a set of stakeholder’s requirements can be organized in a quantitative way. For example, if the evaluated object is a home (house or condo), then its users/stakeholders (members of homebuyer’s family) may specify quantitative requirements for area, location, number of bedrooms, number of bathrooms, garage, backyard, kitchen and household appliances, neighborhood, availability of public transportation, and so on. Generally, the requirements are specified so that users can achieve desired goals; the achievement of goals depends on the availability and suitability of evaluated objects. We assume that the requirements are realistic and attainable. For example, in the case of home, it is not realistic to require features that are beyond the attainable price range. Generally, if the requirements are realistic, then evaluated objects should be able to satisfy the majority of requirements.

1.2.1 Stakeholders and Their Goals

All requirements reflect the needs of specific stakeholder/user, and evaluation criteria cannot be designed unless it is clear who the user is, and what are user’s goals, interests and needs. Therefore, each evaluation starts by defining the stakeholder and the evaluated system. In this chapter we present a very simplified quantitative evaluation of a home, and a realistic home evaluation is presented in Chapter 4.2. We assume that the stakeholder is a family of five people: two parents, one child in elementary school, and two preschool children. This example will follow the steps of intuitive evaluation from Chapter 1.1.

If the stakeholder/homebuyer is defined as a family with children and the evaluated system is a home (house/condo), then the general goal is to provide an environment for healthy and prosperous life for all family members. In other words, we might think of a group of five stakeholders and each of them has a separate goal (e.g., the proximity to work, the proximity to school, etc.). In addition, a collective goal of the family is to simultaneously satisfy individual goals of all family members. To attain this collective goal, the family first specifies a list of home attributes that will be used for evaluation and the available home price range. In the next step, for each attribute it is necessary to specify a separate requirement that accurately reflects stakeholder’s needs.

1.2.2 Attributes

The derivation of attributes is a process that is illustrated in Fig. 1.2.1. We create an attribute tree by decomposing each node into its components. In this simple example, the suitability of home is evaluated using only two components: the suitability of home location and the quality of home (reduced to the evaluation of available space). Each of these components can be further decomposed. The location is decomposed into the distance from work for parents and the distance from school for children. Similarly, the quality of the home is decomposed and the resulting attributes are the total area of home and the number of bedrooms. The decomposition process is completed when we generate components that cannot be further decomposed.

We will use the following terminology: All attributes that can be decomposed will be called subsystem (compound) attributes, and those that cannot be decomposed will be called elementary (input) attributes. For example, the number of bedrooms cannot be further decomposed, and consequently, that is one of input attributes. In this simplified example, we will use only four input attributes (see Chapter 4.2 for a realistic list of 36 attributes and realistic criteria for home evaluation).

1.2.3 Attribute Criteria

The second step in the evaluation process is to specify requirements that each elementary attribute must satisfy. A sample requirement for the distance from work for parents is shown in Fig. 1.2.2. The degree of satisfaction of requirements is called the preference or suitability score (or, for short, the preference or suitability) and for simplicity it can be interpreted as the percentage of satisfied requirements. If the travel time to work (using any form of transportation) is less than or equal to 10 minutes, the working parent is completely satisfied. If the travel time is greater than 60 minutes, this is not acceptable and the degree of satisfaction is 0. For travel times between 10 minutes and 60 minutes we use for simplicity the linear interpolation. For example, if the travel time is 35 minutes, the corresponding degree of satisfaction is 50%. A soft computing interpretation of this value would be that the degree of truth is 0.5 for the statement that the working parent is completely satisfied with the travel time to work.

The criterion function shown in Fig. 1.2.2 is called the elementary attribute criterion or simply elementary criterion, to emphasize the fact that this is the simplest component requirement that the user can specify. Of course, every effort must be made that each elementary criterion reflects the stakeholder’s needs in the most accurate way. For another stakeholder, the values of 10 and 60 minutes might be different, and we assume that each stakeholder can provide strong arguments supporting the selected parameters of each elementary criterion.

Elementary criteria are frequently presented in the standardized rectangular form shown in Fig. 1.2.3. This rectangular form has three fields: title, preference scale, and description. The title field contains the full name of the evaluated attribute. The preference scale includes the values of breakpoints that correspond to specific degrees of satisfaction. It can have any number of breakpoints, and it assumes a linear interpolation between them. The description field contains comments that are necessary for correct interpretation of the criterion function, the definition of units of measurement, and (if necessary), the method for measurement or computation of the value of attribute.

The requirements for the travel time to school for small children must be stricter than the criterion for the travel to work for parents. A corresponding elementary criterion with four breakpoints is shown in Figs. 1.2.4 and 1.2.5. Let us note that the end values of all preference scales denote the beginning of the constant value of preference. In Fig. 1.2.5, if the time is greater than 25 minutes, the values of preference remain constant (zero). Similarly, if the time is less than or equal to 5 minutes, the preference is again constant (100%). Increasing the number of breakpoints increases the precision of the criterion function.

In this simplified case, the elementary criteria in Figs. 1.2.3 and 1.2.5 completely specify user requirements regarding the location of the home. Of course, in the case of intuitive evaluation, the requirements can be expressed in verbal and not in a quantitative way, but the homebuyer always knows what requirements must be satisfied and can use them in the process of evaluation and comparison of competitive homes.

In the case of the available space, the user must first specify a criterion for the total area of home. This criterion is shown in Fig. 1.2.6. The first property of this criterion is that the minimum acceptable area is specified to be 80 m². This area satisfies 50% of the stakeholder’s requirements. However, the stakeholder is not ready to accept an area that would be less than 80 m². Consequently, the area of 79 m² yields a zero preference. The area of 120 m² completely satisfies the stakeholder’s needs, and larger areas get no extra credit. The reason for this requirement is that the family feels that areas that are greater than 120 m² do not yield significant benefit, but can cause significant increases of the cost of purchase and/or maintenance.

The requirements for the number of bedrooms are shown in Fig. 1.2.7. Because of three children, the ideal home should have four bedrooms. In extreme situations (financial difficulties, temporary solutions, etc.), two bedrooms might still be acceptable, even though such a home satisfies only one half of the family requirements. This is a discrete criterion because the number of bedrooms is a natural number, but the preference scale can be used as before.

With these four elementary criteria, the user systematically and precisely specified basic requirements, but this is not enough for evaluation. Any specific home can now be characterized by four input values: the travel time to work (T_w), the travel time to school (T_s), the total area (A) and the number of bedrooms (B). In addition, the cost of the home (C) will also play a critical role in deciding which home is the best alternative (i.e., the most suitable for the specific family).

Suppose that $ denotes an arbitrary monetary unit, not necessarily the US dollar. Let the available funding be limited to $750,000, and within that constraint we have four competitive homes called Alpha, Beta, Gamma, and Delta, shown in Table 1.2.1. Using the presented elementary criteria, the homebuyer can generate four suitability degrees (preference scores X_w, X_s, X_a, and X_b) shown in Table 1.2.2, that reflect homebuyer’s degree of satisfaction with each of the four individual requirements. Of course, in both intuitive and quantitative evaluation, each individual requirement is separately investigated, but our goal is to determine how good is each home as a whole, not how good are individual components. Therefore, the next step is to find a method to combine (aggregate) the four elementary preference scores (suitability degrees) and the cost of each home to evaluate each of the competitors and to produce their final ranking.

1.2.4 Simple Direct Ranking

It is important to differentiate the evaluation and the relative ranking. The goal of evaluation is to determine the overall suitability of a home expressed as the degree of satisfying the homebuyer’s requirements. The overall suitability can be determined for each home separately and independently of other homes. On the other hand, it is possible to perform simple ranking in a strictly relative way, without any evaluation. For example, we might directly take data from the Table 1.2.1 and perform ranking of each component, so that the best component gets the rank 1 and the worst gets the rank 4. Then we add ranks for each home, and the home with the lowest sum is the best alternative. This process, called direct ranking (a variation of the Borda count method [AA15]), is shown in Table 1.2.3. If some of the analyzed values are equal, then the resulting rank is the mean value of individual ranks; e.g., if two systems share the first place, their rank will be . Using the sum of ranks, we generate the quality (suitability) rank that shows that Gamma is the best system.

The next step is to take into account the cost. We can introduce ranking for costs, where the rank of the cheapest system is 1. After adding the cost ranks and the quality ranks, we get Q + C sum, which is used in Table 1.2.3 to make final ranking that shows that Gamma and Delta share the first place. This is a typical example of trading accuracy for simplicity, and we do not suggest this approach (unfortunately, it is used in politics).

The presented direct ranking technique can sometimes be useful, because it is extremely simple, but it has many serious drawbacks. The first problem is that direct ranking indicates who the best system in a group is, but this does not mean that the best system is able to satisfy user needs. The winner of the competition can still be worthless according to the stakeholder’s requirements. The second problem of direct ranking is that there is no information on how big is the difference between systems in the list; the same ranking applies to both negligible and substantial differences. The third problem is that all components have the same level of relative importance. For example, by adding ranks for home suitability and cost we implicitly consider that the home suitability and cost are always equally important; in many cases this is not true. The fourth problem is that ranking cannot express logic relationships between aggregated ranks. If one of components is so bad that the system becomes unacceptable, this can be ignored by the direct ranking method. For example, for system Delta minutes and the corresponding rank in Table 1.2.3 is 4. However, all results in Table 1.2.3 would remain unchanged for any value minutes. So, even if hours the Delta house would not be rejected from the competition, but it remains one of the two winners. This is meaningless and unacceptable, and shows that in this case, the direct ranking is not a right approach. Indeed, the relative ranking and evaluation are two different things. We can evaluate to perform ranking, but we cannot perform ranking instead of evaluation.

1.2.5 Aggregation of Attribute Suitability Degrees

The process of evaluation can follow the usual pattern of intuitive reasoning that applies in such situations. The intuitive reasoning consists of two steps. First, the evaluator aggregates all elementary suitability degrees (preferences) to get an overall suitability score that indicates the suitability of the system as a whole. Second, the overall suitability score is combined with the total cost to find the most convenient combination (the highest value that combines both a good quality and a reasonable cost).

It is important to note that the cost of home is not an attribute that should be directly merged with other attributes. Indeed, the total cost corresponds to the overall suitability of home and consequently the cost should not be considered an individual attribute and mixed with other individual attributes of the evaluated home. So, our problem is now to find a method for aggregation of four elementary attribute suitability scores and computing the overall suitability scores for all competitive homes.

Following the pattern of intuitive reasoning, the process of aggregating attribute suitability scores is shown in Fig. 1.2.8. We first answer the question about the quality of location by aggregating attribute suitability scores for the travel time to work (X_w) and the travel time to school (X_s) to get the subsystem suitability score for the overall quality of home location (X_loc). Then, we evaluate the subsystem of space (which in this example represents the home quality) by aggregating preference scores of the area (X_a) and the number of bedrooms (X_b) to get the subsystem suitability score for the overall quality of available space (X_space).

Once we know the degree of satisfaction with the location and the space of home, we are ready for the final step: the aggregation of subsystem suitability scores for location and space to generate the resulting overall suitability for the home as a whole (X₀). The inputs for aggregation are X_w, X_s, X_a, and X_b. They are expressed as percentages of satisfied requirements. Consequently, the overall suitability X₀ is also interpreted as an overall percentage of satisfied requirements, taking properly into account all elementary attribute requirements. In other words, X₀ is an indicator of the overall suitability of the evaluated home. It is important to note that the computational structure shown in Fig. 1.2.8 represents a complete evaluation criterion: after selecting aggregators it will generate a meaningful value of X₀ for any combination of the input attribute values T_w, T_s, A, and B.

Since , we must now find what kind of function should be the aggregator H. Let us immediately make clear that this is a hard question and one that is beyond this introductory section—its full answer is in Part Two and Part Three of this book. However, something can be done right now: we can easily see that we must not use the arithmetic mean . There are two questions that we must answer when selecting the aggregator H: (1) “Would you accept a home location if the travel time to work is greater than 60 minutes?” and (2) “Would you accept a home location if the travel time to school is greater than 25 minutes?” According to our elementary attribute criteria, the obvious answer to these questions is no. If any of the two travel times has a zero suitability score, that must yield to indicate a complete dissatisfaction with such home location. Thus, the conditions for the H function are: , and . In Chapter 1.1 we called these conditions the mandatory requirements. Now we see that the arithmetic mean does not satisfy the mandatory requirement conditions and consequently, it must not be used as the aggregator of X_w and X_s.

The reason for rejecting the arithmetic mean is that the homebuyer essentially wants a home that simultaneously has a short distance to work and a short distance to school, but the arithmetic mean is an aggregator that cannot model this level of simultaneity. On the other hand, there are many functions that satisfy the mandatory requirement condition , and one of those that have suitable properties is the harmonic mean illustrated in Fig. 1.2.9. For simplicity, we will assume equal importance of all inputs and use the harmonic mean to compute the subsystem suitability scores. For the location of home, we have: .

In the case of aggregating the suitability scores for the area of home and the number of bedrooms, we have again two mandatory requirements. Homes with unacceptable area or unacceptable number of bedrooms should be rejected. Consequently, our second aggregator can also be the harmonic mean: . In the final step of aggregation, for simplicity we can again use the harmonic mean because homes with unacceptable location or unacceptable space should be rejected:

We used equal weights for all inputs. Not surprisingly, some homebuyers may consider the home location more important than the available space (or vice versa). In quantitative models the level of relative importance is usually expressed using positive normalized weights that have the sum equal to 1. So, if the home location is considered two times more important than the available space, then the corresponding weights would be 0.67 and 0.33. In such a case, we might compute the overall preference as follows:

Using the attribute criteria and three harmonic mean aggregators, we can now evaluate the four competitive homes and get the results shown in Table 1.2.4. The location of home Delta is rather poor—it satisfies less than one half of stakeholder’s requirements. This is the reason why Delta is the least desirable from the standpoint of quality, but it is convincingly less expensive than other competitors. The best quality has the home Gamma, which satisfied 93.3% of homebuyer’s requirements. So, now we know how good our best home is and we also know that the next competitor, Beta, satisfies approximately 10% less requirements than the leading home Gamma. However, the selection of the best home cannot be based only on the overall preference/suitability scores because for most homebuyers the cost is also an important factor for making the final decision.

1.2.6 Using Cost and Suitability to Compute the Overall Value

Our last problem in this example is to combine the overall preference score X₀ and the total cost C to compute an overall indicator of the value of home, V. There are various models of the overall value function V(X₀, C), and they are described in detail as part of the cost/preference (or suitability/affordability) analysis in Chapter 3.6. For simplicity, we use here the formula V = X₀/C that shows the level of satisfaction of user requirements per monetary unit of cost. This model rewards high overall suitability and low cost, assuming that the total cost and the overall suitability have equal role and equal importance. Using this model, we computed the results in Table 1.2.4 and found that the home Gamma is the winner, having the highest overall value. The remaining three competitors have rather similar normalized overall values, more than 16% below the level of Gamma.

The resulting ranking of competitors is shown in Tables 1.2.5. These tables also show what would happen if instead of harmonic mean we made an oversimplification and used the arithmetic mean. The arithmetic mean is unable to sufficiently penalize the Delta home for its poor location. Consequently, even though it has the lowest overall suitability score (only 68%), in the case of arithmetic mean the Delta home would be the winner of this competition because of its low price. This example shows the importance of using a correct logic of aggregation operators, and a significant part of this book is devoted to the development and use of flexible and justifiable aggregators. In particular, the problems of oversimplification of aggregation process caused by simple scoring techniques are analyzed in detail in the next chapter.

The presented simplified home selection example provides an outline of a typical evaluation process for any type of evaluated object. This process is the same for intuitive and for quantitative evaluation. In other words, the quantitative evaluation strictly follows the steps of the intuitive evaluation process. If we take into account that we only have four input attributes and their values summarized in Table 1.2.1, an interesting question is whether intuitive evaluators could generate the results from Tables 1.2.4 and 1.2.5. Regardless of the simplicity of this problem and the possibility to reach similar conclusion using the intuitive approach and the quantitative approach, it is easy to note that intuitive evaluation even with only four attributes and four systems needs substantial concentration and effort, and can yield uncertainty, low precision and justifiability problems.

It is easy to note that the intuitive evaluation process has a complexity ceiling and that ceiling is rather low, perhaps in the vicinity of the presented simple example problem. If the number of attributes and objects is larger, then the intuitive evaluation is not possible (professional evaluation problems can have up to several hundred inputs). On the other hand, the quantitative approach offers a sequence of easily justifiable simple steps and generates results with more accuracy and less effort.

Our goal in this section was to show the main steps and the basic properties of quantitative evaluation. However, the precision and usability of our model was obviously rather low: The number of inputs was too small, all inputs were equally important, and all aggregators were the same. The methodology for creating more complex, precise, realistic, and justifiable evaluation criteria is presented in Part Two and Part Three of this book.