Outline of the chapter

Having given a rough introduction to configurable systems and an explanation of why it is important to stress the focus on quality of those systems, we detail this in Section 9.2. Furthermore, we provide a framework describing how techniques for quality assurance of configurable systems can be implemented. For this purpose, we first introduce the means to integrate the needed flexibility into a systems engineering process (Section 9.2.1). Then, in Section 9.2.2, we describe how a quality assurance concept in this context can be set up.

Afterwards we show how to apply this concept to two different quality assurance techniques: MBT is a common technique used to attain confidence in a correctly implemented functionality. We discuss in Section 9.3 how to apply MBT to configurable systems. Model-based software deployment is another quality assurance technique that provides a means to construct correct and good deployments according to predefined safety and resource constraints. By this, performance, efficiency, and reliability quality aspects can be ensured. How model-based deployment can be applied to configurable systems is discussed in Section 9.4.

We substantiate the explanations with example models that describe the properties of a flight management system (FMS), which is introduced in Section 9.1. To conclude, we give an overview of related work in Section 9.5 and summarize this chapter in Section 9.6.

9.2.1 Configurable models

Before we define how to express configurability in a model, we first have a closer look at which kinds of models we consider and where the distinction of different models or views comes from.

9.2.2 Quality assurance of configurable systems

In the setting of a model-based development process, quality assurance is based on the evaluation of the models that represent the design decisions drawn so far in the process. This might either be an evaluation that is explicitly performed after the model has been created in order to prove that the quality requirements are still respected, or it may be an evaluation that is built into the construction of the model, for instance, when the model construction is machine-supported.

Given the framework described above—an overall model, a feature model, the feature-model-relations, and the model configuration mechanism based on feature selections—we discuss now how quality attributes of configurable systems can be evaluated.

Suppose a method is given that allows the evaluation of a model without variation points. In general, there are two possibilities to extend this method to a model with variation points:

1. Product-centered approaches evaluate all product models or a representative subset of them by

a. deriving all (or sufficiently many) product models.

b. evaluating each model.

c. condensing the results to an evaluation of the product line.

2. Product-line-centered approaches extend the method to the evaluation of variation points, that is, they define how to integrate for example, options, alternatives, and parameters into the evaluation without resolving single product variants.

It should be noted that the evaluation criteria on the product line level may slightly differ from those on the product level. In other words, an optimal solution for the product line does not necessarily consist of optimal solutions for every single product.

Product-centered and product-line-centered approaches are illustrated in a schematic diagram in Figure 9.2. The lower layer in the figure, the product model level, represents single system evaluations: A variability-free product model Model_P is evaluated by the function evaluate_P. The result of this evaluation is denoted by Result_P.

The upper layer in Figure 9.2, the product line model level, shows the variability-aware function evaluate_PL, which is used to analyze the product line model Model_PL and produces the corresponding result Result_PL. The product line model Model_PL includes a feature model that exhibits the supported variability of the product line, base models that represent the realization of the product line (e.g., design models) as well as the mapping model between the feature model and the base models in order to specify the presence conditions of base model’s elements by features.

The function derive_Model denotes the derivation of a product model Model_P from the product line model Model_PL for the feature configuration corresponding to product P. Analogously, derive_Result represents the derivation of a product result from the product line result. The arrow set labeled aggregate_Results stands for the inverse operation of aggregating the set of all product results Result_P and building the single product-line result Result_PL.

9.3.1 What is MBT?

Testing is a widespread means of quality assurance (Ammann and Offutt, 2008; Beizer, 1990; Myers, 1979). The idea is to compare the system under test to a specification by systematically stimulating the system and comparing the observed system behavior to the specified one. The basic approach is to first design test cases based on requirements and to execute them on the system under test afterwards. Complete testing is usually impossible due to loops and a large input data space. So, heuristics like coverage criteria are used to make a quantified statement about the relative quality of the test suite. It has been experienced in many projects that the costs for testing range between 20 and 50% of the overall system engineering costs. This is substantiated by our experiences with industrial transfer projects. For safety-critical systems, testing costs can easily reach up to 80% (Jones, 1991). Nolan et al. (2011) discovered on the basis of data collected at Rolls-Royce that around 52% of the total development effort is spent on verification and validation activities (V&V). They argue that applying a software product line approach will reduce development efforts, but not necessarily the efforts for V&V. They report that V&V can comprise up to 72% of a product’s overall effort in a software product line context, and it could theoretically increase to more than 90% in cases of very low costs because of high reuse. They conclude that “the percentage is higher on a product line not because verification has increased but because the development effort has decreased making verification costs higher in relation to the overall effort,” and that “testability therefore becomes critical for safety-critical product lines.”

Thus, automation is a means to reduce these costs. It can be applied to test execution and to test design. Automating the test design is often implemented by deriving a formal model from requirements and by using test generators that produce a set of test cases based on this model. Corresponding approaches are also called model-based testing (Utting and Legeard, 2006; Zander et al., 2011). There are many different approaches with respect to the chosen modeling language, the modeled aspects of the system, and the algorithms for test generation (Utting et al., 2012).

In the following, we use state machines as defined in Unified Modeling Language (UML) (OMG, 2011) to describe the behavior that is used for test design. We additionally annotate the state machines with variability information and configuration rules in order to use the state machine as the base model for the MBT of product lines. There are several approaches to steer the test generation on the model level (Chilenski and Miller, 1994; Rajan et al., 2008; Weißleder, 2010). We will focus on the widespread approach of applying coverage criteria to the state machine level, for example, such as transition coverage. This criterion defines a type of model elements (here: all transitions of the model) that have to be covered by the generated test cases. There are also coverage criteria for feature models describing the coverage of feature selections and their combinations in the tested products.

9.3.2 Test model for the flight management system

For MBT it is necessary to use models from which test cases can be automatically generated. For our FMS example, we model the application behavior of the FMS product line as a configurable UML state machine. It describes the potential behavior of all products. By including and excluding elements of this model, it can be tailored to describe the behavior of only one product.

The UML state machine is related to the feature model in order to extract product-specific variants of the state machine according to a certain configuration. All variable elements of the state machine have been assigned with a configuration rule consisting of elements of the feature model. Configuration rules define the conditions when those elements need to be included in or excluded from state machine.

By selecting a certain product variant on the feature level (e.g., the defined products in Table 9.1), the corresponding state machine for test generation can be automatically derived using the feature mapping (i.e., by interpretation of the configuration rules).

Figure 9.3 shows the configurable state machine for the FMS as well as the configuration rules using the features from the feature model in Figure 9.1. Variation points have been explicitly marked in the state machine. For example, the transition requestGraphics/showGraphics between the states computing and display is only present in a product that has the MFD feature. Usually, the configuration rule can be any Boolean expression using features as predicates. In case of an optional state (e.g., checkTrajectoryOnly), all transitions to or from this state are removed from the state machine if the state’s configuration rule has not been satisfied by the corresponding configuration (e.g., if feature SurSys has been deselected). Thus, those transitions need not be explicitly marked as optional because they depend on the existence of the states. For the sake of simplicity, we have not modeled the complete behavior in the given state machine. Also, not every feature from the feature model has been used here.

We have sketched here how to design product line models for the test generation using the 150% product line modeling approach and how to link them with the feature models. However, this only describes the static connection between the different modeling artifacts. As the framework in Figure 9.2 already depicts, there are several ways of using this information for test design automation of product lines which is explained in the following.

9.3.3 Applying MBT

We now instantiate the basic framework for quality assurance of configurable systems (see Section 9.2.2) for MBT. We show how MBT can be done for product-centered as well as for product-line-centered approaches. Before going into the details of both approaches, we instantiate the framework’s artifacts in Figure 9.2 for the MBT context.

The product line model Model_PL is the union of the feature model, the base model (i.e., the state machine), and the mapping between the features and base model elements as sketched in Section 9.3.2. Each Model_i consists of a concrete product variant (i.e., a feature model configuration, cf. Table 9.2) and the correspondingly resolved model, that is, a state machine that describes the behavior of product i only.

The direct result of test generation is the set of generated test cases. Thus, Result_i describes the result for just one product. More interesting information, for example, about transition coverage, can be derived for each product. But what is the meaning of a test case on the product line model level? How should Result_PL—the result of the test generation on the product line model level—be represented? There are several possible forms of test case representations on the product line model level.

One extreme form is a direct mapping of product variants to test cases. This solution conforms to a product-wise description and does not show information on the product line model level. The opposite extreme form consists of a direct mapping from single features to test cases. Given that single test cases usually require information about more than one feature inclusion, this approach is not practicable. The form between both extremes is a mapping from feature configurations (i.e., the inclusion or exclusion of some but not all features) to test cases (see, e.g., Table 9.3 below). This solution allows describing the links between test cases and necessarily included or excluded features. Not making any assumptions about features that are unimportant for the test case allows for optimizing the inclusion of concrete product variants for testing. Summing up, Result_PL is the set of pairs consisting each of a feature configuration and a set of corresponding test cases.

In the following, we describe the two mentioned approaches for test generation for configurable systems. The two approaches are focused on different sequences of applying coverage criteria for test case selection on the different test artifacts. Both approaches have been implemented in our prototypical tool chain SPLTestbench. The product-centered approach consists of a concatenation of well-known techniques. Consequently, it uses parts of the available information in a step-wise manner. The product-line-centered approach is aimed at combining all techniques in one step. Thus, by using all available information in one step, it bears the potential for an improved quality of the generated artifacts.

9.3.4 Product-centered MBT

Here, we describe the product-centered approach for MBT of configurable systems. This approach consists of three steps, each of which uses one of the three available sources—the feature model, the base model, and feature mapping. In the first step, a representative set of products is derived using feature configurations. This can be done, for example, by applying heuristics such as feature-model-based coverage criteria (Oster et al., 2011). For our FMS example, we use the feature-based coverage criterion All-Features-Included-And-Excluded, which is satisfied by a set of feature configurations if each non-mandatory feature is at least once included and at least once excluded. Feature configuration generation with SPLTestbench results in two relevant feature configurations (regarding only the features used in the state machine in Figure 9.3). See Table 9.2: The first test candidate is equipped with all features used in the state machine; the second candidate is the minimal FMS product consisting of the mandatory features only. The “(?)” sign in the table denotes features that are irrelevant for the test cases, because they have not been used in the FMS state machine example.

In the second step, the feature mapping is used to derive a resolved state machine model for each configured product variant.

The third step consists of running automated model-based test generation on each of the resolved state machines. There are several influencing factors for model-based test generation (cf. Weißleder, 2009). However, we focus on generating test suites that cover all transitions of each resolved state machine. For All-Features-Included-And-Excluded, satisfying transition coverage results in two test suites with a total number of 12 test cases and 63 test steps.

The product-centered approach can be implemented by standard test generators. This implementation is straightforward. It suffers, however, from the disadvantage that the available information is not used together at once, but in sequence. One consequence of this sequential use of information is that sets of model elements that are common to several product variants are tested again and again for each variant in full detail. While this repetitive approach might be useful for testing single products, it results in unnecessarily high effort for product line approaches that apply the reuse of system artifacts. Another consequence of this sequential approach is that the test effort cannot be focused on the variability. Each product variant is handled separately and the knowledge about the commonalities in their state machines is unused.

9.3.5 Product-line-centered MBT

Here, we describe the product-line-centered approach for MBT of configurable systems. In contrast to the product-centered approach, it is focused on using all available information together at once and not in sequence.

For implementing this approach, one typically has two options: The first choice is the invention of a new test generator; the second is an appropriate transformation of the input model. Our approach is based on the latter, that is, we use a model transformation to combine all the available information within one model. By ensuring that the model transformation results in a valid instance of UML state machines, we take advantage of using existing test generators that can be applied on this combined model.

Our model transformation works as follows: The information from the feature model and the feature mapping is used to extend the transitions of the state machine. This extension alters transitions in a way that a transition can only be fired if the features of its configuration rule have not been excluded yet. Figure 9.4 shows the result of an exemplary transformation of a transition that originally had only the guard [Guard] and no effect. The transformation adds two variables for each feature f: The variable valueFeature_f marks the value of the feature selection. The variable isSetFeature_f marks whether the value of the feature has been set already (i.e., if it has been already bound before). A transition that is mapped to a feature can be fired only if the feature has been included in the product variant already (i.e., if valueFeature == true) or if the feature has not been considered at all (i.e., if isSetFeature_f == false). As the effect of the transition, both values are set to true. This means that for the current path in the model, this variable assignment cannot be undone in order to guarantee consistency of the product feature selection. For more complex configuration rules, there are corresponding transformations.

As a consequence of this transformation, the test generator only generates paths through the state machine for which there is at least one valid product variant. Furthermore, the test generator not only generates a set of paths, but also the conditions for the corresponding valid products on the feature model level (i.e. in form of feature configurations). This allows assigning the generated test cases to the product variants they are run on.

We have applied the transformation on our FMS example. The transformed state machine was used then to generate tests to satisfy transition coverage. As a result, nine test cases with a total number of 47 external steps were generated that covered all transitions. The generated test cases are shown as message sequence charts in Figure 9.5. Each test case is assigned with only the necessarily included or excluded features to make this test executable. This leaves room for optimization of the test suites such as, for example, deriving a big or small number of variants or maximizing a certain coverage criterion on the feature model. The corresponding minimal configurations for the test cases are shown in Table 9.3. The sign “(?)” denotes the features that have not been needed to be bound for running the test case. “(?)” again marks features not used in the statechart.

9.3.6 Comparison

Here, we describe a first comparison of both approaches. Our comparison uses two values: coverage criteria and the efficiency of the test suite to satisfy them. The application of coverage criteria is a widely accepted means to measure the quality of test suites, which is also recommended by standards for safety-critical systems such as the ISO 61508. We measure the test suite efficiency in terms of the numbers of test cases and test steps. The more efficient we consider the test suite, the fewer test cases and test steps are needed to satisfy a certain coverage criterion.

The current results of our evaluations are that both approaches achieve the same degree of base model coverage, which is 100% transition coverage. Furthermore, the product-line-centered approach needed only 9 test cases with a total length of 47 test steps compared to 12 test cases and 63 test steps of the product-centered approach. Based on these results, the product-line-centered approach seems to produce a more efficient test suite than the product-centered approach while guaranteeing the same degree of base model coverage.

For future work, we plan to further investigate the relations of coverage criteria on feature models and base models. For instance, there are other criteria at the feature model level such as All-Feature-Pairs (Oster et al., 2011). The product-centered approach is able to make a quantifiable statement about feature model coverage. Applying the All-Feature-Pairs-and-All-Transitions coverage criteria instead of the All-Feature-Included-And-Excluded-and-All-Transitions coverage in the product-centered approach have resulted in larger test suites: Testing of the FMS product line would need 6 test suites (i.e. 6 product variants) with a total number of 36 test cases and 185 test steps.

The product-line-centered approach leaves feature model coverage to the last optimization step without guaranteeing a certain degree of coverage. The meaning of feature model coverage is not yet clear here. But the most important thing to do is to test specified behavior. And the resulting question is about the impact of the feature model on testing the described behavior.

9.4.1 What is deployment?

Deployment is an important task within the design phase of a systems engineering process. It is concerned with the assignment of resources, for example, CPU time, memory, or access to I/O adapters for application components. The design artifact that is produced by this engineering step is referred to as a deployment model. With resource capabilities of the hardware, network, and computing components on the one hand and resource demands of the application components on the other, the deployment acts as central resource broker. A deployment is often subject to a variety of general and domain-specific constraints which determine its correctness. Generally speaking, a deployment is correct if it assigns the proper type and amount of resources to all application components at any moment during runtime of the system. It is the job of the operating system to ensure the integrity of the deployment at runtime by means of scheduling and isolation techniques.

Due to the complexity of these constraints and the intricacy of the application and resource architectures found in current software-intensive embedded systems, constructing a correct deployment is of NP-complexity. This renders a manual construction a time-consuming and error-prone task. Still, a correct deployment is an essential prerequisite for achieving high levels of system quality because it has a direct impact on the fulfillment of major system requirements. This is especially true for extra-functional requirements of embedded software, such as real-time behavior, safety, and reliability requirements. For instance, a faulty resource assignment may lead to non-deterministic latencies when a resource is erroneously “overbooked,” which may eventually jeopardize the real-time behavior of the entire system.

Beyond its relevance for the fulfillment of extra-functional requirements, a deployment also helps to determine the amount of resources, for example, processing power or memory capacity, which are required. Often it is the starting point for extensive optimizations on the system level because it has direct impact on the cost and performance of the system to be built. A deployment model builder supports the optimization by providing scores for the deployments it constructs. Depending on the quality requirements and evaluation criteria that are given to the deployment model builder, it applies metrics that evaluate the deployments with respect to these requirements and criteria. The results r_i can be weighted with user defined weights w_i, which finally defines the score of a deployment D:

$\mathrm{Score}\left(D\right)={\displaystyle \sum ^i{w}_i.r_i}$

Evaluation criteria may encompass a variety of domain-specific and application-specific aspects of a system architecture, such as hardware cost, penalties for unused capacities, or the load distribution. The construction of a deployment therefore is not only applied to produce correct resource assignments, but it is also a vital part of the design space exploration process to find (cost-)optimal resource designs.

In addition to the relevance of a deployment for the correctness and quality of the system, it is also a central asset and synchronization point for distributed development teams. With the trend toward integrating more and more software components from different vendors into a single electronic control unit, a correct deployment proves to be an essential design artifact to prevent surprises during the integration phase.

9.4.2 Spatial and temporal deployment

In consideration of its NP-complexity, the construction of a deployment should be subdivided into a spatial and a temporal part. A spatial deployment focuses on the entire system architecture and yields a mapping from application components to resources. It is constrained not only by the amount and type of the resources available, but also by safety and reliability requirements. A spatial deployment determines which resources are exclusively used by some application component and which resources have to be shared.

Coordinating the access to shared resources in the temporal dimension is the objective of the temporal deployment. It addresses the question of when to execute an application component on a shared resource, so that all real-time requirements are met. Usually, a static schedule, which correctly executes all application components using shared resources, is produced as a result of a temporal deployment.

Figure 9.6 depicts a common process model for the construction of a deployment model. The spatial deployment should be conducted first because it determines a major part of the system architecture and its properties. However, it is only feasible if a correct temporal deployment can also be constructed. Otherwise, the spatial deployment has to be modified to facilitate a temporal deployment.

In this contribution, we focus on the spatial deployment and its extension to configurable systems. Schedulability tests and tools for generating static schedules for a temporal deployment have also been extensively studied (see Section 9.5).

Their extension to configurable systems, however, is an issue for further research, which might be undertaken using the framework we present here.

9.4.3 Application and resource models for the flight management system

In order to construct a deployment model for the FMS, we first have to provide a resource model. Second, we have to extend the application model by the resource demands of the application components. An extract of the latter is illustrated in Figure 9.7. It shows three of the application components and their resource requirements. In order to simplify matters, we assume that each feature in our feature model (see Figure 9.1) is realized by a single application component. This defines the relation of the feature model and the application model, that is, the feature mapping. Furthermore, resource requirements and safety requirements are assigned to each application component. These requirements constrain the spatial deployment and define the valid assignments of application components to resources.

A simplified resource model is depicted in Figure 9.8. Basically, a FMS may either be built as a single, non-standard, stand-alone unit, or it may use a set of standardized IMA boxes with IMA processing boards. For our example, we assume that in the second case either one or two IMA boxes can be used. Each IMA box may host either one or two processing boards. There are two kinds of processing boards available: type A and type B. The board types differ in their processor (architecture type, vendor, and performance) and in the size of the included memory. Type A boards feature larger memories and faster processors. Boards with type B, on the other hand, are equipped with less memory and slower processors, but they are also less expensive.

At this point we do not define a relation between the variance points of the resource model and the features for two reasons. First, the deployment model that is now constructed induces a feature-to-resource relation by extending the feature-to-application relation with the application-to-resource relation defined by the deployment model. In this induced feature-to-resource relation a resource model element is related to a feature if there is an application model element that is (1) related to the feature and (2) deployed onto the resource model element.

Second, the resource variants are obviously not induced directly by the features we considered for the application. This is indeed a typical situation. Different parts of the system are designed with respect to different specific requirements, constraints, and features. For example, the variance drivers for the resources might be induced by supplier relations, future availability, costs, or other factors that have nothing to do with the functionality of the system. It was possible to represent this as an additional, resource-specific feature model. There is another issue here, however. In fact, the resource variants in this case represent design space alternatives rather than system variants. From the design space alternatives one or several candidates are selected for the realization of the application. The other ones are withdrawn. This is different from the variants of the application: They are all required and must all be realized.

The result of the construction of the deployment model in our example correspondingly consists of two parts: a reduction of the variance of the resource model and a feature mapping to the remaining resource model that is given by the composition of the feature mapping to the application model and the deployment model.

In Figure 9.2 we introduced our general concept for the evaluation of configurable systems. We now apply this general concept to the construction of a deployment of the FMS example and describe a product-centered and a product-line-centered approach for this engineering challenge.

9.4.4 Product-centered software deployment

The product-centered approach consists of three basic steps. First, sufficiently many product models are derived from product line model. These products are then individually evaluated by the deployment model builder in a second step. Finally, the results obtained from these evaluations are aggregated and combined, so that an assessment of the entire product line is achieved. In the following, these steps will be described in more detail in the context of our FMS example.

9.4.4.2 Step 2: Evaluating the deployment candidates

Each pair of a variance-free application model and a variance-free resource model can be supplied to the deployment model builder, which either constructs a deployment for this pair that satisfies all resource requirements and safety requirements or it indicates that no valid deployment can be found. In the latter case, the resource model is deleted from the list of possible resource candidates for this application configuration. As mentioned above, the deployment model builder also attaches a score to each deployment candidate it finds. These will be used in Step 3 to find the product line deployment. Note that all deployment candidates constructed in this way are correct, but not necessarily optimal designs for the overall configurable system.

Already, our small example has many product models, that is, pairs of configured application and resource models. So, we do not compute the whole set of pairs. Instead we use subsets of viable candidates, which is also the way one would proceed practically. There are often external criteria, obtained, for instance, from market surveys, supplier relations, or production constraints, which indicate some configurations as viable, whereas others are excluded from further consideration.

For our example we consider only three different types of flight management applications (products) that have to be developed: Simple FMS, FMS for Small Jets, and FMS for Jumbo Jets. These products and their features have been defined in Table 9.1.

Concerning the possible resource configurations, assume that the following set of viable variants is defined based on the resource model in Figure 9.8. Table 9.4 lists these variants together with their relative hardware cost factor, which is used as a quality evaluation criterion. Only these resource variants shall be considered as candidates for the deployment of the application and to realize the envisioned product line of FMSs.

Table 9.4

Several Variants of the Resource Architecture

Resource Variants	Resource Components	Relative Cost
HWV0	1 Non-standard Box	1
HWV1	1 IMA Box with 2 Boards Type B	10
HWV2	1 IMA Box with 2 Boards Type A	15
HWV3	2 IMA Boxes, each with 2 Boards Type A	30

The result of the deployment model builder for this restricted set of application and resource variants might look as represented in Table 9.5. The numbers indicate the scores that are given to the application and resource pairs by the deployment model builder. The empty set (∅) is used to characterize a pair without a valid deployment.

Table 9.5

Scores of Optimal Product Candidates for All Resource and Application Combinations

FMS Products (Application Variants)	Resource Variants
	HWV0	HWV1	HWV2	HWV3
Simple FMS $\left(S{W}_{P_1}\right)$	100	75	65	35
FMS for Small Jets $\left(S{W}_{P_2}\right)$	Ø	95	90	45
FMS for Jumbo Jets $\left(S{W}_{P_3}\right)$	Ø	Ø	Ø	75

For the Simple FMS, the stand-alone unit (HWV₀) is clearly an optimal choice with a score of 100. The other resource configurations for the Simple FMS are still viable, but they use significantly more resources, which will not be fully utilized and result in lower scores.

However, the HWV₀ is not a viable choice for the FMS for Small Jets as its AirData software component requires a 2x redundant execution, which the stand-alone unit cannot provide. Score differences between HWV₁ and HWV₂ result from different costs for the processing boards of type A and type B. Similarly, the FMS for Jumbo Jets can only be realized with HWV₃, because the Surveillance System requires a 4x redundant execution.

The description of the first two steps illustrates the challenge of dealing with the combinatorial explosion in the product-centered development approach. Finding and assessing all viable product candidates exceeds the capabilities of a manual development—even for simple systems with few variation points. Instead, a tool-supported approach is required to automate these engineering steps. In Hilbrich and Dieudonne (2013), Hilbrich (2012), and van Kampenhout and Hilbrich (2013) we presented a tool that is able to construct an optimal deployment and facilitate Step 1 and Step 2.

9.4.4.3 Step 3: Aggregation of results

In the next step, the results from Table 9.5 have to be aggregated and combined to a deployment model for the configurable application and resource models. Instead of a formal definition of an aggregation, we discuss informally how the results of Table 9.5 for the individual application and resource model pairs without variation points can be interpreted and used to build a single, configurable deployment

The application level has already been modeled as a product line and related to the feature model as presented in Figure 9.1. The initially provided set of possible resource configurations from Figure 9.8 now can be further refined using the results of the deployment model builder. For instance, variant HWV₂ appears to be a suboptimal choice for all FMS products. Therefore, it may be removed from the set of viable resource variants.

However, a systems engineer may decide to use HWV₂ for the implementation the FMS for Small Jets product $\left(S{W}_{P_2}\right)$ instead of HWV₁. This choice would allow for more homogeneity in the hardware components of the entire product line, because processing boards of type B would no longer be required. Taking into account this criterion implies that although using HWV₁ is a locally optimal choice for $S{W}_{P_2}$, the decision to use resource variant HWV₂ for $S{W}_{P_2}$ instead is a globally optimal choice for the entire product line and may be preferred by the systems engineer.

This aggregation of deployments based on their scores and the homogeneity criterion yields the following configurable deployment for the whole product line. First, the resource model is reduced. The boards of Type B are deleted, the XOR-boards that contained the alternative boards of Type A and Type B are replaced by the remaining boards of Type A. Inside the FMS with IMA Boxes we have thus the IMA Box with one Board of Type A and an optional. Additional Board of Type A, and the optional Additional IMA Box with the same internal structure.

The corresponding deployments of the three individual applications are given in Table 9.6. The additional numbers behind the application components indicate their redundant copies that are deployed to different resource components.

Table 9.6

Product Deployments to the Reduced Resources

Application Variants	Resource Variants	Application Components	Resource Components
$S{W}_{P_1}$	HWV0	Primary Flight Display	Board
$S{W}_{P_2}$	HWV2	Primary Flight Display	IMA Box. Board (Type A)
		Air Data(1)	IMA Box. Board (Type A)
		Air Data(2)	IMA Box. Additional Board (Type A)
$S{W}_{P_3}$	HWV2	Primary Flight Display	IMA Box. Board (Type A)
		Air Data (1)	IMA Box. Board (Type A)
		Air Data (2)	IMA Box. Additional Board (Type A)
		Surveillance Systems (1)	IMA Box. Board (Type A)
		Surveillance Systems (2)	IMA Box. Additional Board (Type A)
		Surveillance Systems (3)	Additional IMA Box. Board (Type A)
		Surveillance Systems (4)	Additional IMA Box. Additional Board (Type A)

The mapping defined by the deployment, combined with the feature mapping to the application model, yields the feature mapping indicated in Table 9.7.

Table 9.7

Feature Mapping of the Reduced Resource Model

Resource Component	Configuration Rule
FMS with Non-standard Box	(AD ∨ SurSys)
FMS with IMA Boxes	AD ∨ SurSys
IMABox.AdditionalBoard	AD ∨ SurSys
AdditionalIMABox	SurSys
AdditionalIMABox.AdditionalBoard	SurSys

9.4.5 Product-line-centered software deployment

Deriving all possible products may be a very time-consuming and expensive approach for large and complex systems with many variation points. For some systems, it may not even be a viable approach at all. In these situations, optimization strategies and heuristics are needed that focus more on the product line instead of deriving all individual products. The following two strategies may be beneficial in these situations.

9.5.1 General product line approaches

Beyond the basic product line modeling concepts presented in this chapter are many other approaches that we briefly sketch. They differ in how they represent variability within the various system models, whether they provide explicitly defined variability concepts in the models, and which kind of relationships between feature and system models they allow. In Voelter and Groher (2007), product line approaches have been distinguished in approaches using negative versus positive variability. An example of negative variability approaches is 150% modeling (Czarnecki and Antkiewicz, 2005; Heidenreich et al., 2008; Mann and Rock, 2009). The system models represent so-called 150% models as the base models of the product line and include all the supported variability. Options or alternatives that have been not selected for a specific product will be removed from the base models. In this chapter we have also used a 150% modeling approach. Examples for positive variability approaches are given in Apel and Kästner (2009) and Thaker et al. (2007). Here, a base model exhibits basic features only. A number of transformations are performed on the base model to add specific features and to gain the product-specific variants. Delta modeling (Clarke et al., 2010) can be used for both positive and negative variability. A general syntactic framework for product line artifacts is suggested by the common variability language CVL (OMG, 2012).

A precondition for the quality assurance of single products as well as of the complete product line is that the engineering artifacts have been consistently set up with respect to the variability dimension. Analysis and assessment methods have been developed for the various models in product line engineering approaches. The interpretation of feature models as logical formulas to be used by theorem provers, SAT solvers, or other formal reasoning have been introduced in Batory (2005) and Zhang et al. (2004), for example. A literature overview of the use of such automated analysis of feature models is given in Benavides et al. (2010). Our work is based on this because we need the ability to compute correct feature configurations for the feature coverage criterion and for the model transformations of configurable state machines in order to use standard test generators, for instance.

The idea to make the transition from single evaluation methods to variability-aware methods because of efficiency issues has been already identified within the product line community. An overview of published product-centered and product-line-centered analysis approaches is given in von Rhein et al. (2013) and Thüm et al. (2012). The authors provide a classification model in order to compare these approaches. Their model distinguishes between the dimensions of sampling, feature grouping, and variability encoding. Sampling is about which products will be selected for analysis; feature grouping is about which feature combinations will be selected; and the variability encoding represents the dimension concerning whether analysis methods are able to deal with explicit variability information.

A literature overview of approaches that consider the variability of quality attributes within a product line is given in Myllärniemi et al. (2012). The focus is set on why, how, and which quality attributes vary within a product line.

9.5.2 Product line testing

Testing is discussed in many scientific articles, experience reports, and books (Ammann and Offutt, 2008; Beizer, 1990; Binder, 1999). As testing usually requires much effort during system engineering, there is a trend toward reducing test costs by applying test automation. There are several approaches of test automation for test design and test execution. We focus on automated test design. Some well-known approaches in this area are data-driven or action word-driven testing that strive to use abstract elements in the test design that can be replaced by several concrete elements representing, for example, several data values or action words. The most efficient technique in this respect is MBT, which allows the use of any kind of models in order to derive test cases (Utting and Legeard, 2006; Utting et al., 2012; Weißleder, 2009; Zander et al., 2011). Many reports are available dealing with the efficiency and effectiveness of MBT (Forrester Research, 2012; Pretschner, 2005; Pretschner et al., 2005; Weißleder, 2009; Weißleder et al., 2011).

Product lines have to be tested (Lee et al., 2012; McGregor, 2001). Given that product lines can be described using models, the application of MBT is an obvious solution for testing product lines. Several approaches exist for applying MBT to product lines (Weißleder et al., 2008). For instance, coverage criteria can be applied to the feature models to select a representative set of products for testing (Oster et al., 2011). Most approaches in this area, however, are focused on product-centered evaluation approaches, that is, they apply well-known techniques for single products only. We also started the discussion about applying the product-line-centered approach to MBT (Weißleder and Lackner, 2013). Evaluations of this approach are currently under development.

9.5.3 Deployment

Deployment comprises of a spatial and temporal dimension. The temporal deployment of application software has been discussed extensively in the context of scheduling in real-time systems. Analyzing the schedulability of tasks under real-time conditions has been formally described by Liu and Layland (1973). An overview about scheduling algorithms and a variety of heuristics is presented in Buttazzo (2004), Carpenter et al. (2004), Leung et al. (2004), and Pinedo (2008). Offline scheduling approaches, because they are often used in safety-critical systems, are discussed in Baro et al. (2012) and Fohler (1994). Tools for the construction of temporal deployments, that is, static schedules, are described in Brocaly et al. (2010) and Hilbrich and Goltz (2011). A formal approach to analyze the schedulability of real-time objects based on timed automata families is introduced in Sabouri et al. (2012). Their approach fits to our category of product-line-centered approaches. Similar to our product-line-centered MBT approach, the authors transform timed automata to reuse the common state space during schedulability analysis by late decision making for feature bindings.

Spatial deployment and its challenges have been discussed in Taylor et al. (2009). Engineering approaches for a spatial software deployment for safety-critical systems have been investigated in Hilbrich and Dieudonne (2013), Hilbrich (2012), and van Kampenhout and Hilbrich (2013). A deployment of software components based on their energy consumption has been studied in AlEnawy and Aydin (2005). To the best of our knowledge, there are no solutions yet for spatial deployment that deal explicitly with configurability of the application components or the resources.

9.6.1 Model-based product line testing

For MBT, the results of our example show that an efficiency gain is possible by using all available information together on the product-line-model level compared to the approaches of resolving a representative set of products and applying the technique to each selected product. There are, however, several aspects to discuss about such statements. For instance, there are different means to steer test generation for both approaches.

The test generation on the product-line-model level only requires defining a coverage criterion to satisfy at the base model. In contrast to that, test generation on the product-model level requires defining a coverage criterion on the feature-model level and a coverage criterion on the level of the resolved product model. This results in the obvious difference that the product-centered evaluation is focused on evaluating the quality of each selected product variant thoroughly, and the product-line-centered evaluation is focused on evaluating the behavior of all products. The product-line-centered approach bears the potential advantage of avoiding retesting elements that were already tested in other product variants. The impact of this advantage becomes stronger the more system components were reused during system engineering. If the components were not reused, however, and several product variants are developed from scratch or with a minimal reuse, then the product-centered approach has the advantage of considering all product variants independently.

Conversely, the product-line-centered approach has the potential disadvantage of missing faults that are introduced by a lack of component reuse, and the product-centered approach potentially results in testing all system components again and again in each selected variant. Consequently, the choice of the test generation approach strongly depends on the system engineering approach. This aspect is not always obvious to testers. So, the most important question for testers is whether they want to focus their tests on product line behavior or on covering product variants. A compromise is probably the best solution here.

9.6.2 Model-based deployment for product lines

Concerning deployment, it became apparent that the product-centered approach quickly led to a combinatorial explosion of product candidates. Tools may provide some support to automate the determination and assessment of product candidates. However, the product-centered approach is not a viable way to address the design challenges of a complex product line. Instead, powerful product-line-centered approaches are required. These may comprise strategies and heuristics to improve the efficiency of the computation or reduce the complexity of the variability in the system. Still, further research on product-line-centered approaches is necessary to overcome the shortcomings of the product-centered approaches and allow for complex, software-intensive and configurable systems with high quality.