8

Validating Requirements and Assumptions

8.1. Introduction

In this chapter, we will discuss requirement and assumption validation. We recall that the signification of the word “validation” depends on the standard or norm to which it subscribes. Here, we subscribe to the ARP4754A recommendation, which is the most developed standard concerning validation; consequently, in this chapter, the concept of “validation” only refers to the concepts of requirements and assumptions, but never to the physical systems themselves.

Figure 8.1. ARP4754A validation and verification processes connection

Therefore, validating a system does not make sense in the context of ARP4754A, and this option is fully justified by the following reasoning: to develop the right type of systems¹, it is “only necessary” that the set of all requirements characterizing it and the assumptions related to its environment be positively validated and that the systems produced with regard to its requirements and assumptions be positively verified in relation to its requirements and assumptions, that is to say compliant. The set of requirements and assumptions associated with a type of systems is the cornerstone in developing this type of systems.

First, we recall what the ARP4754A states about the validation process, as well as the goal and means that this recommendation advises and implements. We will then see how these goal and means can be declined within the framework of the systems engineering approach, based on the model we propose, that is to say the property-model methodology (PMM).

8.2. The validation process according to the ARP4754A

8.2.1. Goal of the validation

The ARP4754A recommendation defines validation as the process by which we ensure that the set of requirements allocated to the type of systems is sufficiently complete, and sufficiently correct, to characterize the desired type of systems, before supplying or even developing them, without taking too many risks concerning the compliance of the supplied systems with the targeted type. It also covers the assumptions made concerning its environment to specify the type of systems considered.

We recall that the ARP4754A recommendation does not give a definition of what a requirement is, other than an evasive statement, already emphasized in this book, and it defines an assumption as a statement offered without proof [ARP XX]. It can seem strange that a body of technological knowledge will be built thus on a “missing column”, according to the words of the poet Henri Michaux [MIC 29], but this case is not that exceptional. The ISO15288 has also omitted from giving a definition of the concept of requirement.

However, as formalized in section 6.2, Σ designates a targeted type of systems and {Req_i}_1≤i≤n designates a set of the requirements referring to Σ, then the aim of the validation process is to establish that Σ is identical to SATK({Req_i}_1≤i≤n). If this is not the case, a controlled modification process of {Req_i}_1≤i≤n shall have to be executed so that we can eventually obtain SATK({Req_i}_1≤i≤n)= Σ. This is how we will have succeeded in obtaining a specification that is as exact as possible.

Thus, each requirement Reqi is individually sufficiently correct, and if {Req_i}_1≤i≤n is sufficiently complete (it is not missing essential requirements), then we can hope that the specification{Req_i}_1≤i≤n will also be as exact as possible.

An incorrect (respectively, an incomplete) requirement (respectively, specification) is a requirement (respectively, a specification), which can either put aside an intended characteristic of the type of systems targeted, or on the contrary conserve an undesirable characteristic to the type of systems targeted.

According to the ARP4754A, the assumptions made during the development must be explicit and follow the same validation process.

We understand, therefore, the importance of the requirement and assumption validation process, especially if we recall that these requirements constitute the cornerstone of the development process of a type of systems.

8.2.2. Means of validation

The approach suggested by the ARP4754A to validate a set of requirements and assumptions {Req_i}_1≤i≤n allocated to a type of systems involves building a validation matrix (or any other equivalent method) in which each requirement of the set {Req_i}_1≤i≤n is referred to by a line.

At the initial stage, for each of these lines (that is to say, for each requirement or assumption), the stakeholders in charge of the validation process select, on the one hand, a level of the rigor with which the validation process is conducted and, on the other hand, the validation method or methods to be used.

Concerning the level of rigor intended of a validation effort, the ARP4754A associates the severity of risk the crew and passengers are exposed to in case of a failure of the service provided by the system under consideration. The requirement validation level of rigor whose failure would have catastrophic consequences (CAT) for the crew and passengers must be maximal. Then, the level of rigor can be gradually reduced if the consequences are hazardous (HAZ), major (MAJ), minor (MIN) and finally no safety effect (NSE) according to a classification well established by regulation (for severity categorization, CAT, HAZ, MAJ, MIN, NSE, refer to Chapter 10, Table 10.2).

We see, therefore, that the level of rigor intended by the ARP4754A in order to conduct a validation process is entirely polarized by safety considerations, which seems normal and compliant with Authorities objectives and with what we expect in matters of aeronautic safety. However, we also see that in this framework, the requirements without any consequences on security can become the poor relation of the validation process, and it is suitable to adjust the level of rigor for requirements important for the contractor and operators such as the requirements impacting the availability, even if they have no impact of safety.

Figure 8.2. ARP4754A validation process model

The validation plan, which is the entrance point of the validation activities, identifies, on the one hand, the validation methods that shall be used either on their own or in a combined manner. On the other hand, it defines, depending on the intended level of rigor, the combination of methods required or advised to successfully carry out the validation process.

The ARP4754A identifies the following processes as methods of validating requirements and assumptions:

These methods are used to carry out two types of checks: correctness checks and completeness checks; check-lists are established within the validation plan.

Once the initial validation matrix is established and approved, the validation process can be started, that is to say correctness and completeness checks can be carried out in compliance with the selected methods and the results from those checks recorded in the matrix. Additions, corrections or suppressions, of requirements or assumptions, shall be performed according to the rules of a controlled modification management process.

The ARP4754A recommends that this validation process should be carried out gradually in a descending manner, that is to say by starting with the highest level system (the aircraft) and then validating the aircraft system requirements, so on and so forth down to the elementary items or components. The recommendation also emphasizes that the validated system requirements are the base for the validation of requirements specified to its subsystems.

8.3. The validation process according to the property model methodology

Following the ARP4754A recommendation, the PMM introduces the requirements and assumptions model as cornerstone of the system development process. However, unlike the recommendation, which refers to no precise definition of the notion of requirement, the approach of the PMM is based on the concept of property-based requirement (PBR) introduced in Chapter 6.

8.3.1. Goal of the validation

To develop a type of systems, as we will see in Chapter 11, we start by developing a system specification model (SSM) of the type of systems.

This model will then be validated, that is to say, it will have to be established that this specification is as exact as possible for the type of systems targeted.

Ideally, the goal of the validation is to show that all the specification models, which constitute the specification tree, are free of specification errors.

Unfortunately, this goal is, in absolute terms, inaccessible. The claim that a specification model is free of errors can only be justified relative to the level of rigor of its validation, but remains on the existence of an error that is not detected by the validation process. However, the validation task presented here enables the reduction of this risk to acceptable levels with regard to the potential consequences.

Figure 8.3. Specification model tree validation. For a color version of this figure, see www.iste.co.uk/micouin/MBSE.zip

Following this, an architecture of the type of systems targeted can be designed in terms of components (processes, then subsystems) and structure (flows, then links). The architectural choice enables the derivation of the SSM into a set of subsystem specification models. These subsystem specification models will then be validated in relation to the SSM they are derived from.

In other words, depending on the PMM, the specification model validation is a descending process (as suggested by the ARP4754A recommendation) which:

This gradual requirement validation process is, through the methods indicated below, a powerful way of reducing risk and will enable us to establish an important theorem (the “contract” theorem) in Chapter 9, which we could summarize in the following way for now: if the subsystem specifications of a system are valid in relation to the system specifications and if each of the subsystems is consistent with its specification, then the system will necessarily be consistent with its own specification so long as its physical integration is compliant with its design.

8.3.2. Means of validation

In theory, the PMM does not put aside any of the validation methods considered by the ARP4754A (section 8.2.2). It considers them in a complementary manner (the flight test can be one, for example). However, it favors simulation as a validation method, just as it considers simulation to be the main method of cost and deadline control, and consequently the main method to reduce the complexity of a system development.

Indeed, simulation presents two advantages for validation. First, it is an early method of validation in comparison to methods such as tests, which are belated methods of validation². In this case, it enables a priori the validation of the systems specification model, before the design of an architecture, whereas a validation by tests requires the availability of a prototype of the type of systems. Second, it provides an objective means of assessing the exactness of the specification model very superior to the early means allowed by written specifications. An SSM is not necessarily exact at the beginning of the validation process but it is objective, and this objectivity will enable us to establish whether it is exact or not, within the limits we will detail lower down. Experts (test pilots, for example) called to express themselves on the course and results of a simulation process have an objective basis to evaluate³ the intended behavior, which more or less well-written literal specification files do not offer.

Validation by simulation involves setting up a validation workbench with, on the one hand, a system model and, on the other hand, a validation driver as shown in Figure 8.4. The validation driver is an artifact that submits validation cases, chosen from the validation scenarios, to the system whose specification we seek to validate. During each simulation cycle, the system model reacts to the validation cases submitted to it. The design model computes the outputs in function of the submitted inputs and also, more often, in function of previously determined states. Concurrently, an SSM ensures that the inputs are in accordance with the expressed assumptions and the outputs (and observable states) are in accordance with the specified requirements.

The topics related to validation scenarios and the validation effort required will be tackled in section 8.3.5.

8.3.3. Exactness of a system specification model

If the design model is an equation design model (EDM), as defined in section 7.4.2, then this behavioral model is an “embodiment” of the specification model. It behaves objectively and ideally in relation to the specification, and no discrepancy can be observed between the behavior that is specified and intended, and the effective model behavior (see section 2.3). It is, therefore, possible to judge the validity of the specification by observing the effective behavior of the EDM.

Figure 8.4. System specification model exactification process by simulation

If the effective behavior of the EDM satisfies the stakeholders in charge of the validation of the specification model (SSM), we can conclude that the specification model exactly characterizes the type of systems targeted.

However, the claim that the specification model is exact remains limited and approximate. This limitation is due to the fact that a model, as representative as it may be of a concrete object, cannot entirely characterize it (the real object exceeds its models in all respects), rather than due to the fact that it is conditioned by the wealth of validation scenarios, and the operational representativeness of these scenarios (see section 8.3.5). The more a specification model is validated through a large number of validation scenarios, the more exact it will be, without ever being able to attain an absolute exactness.

8.3.4. Validating the derivation of system requirements

Figure 8.5. Subsystem specification model validation process by simulation

Just as the ARP4754A recommends, the validation process implemented here is gradual. Once the exactness of the SSM is established at the highest level, we will stick to demonstrating that the requirement specified to the first rank subsystems correctly and completely derives the system requirements.

This demonstration is carried out by a simulation, which is based on validation scenarios that are defined with regard to the required validation effort. We note, however, that this validation cannot be carried out individually, requirement-by-requirement, unlike what the validation matrix (validated line-by-line) could suggest, but requires a global awareness of each derivation operation.

In fact, we do not directly validate the requirements derived and allocated to the subsystem models, but rather the derivation task carried out in a context defined by the structural design model (SDM), which interconnects the subsystem models. It is just a misuse of language to declare that derived requirements are valid when the derivation is validated. Furthermore, we will note that an SDM is not subject to validation.

8.3.5. Scenarios and validation cases, efforts and rigor in validation

As we have already indicated, a specification model can be represented graphically as shown in Figure 8.6.

Figure 8.6. Graphical representation of a specification model

Each of the PBRs presented above is an expression in the form of PBR: [when C] → val(O.P) ∈ D. Such an expression is logically equivalent to the assertion [val(O.P) ∈ D] or not (C), or even to the expression O.P(C,D) where O.P(C,D) is a Boolean function P: E_1xE_2x..xEn →{false, true}, and so the set of PBRs specifying the type of systems is a set of Boolean functions with parameters selected from the inputs or outputs (or observable states) of the SSM. E_i for 1≤i≤n is the definition domain (still named: the type) of an input, output or observable state belonging to the SSM.

During a simulation process, assertions P(x1,x2,..,x_n) are evaluated simultaneously, that is to say that at the start of each simulation cycle, all those that require reevaluating (because the values of certain of their pertinent parameters⁴ (x1,x2,..,xn) are changed) are reevaluated “in parallel”. Under these conditions, one or more of the reevaluated assertions can become untrue, thus reporting the non-respect of requirements.

We name a validation case (x1,x2,..,xn) any element of the definition domain of a requirement P: E=E_1xE_2x..xEn →{false, true}. Depending on the types involved in the definition domain E_1xE_2x..xE_n, there is potentially an infinite number of validation cases. An exhaustive simulated validation of a specification model is, therefore, unattainable. This leads to the partitioning of the definition domain P into equivalence classes P/R such that all the validation cases belonging to the same class will be supposed to be equivalent (i.e. having the same truth value). It is then enough to choose just one validation case within each class to have a systematic validation strategy and a finite and representative set of validation cases at our disposal.

Let us use the following example regarding the indication of the barometric altitude on an aircraft (airworthiness requirement CS2x 1303,b). This requirement states that the altimeter of an aircraft must give an indication of the barometric altitude (ADC.Indicated Alt) with an error inferior or equal to 25 ft when the aircraft is at a real altitude (AC.Alt) between 0 and 5,000 ft, which is translated by the following PBR:

PBR₁: when AC.Alt [0.0, 5,000.0] → | ADC.Indicated _Alt - AC.Alt | ≤ 25.0 ft

The rest of the domain [5,000.0 and 50,000.0 ft] is covered by the PBR₂ up to PBR₉ and PBR₀ requirements stated in section 6.6.1.

This PBR₁ requirement is translated, within the PMM methodology, by a Boolean function P: [0.0, 5,000.0]xR →{false, true} where R is the real type (field of real numbers) and [0.0, 5,000.0] is the interval of real values between 0.0 and 5,000.0 and formally defined by the predicate P: ∀ Alt ∈ [0.0, 5,000.0], ∀ Indicated_Alt ∈ R, P(Alt, Indicated_Alt) =true if | Alt - Indicated_Alt | ≤25.0; otherwise, P(Alt, Indicated_Alt)= false.

The exhaustive validation of this Boolean function P is impossible within a simulation process; however, the validation cases described in Table 8.1 can provide an indication of the exactness of the Boolean function.

Table 8.1. Validation case for a level of rigor consistent with no safety effect (NSE)

If we use this Boolean function in simulation by injecting the validation cases above as parameters, and if we obtain the intended result in each case, the corresponding requirement could be presumed to be exact.

We can see, considering the circumstances, the lack of evidence produced to claim the requirement is exact. This lax attitude toward the validation effort could be acceptable if the erroneous altitude indication had NSEs. In fact, for the Altitude Indication, this is not the case. Consequently, we must harden the demonstration of the exactness of the requirement by increasing the quantity of evidence produced, in a reasoned and systematic manner.

It is not within the scope of this work to set the level of rigor and the quantity of evidences to produce to be able to declare a requirement to be as exact as possible. This responsibility belongs to the developing team, and it must fall within the scope of their validation plan. However, we can indicate that as the Boolean functions express themselves as sequential algorithms in simulation language, all the software-testing techniques are available to select the validation cases, pertinently and systematically. We can, for example, refer ourselves to Chapter 4 of the software engineering body of knowledge (SWEBOK V3.0) [BOU 14] dedicated to software testing.

We can, for example, harden the demonstration of requirement validation by considering the set of validation cases of the table above.

We could of course add many others, but we must ask ourselves the question of the benefit expected from a multiplication of the validation cases. From this point of view, software-testing technologies provide objective selection criteria.

Table 8.2. Validation case for a hardened level of rigor

Once the validation cases enabling the exactification of a requirement are selected, we will have to inject these cases into one or more simulation scenarios, corresponding, if possible, to system usage scenarios (operational scenarios). Injecting these validation cases into operational scenarios enables the detection of situations insufficiently covered by the validation cases, and therefore the correction of this situation by enhancing the coverage of some critical phases.

Table 8.3. Validation scenarios

We can see, in this example, that the takeoff and landing situations, which are the most delicate, are not covered much and that there might be benefits in covering them more.

8.4. Conclusion

The property-model methodology enables the development of a requirements and assumptions validation strategy that is entirely in accordance with the ARP4754A recommendation.

It enables the validation of all the specification models of a type of systems from the highest level to the most detailed levels, in a descending manner by ensuring, on the one hand, the exactness of the head specification and, on the other, by establishing the coherence, correction and completeness of the inferior level specification models in relation to the highest level specification model.

A simulation process establishes the evidence of this early validity, well before the components of the corresponding physical systems are designed and then produced.

The quantity of proofs can be of varying size and the rigor in the demonstration can be more or less hardened, depending on the stakes, particularly safety, carried by the type of systems targeted, while the validation coverage is determined systematically and adaptively to usage models.

1 Right systems are those that are targeted and accepted by all the stakeholders.

2 As they are belated in the sequence of tasks, when the time comes for tests, most of the initial budget has already been used, leaving no room for change.

3 Equivalent to those offered later by laboratory tests.

4 All the parameters are not necessarily pertinent; for example, the change of an input must not necessarily automatically launch a reevaluation of P that can only be reevaluated on an output after the propagation of the change in the system model. The specifier determines the moments during which a requirement must be evaluated.