9.6.2.1 Requirements-Based Test Methods

DO-178C section 6.4.3 promotes three requirements-based test methods.

1. Requirements-based software/hardware integration testing [1]: This test method executes tests on the target computer to reveal errors when the software runs in its operating environment, since many software errors can only be identified in the target environment. Some of the functional areas verified during software/hardware integration testing are interrupt handling, timing, response to hardware transients or failures, databus or other problems with resource contention, built-in-test, hardware/software interfaces, control loop behavior, hardware devices controlled by software, absence of stack overflow, field-loading mechanisms, and software partitioning. This method of testing is normally performed by running tests against the highlevel requirements, with both normal and abnormal (robustness) inputs, on the target computer.

2. Requirements-based software integration testing [1]: This method of testing focuses on the interrelationships of the software to ensure that the software components (typically, functions, procedures, or modules) interact properly and satisfy the requirements and architecture. This testing focuses on errors in the integration process and component interfaces, such as corrupted data, variable and constant initialization errors, event or operation sequencing errors, etc. This testing is typically accomplished starting with the high-level requirementsbased tests for hardware/software integration testing. However, requirements-based tests may need to be added to exercise the software architecture; likewise, lower level testing may also be needed to supplement the software/hardware integration testing.

3. Requirements-based low-level testing [1]: This method generally focuses on compliance with the low-level requirements. It examines lowlevel functionality, such as algorithm compliance and accuracy, loop operation, correct logic, combinations of input conditions, proper response to corrupt or missing input data, handling of exceptions, computation sequences, etc.

9.6.2.2 Normal and Robustness Tests

DO-178C also promotes the development of two kinds of test cases: normal and robustness.^*

9.6.3.1 Equivalence Class Partitioning

Pressman writes:

Equivalence partitioning is a black-box testing method that divides the input domain of a program into classes of data from which test cases can be derived. An ideal test case single-handedly uncovers a class of errors (e.g., incorrect processing of all character data) that might otherwise require many test cases to be executed before the general error is observed [12].

Equivalence partitioning requires an evaluation of equivalence classes. The DO-178C glossary defines equivalence class as: “The partition of the input domain of a program such that a test of a representative value of the class is equivalent to a test of other values of the class” [1]. Equivalence class testing considers classes of input conditions to be tested, where each class covers a large set of other possible tests. It “enables the tester to evaluate input or output variables systematically for each parameter in a feature” [9]. It is typically applied to situations where valid (normal) and invalid (robustness) inputs over a range are identified. Rather than testing every value of the range, only representative values are needed (typically on both sides of the boundaries). For large ranges of data, a value in the middle and at the extremes of each end are also typical. In addition to ranges of values, equivalence classes consider similar groups of variables, unique values that require different handling, or specific values that must or must not be present.

Even though equivalence class partitioning may seem trivial, it can be quite challenging to apply. Its effectiveness relies on “the ability of the tester to decompose the variable data for a given parameter accurately into welldefined subsets in which any element from a specific subset would logically produce the same expected result as any other element from that subset” [9]. If the tester does not understand the system and the domain space, he or she may miss critical defects and/or execute redundant tests [9].

The following recommendations may help when looking for equivalence classes [7,10]:

• Consider the invalid inputs. These are often where the vulnerabilities in the software and the bugs are found. These also serve as robustness tests.

• Consider organizing classifications into a table with the following columns: input or output event, valid equivalence classes, and invalid equivalence classes.

• Identify ranges of numbers. Typically for a range of numbers, the following are tested: values within the range (at both ends and in the middle), a value below the smallest number in the range, a value above the largest number in the range, and a non-number.

• Identify membership in a group.

• Consider variables that must be equal.

• Consider equivalent output events. This can sometimes be challenging because it requires one to determine the inputs that generate the outputs.

9.6.3.2 Boundary Value Testing

Boundary value analysis (BVA) is a test-case design technique that complements equivalence partitioning. Rather than selecting any element of an equivalence class, BVA leads to the selection of test cases at the “edges” of the class. Rather than focusing solely on input conditions, BVA derives test cases from the output domain as well [12].

BVA is closely related to equivalence class partitioning with two differences:

(1) BVA tests each edge of the equivalence class and (2) BVA explores the outputs of equivalence classes [8]. Boundary conditions can be subtle and difficult to identify [8].

Boundary values are the biggest, smallest, soonest, shortest, loudest, fastest, ugliest members of the equivalence class (that is, the most extreme values). Incorrect equalities (e.g., > instead of >=) are also considered [10]. Programmers may accidently create incorrect boundaries for linear variables; therefore, the boundaries are one of the best places to find errors. Boundary testing is especially effective for detecting looping errors, off-by-one errors, and erroneous relational operators [9]. It is important to consider outputs as well as inputs. Midrange values should also be examined.

A classical programming error is to be off by one in a loop count (e.g., the programmer may check for < when he or she should check for <=). Therefore, testing the boundaries of loops can be a good place to find errors. When testing the loop, one first bypasses the loop; then one iterates through the loop once, twice, maximum number of times, maximum minus one times, and maximum plus one times. This type of testing is often performed as part of low-level testing, since looping is normally a design detail rather than a functional requirements detail.

9.6.3.3 State Transition Testing

When state transitions are used, state transition testing considers the valid and invalid transitions between each state. Invalid state transitions that could impact safety or functionality should be tested in addition to the valid transitions.

9.6.3.4 Decision Table Testing

Requirements sometimes include decision tables to represent complex logical relationships. Since the tables function as requirements, they need to be tested. Typically, conditions in the table are interpreted as inputs and actions are interpreted as outputs. Equivalence classes may be present in the table as well. Testers need to show that they have thoroughly exercised the requirements presented in the table. Testing may reveal deficiencies in the table itself (such as, incorrect logic or incomplete data), as well as incorrect implementation in the code.

9.6.3.5 Integration Testing

Integration is the process of putting the software components together. The components may work beautifully when performing the low-level testing on each component; however, the process of integrating the components may result in unexpected results. When components are integrated, several things can go wrong, including data may be lost across an interface, one component may adversely affect another component, subfunctions may not properly contribute to the higher level functionality desired, global data may become corrupted, etc. [12]. To avoid such integration hurdles, an integration strategy is typically applied. Some common integration approaches are discussed in following.

Big bang integration: This approach throws the software components together to see if it works (it usually does not). Even though conventional wisdom tells us that big bang is a bad idea, it happens. I have seen multiple companies unsuccessfully attempt it. One of the main problems with big bang is that when a problem exists, it is difficult to track it down, because the integrity of the modules or their interaction is unknown. Some more effective integration strategies are now discussed.

Top-down integration: With this approach components are integrated moving downward through the hierarchy—typically starting with the main control module. Stubs are used for components that are not yet integrated. Stubs are replaced with one component at a time. Tests are conducted as each component is integrated.

Bottom-up integration: With this approach, the lower level components are tested using drivers. Drivers emulate components at the next level up the tree. Then the components are integrated up the hierarchy.

Combination of top-down and bottom-up integration (sometimes referred to as sandwich integration or hybrid integration): Oftentimes one branch at a time is integrated. With this approach not as many stubs and drivers are needed.

The preferred approach depends on the architecture and other programmatic details. Typically, it is best to test the most critical and high-risk modules first, as well as key functionality. Then functionality is added in an organized manner.

9.6.3.6 Performance Testing

Performance testing evaluates the run-time performance of the software within the integrated system and identifies bottlenecks in the system. Performance tests are used to demonstrate that performance requirements (such as, timing and throughput) are satisfied. Race conditions and other timing-related issues are also examined. Performance testing typically occurs throughout the testing process and may lead to redesign or optimization setting changes, if results are inadequate. As noted in Section 9.5, these tests are often also used to help determine the margins of the software in the target environment.

9.6.3.7 Other Strategies

There are several other strategies for testing that may be employed, depending on the system architecture and the requirements organization. For example, loop testing may be applied for simple loops, nested loops, and concatenated loops. Also, if a model is used in the requirements, model-based testing techniques may be applied (model-based development and verification are discussed in Chapter 14). Additionally, for algorithms, special testing may be needed, depending on the nature of the algorithm. For example, when testing the digital implementation of a first order lag filter, enough iterations need to be run to be able to show that the first order response actually curves instead of just being a linear ramp.

9.6.3.8 Complexity Measurements

Experience shows that complex code and interactions of complex portions of code are generally more prone to mistakes [9]. Complex code has a tendency to be buggy and difficult to maintain. Complexity measurements are often applied to code during the development phase to alert coders of the risky code constructs. When possible, the software is redesigned or refactored to make it less complex. Coding standards often include a guideline to measure and minimize the complexity (e.g., McCabe’s cyclomatic complexity measure is often applied). However, if the code complexity is not resolved, it can be a good indicator to testers of where the code vulnerabilities exist. Complex software should be pounded hard (that is, more normal and robustness tests), because complex code will typically have more errors. The errors often creep in when the code is modified, because the code is difficult to understand and maintain.

One project that I lived through had a particularly complex feature. It was called the fragile code. Any time it was modified, the entire subsystem had to be completely retested, because the impact of the change was unpredictable. It ended up being an unacceptable situation for a safety-critical system and had to be redesigned late in the project. Highly complex or fragile code should be redesigned as early as possible. It is less agonizing to fix it up front than to try to track down phantom symptoms at the 11th hour.

9.6.3.9 Summary and Characteristics of a Good Test

Section 9.6.3 has surveyed several testing approaches. Most projects use a combination of many or all of the aforementioned approaches. There are others we could cover. Kaner et al. explain that regardless of the test approach, there are some common characteristics of a good test. Good test cases satisfy the following criteria [10]:

• “It has a reasonable probability of catching an error.” Tests are designed to find errors—not merely to prove functionality. When writing a test one must consider how the program might fail.

• “It is not redundant.” Redundant tests offer little value. If two tests are looking for the same error, why run both?

• “It is the best of its breed.” Tests that are most likely to find errors are most effective.

• “It is neither too simple nor too complex.” Overly-complex tests are difficult to maintain and to identify the error. Overly-simple tests are often ineffective and add little value.

9.6.5.1 Test Cases

DO-178C defines a test case as: “A set of test inputs, execution conditions, and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement” [1]. Typically, a template is provided for the test team to ensure that the tests cases are laid out in a common manner. If the tests are automated, the format is particularly important. Additionally, a tool is sometimes used to extract information from the test case headers to generate a test summary or traceability report. Whether the tests are manual or automated, each test case usually includes the following:

• Test case identification

• Test case revision history

• Test author

• Identification of software under test

• Test description

• Requirement(s) tested

• Test category (high-level, low-level, or both highand low-level)

• Test type (normal or robustness)

• Test inputs

• Test steps or scenarios (the actual test)

• Test outputs

• Expected results

• Pass/fail criteria

9.6.5.2 Test Procedures

In addition to test cases, DO-178C mentions the need for test procedures. A test procedure is “detailed instructions for the set-up and execution of a given set of test cases, and instructions for the evaluation of results of executing the test cases” [1]. Test procedures come in different shapes and sizes. Some are actually embedded in the test cases themselves; others are separate high-level documents that explain how to execute the test cases. The test procedures are the steps used to execute the test cases and obtain the test results. Sometimes they are manual and sometimes they are automated.

Each test step and how to verify pass/fail should be clear and repeatable. The person who executes the test is often not the one who wrote the test. In fact some companies and certification authorities insist on independent test execution. Vague test procedures are a common problem; the steps may be clear to the test author, but someone who is not familiar with the test or functionality may not be able to execute the test successfully. To ensure clarity and repeatability, it is a good practice to have someone who did not write the tests perform a dry run of the tests as early as possible. This may actually be part of the test review process, which is discussed later.

9.6.5.3 DO-178C Requirements

Table 9.1 summarizes the DO-178C minimum requirements for highand low-level requirements testing, as well as normal and robustness testing.

9.6.5.4 Low-Level Requirements Testing versus Unit Testing

Before exploring test execution, I want to briefly discuss unit testing. Some companies use unit testing for their low-level requirements testing, but this needs to be handled with care. Traditional unit testing is performed by writing tests against the code. This certainly is not prohibited and is a good practice for the code developers to perform to ensure their code does what they want (that is, debugging). However, DO-178C emphasizes that test ing for certification credit is to be against the requirements. The concepts of unit testing can still apply when each module (or group of modules) is tested against the low-level requirements, but care must be taken to write tests against the requirements and not the code.^* Additionally, testing module by module may create some issues for integration. If low-level testing is performed at the module or function level, a concrete plan for how the modules are integrated and tested together is needed. As discussed earlier, big bang integration is not advised and is normally unacceptable. Also, at the highest level of integration it may be difficult to confirm the correctness of interfaces between software modules or functions. When module-level testing is used, there is generally some additional software/software integration testing required in addition to the high-level software/hardware integration testing. When data coupling and control coupling analyses are discussed later (in Section 9.7), it should become more apparent why the software/software integration is needed.

9.6.5.5 Handling Requirements That Cannot Be Tested

During the test development phase, it may be discovered that some requirements are not testable. In some cases, the requirement may need to be re-written (e.g., negative requirements, requirements without tolerances, or ambiguous requirements). In other cases, an analysis or code inspection might be used in lieu of a test. Such situations should be handled carefully, because in general, analysis and code inspection are not considered as effective as testing.^* If an analysis or code inspection is used instead of a test, it must be justified why the analysis or code inspection will find the same kinds of errors as a test and why the analysis or code inspection is equivalent to (or better) than a test. I recommend including the justification in the analysis or code inspection itself, so it is recorded. The analysis should also be well documented and repeatable (as noted in Section 9.5). An analysis or code inspection that is used in lieu of a test needs the same level of independence as a test would.

9.6.5.6 Obtaining Credit for Multiple Levels of Testing

In some situations a project may be able to write some tests so that they exercise both highand low-level requirements. Generally, it is more likely that a high-level test to can satisfy low-level requirements, rather than low-level tests satisfying high-level requirements, but it depends on the integration approach and the granularity of the requirements. If a single test or group of tests is used to claim credit for multiple levels of requirements, there needs to be documented evidence that both levels of requirements are satisfied, including trace data and review records. In some cases an analysis may be needed demonstrate how the test(s) fully exercise both levels of requirements, particularly if the tests were not originally written to cover multiple levels of requirements. Such an analysis should be included as part of the test data.

9.6.5.7 Testing Additional Levels of Requirements

In some projects, in addition to highand low-level requirements, there may be other intermediate levels of requirements. These requirements will also need to be tested, with the possible exception of level D software. As noted in the previous section, it might be possible to use one set of requirementsbased tests to exercise multiple levels of requirements.

9.6.6.1 Performing Dry Runs

Most programs do a dry run of the tests, in order to work out any issues and resolve problems prior to for-score or formal runs that are used for certification credit. A dry run of tests is highly recommended. Trying to go straight from a paper review to formal execution typically results in some unexpected and undesired surprises. For higher software levels (levels A and B), it is a good practice to have an independent person who did not write the procedures execute the dry run. This helps work out issues with the procedures, as well as uncovers unexpected failures.

9.6.6.2 Reviewing Test Cases and Procedures

Prior to test execution, the test cases and procedures should be reviewed (for levels A, B, and C) and put under change control. Oftentimes, the tests are informally run prior to the review, so that the results can be referenced during the review. The review process is further discussed in Section 9.7.

9.6.6.3 Using Target Computer versus Emulator or Simulator

Tests are typically executed on the target computer—particularly the hardware/software integration tests. Sometimes a target computer emulator or a host-based computer simulator is used to execute the tests. If this is the case, the differences between the emulator/simulator and the target computer need to be assessed to ensure that the ability to detect errors is the same as on the target computer. Showing the equivalence may be achieved by a difference analysis^* or by qualifying the emulator or simulator (using the tool qualification approach described in DO-178C and DO-330, which will be discussed more in Chapter 13). It should be noted that some tests will likely need to be rerun on the target computer, even if an emulator or simulator is used, since there are some types of errors that may only be detected in the target computer environment.^†, ^‡

9.6.6.4 Documenting the Verification Environment

The Software Life Cycle Environment Configuration Index (SLECI) (or equivalent) should be up-to-date prior to executing the tests for certification credit. The SLECI documents the details of the verification environment and is normally used to configure the test stations before tests are run for certification credit. Chapter 10 provides information on the SLECI contents.

9.6.6.5 Test Readiness Review

DO-178C does not discuss the TRR; however, most projects perform a TRR prior to executing the tests to ensure the following:

• The requirements, design, source code, and executable object code have been baselined.

• The tests are aligned with the baselined version of the requirements, design, and code.

• The test cases and procedures have been reviewed and baselined.

• Any issues from dry runs have been addressed.

• The test builds have been completed and the part-number of the software under test is documented.

• The approved test environment is used.

• The test stations have been configured per the approved SLECI (or equivalent).

• The test trace data is up-to-date (traceability is discussed later in this chapter).

• The test schedule has been provided to those who are required to witness or support the test (for example, software quality assurance [SQA], customer, and/or certification liaison personnel).

• Any known problem reports against the software under test have been resolved or agreed upon for deferral (if there are tests that are expected to fail, it is important to know the expected test failures going into the formal test run).

It should be noted that for some larger projects, the tests are executed in groups. In this situation, there may be a TRR for each group of tests. When this occurs, care must be taken to ensure that all requirements are fully evaluated and that any dependencies between the test groups are considered.

9.6.6.6 Running Tests for Certification Credit

Test execution is the process of executing test cases and procedures, observing the responses, and recording the results. Test execution is often run by someone who did not write the tests and is frequently witnessed by SQA and/or the customer.

During the formal test execution, the test procedures are followed and checked off as they are completed (they should already have been reviewed and baselined).^* A test log is typically kept to record who performed the test, when it was performed, who witnessed the test, the test and software version information, and what the test results are. Oftentimes pass/fail will also be determined during the test execution. However, if the data needs to be analyzed, the pass/fail determination might occur later.

As the tests are run for certification credit (formal test execution) there may be some minor redlines^* to the procedures. If a good dry run is performed, redlines should be minimal. Too many redlines tends to be an indicator of immature test procedures. When redlines are minimal, they are typically recorded on a hard copy or on a test log and will be approved by engineering and SQA prior to release of the results. The redlined data are normally approved by SQA and are included in a problem report, so the tests are updated in the future.

When executed for certification credit, the tests may uncover some unexpected failures. When this happens, the failures are analyzed and appropriate action taken. Oftentimes, the tests and/or requirements are updated and tests are rerun. Test failures and resulting updates should be handled via the problem reporting process. For tests that are rerun, a regression analysis may be needed. See Section 9.6.9 for additional discussion on the regression test process.

9.7.4.1 Statement Coverage (DO-178C Table A-7 Objective 7)

Statement coverage is required for levels A, B, and C; it ensures that “every statement in the program has been invoked at least once” [1]. This is achieved by evaluating the coverage of the code during requirements-based testing and ensuring that every executable statement has been invoked at least once. Statement coverage is considered a relatively weak criterion because it does not evaluate some control structures and it does not detect certain types of logic errors. An if-then construct only requires the decision to be evaluated to true to cover the statements. An if-then-else requires the decision to be evaluated both true and false. Basically, statement coverage ensures that the code has been covered, but it might not be covered for the right reason.

9.7.4.2 Decision Coverage (DO-178C Table A-7 Objective 6)

Decision coverage is required for levels A and B and is commonly equated with branch or path coverage. However, decision coverage is slightly different. DO-178C defines decision and decision coverage as follows [1]:

• “Decision – A Boolean expression composed of conditions and zero or more Boolean operators. If a condition appears more than once in a decision, each occurrence is a distinct condition.”

• “Decision coverage – Every point of entry and exit in the program has been invoked at least once and every decision in the program has taken on all possible outcomes at least once.”

The confusion centers around the traditional industry understanding of branch point (which is typically an if-then-else, do-while, or case statement) versus the DO-178C literal definition of decision. Basically, using the literal definition of DO-178C, a decision is not synonymous with a branch point. The main difference is that a Boolean value that is output on an input/output (I/O) port may have a binary effect on the behavior of the system (e.g., wheels up when TRUE and wheels down otherwise). It is important to make sure that this value is evaluated both TRUE and FALSE during test. This could be treated as a boundary value problem or a decision coverage problem.

In general decision coverage ensures that all paths in the code are taken and if there are any Boolean assignments, both the true and false conditions are exercised either when the value is used directly externally or when the value is used in a construct that results in a branch.

9.7.4.3 Modified Condition/Decision Coverage (DO-178C Table A-7 Objective 5)

Modified condition/decision coverage (MC/DC) is required for level A software and has been the subject of much debate most of my career and will probably continue to generate heated discussions after my retirement.

DO-178C defines condition, decision, and modified condition/decision coverage as follows [1]:

• “Condition – A Boolean expression containing no Boolean operators except for the unary operator (NOT).”

• “Decision – A Boolean expression composed of conditions and zero or more Boolean operators. If a condition appears more than once in a decision, each occurrence is a distinct condition.”

• “Modified condition/decision coverage – Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken all possible outcomes at least once, every decision in the program has taken all possible outcomes at least once, and each condition in a decision has been shown to independently affect that decision’s outcome. A condition is shown to independently affect a decision’s outcome by: (1) varying just that condition while holding fixed all other possible conditions, or (2) varying just that condition while holding fixed all other possible conditions that could affect the outcome.”

Even though an MC/DC criterion has been applied to aviation projects since the late-1980s/early-1990s, it is still rarely discussed in the mainstream software engineering literature. It was developed for the aviation industry to allow comprehensive criteria to evaluate test completion, without requiring that each possible combination of inputs to a decision be executed at least once (i.e., exhaustive testing of the input combinations to a decision, known as multiple condition coverage). While multiple condition coverage might provide the most extensive structural coverage measure, it is not feasible for many cases, because a decision with n inputs, requires 2n tests [13]. MC/DC, on the other hand, generally requires a minimum of n + 1 test cases for a decision with n inputs.

Most projects apply MC/DC at the source code level, because the source code coverage tools are more available. However, there have been a few projects that have applied the structural coverage criteria at the object code or executable object code levels. There has been some debate on this approach, as noted in the Certification Authorities Software Team (CAST) position paper CAST-17, entitled Structural coverage of object code. The position paper provides a summary of the motivations behind object code coverage and some of the issues to be addressed [14]. The CAST-17 paper has served as the foundation for project-specific Federal Aviation Administration (FAA) issue papers. DO-178C section 6.4.4.2.b modified the wording from DO-178B to make it clear that object code or executable object code coverage are acceptable approaches: “Structural coverage analysis may be performed on the Source Code, object code, or Executable Object Code” [1]. DO-248C frequently asked question #42, entitled “What needs to be considered when performing structural coverage at the object code level?” also clarifies this topic [3].

It is tempting to write more about structural coverage—especially MC/DC, since I have spent considerable time investigating, debating, and evaluating it over the last several years. However, because it has been so widely discussed and debated in the aviation community, there are publicly available documents that cover the topic well. Items 1–5 in the “Recommended Reading” section of this chapter identify some particularly helpful resources.

9.7.4.4 Additional Code Verification (DO-178C Table A-7 Objective 9)

DO-178C Table A-7 objective 9 is summarized as follows: “Verification of additional code, that cannot be traced to Source Code, is achieved” [1].

This objective was added for DO-178C, although the text in DO-178B also required it. Most projects perform structural coverage on the source code (although a few do perform object code coverage or machine code coverage). However, the executable object code is what actually flies. For level A software, there needs to be some kind of analysis to ensure that the compiler does not generate untraceable or nondeterministic code.

For projects that perform their structural coverage on the source code, a source-to-object code traceability analysis is normally performed. In my experience, this includes a comparison between source code and object code to ensure that the compiler is not adding, deleting, or morphing the code. The analysis is usually applied using a sample of the actual code, rather than 100% of the code.^* The sample used should include all constructs that are allowed in the source code and comprise at least 10% of the actual code base. It is also recommended that combinations of the constructs be evaluated. The analysis should be performed using the compiler settings that will be used for the actual code generation. The analysis normally involves a lineby-line comparison of the source code and the object code (or machine code). Any object code (or machine code) that is not directly traceable to the source code is analyzed and explained. In some situations, the analysis may identify unacceptable compiler behavior, which requires action. Compiler features such as register tracking, instruction scheduling, and branch optimization can lead to problems with the analysis. Highly optimized compiler settings and certain languages (e.g., languages with object-oriented features) can also result in untraceable code, and hence would not be certifiable. The analy sis requires an engineer with knowledge of the specific language, assembly, machine code, and compilers. If the source code changes, there will need to be an assessment to ensure that no additional code constructs are added and that the analysis remains valid. Once this analysis is performed, it may be used for multiple projects, as long as the compiler and its settings are unchanged and the code uses the same constructs.

For the majority of projects that I have worked on, the source-to-object code analysis has not uncovered significant issues, as long as the optimization is kept minimal and mature languages and compilers are used. However, the analysis is still needed and should be done thoroughly.

9.7.4.5 Data Coupling and Control Coupling Analyses (DO-178C Table A-7 Objective 8)

I have saved the best for last in this section on structural coverage. The data coupling and control coupling (DC/CC) objective (Table A-7 objective 8) was included in DO-178B but was not clear. In particular, DO-178B varied from the overall software engineering definition of data coupling and control coupling but did not clearly explain the intent. According to members from the RTCA Special Committee #167 that developed DO-178B, the data and control coupling objective was added late in the committee deliberations and did not have extensive discussion. Consequently, there has been confusion about what was actually meant by the objective. Thankfully, DO-178C has clarified it. However, the clarification will likely cause some challenges throughout the industry, because many companies do not comply with the clarified objective (i.e., their interpretation of the original DO-178B objective is not consistent with the DO-178C clarification).

DO-178C defines data coupling, control coupling, and component as follows [1]:

• “Data coupling—The dependence of a software component on data not exclusively under the control of that software component.”

• “Control coupling—The manner or degree by which one software component influences the execution of another software component.”

• “Component—A self-contained part, combination of parts, subassemblies, or units that performs a distinct function of a system.”

According to DO-178C, the intent of Table A-7 objective 8 is to perform an “analysis to confirm that the requirements-based testing has exercised the data and control coupling between code components” [1]. CAST-19 states that the purpose of data and control coupling analyses is as follows:

To provide a measurement and assurance of the correctness of these modules/components’ interactions and dependencies. That is, the intent is to show that the software modules/components affect one another in the ways in which the software designer intended and do not affect one another in ways in which they were not intended, thus resulting in unplanned, anomalous, or erroneous behavior. Typically, the measurements and assurance should be conducted on R-BT [requirements-based tests] of the integrated components (that is, on the final software program build) in order to ensure that the interactions and dependencies are correct, the coverage is complete, and the objective is satisfied [16].^*

Unfortunately, this is not how many developers have interpreted the criteria. Many have applied it as a design and code review activity to ensure that the code accurately implements the control and data flow of the design. This activity during development is important to ensure that the final DC/CC analysis will be successful and to satisfy DO-178C Tables A-4 and A-5 objectives; however, this is not the intent of the Table A-7 objective. In addition to design and code reviews, those who comply with DO-178C also need to ensure that their tests cover the data couples and control couples between components (such as, modules or functions). Some organizations have applied the DC/CC criteria as intended, and it can be quite challenging—particularly, if design and code reviews were inadequate or if the integration approach is not sound. If the requirements-based testing and structural coverage measurements are performed on an integrated system (rather than module by module) without instrumenting the code, this helps improve confidence.

In order to successfully perform DC/CC analysis using requirementsbased tests, it is important to ensure during development that the architecture is consistent with requirements and that code complies with architecture. Satisfying DO-178C’s guidance on DC/CC analysis normally involves four steps as described in the following.

1. First, the software architecture must be documented (DO-178C Table A-2 objective 3). DO-178C section 11.10 provides guidance on what to document in the software design description, including the following related to data or control flow: data structures, software architecture, internal and external input/output, data flow and control flow of the design, scheduling procedures and inter-processor/inter-task communication mechanisms, partitioning methods, and descriptions of software components [1]. Chapter 7 discussed the design process.

2. Second, the software architecture and code are reviewed and/or analyzed for consistency. DO-178C Table A-4 objective 9 references DO-178C section 6.3.3., which explains that one purpose of the design review and/or analysis is to ensure the correct relationship between “components of the software architecture. This relationship exists via data flow and control flow…” [1]. DO-178C Table A-5 objective 2 references DO-178C section 6.3.4.b, which explains that one purpose of the code review and/or analysis is to “ensure that the Source Code matches the data flow and control flow defined in the software architecture” [1]. Normally, both the design and code review/analysis activities involve a detailed checklist or questionnaire to consider common data coupling and control coupling issues. Table 9.2 provides some example issues to consider during the reviews and analyses; the specifics will vary depending on the architecture, language, environment, etc. These are merely examples.

3. Third, requirements-based integration tests are developed (DO-178C Table A-5). DO-178C section 6.4.3.b explains that requirementsbased software integration testing is performed to ensure that “the software components interact correctly with each other and sat isfy the software requirements and software architecture” [1]. The integration testing is intended to identify the following kinds of errors [1]: incorrect initialization of variables and constants, parameter passing errors, data corruptions (especially global data), incorrect sequencing of events and operations, and inadequate end-to-end numerical resolution. Some argue that including the requirementsbased emphasis in the DO-178C explanation of DC/CC analysis weakens the criteria, because the architecture also needs to be exercised. However, during design review the consistency between architecture and requirements should be verified. If that occurs, then the requirements-based testing should also exercise the architecture. Also, as noted in DO-178C section 6.4.3.b the architecture should be considered during the DC/CC analysis.

   Table 9.2 Example Data and Control Coupling Items to Consider During Design and Code Review/Analysis

   Example Data Coupling Items to Consider

   Example Control Coupling Items to Consider

   All external inputs and outputs are defined and are correct All internal inputs and outputs are defined and are correct Data is typed correctly/consistently Units are consistent and agree with data dictionary Data dictionary and code agree and are both complete Data is sent and received in the right order Data is used consistently Data corruption is prevented or detected Data is initialized or read-in before being used Stale or invalid data is prevented or detected Data miscompares or data dropouts are prevented or detected Unexpected floating point values are prevented or detected Parameters are passed properly Global data and data elements within global data constructs are correct I/O is properly accessed from external sources All variables are set (or initialized) before being used All variables are used Overflow and underflow is identified and correct Local and global data are used Arrays are properly indexed Code is consistent with the design

   Order of execution is identified and correct Rate of execution is identified and correct Conditional execution is identified and correct Execution dependencies are identified and correct Execution sequence, rate, and conditions satisfy the requirements Interrupts are identified and correct Exceptions are identified and correct Resets are identified and correct Responses to power interrupts are identified and correct Foreground schedulers execute in proper order and at the right rate Background schedulers are executed and not stuck in infinite loop Code is consistent with the design

4. Fourth, tests are analyzed to confirm that the data and control coupling between components are exercised by the requirementsbased tests (DO-178C Table A-7 objective 8). If Steps 1–3 are not performed adequately, it will be difficult (probably impossible) to adequately complete Step 4. As far as I know, there are no commercially available tools to perform the data coupling and control coupling analysis. Most companies either development their own tool(s) or perform the analyses manually. Hopefully, there will be some expansion of the better commercially available tools in the future to help with the data and control coupling analyses.

Item 6 in the “Recommended Readings” section of this chapter pro vides some additional insight into data coupling and control coupling analyses. Additionally, since partitioning is closely related to data and control coupling, Chapter 21 discusses some additional items worth considering when developing an approach to satisfy DO-178C Table A-7 objective 8.

9.7.4.6 Addressing Structural Coverage Gaps

When measuring code coverage obtained through requirements-based tests, some gaps in coverage may be identified. An analysis is performed to determine the reason for the gaps. The following actions may be taken:

• If the coverage gap is caused because of missing tests, additional test cases are added and executed.

• If the gap is caused because of a missing requirements (an undocumented feature), the requirements are updated and tests are added and executed.

• If the gap identifies dead code or extraneous code, the dead or extraneous code is removed and an analysis is performed to assess the effect and the need for reverification. Typically, some reverification is needed. See Chapter 17 for information on dead and extraneous code.

• If the gap is caused by deactivated code, the deactivation approach is analyzed to ensure it is adequate and consistent with requirements and design. See Chapter 17 for discussion of deactivated code.

• If the gap is not addressed by any of the aforementioned, an additional analysis is needed to ensure that the code is adequately verified and works as intended. Typically, this analysis is either included in or summarized in the SVR.

9.7.4.7 Final Thoughts on Structural Coverage Analysis

Before we leave the subject of structural coverage, I want to summarize four important things.

First, when structural coverage credit is claimed, it should be based on requirements-based tests that pass (and satisfy the requirements they are traced to). If the test fails, the coverage may not be valid.

Second, structural coverage is typically measured by instrumenting the source code to identify where the source code has been exercised during the test execution. As the source code is changed it is important to assess if the changes introduced by the instrumentation are benign (as they should be). The resultant code optimization is also affected by the instrumentation because the information recording coverage features changes information flow. Therefore, it is necessary to rerun the tests without the instrumentation and compare the instrumented and non-instrumented results before claiming structural coverage credit.

Third, not all structural coverage tools fully satisfy DO-178C’s criteria. In particular, be cautious of tools used to measure coverage in concurrent code that have not been designed to measure coverage in the presence of tasking constructs. Tools should be carefully evaluated and thoroughly understood before investing time and money. The FAA research report, entitled Software Verification Tools Assessment Study, provides some interesting insight. The FAA-sponsored research proposed a test suite to assess commercially available structural coverage tools. Anomalies were found in each of the three evaluated tools. This research demonstrates that care must be taken when selecting a tool, since some tools may not meet the DO-178C definition of structural coverage [17].

Lastly, the structural coverage analysis results are typically either included in or referenced in the SVR. Any structural coverage gaps and their analysis will be evaluated closely by certification authorities and/or their designees; therefore, they should be clearly stated and thoroughly justified.