CHAPTER 77

THE FUTURE OF INFORMATION ASSURANCE¹

Peter G. Neumann

77.1 INTRODUCTION

77.2 VIEW OF THE FUTURE

77.3 FOUNDATIONS OF ASSURANCE

77.3.1 Methodology

77.3.2 Guarantees

77.3.3 Pervasively Integrated Assurance

77.3.4 Analysis of Requirements

77.3.5 Analysis of Compositions

77.3.6 Analysis of Property Transformations

77.3.7 Analysis of Dependencies

77.3.8 Detecting and Eliminating Vulnerabilities

77.3.9 Software and Hardware Consistency Analysis

77.3.10 System-Oriented Analyses

77.3.11 Development Tools

77.3.12 Measures of Assurance

77.3.13 Risk Analysis and Risk Abatement

77.3.14 System Evaluation and Certification

77.4 BEST PRACTICES FOR INCREASING ASSURANCE

77.5 ASSURANCE-BASED RISK REDUCTION

77.5.1 Security

77.5.2 Human Safety

77.5.3 Reliability, Availability, and Survivability

77.5.4 Operational Assurances

77.5.5 Sound User Interfaces

77.6 ILLUSTRATIVE APPLICATION: COMPUTER-AIDED VOTING

77.6.1 Election Process

77.6.2 Voting-Related Requirements

77.7 CONCLUSIONS

77.8 FURTHER READING

77.9 NOTES

77.1 INTRODUCTION

Although this chapter is at the end of the Handbook, we are still only at the beginning of the quest for meaningfully trustworthy systems. We begin by asserting that there are no easy answers, although some potentially worthy approaches are outlined here. Nevertheless, the reader must bear in mind that the problems of obtaining trustworthy systems are inherently complex, and thus it is important to approach this complexity constructively rather than with conventional ad hoc approaches (which usually produces seriously flawed systems).

This chapter is more concerned with long-term issues than most of the rest of the Handbook. It is intentionally less of a hands-on guide than an assessment of what is possible. As such, it tends to open up many questions and possible future directions. It is intended to serve as a harbinger of what will be important in the future if the information assurance community is to contribute to the development and use of meaningfully secure computer systems, networks, and enterprise-based application software.

Webster's generic dictionary definition of assurance is relevant in the present context: “something that inspires or tends to inspire confidence.” Assurance can mean many different things to different people, especially in the context of environments dependent on information systems—which historically have experienced many failures and unexpected deviations from expected behavior on the part of the technology, the people involved, and in some cases the natural environments in which those systems operate. Thus, useful notions of assurance should encompass all factors that are relevant in any particular enterprise.²

The first question that must be considered by a system developer or acquirer is this: With respect to what requirements, for which attributes, of which systems, is assurance desired? Without well-defined requirements, most considerations of assurance are relatively meaningless.

Requirements typically include local to enterprise-wide properties relating to system and network security, reliability, system survivability, human safety, system monitoring and accountability, guaranteed real-time performance, facile interoperability, ease of use, ease of system administration, ease of maintainability, and ease of long-term evolvability in the face of changing circumstances. These requirements may be generic or specific to particular enterprises, and they tend to be associated with particular layers of abstraction, such as hardware, operating systems, application software, networking, or total systems. In the absence of well-defined requirements for the desired attributes, any concept of assurance must be considered suspect, because it is likely to be ill founded.

Although this is ostensibly a handbook on security, a system that is not reliable is not likely to be secure. A system that is not secure is not likely to be safe or survivable. Thus, we stress the notion of trustworthiness that encompasses security as just one of its requirements. By focusing on trustworthiness with respect to the broader spectrum of requirements, it is likely that a system would be more likely to satisfy those requirements—rather than the usual approach of trying to add the other requirements retroactively.

Another fundamental question for system developers and users is this: What kind of assurance is required? A wide range of approaches to assurance is available. Consequently, the response in any particular application environment depends on the nature of the requirements and the criticality of the intended applications, as well as on many other considerations, such as cost, development schedules, and risks of producing systems whose failures result directly from the inability of those systems to satisfy their intended requirements. Those risks may themselves have associated costs and long-term implications.

What is desired here are some measures of assurance that a system will, with acceptable likelihood, satisfy its requirements under specified (and, we hope, realistic) assumptions. That is, a system should be demonstrably trustworthy—in other words, worthy of being trusted to satisfy the given requirements under the specified assumptions and not to do something too horrible if those assumptions are violated. Note that if a system is not reliable and its resources are not adequately available, it is also not likely to be secure, particularly in the presence of reliability failures. Therefore, it is typically not enough for a system merely to satisfy a set of requirements for integrity, confidentiality, and availability. For that reason, we prefer the more encompassing concept of trustworthiness when discussing assurance, presumably with respect to a more realistic and comprehensive set of requirements.

Assurance is a concept that is meaningful throughout the system life cycle. In particular, we want a priori assurance that stated requirements are complete, mutually consistent, actually relevant, and realistically implementable. Next, we need assurance that system architectures and detailed design specifications can satisfy the given requirements—ideally, before any software is written, and certainly before systems are deployed for any particular purposes. We then need assurance that hardware and software are consistent with their specifications, and indeed that the systems can do what is expected of them. In cases in which specifications are either inherently incomplete or indeed nonexistent, determining whether implemented systems actually can satisfy their given requirements with any meaningful assurance is very difficult, and likely to be unsatisfying. These concepts are generally static in nature, although we must potentially reevaluate them whenever changes occur. Further assurance could demonstrate dynamically that operational configurations and administrative practice are consistent with operational expectations.

Assurance can have widely varying degrees of rigor, from rough estimates based on vague assumptions at one extreme, through programming language annotations that can be interpreted by preprocessors, compilers, and static analysis tools, to system testing, and at the high end to detailed mathematically based formal methods that permeate the development cycle. Assurance can be qualitative or quantitative, objective or subjective, formal or informal. Unfortunately, informal analyses may inspire confidence without actually being trustworthy. Furthermore, even the most rigorous formal methods may give meaningful assurances only within the context of the assumptions implicit or explicit in the analyses, ignoring overall system vulnerabilities. Further details are given in Section 76.3.

As noted, we must reconsider assurance arguments whenever any changes are made. Clearly, changes in requirements can necessitate consequent changes in design and implementation. Similarly, even small changes in software or execution environments can invalidate previously sound assurance arguments and have harmful effects. As a consequence, assurance is typically a moving target.

77.2 VIEW OF THE FUTURE.

Whether or not information systems are adequately trustworthy, we often must trust them. As our enterprises become increasingly dependent on information systems, it is imperative that we radically improve our collective ability to develop and operate trustworthy systems. As a consequence, greater emphasis must be placed on assurance that is pervasively integrated into the system development and procurement processes. This represents a significant change in the way computer systems and applications are developed, and creates challenges for everyone involved—conceptual thinkers, people who establish the requirements, system designers, implementers, development managers, and system administrators, as well as government and corporate personnel involved in procurement, legislation, oversight, and enforcement. User communities also need to be more aware of the myriad risks that result from the absence of assurance, and should be involved in assessing the viability of would-be systems. Improvements in education, training, and general awareness are also essential to foster a culture in which assurance is deemed much more important.

A primary goal is to achieve trustworthy systems and networks, with some demonstrably sound measures of assurance—that is, rigorously addressing how well the intended trustworthiness can be achieved. Measures of assurance can be sought in a variety of ways, throughout the development cycle, and thereafter as well. For example, they might involve analyses applied to requirements, architectures, and detailed system designs of operating system and application software, compilers, hardware, and operational practices.

Potentially important challenges relating to assurance suggest many fruitful areas for research and development. These span the entire development cycle from requirements, to system architecture, to detailed design, to implementation. Some approaches might involve relatively short-term development of analysis tools, whereas others might require major rethinking of how to design, develop, assure, procure, and use computer systems and networks.

Imagine a future world in which we have developed techniques and tools that enable us to build trustworthy systems. Further imagine that these techniques are cost effective, widely used in mainstream developments, and demonstrably capable of producing systems that satisfy well-defined requirements. This is the world that we need to attain.

Two possible paths toward this end come easily to mind.

1. Placing ill-founded trust in faulty, insecure, and fundamentally broken systems, society would experience failures with increasing frequency and negative repercussions, leading to a crescendo of disasters publicly viewed as completely unacceptable. At that point, perhaps, serious effort would be devoted to the challenges just discussed.
2. In a proactive alternative, placing emphasis now on research, education, training, cultural change, and recognized importance of a priori understanding of the consequences, we could build the systems of tomorrow that are trustworthy from the outset because of the way in which they are developed and operated.

Clearly, much greater commitment is needed to provide assurance of trustworthiness with respect to essential properties, especially for properties of entire systems rather than just of a few components in isolation.

At the higher ends of assurance, the potential benefits of formal methods remain considerable, particularly with respect to hardware and software. Their usefulness in the specification and analysis of critical system components remains enormous. Such methods have been adopted in recent years with respect to hardware development (particularly subsequent to the 1994 Intel Pentium Floating Divide processor problem, which resulted in Intel hiring a large group of formal methodists), where formal assurance techniques have had demonstrably tangible value in assuring that the hardware designs satisfy stringent criteria. Assurance applied to operational practice lags far behind either hardware or software assurance. In the light of past events—including rampant system flaws and detected vulnerabilities, system failures, experienced penetrations, flagrant system misuses, deaths and injuries, financial losses, and other problems—formal methods remain a potentially important part of the process of developing trustworthy systems with meaningful assurance. An ongoing effort to stimulate further advances in realistic uses of formal methods is found in “Verified Software: A Grand Challenge.”³

With respect to software developments, formal analyses selectively applied throughout the development cycle can provide some significant levels of assurance, although informal techniques such as code inspection, testing, and red-teaming are complementary techniques that can also be useful. Of some potential help, but by no means sufficient by themselves, are measures of institutional goodness (as in the Capability Maturity Model) and individual programmer competence (as in certification of software engineers, such as the CISSP process described in Chapter 74.) Note that highly rated organizations and certified system developers can nevertheless produce bad systems. Overall, no single assurance technique is adequate by itself; each has inherent limitations that must be recognized and surmounted. Thus, a comprehensive combination of assurance techniques is highly desirable.

77.3 FOUNDATIONS OF ASSURANCE

Several basic issues immediately come to mind in seeking increased assurance. This section considers some approaches that can contribute thereto.

77.3.1 Methodology.

The choices of methodologies, requirements, specification languages, programming languages, and assurance techniques are important to the success of a development effort. Many system development failures have been attributed to poorly defined requirements, an inappropriate system architecture, seriously incomplete system descriptions, or perhaps specifications that are too abstract, or badly overconstrained, or too low level. Thus, we may justify devoting considerable effort to establishing sound requirements and sound architectures.

However, there is also a danger that inappropriate choices of methodology, approach, and programming languages will lock the development into a path that will later require massive changes in the systems being developed. Thus, it is essential to consider all these factors wisely early in the system development process.

77.3.2 Guarantees.

Although one can make carefully couched conclusions based on tightly constrained assumptions about paper designs and software, there can never be any absolute guarantees with respect to the behavior of computer-communication technology in actual use. No matter how careful one may be, it is impossible to anticipate and compensate for all possible deleterious events, such as hardware malfunctions, software flaws, human mistakes, malicious human actions, and environmental disruptions. We should never use assurance techniques to create a pretense of perfection. Instead, these techniques should be used to identify imperfections and potential problems in requirements, architectures, and implementations, to reason about the implications of what might go wrong, and to iterate thereupon to transcend those obstacles.

77.3.3 Pervasively Integrated Assurance.

Techniques to provide assurance related to all relevant aspects of trustworthiness should be an integral part of the development cycle. The notion of building assurance into the system development process is akin to the classical notion of addressing security architecturally as a by-product of the development.⁵ Such assurance techniques should be invoked persistently during operation, maintenance, administration, and long-term evolution.

77.3.4 Analysis of Requirements.

In general, efforts to increase assurance are meaningful only when meaningful requirements have been established. Having a well-defined relevant and extensive set of requirements is an important beginning to increasing developmental and operational assurance. Anything we can do to increase the soundness, completeness, and specificity of the stated requirements is likely to be valuable. In particular, assessing the realistic implementability of the requirements long before any development is undertaken can have significant payoffs later in development. Recognition of requirements that are likely to change over time can provide an impetus for more open-ended system designs.

Requirements are generally incomplete, erroneous, and even infeasible; developing a system based on such requirements is clearly counterproductive, costly, and time consuming. In contrast, detailed analyses of requirements (in particular, formal analyses where the requirements are formally defined) can be enormously beneficial.

77.3.5 Analysis of Compositions.

In the present context, the term “composition” has a range of meanings relating to how a system is constructed out of its components. It includes conventional computer science usage involving serial and parallel composition. It also includes the coexistence of multiple modules at a given layer of abstraction (horizontal composition) and the synergy among different layers of abstraction (vertical composition).

The concept of predictable composition is meaningful with respect to many entities, including requirements, specifications, protocols, implemented components, and proofs. Combinations of these entities often have surprising results. Consequently, as a result of a composition, analytic techniques can significantly increase assurance that existing properties will be preserved (composability), that existing properties will be suitably transformed, and that newly emergent properties will be acceptable (compositionality). It is desirable to be able to determine the existence of unspecified side effects, as well as the satisfaction of higher-layer properties resulting from composition, with respect to both designs and implementations of components and entire systems. Compositions can be evaluated with respect to many attributes, such as interoperability, sound information flow, the absence of covert channels, reliable and secure intercommunications, absence of deadlocks, serializability, and the avoidance of characteristic flaws. Resilience is also an important property, including identifying potential damage and confining it or otherwise remediating its effects.

In the vision of the future considered here, special-purpose and general-purpose systems and system applications might become routinely composable out of more or less compatible, demonstrably trustworthy components. To make such composition practical, a new type of analysis tool would be highly desirable. Such a tool would analytically determine the effects of composing software components (ideally, well designed and well specified), not just for the initial creation of a composed system, but also in subsequent reconfigurations, upgrades, and even in the installation of mobile code. Such an approach could, for example, take advantage of software formally shown to be consistent with its specifications, including descriptors relating to previously evaluated characteristics of the components, such as internal locks that might cause mutual deadlocks in certain contexts, assumptions regarding relative dependencies, identified interface limitations, and other attributes that might affect the effectiveness of the composition. Ideally, this approach could then be used iteratively: for example, initially pairwise or n-wise, and then over successively wider scopes, possibly ascertaining obstacles to the desired compositions or even potential failure modes that would suggest that a specific composition should not be permitted in certain types of environments because of the identified deficiencies. Other properties could also be included in the analysis, such as dynamic trustworthiness, configuration stability, and operational factors. There are practical difficulties with such an approach, but even small steps forward could be useful.⁶

77.3.6 Analysis of Property Transformations.

Composition may possibly alter the properties of the combined abstractions. Considerable value can accrue from transformations that demonstrably preserve desired properties—for example, transformations that maintain the consistency of specifications with requirements despite changes to requirements, or maintain the consistency of code to specifications despite the machinations of optimizing compilers, or maintain the satisfaction of evaluated properties after subsequent system modifications and reconfigurations. Demonstrating that a composition preserves certain essential properties, and that newly developed properties are satisfactory, can significantly simplify analysis of systems in the large. Transformations that result from nontrivial compositions, hierarchical layerings, and interpositions of mediators, such as trustworthy firewalls and guards, are of considerable value but may require further analysis because the resulting emergent properties may have unforeseen consequences. For example, joining two seemingly secure components via an unencrypted link may result in both of the components being compromisable. The analytic approach should also permit parametric architectural representations—for example, capable of being analyzed for what simplifications or complications might result when an architecture undergoes a particular change in design, such as making a file server multilevel secure instead of having multiple single-level file servers.

77.3.7 Analysis of Dependencies.

In any system, it is desirable to be able to identify explicitly all potential dependencies and, in particular, all potentially adverse dependencies—that is, components or subsystems that rely on other entities that are less trustworthy or of unknown trustworthiness. It is also useful to identify cyclic dependencies, especially those that can result in indefinite looping or deadlocks. Analysis of dependencies would be useful in avoiding many failures arising in large software systems, especially those with distributed control. The use of structured architectures with modular encapsulation that eliminates certain side effects can also avoid such problems in the first place, rather than having to rectify them later. Similar analytical techniques apply to undesired dependencies among policies, specifications, and assurance techniques themselves.

77.3.8 Detecting and Eliminating Vulnerabilities.

Assurance techniques in general, and formal methods in particular, are most fruitfully applied when they can dramatically improve the process of detecting and eliminating significant system flaws and vulnerabilities, and also improve the quality of the resulting systems. To this end, it is appropriate to ascertain that critical requirements soundly represent their intended purpose and to demonstrate, by formal, semiformal, or informal reasoning, that critical properties are satisfied by particular components or subsystems with respect to their specifications. It is also appropriate to examine critical aspects of the resulting system and of the implementation process itself, seeking to derive or prove properties of the system in terms of properties of its subsystems. The essence of this process is to provide cumulatively increasing confidence in the system design and its implementation by first identifying inadequate requirements, flawed designs, and bad implementations and then overcoming them. Steps that do not add substantively to this process are generally less effective. For example, if the requirements are flawed and the specifications are flawed, then attempting to carry out analyses that demonstrate consistency between code and specifications would be of questionable value. Once requirements are properly established, analyses and supporting tools can be helpful if they can identify certain types of flaws in designs, source code, and executables. In particular, formal techniques and other analyses can be useful in identifying flaws in authentication algorithms, access control mechanisms, network protocols, cryptographic embeddings, trustworthiness in general, and sets of requirements themselves.

77.3.9 Software and Hardware Consistency Analysis.

The term “correctness” is often used in a vague sense that raises the question of “correctness with respect to what?” Instead, this chapter refers to more precise statements such as “consistency of specifications with respect to requirements” or “consistency of code with respect to specifications” (assuming that there are any specifications!). It is generally unwise to overemphasize the process of trying to prove “correctness” of software and hardware: Correctness is meaningful only with respect to a set of presumed conditions that must be met—and those stated conditions themselves can often be wrong or incomplete. (The quote at the beginning of Section 76.3 is particularly pithy and also applies to requirements for trustworthiness as well as to specifications. It is also true of hardware.) Thus, efforts to increase the assurance that can be attained by an implementation should be up-front concerns, but not overly stressed, at least until the rate of discovery of flaws in requirements and design has dwindled to some agreed-on level of acceptability. However, at that point, code analyses can be valuable, especially if they are able to show the existence of design flaws and software bugs as well as the absence of certain types of harmful effects, such as Trojan horses or potential timing faults. In particular, formal methods in hardware implementations may prove effective—for example, in specifications, mask layouts, and fabrication, and especially for critical coprocessors or chips.

77.3.10 System-Oriented Analyses.

Overall, we need a system-oriented view of assurance that coherently encompasses temporal beginning-to-end life-cycle issues as well as analyses across and among layers of abstraction. Whether the analyses are linearly continuous (e.g., top to bottom, or bottom to top, or strictly beginning to end) or merely done piecewise and then joined together as appropriate is not so important as whether the separate analyses enable the pieces to fit together coherently. Higher-layer analyses, however, may be rendered useless if the integrity of the lower layers cannot be assured because of irreparable flaws.

Continuing on the subject of composition, horizontal (modular) composition and vertical (hierarchical abstraction) composition are both important subjects of analysis for the purpose of increasing assurance. Various past efforts are of considerable theoretical and practical interest.

Two relevant historical examples involve proving consistency within a successive refinement thread throughout the development effort, and proving consistency from top-level requirements through detailed design specifications to software implementation, and perhaps even subsequent operation and evolution, particularly when applied to large and complex systems and even when dealing with vertical and horizontal abstractions. One is the formally specified Provably Secure Operating System hardware/software design and analysis based on the Robinson-Levitt⁷ Hierarchical Development Methodology approach. The second is the Moore-Hunt-Young CLInc stack.⁸ Both showed the functionality of higher-layer abstractions to be related explicitly to the functionality of lower-layer abstractions, iteratively across all layers as desired—for example, from applications to hardware—as well as the relationships among different modules at the same layer. These techniques facilitate analysis of systems in the large, based on properties of the components. However, once again, those efforts must be considered as overkill unless there is proactive attention to requirements, system and network architecture, concise and complete specifications, and other good practices—in which case the comprehensive analysis can be compelling, even if labor intensive.

In an important sense, assurance is just as important an artifact as the code itself. Assurance is built in layers so that each layer is crafted from the assurance provided by the lower layers. Layering provides a way of isolating components from the effects of other components. Explicit assurance is also needed for managing incremental changes and adapting a system to new circumstances.⁹

77.3.11 Development Tools.

Various development tools can be useful for the modeling and analysis of requirements, system and network architecture, testing, verification, source and object code, the various types of composability, configuration management, and operation. Some tools with interesting capabilities have been developed recently. However, they are too numerous and rapidly evolving to be itemized and contrasted here. A major challenge for their developers is to make these tools more general, more compatible with one another, and more rigorously based.

77.3.12 Measures of Assurance.

To develop any concrete sense of assurance, it is necessary to establish well-defined standards and criteria against which requirements, architectures, specifications, software, tools, and operational practice can be evaluated. This is a complicated area. However, it is unrealistic to consider assurance without any measures of assurance itself. Above all, security—and even more so trustworthiness that encompasses other attributes such as reliability and human safety as well—is intrinsically multidimensional. Different applications may require different requirements, and evaluated systems typically do not satisfy all requirements equally. Reducing an evaluation to a single one-dimensional measure (e.g., a weighted average of the component measures) is clearly a bad idea unless the weights actually reflect the relative importance of the various attributes. Such a measure, however, may seem to be a wonderful idea for users and procurers, who seek (as in the case of system certification) simply a yes-or-no answer. With respect to trustworthiness, such measures are seldom black or white, especially if the requirements, the assumed range of threats and attacks, and evaluations are incomplete.

77.3.13 Risk Analysis and Risk Abatement.

Allocation of resources devoted to security should generally have some logical basis related to the risks that can thereby be avoided. However, beware of overendowing quantitative risk analyses that are based on incorrect assumptions and inadequate analyses.¹⁰

The notion of risk management (see Chapter 62) is somewhat misleading, possibly suggesting that certain risks can be lived with or compensated for externally—for example, through legislation, liability, insurance, and penalties. That is often a dangerous concept when an alternative of simply avoiding those risks could be achieved, through a combination of sound system architecture, good software development practice, and other approaches discussed herein.

77.3.14 System Evaluation and Certification.

The creation of comprehensive evaluation criteria (e.g., within the Common Criteria framework) and thorough independent and open evaluations can also significantly increase assurance levels. However, in the absence of well-defined requirements, standards, and evaluation processes, all such evaluations may be of questionable value: costly, time consuming, and ultimately not definitive.

Certification generally tends to be a black-and-white assessment of the evaluations, which means that the certification is more of a blessing than an assurance mechanism. With respect to systems that must be trustworthy, certification may give the appearance of goodness without any real substantiation. Beware of certifications without adequate evaluations. However, certifications based on detailed evaluations could be very useful, especially if accompanied by some sort of good-housekeeping seal that meaningfully attests to some measures of trustworthiness.

Evaluations should be conducted throughout a development process, not just when it supposedly is all done. Except in certain special-purpose dedicated systems, developments are seldom ever finished until the system is decommissioned.

A distinction must be made between total-system evaluations and partial evaluations that are done in the absence of the intended applications and enterprises that use those systems. This distinction suggests a need for the composability of evaluations of systems that can be extended incrementally to enterprises.

77.4 BEST PRACTICES FOR INCREASING ASSURANCE.

Providing meaningful assurance of trustworthiness is itself a complex problem, and needs to be spread out across the development process as well as into operational practice. Together with those concepts just outlined, these approaches can be used in combination with one another to enhance assurance:

• Assurance-enhancing principles. Disciplined adherence to certain useful assurance-enhancing principles¹¹ can be beneficial throughout system development. Some of these principles (such as modular abstraction and encapsulation, separation of concerns, hierarchical layering, least privilege, and minimization of what must be trustworthy) have been long established in academic communities, although they are not consistently used in practice.
• Sound system architectures. Considerable assurance can result directly from the establishment of an inherently sound conceptual architecture and analytic techniques for determining whether it is capable of satisfying its given requirements. One of the most fundamental but often ignored concepts involves architectures that effectively minimize what has to be trusted by suitably decomposing the system and ensuring that critical functions do not depend adversely on less trustworthy components. One example of that concept would be a high-assurance separation kernel on top of which a virtual machine monitor could be built. The monitor would allow multiple applications, possibly running on multiple virtualized operating systems, to execute simultaneously without interfering with one another.¹² Another approach involves creating systems that are more trustworthy than the subsystems out of which they are composed. This is a well-established approach for reliability, but also has some possibilities for security.¹³ Such techniques can have enormous payoffs, as already noted, simplifying developments, reducing costs and overruns, further simplifying assurance analyses, and greatly increasing the likelihood of development success.
• Sound software engineering. Soundly based development methodologies with appropriate development tools can encompass techniques for many measures of assurance. Use of programming languages that systemically avoid characteristic flaws, together with supporting analytic tools that are formally based, and about which properties can be proven, are also desirable. It seems silly that we are still beset with buffer overflows, missing bounds checks, wild pointers, type mismatches, inadequate handling of exceptions, and other common bugs. Developments based on specific understanding of the risks to be prevented are likely to be more effective than other developments.¹⁴
• Sound algorithms and protocols. Analysis can determine consistency with requirements for trustworthiness (security, reliability, network resilience, and so on), with respect to both designs and implementations.
• Robust system accountability. Accountability is a broad topic that encompasses the collection of audit trails with appropriate accuracy, timeliness, and tamper-resistance, as well as dynamic analyses (in real time or post hoc) that can coherently detect system anomalies, misuse, and failures, and lead to intelligent remediation. Ideally, the audit trails should be sufficiently secure, robust, and tamper-resistant to permit their relatively indisputable evidentiary use in forensic applications. Thus, the assurance of the accountability mechanisms themselves can be important.
• Analysis of properties. Functional and nonfunctional properties of subsystems, systems, and networks are subject to analysis at different layers of abstraction. For example, subsystems, operating systems, application packages, and entire enterprises can all be modeled and analyzed, formally and informally, with respect to designs and implementations. Hierarchical dependence analyses can uncover design flaws, implementation bugs, and architectural inconsistencies in adverse trust relationships. Higher-layer properties—including emergent properties—can be analyzed hierarchically (e.g., as in the Robinson-Levitt approach).
• Static and dynamic code analysis. Ideally, the strategy of doing constructive design and analysis up front wherever it is advantageous suggests that discipline embedded in the software development process can have considerable payoff, augmenting what programming languages can do to encourage greater discipline. Compilers and related pre- and postprocessor tools that provide rigorous checking would also be useful. However, the integrity that can be provided by the best methodologies, programming languages, and compiler tools is potentially compromised by people involved in design and implementation, debugging, integration, maintenance, and evolution. Examples of relevant tools include checkers for buffer overflows and other types of flaws, as well as for dynamic configuration errors. In cases where static analysis is not effective, run-time checks may be desirable. Also of considerable interest is proof-carrying code¹⁵ and cryptographic integrity seals that can detect and hinder undesired modifications of data and programs.
• Code inspection, testing, and debugging. The use of formally and informally based tools can also be highly beneficial for code inspection, testing, and debugging. Conventional uses of code inspection and testing are labor intensive, subjective, and very much dependent on the intelligence, education, training, and experience of the individuals involved; although those techniques can be aided by automated or semiautomated tools, there is always a risk that the weaknesses of the tools will interact with limitations of the people using those tools to result in a false sense of satisfactory results. Nevertheless, these approaches can be beneficial.
• Red-teaming. Red teams (penetration testers; see Chapter 46 of this Handbook) can also be used to increase assurance, if they provide truly independent and objective assessments. Although red teams have tended to be rather ad hoc in the past, the use of disciplined tool sets could actually be helpful (to the extent those tools are effective). The notion of pervasively integrated assurance suggests that the red-team concept should be substantially broadened to include pervasive interactive assessments throughout development rather than after premature decisions are cast in concrete. However, despite the desire for pervasive integration, it is important that the red team be independent of the development efforts, although still capable of providing valuable feedback.
• Composability within the evaluation process. Paul Karger and Helmut Kurth address the essential needs for cooperation and communication within various components of the evaluation process, particularly as it relates to the notion of composite evaluation within the Common Criteria.¹⁶ (This is an important paper, and deserves considerable attention. One major point of the paper is that the results of hardware evaluations must flow unimpeded to the software developers and software evaluators, irrespective of proprietary constraints. The consequences of interfering with that information flow can be quite dire.) Note that the same conclusion also applies to red-teaming, where the red team needs to have effective communication paths to project management. Whether the concerns involve functional assurance, security assessments, quality assurance, or anything else, it is important that constructive feedback not be stonewalled.
• Operational practice. At present, system administration and operational practice seem generally less amenable to assurance techniques than to technical defense measures. However, there are some potential opportunities for progress in this area, particularly in handling software upgrades, in configuration management, in emergency recovery, and in planning for these activities in system design.
• Research and development. Consistent with the ability to analyze critical aspects of complex systems, research must continue to explore the frontiers of the technology, and development must be carried out with much greater discipline than is generally invoked today. Some readers will consider assurance issues to be pie in the sky, and unrealistic from the perspective of potential consequences of increased costs, project delays, and increased needs for education and training, the spectrum of assurance techniques does have something for everyone. At the low end, anything that reduces design flaws, implementation bugs, cost overruns, scheduling delays, and resulting operational risks at acceptable costs is beneficial. At the high end, judicious use of formal methods can add significantly to development and operation—but also can add complexity, delays, and cost overruns if used unwisely. Although formal models and specifications may seem to complicate the design process with delays, increased costs, and greater intellectual demands, they can also substantively improve assurance. Formal methods can also lead to earlier identification of problems that might otherwise be uncovered only in later stages of development and use. However, such methods need to be used with considerable care and applied primarily where they can accomplish things that design reviews, testing, and operational discipline cannot. Because errors in requirements, design, and specification are perhaps the most difficult and costly to repair, formalisms can be particularly valuable in the early stages of development.
• Holistic approaches to systems. Overall, assurance must be considered as an intrinsic part of the development cycle. Its costs and benefits must be evaluated not just locally and in the short term, but rather globally and in the long term—including costs of ongoing maintenance, administration, and system evolution. When considered holistically, the benefits may well eventually outweigh the shortcomings.¹⁷

77.5 ASSURANCE-BASED RISK REDUCTION.

Chapter 62 in this Handbook considers risk assessment and risk management. The assurance techniques summarized in the previous sections of this chapter can have significant effects in identifying and reducing risks, particularly with respect to the extent to which critical system requirements are likely to be satisfied by system designs and implementations. In general, it is better to anticipate the potential risks and avoid them through sensible system architectures and development, rather than to attempt to manage the risks in a badly designed and poorly implemented system. The techniques discussed in this chapter may be applicable in many different ways, all of which are potentially relevant here. In particular, analysis can contribute at all development stages and all layers of abstraction within a development. Furthermore, it is unwise to consider security in the absence of other requirements, especially where security interacts with reliability, safety, and other critical properties that are also encompassed by trustworthiness.

There have been many advances in assurance techniques over the past decades. Many of the pieces of the puzzles relating to the development and continued operation of trustworthy systems—including theories, methods, and tools—are now reasonably well understood and advancing in relevance. It is generally unwise to put all your eggs in one basket (such as a single development methodology of testing or penetrate-and-patch). Thus, a comprehensive combination of approaches is generally desirable, including many of the approaches discussed earlier, especially if the desired paradigm shifts are observed.

Several examples of past cases that caused serious risks are given in this section. These cases illustrate how assurance techniques might be applied to systems that must be trustworthy (particularly with respect to security, safety, and reliability). Even though these examples consider what might have been done to prevent effects that actually resulted, they are explicitly not intended as a “wise-after-the-fact” exercise in hindsight. Instead, consider them as explicit representations of what types of assurance might be applicable in future developments of a similar nature.¹⁸

77.5.1 Security.

The past and present are littered with security and integrity vulnerabilities and exploitations thereof, including privacy violations. Buffer overflows, erroneous or missing bounds checks, and other annoying types of security flaws that seem to recur much too frequently can be avoided with sensible system architectures, good programming practice, and a variety of static and dynamic assurance techniques. Similarly, the insertion and execution of malicious code such as Trojan horses, Web- and e-mail–based executables, and so on, need to be countered by sound architectures and operational practices that enforce isolation, confined execution, separation of concerns, and so on. Authentication can be made more trustworthy by incorporating demonstrably robust cryptography or possibly biometrically based authentication, and analyses that demonstrate the security of their embedding into systems, thus avoiding overreliance on porous perimeter defenses without additional security mechanisms. Real-time misuse detection and response can also increase assurance of trustworthiness, especially with respect to integrity violations, insider misuse, denial-of-service attacks, and other threats. Unfortunately, today's commercial systems tend to be rather limited; some of them could be supplanted by better system architectures and implementations. However, all of the problems and would-be solutions described in this Handbook are in need of some sort of assurance analysis.

77.5.2 Human Safety.

Historically, many seemingly easily avoidable accidents have occurred in healthcare, aviation, and other areas. Two examples are illustrative.

A computer-controlled therapeutic radiation machine called the Therac 25, manufactured by Atomic Energy of Canada Limited (AECL) and CGR MeV of France, was implicated in the deaths of six patients between 1985 and 1987 because of fundamental errors in management and systems engineering including inadequate code review, failure to analyze failure modes, and delays in responding to user complaints. This case is routinely used in systems engineering courses and papers as an example of the disastrous results of quality assurance failures in both the design and implementation phases of production. Testing a medical device such as the Therac 25 for an asynchronous time-dependent race condition (which resulted from poorly implementing the Therac 20 hardware interlock in software), and testing a heart pacemaker or defibrillator for electromagnetic interference would seem to be standard practice; yet these failure modes were not considered until after people had died.¹⁹ Once again, better attention to requirements, architecture, and good software engineering practice could have avoided the problems altogether. Furthermore, electromagnetic interference problems clearly must be addressed through explicit requirements, threat models, and testing. However, testing generally cannot detect many failure modes, and needs to be supplemented with other assurance techniques.

On May 26, 1991, Lauda Air Flight 004, a B767, left Bangkok, Thailand, for Vienna, Austria, after a planned stop in its flight plan from Hong Kong to Vienna. Fifteen minutes into the flight, the plane disintegrated after reverse thrusters, normally used on landing, deployed in error. The death toll included 213 passengers and 10 crew members.²⁰ Design reviews and testing could have detected the missing interlock that allowed the Lauda Air thrust reverser to be deployed during flight, although more careful and proactive requirements, design, and implementation might have prevented the problem from occurring at all.

77.5.3 Reliability, Availability, and Survivability.

The ACM Risks Forum archives are full of reliability-related failures such as these:

• Excessive clock drift in the Patriot missile system caused the intended targets to be outside of the expected computational frame.²¹ It is surprising that a demonstrably more robust clock algorithm was not used, irrespective of the requirements. Apparently the implementers tacitly assumed the launch platform would be moved frequently—perhaps once a day—to reduce the likelihood of its being targeted by the enemy. This design flaw could have been prevented through any of several assurance measures. For example, requirements could have explicitly demanded that the systems should operate correctly for several weeks, which would have necessitated a better clock mechanism. Alternatively, given the realization of an obviously unreliable clock, an operational fix could have required the launch platform computers to be rebooted daily.
• The Yorktown Aegis missile cruiser was disabled for 2.75 hours in September 1997 as a result of a divide-by-zero error in an application that crashed the operating system (Windows NT Smart Ship). This failure could have been prevented simply by observing the shrink-wrap disclaimers, which suggested that the software not be used in critical applications.²² However, making the engine controls dependent on a system with recognizably poor survivability was an architecturally unwise decision. The implications should have been understood during requirement specification, development, and system certification. (One obvious approach to greater assurance is that critical applications need trustworthy operating systems that adhere to established principles, and that can inherently protect themselves against their own flaws. For example, the 1965 Multics ring-structured architecture effectively prevented this kind of flaws.)
• The backup computer synchronization problem on the very first Columbia Shuttle launch was apparently known to the designers, but not to the launch support team. The two-day launch delay could have been avoided by a simple retry that would have skipped over the 1-in-64 transient failure mode—if the documentation had noted this problem, or if the launch crew and the developers had been in closer contact. However, the synchronization problem could also have been prevented altogether by a more robust design, or explicitly diagnosed by a variety of analysis tools.²³

Over the past five decades, there have been multiple cases of severe outages in which a single-point failure propagated widely, beginning with the 1965 Northeast power blackout. The 1980 ARPANET total network outage,²⁴ the 1990 AT&T long-distance collapse, and various electrical power outages—including the 2006 almost complete shutdown of Amtrak's New York–to–Washington corridor for an entire day—all share a common failure pattern: the propagation of a local failure into a widespread outage. In most of these cases, it had been widely believed by the developers and operators that single-point failures could not cause such propagating effects. At the least, some detailed modeling of the interactions among components could have demonstrated the possibility of mutual contamination and propagating outages, enabled further analysis to determine what conditions might trigger such effects, and inspired resulting architectural changes necessary to prevent such problems. Some combination of analysis techniques (such as fault-tree analysis and model checking) would presumably have detected specific failure modes that could trigger such outages. In the case of one of the extended AT&T System 7 telephone system outages, detailed laboratory simulation environments were normally invoked, but because a particular change affected only a few lines of code, it was not thought to be necessary to re-cycle through the entire analysis process.

77.5.4 Operational Assurances.

More careful analysis of human interfaces would be particularly valuable with respect to system administrators, both statically (e.g., leading to warnings before permitting certain questionable upgrades and configuration changes) and dynamically (in anticipating needs for emergency response). Furthermore, many of the concepts of this chapter could be applied to operational practices.

77.5.5 Sound User Interfaces.

Numerous serious accidents have resulted from poorly designed human interfaces, as well as from misguided internal system interfaces. Assurance techniques analyzing human interfaces could perhaps have avoided many of the cases noted in the ACM Risks Forum.

• Testing should have caught the sensor cross-wiring in maintenance of the British Midland 737, which caused the pilot to shut off the good engine rather than the failing one.²⁵
• Testing might have detected the missing interlock in the F-16 landing gear that allowed retraction on the ground.²⁶
• Sensible interface design might have proactively recognized the risk of using a GPS unit that after a battery change reset the coordinates of an intended Taliban target to the GPS unit itself, which killed the U.S. Special Forces launch crew. The designers of the weapons system might have anticipated the need for a warning to users.²⁷
• Heart-monitoring devices should have had standards that could have prevented a developer from using an ordinary electrical plug as a connector that could mistakenly be plugged into an electrical socket, thus electrocuting the patient; in addition, the certification process should have prevented that device from being used.²⁸
• A failure of a shuttle Discovery laser-beam experiment resulted from the target elevation over Mauna Kea being input in feet rather than miles (directing the laser up rather than down). That error could have been avoided by a human interface that explicitly insisted on an input in feet. Self-defining input types may seem like a low-assurance programming technique, but it would have avoided several past problems.²⁹

This illustrative enumeration suggests the need for total-system development approaches and evaluations, with assurance as an integral factor throughout development and operation. Among the wide variety of assurance techniques (some almost obvious, some more subtle), each potential system risk can benefit from the application of some subset of the total collection of approaches to increasing assurance. Establishment of sound requirements, sensible architectures, and good software development practice would undoubtedly have avoided many of the problems discussed throughout this chapter, and could be significantly aided by careful requirements analysis, model-based design, model checking, formal test-case generation, static analysis, and so on. However, there is no one size that fits all; the particular techniques must be used in various coherent combinations, according to the circumstances, the development challenges and risks, and the competence of the developers and analysts. In addition, the amount of resources and the effort to be devoted to assurance need to be commensurate with the overall long-term and enterprise-wide risks. Unfortunately, most risk assessments relating to how much effort to devote to assurance tend to be shortsighted and locally scoped.³⁰

77.6 ILLUSTRATIVE APPLICATION: COMPUTER-AIDED VOTING.

To illustrate many of these concepts related to assurance, consider the election process as a specific application area, with particular emphasis on computer-based voting systems. Elections represent a problem area that involves technology, legislation, standards, independent evaluations, operational procedures, and people in many different capacities. Furthermore, they are in a sense a paradigmatic example of a system that must provide assurances to voters that their votes are correctly treated throughout. The security, integrity, and privacy requirements need to span the entire process in an end-to-end manner. Each step in the process typically includes many vulnerabilities, and each step is a potential weak link.

77.6.1 Election Process.

The basic steps in the election process include:

• Registration of voters, ongoing throughout the year
• Preparation of ballot materials for a particular election
• Definition of ballot appearances for a particular election
• Preparation of voting machines for particular precincts
• Authentication of each voter for an election
• Authorization of each voter to cast his or her ballot
• Ballot-casting process
• Ballot-tabulation process
• Collecting local results into aggregate totals, including absentee ballots (see next list)
• Distributing the partial and final results as they emerge
• Verification and certification of the results
• Recounting in case of close or disputed elections

The basic steps associated with absentee ballots are these:

• Authentication and validity of absentee ballot requests
• Distribution of absentee ballots to voters
• Voter completion of absentee ballots and delivery as indicated
• Receipt, verification, and maintaining privacy of absentee ballots
• Counting of absentee ballots (different from mainstream ballots unless the same system is used for both)
• Merging the results into the mainstream aggregation (see list above)

There is no one-size-fits-all approach to elections. Some methods introduce additional complexities, such as preferential balloting that seeks to avoid runoff elections by iteratively discarding the candidate with the lowest totals and reranking accordingly. This method introduces its own set of problems.

Essentially, each step in the overall process has potential risks that must be addressed, detected, and remediated. The registration and voter certification processes need to be nondiscriminatory. Considerable care must be taken with voter registration databases.³¹ The ballot appearance needs to minimize any potential voter confusion. Ballot-preparation systems must be unbiased. Ballot-casting systems must have sufficient independent audit trails to ensure that votes cast are consistent with voter intent and that recounts are meaningful in case of irregularities. Casting and processing ballots needs independent assurances of integrity. Today, instead of strength in depth, there is primarily only weakness in depth. Every step is a potential weak link. Clearly, assurance efforts throughout the process, from beginning to end, can help reduce computer-related risks. For example, a serious attempt to cast the computer-related voting part of the election process into the formulation of the Common Criteria is given by Rebecca Mercuri.³² Significant risks are likely to be associated with failures to satisfy each of the next highly oversimplified illustrative sets of requirement topics, each of which would need much greater detail to be relevant to evaluation of a specific system and its intended uses.

77.6.2 Voting-Related Requirements.

An extremely simplified enumeration of some of the main requirements for computer-based elections follows; of course, the requirements would actually be rather complex when fully explicated. (Some of these requirements would also be applicable to entirely paper-based methods.)

System Requirements

• System integrity. Tamper-resistance and tamper-evidence; absence of self-modifying code and run-time changes, trusted bootload; physical security of machines.
• Data integrity and reliability. Correct recording and processing of votes throughout; uncompromised voter inputs; detection of accidentally and intentionally altered votes; predictable and reproducible correctness.
• System availability and survivability. Availability of election systems during required periods of operation; ability to withstand denial-of-service attacks and other types of failures.
• Voter anonymity and vote confidentiality. Absence of associations between votes and specific voters; system-based protections against vote selling and voter coercion to the extent possible. Note that absentee ballots and voting remotely over the Internet or by telephone are typically subject to fraud and misuse.
• Voter verifiability of individual votes. Independent means of ensuring that each vote is correctly handled throughout; immediate disabling or alternative treatment of noncompliant systems.
• Accountability. Uncircumventable and definitive audit trails recording all pertinent information, with tamper resistance and tamper evidence of the audit process, sufficient for independent recounts based on demonstrably true input data.
• Nonspoofability. Nonspoofable identification and authentication of all personnel with any access to relevant computer systems; for example, shared passwords are a poor choice, especially if never changed. (Fixed cryptographic keys are similarly risky.)
• Voter ease of use. Usability is a serious concern for all voters, but especially for those with physical or other disabilities, or with language problems, and for election officials, subject to security constraints; avoidance of confusing and misleading ballot layouts.

System-Related Requirements

• Chain of custody. Provenance of equipment, software, and data throughout the end-to-end process, including system development, trustworthy software distribution, ballot image preparation, system initialization, preventive maintenance, final testing, configuration changes, actual operation, and post-mortem analyses. For example, insertion of malware and run-time changes must be prevented, and insider misuse must be detected when it occurs.
• Testing, evaluation, and certification. This is a huge problem area, and needs much greater attention to increase its relevance and value. The Election Assistance Commission's Voluntary Voting System Guidelines³³ are inherently incomplete. Testing is inherently incomplete. Evaluation does not adequately cover or prevent subsequent system changes, and the results in the past have generally been proprietary. Certification is based on incomplete knowledge. The California Secretary of State's Top-to-Bottom Review of voting systems in 2007³⁴ has inspired a much greater awareness of the depth of the existing problems, and is being emulated elsewhere as well.
• System disclosability. Considerable benefits can result if all relevant software, hardware, firmware, custom circuitry, election results, error indications, irregularities, and associated documentation are open for inspection and analysis, at least by unbiased qualified experts. Documentation must be truthful, and should include development practices, operational procedures, and testing procedures, all unambiguously and consistently described. Documentation must also elaborate on assurance measures that have been applied to each of the above system aspects.

Operational Requirements

• Evaluation of the competence, integrity, and trustworthiness of system developers and vendors
• Evaluation of the competence and integrity of election administrators and operators.
• Procedural and operational enforcement of requirements. For example, protections against tampering, loss of privacy, vote selling, voter coercion, adverse physical access, disenfranchisement, bias, conflicts of interest, preferential treatment of certain voters or votes, and so on, throughout the election process, within and beyond the purview of any computer-based systems.
• Detailed oversight on the satisfaction or violation of all of the above requirements.

Many assurance issues are relevant here, including essential functional components in the small and their compositions in the large—encompassing, for example, voting-specific software, operating systems, cryptography, and auditing functionality. It should be possible to reason about properties of the compositions of subsystems, interdependencies among the subsystems, and ultimately critical properties of the entire system and the end-to-end process.

The problems of election integrity are rapidly gaining public awareness and governmental visibility. For a more detailed view of the challenges in designing high-assurance voting machines, together with a system architecture that succeeds in considerably reducing what must be trustworthy in order to achieve high-assurance voter confirmation without having to trust the ballot-casting component, see Sastry, Kohno, and Wagner's analysis of how to design voting machines for verification.³⁵ Another approach pursues the principle of starkly minimizing what must be trustworthy, as exemplified by the remarkably small pVote system described in Ka-Ping Yee's doctoral thesis.³⁶ Another approach to confronting the requirements of privacy and integrity is that of-voter-verified audit trails proposed by Rebecca Mercuri,³⁷ which has now been adopted into several commercial systems. In addition, several novel approaches proposed by Ronald Rivest, David Chaum, and others satisfy almost all of the main functional requirements and could conceptually be used entirely without the use of computer systems.³⁸

77.7 CONCLUSIONS.

Perhaps the most important conclusion of this chapter with respect to developing and operating trustworthy systems and networks is that assurance must become a more integral part of the development cycle, subsequent operational use, and long-term evolution of systems and networks, and that it must become better able to deal with entire systems in the large. To this end, building security, reliability, and—more broadly—trustworthiness and its assurance into the entire life cycle is extremely important. Furthermore, usability must be considered as an integral part of the development process.

Attaininging some nontrivial measures of assurance is seemingly a labor-intensive process, but then so is conventional software development, which includes testing, debugging, integration, red-teaming, maintenance, and evolution. Ideally, techniques for enhancing assurance should be incorporated into existing tools for software and hardware development. However, there are grave dangers in believing in the infallibility of technological solutions, procedures, automated tools, and assurance arguments that can be easily misused, compromised, or ignored. Once again, assurance ultimately depends on the integrity, intelligence, training, and experience of our system architects, designers, implementers, application system operators, system and network administrators, and—in many cases—end users themselves.

From a retrospective viewpoint, many of the approaches discussed in this chapter have their roots in the past, and indeed some of them could have been pursued more successfully years ago if there had been sufficient dedication. For example, stack overflows were essentially eliminated in Multics in 1965. Formal methodologies were available in the 1970s. Strongly typed safe programming languages have been developed. However, the disciplined use of such approaches in the development mainstream has been lacking.

Typically, there are enormous benefits from techniques that can be applied from the beginning of the development process, such as specifications for critical requirements, principled architectures, and formal, semiformal, or even informal design specifications. It is clearly preferable to prevent flaws early that would otherwise be detected only later on in the development process, or much worse, in operations. However, many flaws cannot be detected early—for example, those introduced during implementation, debugging, and maintenance—which can negate earlier assurance techniques. Consequently, assurance must be a distributed and temporal concept throughout development, maintenance, and operation, where constituent assurance techniques and isolated analyses must themselves be consistent, composable, and carefully coordinated. For example, careful documentation, disciplined development methodologies, coding standards, and thoughtful code inspection all have a place in helping increase assurance—as well as having secondary effects, such as reducing downstream remediation costs, and improving interoperability, system flexibility, and maintainability. However, there are of course risks in blindly believing in the results of specific assurance methods in isolation. Significantly greater emphasis needs to be devoted to developing and operating complex systems with appropriate assurance and to managing those activities. It follows that greater efforts need to be made to encompass the concepts of this chapter within mainstream university curricula and corporate training programs. It is always easy to speculate on what will happen in the future. It is also difficult to do so with any expectation of being accurate. Nevertheless, here are some glimpses of possible futures.

• In the absence of some significant cultural changes in education (at all ages), development practice, management awareness and training with respect to technical and nontechnical issues, legislation, enforced liability, social awareness and responsibility, altruism, and so on, it is likely that business as usual will continue more or less unabated, with the further prevalence of development failures, disasters, risks, unnecessary expenditures, and recriminations. Even a gradual evolution toward some improvements is not enough.
• In the presence of major cultural shifts, such as those outlined here, we might hope for some major improvements. Run-time errors could largely disappear, through carefully established parametric requirements, better system architectures, principled software engineering, use of sound programming languages, systematic uses of better analysis tools, fundamental attention to usability (especially with respect to security and system administration), and a pervasive commitment to higher assurance throughout. Predictable compositions utilizing libraries of trustworthy components could help significantly. The future is ready for great progress, but only with pervasive commitment to attaining it.

When it comes to providing meaningful assurance for security and other attributes of trustworthiness, the usual dictum applies: There are no easy answers. With respect to information security, trustworthiness, and assurance, nothing is absolutely certain (except perhaps this sentence). Some of the approaches discussed in this chapter may be mutually conflicting in some contexts. Some of them today are perceived (perhaps, in the long term, erroneously) to be impractical. Conventional wisdom often ignores assurance in the rush to market. Nevertheless, there are many critical applications that cry out for greater assurance, and we need to find ways to make it a pervasive, effective, and efficient part of the development process. If the critical needs of high-assurance systems cannot be used to drive system development, operation, and ongoing improvements more systemically, the overall development practice will continue to suffer seriously.

77.8 FURTHER READING

Neumann, P. G. “Reflections on Trustworthiness,” In M. Zelkowitz, ed., Advances in Computers, vol. 70, pp. 269–310, Academic Press imprint of Elsevier Science Publishers, 2007.

Neumann, P. G., and R. J. Feiertag. “PSOS Revisited,” Proceedings of the 19th Annual Computer Security Conference, pp. 208–216. Available: http://www.acsac.org/ and http://www.csl.sri.com/neumann/psos03.pdf.

77.9 NOTES