1.1.1 What Is EROM?

Enterprise risk and opportunity management (EROM) refers to the methods and processes used by organizations to manage risks and seize opportunities related to the achievement of their objectives. It is a means by which organizations identify and implement their strategic goals, objectives, and priorities, subject to imposed constraints, through a process of strategic planning, execution, and performance evaluation.

Quoting from a report by the Committee of Sponsoring Organizations (COSO) of the Treadway Commission (2004), “Enterprise risk management encompasses:

• “Aligning risk appetite and strategy—Management considers the entity's risk appetite in evaluating strategic alternatives, setting related objectives, and developing mechanisms to manage related risks.¹
• “Enhancing risk response decisions—Enterprise risk management provides the rigor to identify and select among alternative risk responses—risk avoidance, reduction, sharing, and acceptance.
• “Reducing operational surprises and losses—Entities gain enhanced capability to identify potential events and establish responses, reducing surprises and associated costs or losses.
• “Identifying and managing multiple and cross-enterprise risks—Every enterprise faces a myriad of risks affecting different parts of the organization, and enterprise risk management facilitates effective response to the interrelated impacts, and integrated responses to multiple risks.
• “Seizing opportunities—By considering a full range of potential events, management is positioned to identify and proactively realize opportunities.
• “Improving deployment of capital—Obtaining robust risk information allows management to effectively assess overall capital needs and enhance capital allocation.”

The overall objectives of EROM are to facilitate the successful development of the strategic plan, to promote an overall best approach for implementing the plan, and to evaluate performance with respect to the plan. The means for doing this is to seek an optimal balance between minimizing the potential for loss (risk) while maximizing the potential for gain (opportunity) with respect to the organization's overall mission. The focus on the overall mission is the reason for the “E” in “EROM.” It implies an integration of risk and opportunity management over all programs, projects, initiatives, and activities in the organization's portfolio. Achievement of an optimal balance implies the involvement of the decision maker(s) in setting maximum tolerable levels for risk, minimum desirable levels for opportunity, and the trade-offs between them.

1.1.2 Why Is EROM Important to TRIO Enterprises?

Organizations that perform pioneering technical work must continually assess whether their strategic objectives continue to be achievable as conditions evolve, whether the balance between the risks and the opportunities has changed with time so as to require a recalibration of the strategic plan or a reassessment of how it is being implemented, and whether the funding agencies have introduced new requirements or constraints that need to be addressed.

For example, NASA, in response to new directions advocated by the executive branch of the US government, announced its intentions in 2013 to embark on new space exploration missions that necessitate a change in philosophy from strict risk minimization to a balanced combination of risk control and opportunity exploitation. This direction was enunciated in the following statements made by NASA Administrator Charles Bolden in a letter addressed to all NASA employees (Bolden 2013):

This change in philosophy has infused not only NASA but also other TRIO enterprises. Because of it, there is a need to expand our thinking regarding enterprise risk management from one that is centered on reducing risks to one that includes recognizing, cultivating, and exploiting opportunities. EROM is a rational, structured approach toward reaching an optimal balance between minimizing the potential for loss (risk) while maximizing the potential for gain (opportunity).

Finally, EROM is important to government technical organizations because the July 2016 update of OMB Circular A-123 specifically requires that all federal agencies use enterprise risk management as an integral part of deriving, implementing, and managing internal controls.

1.1.3 What Kinds of Risk and Opportunity Are Considered within EROM for TRIO Enterprises?

EROM in general is concerned with the enterprise-wide management of strategic and performance risks, which for purposes of this book are characterized as follows:

• Strategic risk refers to shortfalls in the ability of an organization to adequately achieve the long-term goals of its stated mission. In part, strategic risk may be equated with the potential for an organization to fail in accomplishing one or more of its strategic objectives. Inferentially, it additionally includes the potential for an organization to fail to formulate its strategic objectives in a manner that best serves its overall mission.
• Performance risk refers to shortfalls in the ability of an organization to achieve its shorter-term performance plan. Performance risk in part concerns the potential for an organization to fail to accomplish one or more of the performance objectives in its performance plan. It additionally includes the potential for an organization to fail to formulate its performance objectives in a manner that best serves its strategic objectives.

Strategic and performance risks are considered to consist of the enterprise-wide aggregation of several categories of risk, including (for purposes of this book) program/project risks, institutional risks, requirement risks, and reputational risks. These risk categories may be defined as follows (COSO 2004; International Standards 2008; NASA 2008, 2016a):

• Program/project risk is the potential for performance shortfalls, which may be realized in the future, with respect to achieving explicitly established and stated program/project performance requirements. Performance shortfalls for programs/projects may be related to any or all of the following mission execution domains: safety, technical, cost, and schedule.
• Institutional risk concerns risks to infrastructure, information technology, resources, personnel, assets, processes, occupational safety, environmental management, or security. They affect capabilities and resources necessary for mission success, including institutional flexibility to respond to changing mission needs and compliance with external requirements such as government regulations.
• Requirement risk is the risk of not satisfying the requirements of the organization's stakeholders and regulators. Requirements to be satisfied may include environmental safety and health (ES&H) protection, protection against fraud and misconduct, equal opportunity and other labor requirements, and in the case of federal agencies, federal mandates directed at achieving specific goals in the areas of public education, international cooperation, and commercial partnerships.
• Reputational risk concerns risks that could jeopardize the viability of the organization, and includes risks to financial health, legal risks, and public confidence risks. The latter category includes the risk of a catastrophic accident or other high-profile loss attributable to mismanagement or malfeasance.

1.1.4 How Does EROM for Nonprofit and Government TRIO Enterprises Differ from EROM for Typical Commercial Enterprises?

The last 10 to 15 years has seen a steadily expanding development of processes and standards for conducting EROM within commercial enterprises, for example, COSO (2004) and ISO-31000 (2008). While these frameworks have undoubtedly provided impetus for the acceptance and practice of EROM, they have tended to emphasize monetary risks and opportunities as would be paramount for profit-making companies. EROM to this point has been used less widely for nonprofit or government TRIO enterprises. For EROM to be effective at such enterprises, it must focus on the more qualitative, multidimensional objectives and constraints that noncommercial TRIO enterprises are required to satisfy, including:

• Achievement of scientific and technical gains in the public interest, over both short-term and long-term horizons
• Exploration of new frontiers and knowledge development
• Partnerships with other nations, commercial enterprises, and academia
• Public education and outreach
• Objectives common to both commercial and nonprofit enterprises, including institutional development and maintenance, legal and reputational protection, and financial health
• Specific annual outcomes mandated by funding entities (e.g., in the case of federal agencies, Congress, and the White House)
• Outcomes specified by oversight bodies such as independent advisory groups and inspectors general
• Satisfaction of government requirements and policies such as, for federal agencies, those prescribed within GPRAMA (2011), OMB Circular A-11 (OMB 2016a), and OMB Circular A-123 (OMB [2004, 2016b]), among others²

In addition, these objectives must be met within financial, schedule, and political constraints that are subject to periodical change due to changing administrations and changing public priorities.

Thus, the EROM framework for TRIO enterprises may utilize ideas from COSO, ISO-31000, and standardized quality management systems where applicable, but also must include the capability of addressing strategic objectives that are fundamental to the mission of the organization and should build on its culture and history of performance management and risk management. Furthermore, it should adhere to the basic principles in its directives, requirements, and standards. These documents typically address roles and responsibilities pertaining to risk management and the functions to be addressed by risk-informed decision making (RIDM) and continuous risk management (CRM).

1.1.5 To What Extent Does EROM Work within the Existing Management Structure of a TRIO Enterprise?

For any well-established organization, the EROM approach is framed and structured to synchronize with and facilitate the philosophy and management processes that already exist within that organization. EROM does not fundamentally alter the existing management approach for setting strategic direction, goals, architectures, requirements, and policies, establishing metrics, setting mission and budget priorities, and approving major new initiatives, although it may result in adjustments to some of the processes. Rather, it generally supports the existing approach for overseeing and approving risk plans and mitigation strategies, reviewing progress, overseeing internal controls, identifying deficiencies, and reviewing corrective actions.

Over time, TRIO enterprises evolve a set of processes for establishing enterprise-level strategic objectives and desired outcomes while developing their core institutional and technical capabilities and tailoring their programmatic initiatives to support these objectives. In facilitating these processes and helping make them more effective, the EROM framework for TRIO enterprises should support decisions made within the strategic management, mission support management, and program management functions of the organization. Simultaneously, it should support existing high-level reviews and decision forums conducted within the organization, such as meetings of management councils, acquisition planning and procurement meetings, and portfolio performance review meetings.³

The EROM process facilitates management activities by providing some of the key data and insights needed to make informed decisions. These processes are guided by information obtained from both external and internal sources. The needed information includes knowledge and understanding of the constraints that are imposed by government and other sources, as well as recognition of the problems that occur during the execution of the strategic plan, the opportunities that present themselves, the risks from potential adverse events that have not yet occurred, and the leading indicators that portend emerging problems, opportunities, and risks.⁴

1.1.6 How Does EROM Facilitate Negotiations between a TRIO Enterprise and the Entities That Provide Funding and Governance?

Although strategic planning is performed within the enterprise that is responsible for executing the strategic plan, external stakeholders often mandate many of the strategic objectives that the executing enterprise must achieve. EROM has a role to play in informing external stakeholders and funding entities about the achievability of various strategic objective alternatives so that these stakeholders can make informed decisions about which objectives to mandate. EROM does this by determining the overall risk of not being able to meet each strategic objective, taking into account all the individual risks and opportunities that accompany the objective. While stakeholders like Congress, the White House, and nongovernment funding entities may have different views from the TRIO enterprise about what constitutes gain and what level of opportunity is significant, a majority can agree on whether the risk of not being able to achieve an objective is intolerably high so long as the case is laid out plainly and accurately. The justification of the case is the role that EROM plays. When a TRIO enterprise determines through EROM analysis that the aggregate risk of not being able to achieve an objective is steep and there are few opportunities for reducing it, it makes these findings known to all stakeholders to help discourage them from mandating unachievable objectives and from having unrealistic expectations.

1.1.7 Can Various Management Units within the Organization Separately Apply EROM as Though Each Were an Enterprise?

Although EROM is intended to apply to an autonomous, self-contained enterprise such as an agency, an institution, or a company, it can also be applied separately to management units within an enterprise so long as the objectives of each management unit are consistent with the objectives of the enterprise as a whole, and the cross-cutting risks and opportunities are handled consistently. For example, a typical TRIO enterprise management structure may consist of its administration and supporting offices providing its executive management, a set of program directorates providing its programmatic management, and a set of technical centers and facilities providing its institutional and technical management as well as program/project support. Each of the program directorates, technical centers, and facilities has its own top objectives and lower-level performance objectives, each with its own set of risks, opportunities, and associated indicators. Therefore, the EROM framework can be applied to each unit separately. However, the EROM processes applied for management units will not be successful unless there are both formal and informal communication channels to ensure that the top objectives of each program directorate, technical center, and facility support the strategic objectives developed at the executive level, and that the technical performance objectives of the technical centers and facilities support the program/project performance objectives of the program directorates. Such communication channels must also ensure that risks, opportunities, and associated indicators that cut across management units are identified and accounted for by all affected parties in a consistent manner.⁵

1.1.8 In What Areas Does EROM Facilitate Strategic Planning, Implementation, and Evaluation of Performance for TRIO Enterprises?

Following are examples of the planning, implementation, and evaluation processes that benefit from an EROM approach:⁶

• Developing the organization's strategic plans and performance management plans by selecting options that maximize the likelihood of successfully advancing the organization's fundamental mission. In the case of federal agencies, EROM provides traceable and documented evidence for justifying the selections of objectives in a manner that is consistent with the constraints placed by the government.
• Developing a portfolio of programs, projects, research initiatives, institutional assets, and other activities and resources by selecting alternatives that maximize the likelihood of successful achieving the strategic objectives. EROM uses a risk- and opportunity-informed decision making process to help the decision makers within the enterprise select the most viable portfolio.
• Promoting creative technologies and new processes and/or leveraging legacy systems for advancing the organization's mission in a manner that promotes a more optimal trade-off between risk and opportunity while working within the reality of a limited and sometimes shrinking budget.
• Allocating the organization's budgets, facilities, infrastructure, and human resources in a manner that promotes a more optimal balance between the probability of success and the cost of implementation. In concert with the organization's ongoing technology capabilities assessment processes, EROM identifies enterprise-level risks and opportunities that pertain to staffing requirements, the qualifications of the staff, test facility requirements, information technology needs, and other program/project support needs, thereby providing focus for institutional and mission support functions and initiatives.
• Tracking and controlling risks, opportunities, and leading indicators so as to facilitate evaluation of performance relative to the strategic and performance management plans. EROM provides traceable and documented evidence of how well the programs, projects, and other portfolio items are being implemented and the degree to which that implementation is satisfying the strategic and nearer-term objectives.
• Updating and amending the strategic and performance management plans at selected (usually different) intervals to reflect status changes and the emergence of new risks and opportunities.
• Complying with federal and other regulations on risk and internal controls, and in the case of federal agencies, producing the Statement of Assurance required by the Federal Managers Financial Integrity Act (FMFIA). For federal agencies, EROM also supports the requirements and guidelines contained in the GPRA Modernization Act (GPRAMA) and in OMB Circulars A-11 and A-123.
• Informing portfolio performance reviews (PPRs) and strategic assessment reviews (SARs).⁷ EROM interacts with PPRs and SARs by helping to identify risk and opportunity indicators that each program needs to track and internal controls that each program needs to manage; by informing these reviews about how the indicators and controls cross programmatic boundaries; by helping to provide a logical basis for self-assessing performance relative to the strategic plan; and by helping to generate results that are required by external entities, including self-assessment results and rankings.
• Enabling an agile response to pervasive new conditions, either positive or negative, that require immediate action. By treating risks and opportunities that cut across programs, projects, entities, and organizational units in a consistent and integrated fashion, EROM helps ensure that the means are in place to develop timely responses to newly developing cross-cutting issues that require an integrated response.
• Facilitating risk acceptance decisions at key decision points. Results obtained from EROM include an aggregation of risk and opportunity information from lower to higher levels, allowing decision makers to obtain insight into the overall level of concern or confidence attributable to the organization's chances of satisfying each of its top objectives.

The benefits that derive from using an EROM approach are particularly significant for complex missions that involve difficult choices between alternative pathways.

1.2.1 What Is Meant by Risk and Opportunity within the Context of EROM?

Within the context of EROM, we define risk and opportunity as follows:

• Risk is the possibility of future performance shortfalls with respect to achieving explicitly established and stated objectives at all organizational levels, including the organization's strategic objectives.
• Opportunity is the possibility of future performance improvements with respect to achieving the explicitly established objectives and accomplishing the mission of the organization.

Risks and opportunities are always possible occurrences that may take place in the future. Once a risk is realized, it becomes a problem and is no longer a risk. Once an opportunity is realized, it becomes a gain and is no longer an opportunity.

Although the realization of a risk is viewed as negative and the realization of an opportunity is viewed as positive, risk and opportunity are two sides of the same coin. We speak of “the risk of missing an opportunity” to emphasize that missing an opportunity is a form of risk. In the same way, we speak of “the opportunity of mitigating a risk” to emphasize the fact that mitigating a risk is a form of seizing an opportunity. Both risk and opportunity require an action to achieve the best possible outcome (i.e., mitigate a risk or seize an opportunity). The actions must occur within an acceptable time frame to be effective.

That said, the fundamental difference between a risk and an opportunity is that the action is intrinsic to the definition of an opportunity but extrinsic to the definition of a risk. The potential negative outcomes that are the basis for identifying a risk exist as concerns prior to any intervention, whereas the potential benefits of an opportunity that are the basis for identifying a circumstance as an opportunity only exist in the context of some action(s) that could be taken to realize those benefits.

In the present context, opportunity has two dimensions. The first applies to the potential to reduce the risk of not meeting one or more already-stated strategic goals or desired outcomes. For example, an emerging opportunity for an organization that has begun execution on a project to share a research and development task with a partner organization that has specialized expertise in that area might result in a reduction of the risk of the originating organization failing in that task. The event that leads to the possibility of a partnership (e.g., the partnering organization expressing a willingness to participate) is an opportunity because it offers the promise of leading to a positive outcome. (In contrast, a risk leads to the possibility of a negative, or unwanted, outcome.)

The second dimension applies to an opening for changing strategic objectives or desired outcomes to align them better with the TRIO enterprise's vision and mission. For example, the emergence of a new technology might open up possibilities for the originating organization to achieve strategic benefits that were not previously considered possible. The latter type of opportunity pertains to promoting accomplishment of the TRIO enterprise's mission through strategic re-planning, rather than reducing the risk of not meeting its existing strategic objectives.⁸

Risks and opportunities may both have a time frame associated with them, a window of opportunity, after which response to the risk or seizure of the opportunity is no longer possible. This is one reason that an enterprise must be agile.

Significant gains in advancement or progress may involve proactively searching for opportunities, such as putting resources into basic or applied research, with the expectation that on the whole these efforts will bear fruit and speed the rate of progress toward long-term goals. In the words of Francis Bacon (1612): “A wise man will make more opportunities than he finds.”

1.2.2 How Do We Differentiate between Risks and Opportunities during Strategic Planning versus during Plan Implementation and Performance Evaluation?

EROM is concerned with enterprise-wide risks and opportunities during strategic planning, during development of the TRIO enterprise's portfolio of programs, projects, initiatives, and other activities, and during evaluation of performance. Strategic planning often occurs when the functions to be performed have been conceived but the specifics of the system design, and even the system architecture, have not yet been decided on. In that case, the identification of risks and opportunities derives from historical experience, tempered with expert judgment, gained from missions that have preceded the present one but are in some ways similar to it. For example, in the case of space exploration, the identification of risks for a low-earth-orbit mission using some future, as-yet undefined system may, for preliminary purposes, be considered to be informed by the risks that were identified for the space shuttle. These are risks that may or may not remain applicable as the system design matures, but that the organization needs to be aware of in making strategic decisions.

Obviously, the state of definition of risks and opportunities for future missions without a specific system design will be less mature than for missions that have well-defined system designs. Correspondingly, the state of risk and opportunity definition during strategic planning will generally be less mature than during implementation and performance evaluation.

1.2.3 How Does EROM Help Achieve an Optimal Balance between Risk and Opportunity?

The concept of balancing risk against opportunity is illustrated schematically in Figure 1.1. As shown in the figure, the balance is a reflection of the decision maker's sense of the risk relative to his/her sense of the opportunity. In this context, sense of the risk is equivalent to one's tolerance for the risk as presently perceived, and sense of the opportunity is equivalent to one's appetite for the opportunity as currently perceived. Factors such as the availability of resources or assets, together with other fixed constraints, enter into the decision maker's sense of risk or opportunity.

The balance between tolerating risks and seizing opportunities is informed by guidance provided at the executive level, such as the NASA Administrator's comments cited in Section 1.1.2, which imply that the organization must manage risks and opportunities in a graded manner across its portfolio of activities. As shown in Figure 1.2, most organizations have stricter standards (low tolerance for risk) relative to preserving their core capabilities and human lives and safety, while at the same time having more lenient standards (tolerating higher risk) relative to accepting the possibility of losing hardware in the pursuit of pioneering or capability-expanding activities that create new opportunities to more effectively advance the organization's mission. This considered grading of risk tolerance during strategic planning and during execution of the plan sets the ground rules for strategic risk taking that is essential for progress and success over the long term. It creates areas where the organization learns rapidly, in part through acceptable setbacks, as well as promoting areas where the gains made through high-risk activities are consolidated and institutionalized into a more capable organization.⁹

There is a well-known tendency for such balances to be made based on psychological factors that are not always in the interest of making the optimum decision. A variety of treatises on risk aversion point out that when people are confronted with two choices where the balance between opportunity for success and risk of loss is neutral or even moderately favorable to the opportunity, they will tend to choose the path with lower risk. This aversion is related to the so-called Ellsberg paradox (Ellsberg 1961), which concerns people's choice between situations that exhibit different levels of certainty (they have ambiguity aversion). Use of EROM in a structured approach helps to counter risk aversion and ambiguity aversion by ensuring that strategic decisions are made more objectively.

The decision to pursue an opportunity in one area invariably involves exposure to risk in another area. For example, a major revision to a design may provide an opportunity to increase technical performance but simultaneously introduce risks to cost and schedule. EROM provides an objective means for determining the break even point between the opportunity and the risk. It does this by examining the degree to which the opportunity meets or exceeds the decision maker's minimum expectation for an opportunity to be worthwhile, and comparing it to the degree to which the concomitant risk meets or exceeds the decision maker's tolerance for risk. In other words, EROM makes an objective assessment of the likelihood and magnitude of benefit and the likelihood and magnitude of loss relative to each of the agency's strategic objectives, and the decision maker's stated risk tolerance and opportunity appetite determine whether the former justifies the latter.

Ultimately, the decision maker has the responsibility to define risk tolerance levels rather than simply accept a risk-averse stance.

1.2.4 What Is Meant by the Terms Risk Scenario, Opportunity Scenario, Cumulative Risk, and Cumulative Opportunity?

The EROM process identifies specific concerns that are perceived as presenting a risk to the ability to achieve one or more strategic objectives. Each concern implies a scenario of events that must happen in order for the risk to come true. Collectively, these individual scenarios comprise the cumulative, or aggregate, risk of not being able to achieve the objective.

It is common practice to use the term risk to denote both the individual concern, or scenario, and the cumulative likelihood of not meeting the objective. The differentiation between the two is provided by the context, but sometimes, this dual usage leads to confusion when the context is not clear. In such cases, we refer to the specific concerns as being risk scenarios and the effect on the strategic objective as being cumulative risk or aggregate risk. For example, the possibility of staffing shortages in a crucial technical area due to higher-than-expected retirements is a risk scenario, and the likelihood of not being able to complete the projects that are critical to a strategic objective or goal as a result of this and other risk scenarios is a cumulative risk.

Likewise, the EROM process identifies specific scenarios that, if they should occur, would lead to an opportunity to either increase the likelihood of achieving a strategic objective or open the possibility of defining a new objective that coincides with the TRIO enterprise's mission. Therefore, we sometimes use the term opportunity scenario to differentiate the individual context for opportunity from the cumulative context. For example, the possibility of a breakthrough in the development of a new technology, opening the possibility of taking a positive action to reap the benefit, is an opportunity scenario. The prospect of translating that development, along with other opportunistic developments and directed actions, into higher performance for strategically critical programs and projects is a cumulative opportunity.¹⁰

1.2.5 How Does EROM Incorporate Risk-Informed Decision Making and Continuous Risk Management within the Organization as a Whole and within Different Management Units?

EROM is operationalized within a TRIO enterprise through the introduction of risk- and opportunity-informed decision making and continuous risk and opportunity management into the organization's management processes. In both the program/project domain and the institutional/technical domain, they are denoted as risk-informed decision making (RIDM) and continuous risk management (CRM). The RIDM and CRM processes are documented, for example, in NASA (2011) and Alberts et al. (1996), and as shown in Figure 1.3, they are executed at each of the management levels of the organization.

For the TRIO enterprise as a whole, risk- and opportunity-informed decision making is applicable to strategic planning activities and the selection of the organization's portfolio of programs, projects, and other initiatives. It is similar to its counterpart for programs/projects, RIDM, but it is expanded to make opportunity a more major component of the decision-making process. It is used first to help executive management select from among various alternative sets of long-term strategic objectives and nearer-term programmatic objectives in formulating a strategic plan, subject to external constraints, that supports the mission of the TRIO enterprise. It is then used to help executive management select from among various alternative portfolios of programs, projects, institutional initiatives, and other major initiatives to support the achievement of the strategic objectives. Like the RIDM process that it is derived from, it is composed of the following three steps: (1) identification of alternatives, (2) analysis of alternatives, and (3) the selection of an alternative.

Continuous risk and opportunity management, for the TRIO enterprise as a whole, is applicable to implementation of the portfolio approved at the executive level and to evaluation of the organization's performance relative to the strategic objectives. The process of managing risks and opportunities on a continuing basis is similar to the CRM process exercised for programs/projects, except again for the expansion to make opportunity a more major component in the management process. Like its CRM counterpart, it consists of the following five basic actions: (1) identify, (2) analyze, (3) plan, (4) track, and (5) control. This five-step process is supported by robust communication and documentation.

In incorporating RIDM and CRM into EROM for different management units, the areas of emphasis tend to differ according to the responsibilities assigned to each unit. At the executive level, emphasis is on strategic objectives and meeting the overall goals of the TRIO enterprise. For management units within the programmatic level (e.g., program directorates), the emphasis shifts to programmatic objectives and meeting project milestones within established schedules and costs. For management units within the institutional/technical level (e.g., technical centers), there is an increased emphasis on the development and maintenance of the workforce, facilities, and support systems. While the areas of emphasis may differ, however, the general approach for incorporating RIDM and CRM into EROM is basically the same whether applied at the executive level, the programmatic level, or the institutional/technical level.

1.2.6 Is the Analysis in EROM Principally Qualitative or Quantitative?

EROM uses a mixture of qualitative and quantitative methods. On the one hand, quantitative models are used for assessing and predicting specific outcomes that are amenable to quantitative analysis (e.g., matters of budget and schedule). On the other hand, there is a greater reliance on qualitative methods for EROM than there is for program/project risk management. That is because EROM involves assessments of strategic goals and objectives that are largely subjective in their interpretation and for which there are no easily formed quantitative models (e.g., increase human knowledge; promote the development of groundbreaking new technology; etc.). To assess the status or potential for achieving such goals and objectives, EROM relies on risk and opportunity leading indicators,¹¹ which serve as surrogates for the identified risks and opportunities. Although the leading indicators are in themselves quantifiable, their relationship to the actual risks and opportunities is qualitative, and hence the EROM analysis itself is more qualitative than quantitative.

1.2.7 Can EROM Account for Unknown and Underappreciated (UU) Risks?

Unknown and underappreciated (UU) risks are risk scenarios that either have not been identified and are therefore unknown at the time of analysis, or have been correctly identified but for which the likelihood of occurrence and/or potential severity of harm or loss are underestimated. By definition, it is not possible to identify unknown scenarios before they are revealed, or to be aware that a known scenario is underappreciated before it has occurred. It is possible, however, to be aware of various types of indicators that can be correlated with the likelihood of unknown and underappreciated risks, based on experiences that have been reported in the literature. These indicators tend to be associated with organizational shortcomings, questionable managerial practices, and certain design approaches. As will be discussed shortly, EROM analyses are able to include these indicators in the assessment of whether UU risks are likely to be a large contributor to the overall risk of not achieving the organization's objectives.

Recent work reported in NASA (2015) and Benjamin et al. (2015) has demonstrated that for complex systems, the probability of loss from UU risks early in a program/project or during the initial stages of operation can be several times greater than the probability of loss from known risks, not only for space systems but also for other systems such as commercial nuclear and military. The presence of UU risks can therefore significantly affect the ability of an organization to achieve its strategic objectives.

In addition, sizable UU risks extend not only to safety concerns but also to concerns related to technical performance, cost, and schedule (NASA 2015; Benjamin et al. 2015). An understanding of the potential magnitude of UU risks in each area of concern, and the factors that are causing them to be of concern, is important for at least the following two reasons:

1. It helps inform external stakeholders about the achievability of various strategic objectives and portfolio alternatives so that these stakeholders can make informed decisions about how to allocate funding.
2. It helps identify ways for mitigating the design-related, organizational and programmatic causes of UU risks, thereby increasing the potential for achieving the agreed-upon strategic objectives.

It has not been common practice for UU risks to be considered as a part of an EROM analysis, but the approach described in this book goes beyond present practice by considering the organizational, programmatic, and design factors that can lead to UU risks. These factors, obtained largely from NASA (2015) and Benjamin et al. (2015), are treated as leading indicators of UU risk, and are included in the roll-up of leading indicators that is performed to estimate the aggregate risk of not being able to meet each strategic objective. The treatment of UU risks is itself qualitative, in keeping with the overall qualitative nature of EROM. The potential effects of UU risks are included both in the strategic planning, RIDM-based aspect of EROM, and in the performance evaluation, CRM-based aspect of EROM.¹²