Given the complexity of the risks and opportunities that attend TRIO enterprises and the federal government's recent emphasis on applying EROM to the development, validation, and management of internal controls, independent evaluation of EROM processes and results is highly recommended. Such independent evaluations serve several purposes:
The updated version of OMB Circular A-123 (2016), in a subsection entitled: “Role of Auditors in Enterprise Risk Management,” states that: “Internal or external auditors conduct independent and objective audits, evaluations, and investigations of an Agency's programs and operations, which includes aspects of the internal control and risk management systems.” Independent evaluation is stated as having special value, as follows: “Management and external auditors might have different interpretations of risks based on their respective roles and responsibilities. The agency risk function should seek to coordinate their roles so that the independence and scope of the external auditor's role is preserved while ensuring the continuing flow of risk information to the risk management function.” In a later section, the updated Circular amplifies the importance of evaluating internal controls through the lens of ERM: “Agency managers must continuously monitor and improve the effectiveness of internal control associated with significant risks identified as part of their risk profile. This continuous monitoring, and other periodic evaluations, should provide the basis for the Agency Head's annual assessment of and report on internal control as required by the FMFIA.” Through these statements, the Circular endorses independent periodic evaluations to ensure the integrity of the EROM approach and the completeness and accuracy of its analyses as they relate to the selection and implementation of internal controls and the associated required annual assurance report.
The risk and internal control processes that the Department of Energy (DOE) uses are subject to independent evaluation through the financial statement audit conducted by DOE's external auditor and through normal quality assurance and peer review processes, according to the DOE FY 2014 guidance document on internal control evaluations (DOE 2014).
Also according to DOE, the determination of risk should drive not only the selection and placement of controls, but also the prioritization of controls testing. Controls designed for what would otherwise be intolerable risks should be tested more frequently than controls designed for marginal or tolerable risks. Example risks cited as being of concern to DOE in the context of internal controls (DOE 2014) are similar to those for other agencies. They fall within the following categories:
The United Kingdom's Institute of Internal Auditors (IIA, 2009) provides specific guidance on the desirable content of independent evaluations of ERM within an organization. According to IIA, audits of ERM practices should be performed to “provide objective assurance to the board [of directors of a company] on the effectiveness of risk management. Indeed, research has shown that board directors and internal auditors agree that the two most important ways that internal auditing provides value to the organization are in providing objective assurance that the major business risks are being managed appropriately and providing assurance that the risk management and internal control framework is operating effectively.” The IIA report divides ERM activities into three categories: (1) those that fall under core internal audit roles, (2) those that fall under legitimate internal audit roles with safeguards, and (3) those not subject to internal audit, and it defines the ERM activities within each category as follows:
ERM Activities Falling under Core Internal Audit Roles
ERM Activities Falling under Legitimate Internal Audit Roles with Safeguards
EROM Activities Not Subject to Internal Audit
“In the case of ERM,” according to the IIA paper, “internal auditing can provide consulting services so long as it has no role in actually managing risks—that is management's responsibility—and so long as senior management actively endorses and supports ERM.”
For an EROM approach based on the principles, recommendations, and templates provided in this book, an independent evaluation would need to be concerned with all the activities leading to the selection and implementation of risk mitigations, opportunity actions, and especially, internal controls informed by risk and opportunity drivers. Since there is a requirement for a statement of assurance regarding internal controls, the evaluation would also have to be concerned with whether the residual cumulative risks and opportunities after implementation of mitigations, actions, and controls are acceptable. Because these selections and decisions ultimately depend on the execution of all the processes discussed in the preceding chapters, the independent evaluation would have to be concerned with all of the following subjects:
Table 8.1 itemizes the queries that need to be addressed by the appraisal team for each evaluation category. Underneath each category, the template provides a list of queries and, for each query, results of the evaluation with respect to the subject of the query, recommendations for improvement (if any) in the treatment of the subject, and status of resolution if any is requested.
Table 8.1 Template for Evaluating EROM Process and Results
Item No. | Evaluation Item Description | Evaluation Result | Recommendation | Resolution Status |
EROM Team Structure | ||||
1 | Are the scope and tasks of the enterprise-wide EROM team and each of the subteams appropriately defined? | |||
2 | Do the enterprise-wide EROM team and each of the subteams have the proper depth and diversity of skills and experience to succeed in their tasks? | |||
3 | Are the communications between the enterprise-wide EROM team and each of the subteams regularly scheduled, sufficiently frequent, and effective? | |||
4 | Is there an enterprise-wide database of EROM information and is it sufficiently available to all participants, accounting for the need to protect sensitive and proprietary information where appropriate? | |||
5 | Does the top-to-bottom management of each participating entity actively and vocally support the EROM effort? | |||
Development of Objectives Hierarchy and Identification of Interfaces | ||||
6 | Have all important sources of information pertaining to the definition and intent of the organization's objectives been identified and properly interpreted? | |||
7 | Have all important organizational objectives been included in the hierarchy? | |||
8 | Have all important interfaces between the objectives been identified and accurately represented? | |||
9 | Has the rationale for identifying and interpreting interfaces between the objectives been clearly, completely, and accurately stated? | |||
Derivation of Risk Tolerances and Opportunity Appetites | ||||
10 | Have all significant stakeholders and decision makers been identified and queried to establish risk and opportunity parity statements for each top organizational objective? | |||
11 | Have the responses of the stakeholders and decision makers been correctly interpreted and accurately converted into risk and opportunity watch and response boundaries for each objective? | |||
12 | Has the rationale for establishing watch and response boundaries for each objective been clearly, completely, and accurately stated? | |||
Identification of Risk and Opportunity Scenarios | ||||
13 | Have all important sources of information pertaining to the organization's risks and opportunities been identified and correctly interpreted? | |||
14 | Have all important risk and opportunity scenarios been included in the EROM analysis, including those that affect program/project success, core competencies, and organizational health? | |||
15 | Have all significant risks that would be introduced by availing each identified opportunity been included in the EROM analysis? | |||
16 | Have all important interfaces between the risk and opportunity scenarios and the organization's objectives been identified and accurately represented? | |||
17 | Has the rationale for identifying, interpreting, and assigning risk and opportunity scenarios to objectives been clearly, completely, and accurately stated? | |||
18 | Have cross-cutting risk and opportunity scenarios been identified as such, and are they defined and handled consistently across the affected organizational units? | |||
19 | Are there additional opportunities (not currently considered) to establish new objectives that significantly promote the organization's mission? | |||
Identification of Risk and Opportunity Leading Indicators | ||||
20 | Have all important leading indicators for each known risk and opportunity scenario been identified and included for consideration? | |||
21 | Have the leading indicators that promote unknown and underappreciated (UU) risks been included for consideration? | |||
22 | Have the functional relationships between the leading indicators and the objectives they pertain to been identified and correctly interpreted? | |||
23 | Have cross-cutting risk and opportunity leading indicators been identified as such, and are they defined and handled consistently across the affected organizational units? | |||
Evaluation of Risk and Opportunity Leading Indicators | ||||
24 | Have correlations been established between the leading indicator values and the likelihood of success of each objective, and are these correlations transparent and verifiable? | |||
25 | Have watch and response trigger values been established for all the leading indicators that affect each objective, and are they consistent with the risk and opportunity watch and response boundary values? | |||
26 | Has the rationale for the leading indicator trigger values been clearly, completely, and accurately stated? | |||
27 | Have all important sources of information pertaining to the status and trends of the leading indicators been identified and correctly interpreted? | |||
28 | Have the status and trends of the leading indicators been accurately evaluated? | |||
29 | Has the rationale for the evaluation of the leading indicator status and trends been clearly, completely, and accurately stated? | |||
30 | Have cross-cutting leading indicators been evaluated consistently across the affected organizational units? | |||
Roll-Up of Risks and Opportunities | ||||
31 | Has there been a systematic roll-up of the risks and opportunities from the bottom to top level of the objectives hierarchy to determine aggregate risks and opportunities? | |||
32 | Have the roll-ups accounted for all identified significant leading indicators and all identified significant interfaces between objectives? | |||
33 | Have all important sources of information pertaining to the importance of each objective on other objectives and the mitigating effects of redundancies and workarounds been identified and correctly interpreted? | |||
34 | Have the risk and opportunity roll-ups accurately reflected all important interfaces, redundancies, and workarounds? | |||
35 | For commercial enterprises, are results from the quantitative and qualitative roll-ups of monetary risks and opportunities consistent with one another? | |||
36 | Has the rationale for the roll-ups been clearly, completely, and accurately stated? | |||
Identification and Evaluation of Risk and Opportunity Drivers | ||||
37 | Has the derivation of risk and opportunity drivers included consideration of hardware response, software response, human response, controls, assumptions, and organizational factors, singly and in combination, as opposed to just hardware and software responses? | |||
38 | Is each derived risk and opportunity driver responsible for a change in level of importance of the aggregate risk or opportunity of a top objective (e.g., a change from a green/tolerable risk to a yellow/marginal or red/intolerable risk)? | |||
39 | Do the identified risk and opportunity drivers accurately reflect the stated rationale in the risk and opportunity identification template, the leading indicator identification and evaluation template, the objectives interface template, and the risk and opportunity roll-up templates? | |||
40 | Does the risk and opportunity driver list comprise a complete set of drivers for each top objective? | |||
Identification of Risk Mitigations, Opportunity Actions, and Internal Controls | ||||
41 | Have all existing internal controls been identified and correctly characterized? | |||
42 | Have all significant flaws in the existing internal controls been identified? | |||
43 | Have alternative sets of risk mitigations and opportunity actions been suggested? | |||
44 | Do the suggested risk mitigations and opportunity actions address all the risk and opportunity drivers? | |||
45 | Have all significant assumptions in the assessment of risk mitigations and opportunity actions been identified and correctly characterized? | |||
46 | Have alternative sets of new internal controls and/or modifications to existing internal controls been identified? | |||
Preliminary Evaluation of Risk Mitigations, Opportunity Actions, and Internal Controls | ||||
47 | Is each suggested set of risk mitigations, opportunity actions, and internal controls practicable? | |||
48 | Do the suggested new/modified internal controls protect the viability of all significant assumptions and correct or obviate all significant current flaws? | |||
Optimization Analyses and Associated Implementation Planning | ||||
49 | Have sensitivity analyses or iterations been conducted on the risk and opportunity roll-ups using risk and opportunity driver results as a guide? | |||
50 | Has a near-optimal distribution of human, physical, and instructional assets been derived from these analyses? | |||
51 | Has a near-optimal selection of risk mitigations, opportunity actions, and internal controls been derived from these analyses? | |||
52 | Has a plan been prepared to implement the near-optimal distribution of human, physical, and instructional assets and the near-optimal set of risk mitigations, opportunity actions, and internal controls? | |||
Risk Acceptance Decision-Making Support | ||||
53 | Is the cumulative risk and opportunity for each objective acceptable at the present time based on the stakeholders' risk tolerance and opportunity appetite? | |||
54 | Is it possible to make the cumulative risk and opportunity even more acceptable over all objectives by introducing new risk mitigations, opportunity actions, and/or internal controls? | |||
55 | Have processes for monitoring all important leading indicators been identified and are they being implemented? | |||
56 | What is the recommendation for proceeding forward? |
3.145.51.233