9.1 Technical Capability Assessment (TCA)

TCA teams are empaneled to examine the alignment of the TRIO enterprise's technical capabilities with the enterprise's long-term strategic needs, the program directorates' near-term needs, and the technical centers' identity and values. The purpose of the TCA team is to establish a more efficient operating model for maintaining a minimum set of technical capabilities that meets current and future mission needs while accommodating portfolio changes that occur periodically.

As shown on the upper part of Figure 9.1, the TCA approach may be based on a three-dimensional model that includes technical capabilities as one dimension, organizational entities as a second dimension, and solutions as a third dimension. In this context, technical capabilities include all of the human resources (workforce and contractors) and physical assets (equipment and facilities) utilized to meet the TRIO enterprise's technical objectives. Solutions refer to the current and future portfolio content (programs, projects, systems, subsystems, activities, initiatives) that results from decomposition and implementation of the enterprise's objectives. Analytical models may be used to assist the TCA team in the right-sizing of the enterprise's workforce and physical assets.

There are two ways in which the EROM approach described in this book can interact synergistically with a typical TCA approach. One involves a transfer of information from TCA to EROM, wherein the TCA team provides the EROM team with an understanding of all the interfaces that exist between the technical centers' institutional initiatives, the technical centers' mission support activities, the program directorates' programs and projects, and the TRIO enterprise's strategic objectives. The other involves a transfer of information in the opposite direction, wherein EROM provides TCA with an assessment of how risks and opportunities contribute to the right-sizing of the workforce and other assets. A high-level schematic view of the mechanics of these two interactions is provided in Figures 9.1 and 9.2, respectively.

The EROM focus on incorporating a means for balancing risks and opportunities into the right-sizing process is unique to EROM. The planning for future workforce and other assets should include a reasonable assessment of the uncertainties reflected in risks and opportunities, and therefore, EROM should be an essential part of this TCA function.

9.2 Strategic Annual Review (SAR)

For federal agencies, the Strategic Annual Review (SAR) process is the TRIO enterprise's response to the GPRA Modernization Act requirement that each agency conduct an annual review of its performance goals and objectives. Based on its strategic plan, each agency assesses progress toward the accomplishment of its strategic objectives by considering the status of multiyear and annual performance objectives, performance indicators, risks and associated risk indicators, external factors, and other events that may have affected the outcomes or threaten to affect them in the future.

As stated in OMB Circular A-11, Section 270.10 (2014): “Progress toward achieving individual quantitative performance goals related to the strategic objective[s] is one important consideration, but alone is not representative of the scope, complexity, or external factors that can influence program results and outcomes toward which Federal agencies are working.” It goes on to state that agencies should consider, among other things:

• Whether desired changes have occurred in the ultimate outcomes the agency seeks to improve and whether these outcomes are directly measureable or must be assessed through proxies or other means of evaluation
• Evaluations, research studies, data and policy analysis or other assessments relevant to the strategic objective or the related programs
• Lessons learned from past efforts to continuously improve service delivery and resolve management challenges, especially in coordinating across organization components and with delivery partners
• Identification, assessment, and prioritization of probable risks that may impact program delivery or outcomes significantly in the coming year or two
• Budgetary, regulatory, or legislative constraints that may have an impact on progress

In addition, as stated in Section 270.11:

• “To support the identification, assessment and prioritization of probable risks that may impact program delivery or outcomes and are likely to impact the strategic objectives, agencies are encouraged to leverage any existing Enterprise Risk Management efforts when conducting strategic reviews.”

A reasonable interpretation of this last bullet is that OMB, consistent with the use of an EROM approach, is encouraging agencies to look not only at probable risks that may impact program delivery or outcomes significantly in the coming year or two, as stated in one of the above bullets, but also at longer-term risks that may impact the strategic objectives farther into the future. It would be fair to say, however, that at the present time, the development of a longer-term perspective is encouraged but not mandated.

The EROM templates in Chapters 4 through 7 contain information on individual risks and opportunities, as well as on key leading indicators, which can significantly facilitate the SAR evaluation of strategies, performance goals, risks, opportunities, and associated indicators. In particular, the templates provide reports both for the near-term performance objectives that are required by the GPRA Modernization Act and by OMB Circular A-11, and for the longer-term objectives that are important to a TRIO enterprise's strategic evaluations.

In addition to improving the evaluation of status at each level of the objectives hierarchy, the EROM roll-up templates offer the capability of providing insights into how the risks and opportunities at lower levels affect the likelihood of success in meeting the strategic objectives. Because the roll-up process accounts for the plethora of individual risk and opportunity scenarios that affect each strategic objective in a rational and transparent way, the information in the roll-up templates can lead to significantly greater assurance that the SAR evaluation of the success of each strategic objective is defensible.

Figures 9.3 and 9.4 schematically illustrate the manner in which the EROM templates provide information useful to the SAR process.

9.3 Portfolio Performance Review (PPR)

The portfolio performance review (PPR) is a recurring senior performance management review of programs, projects, and activities within the TRIO enterprise. Its intentions are to integrate TRIO enterprise-wide communication of performance metrics and analysis results, to highlight cross-cutting issues that impact performance and affect risk, and to enable senior management to quickly address issues. For federal agencies, the PPR meets requirements for quarterly progress reviews contained in GPRAMA and in OMB Circular A-11 Section 6.

For most TRIO enterprises, the PPR tends to concentrate on current issues much more than risks (potential future problems), and accordingly it tends to have a shorter-term focus than the TCA initiatives and the SAR reviews. According to Smalley (2013), “The BPR [NASA's equivalent of a PPR] is the culmination of all of the agency's regular business rhythm performance monitoring activities, providing ongoing performance assessment between key decision points.” The BPR/PPR “is ‘action-oriented’ to improve performance and inform agency decision authorities of issues needing attention.”

EROM tends to have a longer-term focus than the PPR, but there is also a component of it that deals with short-term performance objectives, as measured by success or failure in achieving annual performance goals (see Section 3.1.1). Therefore, the relevance of EROM to PPR as traditionally practiced is mainly directed at short-term performance. For TRIO enterprises where the PPR process has a longer-term, more strategic performance evaluation perspective, the interfaces between EROM and PPR have an accordingly larger scope.

Within the more tactical approach to performance evaluation, there are several areas where EROM and PPR can interface. The activities of the PPR process include information gathering, generally obtained both through questionnaires sent to all areas within the TRIO enterprise and through person-to-person interactions at the actual review meetings. These data can help populate the EROM templates dealing with risk, opportunity, and leading indicator identification and evaluation as well as the risk and opportunity roll-up process. Most importantly, information about the status of leading indicators and about existing margins within each mission execution domain (cost, schedule, technical, and safety) and their trends over time are valuable to the EROM process.

EROM can also provide useful information to the PPR process in the area of aggregating the assessment from program/project level to program directorate and technical center levels. As discussed throughout this book, the EROM process can be exercised not only for the TRIO enterprise as a whole but also for individual management units within the TRIO enterprise. The EROM roll-up and high-level display templates, when exercised at the program directorate and technical center levels, lead to rankings of the likelihood of success in achieving each top objective of each directorate and center. These rankings can provide useful information to the PPR in its attempt to provide an assessment aggregate roll-up for each directorate and center.

The manner in which the PPR process can provide information to the EROM process, and vice versa, is similar to the schematics of the TCA-EROM interface in Figure 9.1 and EROM-SAR interface in Figure 9.4.

References

1. Office of Management and Budget (OMB). 2014. OMB Circular A-11. “Preparation, Submission, and Execution of the Budget.” https://www.whitehouse.gov/sites/default/files/omb/assets/a11_current_year/a11_2014.pdf
2. Smalley, S. 2013. “Baseline Performance Review: NASA's Monthly Performance Update and Independent Assessment Forum,” Presentation to National Defense Industrial Association, NASA Office of Chief Engineer.