CHAPTER 3
CPIC Control and Evaluation Phases
Engineering is the art of organizing and directing men and controlling the forces and materials of nature for the benefit of the human race.
—HENRY G. STOTT, FORMER PRESIDENT OF THE AMERICAN INSTITUTE OF ELECTRICAL ENGINEERS
The CPIC planning and selection phases discussed in Chapter 2 represent the most intensive times of activity for the ITIRB because it must analyze a large volume of information and then make important decisions. But if the ITIRB does not also follow up throughout the year by monitoring investment performance, or if periodic analyses of investment viability are not conducted, performance problems may arise. For example, if an approved new development effort gets off track, thousands and possibly millions of dollars may be spent—and wasted—before the problems are uncovered and addressed. A robust control process helps identify the problems at an early stage and permits corrective actions to be implemented.
Effectively controlling and evaluating IT investments usually requires project management and evaluation tools and techniques that may be new to many agencies. Instituting more structured and disciplined processes requires a significant change in culture and necessitates the same level of effort to implement as other major management initiatives. For example, many agencies have been working to implement cost-accounting systems in order to determine what it costs to provide their services and benefits. This initiative requires that federal employees, for the first time, track and report how they spend their time on a daily basis. This change is so significant that it has met with enormous resistance and has created a sense of distrust among the workforce, even though the primary motive is to improve the agencies’ ability to report accurate information to Congress and to taxpayers.
Changes associated with the CPIC control and evaluation phases can evoke a similar response. For example, project teams must use EVM techniques, necessitating the development of a comprehensive work breakdown structure, associated baselines, and target cost and schedule estimates. The estimates are independently verified to ensure that they are accurate, and project teams must report on variances from the baseline values on a regular basis. Project managers who are used to operating independently could resent and resist pressures to estimate and monitor project progress more accurately.
Control and evaluation activities must therefore be structured and implemented in a way that is compatible with the agency’s culture but also ensures sufficient information is provided on a regular basis to inform the ITIRB about progress (and any need for possible intervention). In most cases, the ITIRB will not have the time or inclination to micromanage investments. Its role is to ensure appropriate executive-level involvement and participation by remaining informed of investment progress, and to ensure investments are ultimately meeting and achieving performance expectations.
CPIC Control Activities
Control activities focus on the performance and overall health of the IT portfolio. Areas of emphasis include the portfolio’s overall ability to meet the agency’s IT needs, strategic goals, and performance objectives, and the likelihood that individual investments will not exceed cost or schedule estimates. Control reviews provide an opportunity to examine the portfolio composition as well as collective and individual investment performance. Control reviews also provide an opportunity to realign the portfolio, if circumstances warrant due to changes in mission, statutory or business requirements, or other investment-related issues.
From a CPIC perspective, control equates to monitoring activities that assess investment performance. For example, during the control phase, the CPIC-SG is likely to require monthly reports on EVM results. The reports should indicate any investment cost and schedule variances. Any significant variance should cause the CPIC-SG to consult with the project team and, if warranted, to notify the ITIRB. The objective of the control phase is to identify and respond to problems at the earliest possible time, to remediate them, and to avert the possibility that small problems will turn into large ones as time passes.
Control activities include monitoring performance of in-use assets as well as those assets under development. But there is a problem with using outputs and outcomes to control investment performance: these types of measures are usually lagging measures that cannot be evaluated until an investment is deployed and is in use. Despite this shortcoming, however, a mix of EVM and other performance measures is essential for monitoring project performance.
Earned-Value Management as Control Tool
EVM (described in more detail in Chapter 11) is a useful control tool. It provides a straightforward analytical method for monitoring project cost and schedule variances. OMB requires that agencies take action when an investment has a cost or schedule variance in excess of 10 percent. Many agencies, however, use a 5 percent variance threshold for triggering action to understand the situation and take any necessary action to correct it.
Project team performance can be tracked using EVM techniques. Tracking serves as an effective measure of project team actions and encourages positive behaviors. The asset itself also has performance characteristics and traits that can be measured and monitored. In-use investments usually have operational metrics such as response time and failure rates. Most business cases also include asset performance measures that describe expectations in terms of how the investment will affect operational efficiency and program effectiveness. These might include improvements in response time and reduction in program error rates.
Under EVM, project managers are required to periodically report performance information, on a monthly or quarterly basis. Extremely large, risky projects or ones that are behind schedule or over budget should be required to report on more frequently. Performance categories include the following:
Cost: A comparison of baseline costs to actual costs to date; reports variance, explains the cause, and describes what actions the project team will take to mitigate any variance.
Schedule: A comparison of the baseline schedule to the actual schedule; reports variance, explains the cause, and describes what actions the project team will take to mitigate any variance. If the project manager believes that the original baseline was inaccurate, the reporting process can be used to propose a new cost and/or schedule baseline. All re-baseline efforts should involve the ITIRB because re-baseline changes represent a change of “contract” terms and have both budget and schedule implications. Any significant changes in cost should also trigger the recalculation of the alternatives and an associated cost-benefit analysis (CBA). This will not be overly time-consuming if the original alternatives are used to merely update the CBA computations (discussed in Chapter 9).
Investment Performance: An analysis of expectations for meeting performance targets, as presented in the investment business case. Development projects likely will not be able to report investment performance because they have yet to be implemented. Once operational elements have been deployed, investment performance in terms of outputs and outcomes can usually be evaluated.
Scope/Requirement Changes: This category includes the disclosure of any changes in the external environment, user or technical requirements, or other factors that could impact the scope and magnitude of the investment.
Technology Changes/Considerations: Some investments encounter technical challenges that that could significantly affect project success but were not foreseen during the initial planning and investment approval. Such challenges need to be disclosed during the normal performance reporting process. Examples include decisions by the agency to change its technical architecture or difficulties encountered in using a new technology. Such issues should be explained in terms of their impact on project cost, schedule, and performance.
Preparing for ITIRB Control Reviews
To prepare for control reviews, the CPIC-SG must collect performance information from all investment managers and use it to prepare a briefing package for the review meeting. The CPIC-SG may also be tasked with preparing a watch list of risky or troubled projects. As part of the preparation process, the CPIC-SG may prepare recommendations for consideration by the ITIRB, such as the following:
Continue as is: Approve continuation as long as the investment remains within acceptable cost and schedule variances, performance targets and milestones are being met, identified risks are managed, and the technology solution and investment scope remain viable.
Modify: Change the investment scope, budget, or timetable to increase the probability that the investment will meet its objectives; adjust to changes that require modifications to objectives, scope, or cost or schedule estimates.
Accelerate: Speed up the project to accommodate external or internal changes and needs; modify the work breakdown structure, the budget, and/or the schedule to accommodate the acceleration.
Decelerate: Slow down the project to permit the investment to deal with performance variances; adjust the work breakdown structure, budget, and/or schedule estimates to accommodate the deceleration.
Suspend: Temporarily stop work to analyze project context, requirements, and progress, and to address significant performance issues or changes in business needs.
Cancel: End the investment. This may be necessary because the business context has changed or the project has encountered severe problems.
ITIRB Control Reviews
Investments that have performance variance or other significant problems are typically included in an ITIRB briefing. Investment managers participate to describe the status of the project, explain the reasons for variances, and respond to questions. The ITIRB reviews CPIC-SG recommendations and makes final determinations regarding whether to take action, such as modifying the budget or milestones, adding resources, redesigning elements of the project, cutting scope, or, in extreme cases, suspending or ending project work.
Some investments may exceed performance expectations (e.g., by operating under budget or ahead of schedule, by providing greater-than-expected benefits). These investments provide ITIRB with the latitude to accelerate an investment’s funding or schedule, reallocate resources within the overall portfolio, or make some other type of adjustment.
The purpose of performance review meetings is to assess performance across the portfolio, so the focus should be broader than just development projects. In-use assets that have been operating for long periods should be periodically reviewed using operational analysis techniques, and the results should be included in periodic performance reviews. Similarly, the results of post-implementation reviews can be topics for performance review meetings.
To assist in monitoring performance between meetings, some agencies use investment scorecards with the classic green, yellow, and red colors representing project status and progress. Doing so provides a visual representation of the overall portfolio performance. A large number of red-status projects indicates performance issues and a need for ITIRB attention.
Investment-Level Control Responsibilities
The periodic ITIRB review is important, but the investment sponsors/owners and the project manager (PM) are directly responsible for monitoring performance on a day-to-day basis. Investment sponsors/owners should expect to receive a variety of project management and control artifacts, including the initial business case and project plan, requirements documentation, high-level and detailed designs, and regular project status reports that include EVM cost and schedule variance and highlight any anticipated issues or risks that might impact project performance. Investment owners/sponsors are ultimately responsible for investment success or failure, and they should ensure that effective control practices are exercised.
PMs also have a major degree of responsibility for project control, and they ultimately are accountable to the ITIRB for how they exercise that responsibility. When practical, PMs should be able to rely on automated project cost and schedule control systems to manage estimates, collect and report project performance, monitor changes in requirements, and track resource allocations.
PMs have the following responsibilities during the control phase:
Provide ongoing oversight to ensure that the project is meeting its approved baselines and realizing expected benefits, that project risks are effectively managed, and that the investment meets strategic business needs
Manage change to ensure that adjustments affecting project performance are communicated to the ITIRB and effectively managed
Monitor and manage risks that might impact project performance
The PM must continually monitor and manage investment scope, budget, schedule, and performance; identify, assess, and respond to investment risks; and ensure timely delivery and quality of IT products and services.
During scheduled progress reviews with the ITIRB, specific issues might arise that require development of a corrective action plan (CAP). A CAP includes specific tasks, dates, and responsibilities for addressing the identified problem. It also assesses how the corrective actions might impact other areas of the investment. It is in the project team’s best interests to resolve CAP issues quickly and thoroughly to avoid additional ITIRB scrutiny.
CPIC Evaluation Activities
The CPIC evaluation phase makes the heaviest use of targeted studies and analyses. Evaluation is an ongoing process and is a normal part of the management cycle. The purpose of the evaluation phase is to determine how various in-use assets are performing and how well the CPIC process is meeting its objectives.
Operational Analyses and Post-Implementation Reviews
Two methods of evaluation are operational analysis and the post-implementation review (PIR). An operational analysis compares planned and actual performance using a series of prescribed metrics to determine if an investment’s performance is adequate. If an investment is in steady state, the following issues need to be examined through periodic operational analysis:
Maintenance costs: If costs are escalating because the system frequently requires maintenance support, there might be a case for enhancing or modernizing it.
Nonperformance/usability: Some systems, after being operational for a long time, no longer meet user needs. In such cases, a recommendation for enhancement or modernization may be warranted.
Changes in project staffing: If a project experiences significant increases or decreases in personnel, the causes should be examined.
Changes in scope: When requirements change, necessitating substantive additional support, the investment might warrant ITIRB review and attention.
Technical/interface/compatibility issues: In some cases, systems are developed with technology or interfaces that don’t have the capability to interact with other systems. When this occurs, an ITIRB review might be necessary.
Scalability/capacity needs: Older technology did not provide for incremental expansion or diminishment. An evaluation can reveal if an investment needs to be increased or decreased, and if the means exist to do either.
Security upgrades/impact: With today’s emphasis on security, an evaluation of security capabilities and effectiveness is important.
Contractual issues: When systems are developed by vendors or use licensed software/hardware that significantly increases their cost, this can affect investment cost-benefit performance. Contractual factors should be evaluated.
While operational analyses are usually conducted on investments that have been in use for quite some time, PIRs are conducted soon after an investment has completed development and has been operating in the production environment for 6 to 12 months. A PIR analyzes how well the investment is meeting user expectations and performance parameters.
As part of the PIR, the review team conducts a lessons learned analysis to identify issues that were encountered and strategies that were used to deal with them. For investments that exceeded budget or schedule, an analysis is conducted to determine why the issues occurred and how well they were anticipated and identified during the CPIC process.
A PIR produces both short- and long-term benefits. Over the short term, it identifies ways to improve the functional value of an investment and to increase customer satisfaction. Over the long term, a PIR can lead to reduced project cost, reduced development time, increased project quality and performance, improved project delivery time, and improved IT project decision-making. A PIR also provides feedback regarding how well CPIC is working and identifies areas for improvement.
How Control and Evaluation Activities Relate to Planning and Selection Activities
These early chapters have focused on the basic elements of CPIC to describe the context within which IT investment management occurs. For agencies to ensure that their IT investments are providing a maximum return, several things must happen:
Users must be involved in the CPIC process and must have substantive control, via the ITIRB, over IT spending.
CPIC participants must acknowledge their responsibility for overseeing an entire portfolio of assets that are in different stages of their life cycles and that are providing different returns (some systems provide major benefit while others are only marginally beneficial).
ITIRB decision makers must seek to optimize portfolio performance by supporting investments that (1) have a high rate of return at a reasonable cost and (2) fit within the agency’s profile for risk tolerance.
Sponsors of new investments must accept responsibility for ensuring that sufficient decision-making information, including a comprehensive business case, is available to decision makers to justify the investment opportunity and gain support for funding approval.
CPIC participants must accept the fact that performance information is necessary for controlling and evaluating investment and project team behaviors, and that resources must be allocated to manage and report performance information. They must also understand their necessary involvement in analyzing and explaining the project and in taking action, where warranted, to improve performance.
Part I presented a comprehensive overview of the IT CPIC phases. It showed how the management principles for financial investment are applied to IT investments. When an agency institutes CPIC, it creates a distinct and unified group of investors (the ITIRB) and a community of investment representatives interested in attracting investment capital (the agency budget). Given that an agency usually does not have sufficient resources to match the demand for IT investments, CPIC creates ground rules for competition among advocates for competing investment opportunities, forces sound justification of investment business cases, and fosters innovative approaches for meeting business needs.
Part II now shifts emphasis from the CPIC phases to the methods for maximizing IT return on investment for the agency as a whole. The chapters in Part II examine why global methodologies such as enterprise architecture and portfolio analysis are essential for aligning IT investment with an agency’s strategic direction.