CHAPTER 12
Conducting an Independent or Integrated Baseline Review
“There is no use trying,” said Alice, “one can’t believe impossible things.” “I dare say you haven’t had much practice,” said the Queen. “When I was your age, I always did it for half an hour a day. Why, sometimes I’ve believed as many as six impossible things before breakfast.”
—LEWIS CARROLL, ALICE’S ADVENTURES IN WONDERLAND, 1865
Earned-value management (EVM) is a useful methodology for setting expectations and communicating project progress to stakeholders to mitigate the adverse impact of problems that could delay project completion or cause unplanned expenses. One critical way to ensure that EVM tracks useful and accurate information is to validate that the project estimates, presented as the cost and schedule baselines, are accurate and fully represent the scope, breadth, and complexity of the project work.
The process of validating and verifying the accuracy of cost and schedule baselines is known as a baseline review. A baseline review can be classified as either independent or integrated, depending on the composition of the review team. Project team members do not participate in an independent baseline review (IBR)—the review team must be composed of individuals unassociated with the project. An integrated baseline review includes project team members.
The Office of Management and Budget requires federal agencies to conduct baseline reviews for developmental projects to establish standards to measure performance against.1 According to OMB, this requirement applies to both outsourced IT contracts and in-house projects. OMB Technical Memorandum M-05-23 states:
The IBR process enables project managers to effectively use the Performance Measurement Baseline to assess performance, and to better understand inherent risks. IBRs should be conducted using the Program Managers’ Guide to the Integrated Baseline Review Process2 and until the agency has begun conducting IBRs, independent assessments must be performed.3
An IBR is similar in approach to a typical audit of any particular activity. Once a decision is made to conduct an IBR, a review team is organized. The review team engages in a series of planning activities and conducts the review by collecting and analyzing a variety of data and perspectives. After conducting the review, the team prepares and presents a report outlining its experience and findings, suggesting any subsequent action.
IBRs are generally conducted for all major projects that involve substantive development or modernization work. They are also conducted for all major contract awards where contractors are playing a major role in system development. Project managers should ensure that EVM clauses are included in project contracts, that contracts require project compliance with the ANSI/EIA-748 standard described in Chapter 11, and that the contracts explicitly stipulate that IBRs are required prior to final submission of any major development cost and schedule baselines.
An IBR usually takes four to six weeks to complete, including the time spent planning the work, obtaining documents, conducting fieldwork, preparing the report, and communicating results. In some cases, IBR team members may be asked to conduct a follow-up assessment to ascertain whether IBR recommendations were implemented and whether the new baseline reflects the proposed changes.
This chapter explains the purpose of an IBR and how it is conducted. The methodology presented here should be adapted to the particular organization, system, and nature of the review at hand, but with the common goal of producing a thorough and accurate assessment of the project cost and schedule baseline estimates.
IBR Objectives
When deliberating about investment options, the ITIRB and CPIC support group (CPIC-SG) rely on cost estimates, schedules, and risk assessments to make investment decisions. The IBR team supports this decision-making by:
Assessing the quality, accuracy, and completeness of the investment baseline estimates
Identifying potential risks that might affect those estimates
Providing recommendations for revising and improving the estimates, as necessary
To achieve these objectives, qualified personnel conducting the IBR must base their opinions and conclusions on a comprehensive and thorough review of the process used to formulate the original estimates. To provide the appropriate level of assurance, the review should be conducted as objectively and independently as possible. The review must include a rigorous assessment of potential risks or variables that might significantly impact cost and schedule estimates. The review results should either concur with the standing estimates or suggest a plan of action for revising them, as well as the basis for making such recommendations.
Acting on behalf of the ITIRB, the CPIC-SG must ensure that plans for major development investments include a timetable and allotted resources for IBRs prior to submitting a business case to the ITIRB during the CPIC selection phase. The CPIC-SG should periodically develop and publish a schedule indicating which investments must conduct an IBR and include timing guidelines for doing so.
Planning and Preparing for the IBR
Investment managers are responsible for ensuring that an IBR is conducted to validate the project cost and schedule baseline estimates. Based on schedules provided by the CPIC-SG or through self-initiated action, an investment manager should recognize the need to conduct an IBR and initiate appropriate planning and preparatory activities, including:
Identifying personnel and resources needed to conduct the review
Ensuring that the IBR team is qualified and prepared to conduct the review
Delineating roles, responsibilities, and expectations
Preparing a specific IBR project plan that communicates the objectives, scope, methodology, and timetable
Announcing and initiating the review
Certain preparatory activities—notably identifying available participants and ensuring their training and preparedness—can delay the entire review. Selecting personnel who can commit to full participation during the planned timeframe is a critical success factor when the IBR commences.
Identifying IBR Personnel and Resources
The investment manager or other designated official responsible for an IBR must make several key decisions, including decisions about the IBR team’s level of independence, leadership, staff, and schedule.
An independent review requires a team of non-IT project members. The advantages of an independent baseline review are that 1) the team is perceived as more independent and objective and therefore its work is often perceived as more reliable; and 2) the IBR will have less impact on project resources focused on developing the system. The disadvantages are that 1) the IBR team members are likely to have less specific project knowledge and therefore are likely to spend more time collecting data to gain that knowledge; and 2) because they are less knowledgeable about the project, the IBR team members may reach preliminary conclusions that require refinement and additional discussions with the IT project team members. The investment manager must decide which type of review to conduct if the agency does not have a specific IBR policy.
Selecting an effective IBR team leader is critical to the success of the review. The team leader must have sufficient knowledge about how to conduct an IBR and apply earned-value management concepts, as well as an understanding of the agency and its programs in order to effectively assemble the IBR team, plan and prepare for the review, supervise the fieldwork, interact with the IT project team, and coordinate formulation and communication of the results. In addition to the team leader, the team should include a lead analyst, fieldwork specialists, and subject matter experts, as described in Table 12-1.
An experienced lead analyst is invaluable to the team leader because of his or her knowledge about the review process and his or her ability to ensure that any conclusion, finding, or recommendation is supported by sufficient and reliable evidence. As noted in Table 12-1, the lead analyst works with the IBR team leader to prepare a plan that includes the IBR objectives, scope, methodology, and schedule, leads the IBR team training and preparation activities, and oversees all fieldwork and reporting activities. The lead analyst is usually the primary author of the IBR report. The scope of this work requires the lead analyst to participate on the IBR full-time.
Role |
Characteristics and Responsibilities |
Team Leader |
• Understands how to conduct an IBR and has requisite experience • Oversees the entire IBR from initiation through completion • Directs team activities • Participates as an active team member • Serves as liaison with the IT project team and with the ITIRB and CPIC-SG • Ensures that findings are shared with the IT project team, refined based on that interaction, documented, and presented to stakeholders (investment owners, users, CPIC-SG, and the ITIRB) |
Lead Analyst |
• Understands auditing, operational analysis, and IBR methodologies and is able to ensure that review activities are conducted in accordance with GAO’s government auditing standards4 • Serves as the primary advisor to the team leader in terms of planning, fieldwork, and reporting activities • Works closely with the team leader to prepare the IBR plan • Prepares for and leads IBR team training and preparation activities • Leads fieldwork and reporting activities • Serves as primary author of the IBR report |
Fieldwork Specialists |
• Understand agency operations and/or IT practices and procedures • Participate in IBR team training and preparatory activities • Review the IBR plan developed by the lead analyst and approved by the team leader and clearly understand task assignments and expectations • Review and decompose the submitted IBR baseline information, identifying tasks, work packages, and other elements that merit further investigation and examination • Conduct fieldwork, including obtaining documentation (e.g., business case, detailed EVM project plan) and scheduling and conducting one-on-one interviews and facilitated group sessions to more fully understand investment requirements, plans, resources, team structures, potential risks, and other factors that might impact investment cost and schedule • Analyze fieldwork results and help formulate findings and recommendations, share them with the IT project team, and develop draft and final IBR reports |
• Are chosen for their knowledge of the business operation undergoing automation, the system requirements, agency automation practices and techniques, and/or some other specialized, relevant knowledge and expertise • Participate in IBR team training and preparatory activities • Work closely with fieldwork specialists to review the IBR plan and understand task assignments and expectations • Review the submitted IBR baseline information, including tasks, work packages, and other elements, and participate in fieldwork activities • Use knowledge and experience to validate, adjust, correct, and contribute to IBR fieldwork and conclusions as fieldwork specialists analyze fieldwork results and formulate findings and recommendations |
The remainder of the team is composed of fieldwork specialists and subject matter experts who may participate on the IBR full-time or in a part-time capacity, depending on their availability and roles and responsibilities. Both types of team members participate in IBR team training and preparatory activities, review and understand the IBR plan, and participate in fieldwork and reporting activities. Fieldwork specialists are typically familiar with agency operations or IT practices and procedures and are responsible for reviewing and decomposing the EVM cost and schedule details. With assistance from the subject matter experts, the fieldwork specialists identify tasks and activities that merit detailed review and attention and conduct data-gathering and analysis activities to evaluate the validity of the cost and schedule estimates.
Delineating Roles, Responsibilities, and Expectations
All IBR team members are responsible for ensuring an objective review of the investment cost and schedule baseline estimates. Though the specific tasks and activities may vary among team members, everyone must focus on the same outcome and provide constructive feedback regarding any necessary changes to the baseline information needed for EVM purposes.
The IBR team leader and lead analyst spend the most time on the review—it’s usually a full-time job—and are likely to conduct any follow-up activities after the review takes place. The fieldwork specialists and subject matter experts become involved more gradually; their work peaks during fieldwork and diminishes after the conclusions, findings, and recommendations are formulated and written in the final report. During this final IBR stage, fieldwork specialists and subject matter experts review drafts and participate in meetings and briefings, but they have more of a part-time role.
Preparing the IBR Project Plan
An IBR project plan includes the objectives, scope, methodology, and schedule for conducting the review activities. The objectives might be to assess the quality, accuracy, and completeness of the investment baseline estimates, to identify potential risks that might impact those estimates, and to make recommendations for revising and improving the estimates.
The scope sets boundaries and constraints regarding which parts of the IT project plan will be reviewed, what information will be collected, and who will be interviewed, as well as the review timeframe and any other factors that help delineate the nature and breadth of the review. The methodology includes a detailed, step-by-step approach for initiating, preparing for, and conducting the review. It looks similar to a typical project plan and includes specific tasks and the amount of time estimated to carry out those tasks, planned start and end dates, and individual assignments (either by position title or by name).
The methodology includes steps for obtaining and analyzing the IT project plan, including all supporting documentation used to formulate the investment baseline. It also includes steps for analyzing the work packages and for analyzing the work plan to make an initial assessment about the thoroughness, comprehensiveness, inclusiveness, and accuracy of the baseline estimates. The methodology also specifies any additional fieldwork, such as obtaining and reviewing additional documentation, interviewing the IT project manager, team members, system owners, users, and other knowledgeable persons, and collecting any other information about the investment. In addition, the methodology describes the steps that will be taken to report and communicate the IBR results.
By reviewing the methodology, team members and other stakeholders can understand what work will be done, who will do the work, when it will be done, and what the implications are for both the IBR team and others related to the IT project. A comprehensive IBR methodology is critical for ensuring that a thorough IBR is conducted efficiently and effectively.
Once the IBR team leader and lead analyst finalize the plan, they share it with the IT project team, investment owners, and users so they can understand what information will be needed and the timing and duration of interviews and other activities. This ensures that these participants are prepared to fulfill their important role in the IBR.
Training and Preparing the IBR Team
The IBR team members participate in training activities to ensure that each team member understands how to conduct the review and understands his or her role and responsibilities. In most cases, the IBR lead analyst provides the training with assistance from the IBR team leader. Depending on the team’s experience using EVM and conducting baseline reviews, training might be extensive or it might be brief.
The training includes a general briefing about the IBR and how it is conducted and a detailed briefing about the IBR project plan. The objective is to ensure that all participants understand what work will be done, when it will be done, and each team member’s specific roles and responsibilities during the review. If the investment baseline has been obtained, the training and preparation session should include a review and walk-through of the associated details. The IT project team may be invited to attend part or all of the training session, and they may be asked to present the investment baseline to IBR team members at this time.
Training documentation might include information about the investment sponsors and their programmatic functions, IT system development life cycle (SDLC) policies and procedures, investment business case and design documentation, preliminary interview questions, and areas of concern or interest to examine when the review commences. A draft announcement and request for additional documentation should also be included.
Announcing and Initiating the Review
To ensure that all involved parties and stakeholders are aware of the scheduled IBR, the IBR team leader prepares and disseminates an announcement letter describing the IBR objectives, timeframe, and approach. The letter need not include all the details of the IBR plan, but it should provide sufficient information to those who will be expected to provide information or participate in interviews. The announcement should include a date, time, and location for the kickoff meeting, and it should invite all participants and interested parties to attend to learn more about the IBR.
The IBR announcement letter should also request any additional documentation needed to review and analyze the baseline estimates. The documentation will help the IBR team understand the basis for the investment baseline estimates and to prepare questions and plan fieldwork activities necessary to validate those estimates. Requested documentation might include:
System requirements
Business case
Project documentation including the project plan, work breakdown structure, design documents, software specifications, project expense reports, and other relevant project documentation
Vendor contracts
Descriptions of the technical infrastructure and operating environment
Risk assessment plan
Security management plan
Conducting Fieldwork
After announcing the IBR and conducting the kickoff meeting fieldwork activities commence. Fieldwork activities include collecting and reviewing documentation, conducting interviews and focus group sessions, and identifying and examining other forms of evidence that can help the team ascertain the viability of the baseline estimates. The objective of fieldwork activities is to thoroughly and rigorously examine each estimate and 1) determine if all factors were taken into account when the estimate was made, 2) ensure that the basis for making the estimate, such as the number of labor hours and rates, were valid, and 3) ensure that the approach to performing the work necessary to complete the initiative is reasonable and accurate.
The sequence of fieldwork events typically progresses as follows:
1. Obtain and review requested documentation.
2. Use the project work breakdown structure and baseline estimates as the starting point to determine if all project steps and activities are accounted for, that those steps and activities have reasonably accurate cost and schedule estimates, and that the estimates have been risk adjusted and rated for significance and likelihood.
3. Identify any individuals to interview and particular issues to discuss, and identify any additional documentation needed for review.
4. Based on a review of the documentation and relevant knowledge of the agency, program, and systems development environment, prepare a list of potential risks that might not be accounted for by the project team in its baseline estimates.
5. Prepare interview questions and topics, schedule interviews and focus sessions and provide participants with the questions and topics in advance so they can adequately prepare.
6. Conduct the interviews and focus sessions, obtain responses to the questions, ask additional questions as they arise, and determine, based on the conducted interviews, if other individuals or groups should also be interviewed.
7. Document the results of the interviews and focus sessions.
8. Analyze all collected data and determine if the baseline cost and schedule estimates are reasonable or need to be revised. Document the basis and evidence for reaching such conclusions.
Examining Estimates
The IBR team should carefully examine every aspect of the investment to determine if all risks have been identified, if sufficient mitigation strategies are in place, and if the project budget and schedule sufficiently accommodate those risks. Chapter 10 discusses in detail how to identify and mitigate the types of risks that projects typically encounter.
To proceed systematically, the IBR team should follow a prescribed sequence of steps to analyze the baseline estimates. The following sequence is offered as a starting point, but it should be modified to fit the particular system under review and the nature of the IBR:
1. Discuss as a group the critical elements of the SDLC and ensure that every aspect is sufficiently represented in the work breakdown structure, including:
• Requirements analysis
• General system design
• Detailed system design and development specifications
• Infrastructure analysis, design, and workload implications
• Development activities
• Unit, system, integration, and production testing activities
• User, operational, and technical documentation
• Transitional software necessary to migrate data from older data structures to new ones
• Bridge software to exchange data with other systems
• Implementation and transition support
• Ongoing operational support
2. Ensure that requirements represent the full work scope and that investment owners and users do not plan to add or change the requirements until after the system is deployed.
3. Ensure that technical considerations, including development tools, development and test environments, and processing, storage, and communication capabilities will be ready and available when needed.
4. Determine if sufficient resources (e.g., personnel, software, hardware, funding, contracting support, facilities) and expertise are available to the project.
5. Assess the project management and staffing capabilities to ensure that trained and experienced personnel are available and committed for the full time reflected in the work breakdown structure.
There are myriad other questions and issues that can be probed as part of the IBR fieldwork activities. Table 12-2 presents other issues, questions, and factors that the IBR team might consider investigating.
The IBR team engages over a period of approximately two weeks in an intensive effort to collect and analyze data from a variety of sources. The effectiveness of the fieldwork effort depends on asking difficult questions, challenging the work design, labor estimates, cost estimates, risk assessments, and mitigation strategies, and it depends on identifying and probing variables that might cause the baseline estimates to be inaccurate.
Table 12-2 Issues to Probe During IBR Fieldwork
Issues |
Related Questions and Evaluation Factors |
Project Plan and Work Breakdown Structure Issues |
• Is there sufficient evidence that the project plan was derived from the requirements and that all necessary SDLC and other anticipated tasks and activities have been included in the work breakdown structure? • Do the project plan and associated design documents adequately address the use of technology and its maturity level and accommodate any need to conduct prototypes, learn new technology, or anticipate problems as a result of doing so? • Has the project team coordinated with the technology infrastructure and operations areas to assess the impact of the investment on computational, storage, and bandwidth operations capacity and resources? • Has an assessment of user readiness to participate in testing, acceptance, and deployment activities been made and have reasonable cost and schedule accommodations been made in the budget and schedule? |
Cost and Schedule Estimate Issues |
• Has the critical path for the work steps been determined and have any potential critical path problems or obstacles been identified? • Do the work steps follow a logical sequence? • Are the estimates for the work steps reasonably accurate? • Have issues that are likely to impede schedule progress been identified, such as contractual issues, user readiness and availability to test and implement the system, the learning curve necessary for the new technology, or dependency upon new hardware and/or network infrastructure? • Does the work breakdown structure have sufficient levels of detail needed to ensure that work steps were of manageable duration and complexity before the estimates were developed? • Have contracts been awarded and fees been defined with sufficient rigor to ensure that contractor costs are known and manageable? • Have the estimates been examined using techniques such as sensitivity analysis to determine what will happen to the project cost-benefit analysis model if the estimates are not accurate? |
Availability of Resources |
• Are sufficient people with the right knowledge, skills, and experience available to perform the tasks? • Are project roles and responsibilities clear and understood? • Are potential role conflicts between government personnel and contractors identified and have they been addressed? • Are technical resources available, including workstations, services, software licenses, bandwidth, compilers, other development tools, testing resources, and any other technical resources? |
• Is a sufficient supervision and oversight structure in place? • Are all team members aware and trained on earned-value management principles and are earned-value management system tools and procedures in place? • Are all team members familiar with the project plan and work breakdown structure? Have they had a chance to review and comment on it, and do they understand their role and responsibilities in terms of meeting the schedule and monitoring their expenditures? |
Compiling Fieldwork Results
Fieldwork results must be structured in a manner that enables a broad group of constituents and stakeholders, including the ITIRB and CPIC-SG, to easily understand what was done and identify risks and mitigation strategies. Table 12-3 provides a sample format for summarizing and categorizing the work breakdown structure tasks and communicating, for various types of risk, the potential for estimation errors.
Debriefing the IT Project Team, CPIC-SG, and ITIRB
The optimal approach for a baseline review is to conduct it independently to ensure that the results are objective. Doing so, however, increases the importance of communicating frequently with the IT project team to ensure that participants are informed throughout the process, to obtain their input and rationale for the baseline estimates, and to gauge their reaction to potential findings and recommendations.
Table 12-3 IBR Risk Assessment Table
After informal meetings have concluded and the IBR team has reached its final position regarding conclusions, findings, and recommendations, a formal briefing of the IT project team should take place before the results are presented to the CPIC-SG and ITIRB. Depending on the circumstances, the IBR team may also propose that the investment owners attend the briefing with the IT project team or schedule separate meetings, both held before debriefing CPIC-SG and the ITIRB. The briefings should include:
A presentation of the IBR objectives, scope, and methodology
An analysis of the project cost and schedule baseline estimates
Conclusions about the baseline estimates
Findings and recommendations
The IBR team should also provide a subsequent briefing to the CPIC-SG so the CPIC-SG can assess the results and determine if the situation warrants ITIRB attention and action. Although it is unlikely that fully debriefing the ITIRB will be necessary, the CPIC-SG might request that the IBR team leader attend an ITIRB meeting and be prepared to address any questions that arise, especially if the IT project team disputes the IBR findings and recommendations.
Ultimate decision-making authority resides with the ITIRB, which initially approved the project budget. If the ITIRB learns that the original baseline estimates are incorrect, it may decide to cancel the project, replace the project manager, or something else. If the IT project team disagrees with the IBR team’s findings and recommendations, it can make its case to the ITIRB. The ITIRB, however, makes final decisions based on the results of the IBR.
The IBR Report
The IBR results must be incorporated into a written report so all affected parties and stakeholders have access to the results, to incorporate evidence and supporting information, and to communicate recommendations clearly. An effective IBR report includes enough detail to enable the IT project team to implement the recommendations in a timely manner. The report should explain which estimate is being addressed, the team’s findings, conclusions, and the specific steps needed to increase the accuracy of the estimate.
Follow-Up Activities
As noted earlier, the CPIC-SG may request the IBR team leader and possibly the lead analyst to review the baseline revisions by the IT project team and assess the team’s response to the IBR recommendations and provide a brief follow-up correspondence.
In addition to the CPIC groups, the investment owner also has a vested interest in ensuring that the IBR results are acted upon and that a valid and reliable cost and schedule baseline estimate are prepared, since the investment owner will be most impacted by any budget overruns or schedule slippages. Investment owner representatives should actively participate in all briefings and all follow-up activities to ensure that any baseline recommendations are adequately addressed by the IT project team.
Revising the Baseline
After the IBR has concluded and the results are communicated, the IT project team shifts its focus to managing and carrying out system development activities. A key aspect of controlling the project is calculating progress against the baseline using EVM techniques to monitor progress, preferably on a weekly basis and especially during the early stages of the project.
As work progresses, the IT project manager must continue to monitor the project and its environment to identify new potential risks or newly intensified risks that might impact the budget or schedule. This proactive environmental assessment provides a counter-balance to the reactive nature of earned-value reporting, which highlights problems and issues after they occur. Risks might include unanticipated changes to requirements (scope creep), contractual problems, technical problems, personnel challenges, or a multitude of other factors.
When major issues arise that indicate that the project will encounter significant earned-value management variances, corrective actions—like changing the project baseline—must be taken. In some cases, the problems may be legitimate and significant (e.g., a new requirement created by Congressional action) and therefore justify changing the project baseline.
If the baseline change is substantial, the investment manager may elect to have the IBR team leader and lead analyst review the revisions before they are presented to the CPIC-SG and ITIRB for consideration. To have a revised baseline approved, especially when it involves a substantial budget increase or has a major schedule impact, the CPIC-SG must be informed so the ITIRB can be debriefed and can respond to the request.
An IBR can be a useful and important tool in managing the success and controlling the risk of software development projects. It can be particularly useful for large, complex projects where the risk is high and transparency is low.
As with any management or planning process, however, an IBR can turn into a mere paperwork exercise. If this occurs, the process has not only wasted everyone’s time and effort, but it has also failed to identify the key risks and other issues likely to arise during the project. Moreover, it will give management a false sense of security fully grasped only after the damage is done—when the project is behind schedule, over budget, and does not deliver the capability promised.
ENDNOTES
1. Office of Management and Budget, “Technical Memorandum M-05-23: Improving Information Technology (IT) Project Planning and Execution,” August 4, 2005. Online at http://www.whitehouse.gov/omb/memoranda/fy2005/m05-23.pdf (accessed December 2007).
2. Department of Defense, “Memorandum for Service Acquisition Executives: The Program Manager’s Guide to the Integrated Baseline Review Process,” June 4, 2003. Online at http://www.ndia.org/Content/NavigationMenu/Advocacy/Resources/PDFs30/program_managers_guide.pdf (accessed December 2007).
3. Office of Management and Budget, “Technical Memorandum M-05-23: Improving Information Technology (IT) Project Planning and Execution,” August 4, 2005. Online at http://www.whitehouse.gov/omb/memoranda/fy2005/m05-23.pdf (accessed December 2007).
4. Government Accountability Office, “Government Auditing Standards,” July 2007 Revision. Online at http://www.gao.gov/govaud/govaudhtml/index.html (accessed December 2007).