CHAPTER 10

Identifying and Mitigating Project Risk

I know what you’re thinking. “Did he fire six shots or only five?” Well, to tell you the truth, in all this excitement I kind of lost track myself. But being as this is a .44 Magnum, the most powerful handgun in the world, and would blow your head clean off, you’ve got to ask yourself a question: Do I feel lucky? Well, do you, punk?

—FROM THE MOTION PICTURE “DIRTY HARRY,” 1971

Investment risk is a primary consideration for any investor, and IT systems tend to have more than their fair share of problems, suggesting that they tend to be high-risk investments. A critical element of CPIC is the ability to gauge risk when an investment is still just a concept and to manage it if and when that investment is approved and funded. Risk information must be communicated via the business case, and major investments must have a detailed risk-management plan.

On the whole, project managers are optimistic. They believe that their experience, combined with decision-making latitude and control, is sufficient to recognize and mitigate most types of risk. Statistics, however, suggest that their optimism is just that and that most projects face a different reality. David Rubenstein, citing reports published by the Standish Group, which has done extensive research into software failures, notes that in 1994:¹

   Over 80 percent of projects are unsuccessful because they are over budget, late, missing a function, or a combination of those factors.

   Over 30 percent of software projects are so poorly executed that they are canceled before completion.

   Over 50 percent of projects will cost 189 percent of their original estimates.

   For every 100 projects starts, there are 94 restarts.

   Size matters—about half of projects under $1 million succeed, while less than 10 percent of projects over $10 million succeed.

Although more recent reports have noted substantial improvements, problems still persist, illustrating the need to think differently about project oversight and actions that can and should be taken to increase the likelihood of success.

Building Skyscrapers and Software

Organizations can build reliable skyscrapers, so why is it often so difficult for organizations to build software? A number of academics and practitioners have attempted to answer this question by comparing software development to long-standing engineering efforts. Analyzing the difference between building skyscrapers and software development highlights the unique difficulties of software engineering.

Organizations that design and build skyscrapers typically complete the projects on time and within budget, and the buildings usually work as expected—only rarely does a building collapse or otherwise fail. A number of factors explain why:

   Mankind has over 3,000 years of experience erecting buildings, and the principles and methods for designing and constructing them are well-known and regularly practiced.

   Building skyscrapers is based on detailed knowledge of material properties and other engineering characteristics, and is done using a reliable structure and process.

   The building’s requirements are identified, and the design is finalized and locked down early in the project. All parties recognize that any subsequent changes will add to the cost and schedule, so subsequent and last-minute design changes are uncommon.

   In building skyscrapers, there are commonly accepted development and problem-solving patterns. In the very rare event of a problem, the cause is investigated, diagnosed, and communicated so that the mistake won’t be repeated.

By contrast, organizations have been developing and implementing software applications and systems for only about 50 years, and the methods for doing do are still evolving. Many efforts have been undertaken over the years to improve the process, including structured programming, engineering, computer-aided design and development, and software engineering (which embraces a wide range of methodologies such as the Software Engineering Institute’s Capability Maturity Model for Software).

Recent efforts to improve the success of software design and development include the Rational Unified Process (RUP) and eXtreme Programming (XP). RUP is an iterative software development process framework rather than a single, monolithic prescriptive process. Software development is seen as an adaptable process where specific software development elements are selected by the development team based on the needs of the particular project. XP is a software engineering methodology where traditional software engineering practices are taken to extreme levels in order to make the process more agile and adaptable to customer needs.

Both of these approaches and others like them seek to instill a more structured and repeatable approach to software development. Some approaches are not widely practiced, while others continue to evolve and mature.

Another difference between constructing buildings and constructing software is the difficulty of defining clear, concise system requirements and reaching agreement about what the end product should look like and what it should do for its users and stakeholders. System requirements tend to evolve over time, a tendency that is commonly referred to as “scope creep.” The system development life cycle includes overlapping and sometimes unclear steps for conducting feasibility studies, collecting and finalizing requirements, and developing a system and application design.

Further, unlike a skyscraper, you cannot “see” software as it is being developed, and therefore it is difficult to evaluate progress or identify bottlenecks and errors. Schedule slippages are difficult to detect. The larger the project, the more difficult it is to manage, coordinate, and synchronize the many parts that must work together.

Finally, unlike skyscraper projects, forensic examinations of failed software projects are rarely done, and when they are done, the lessons learned do not seem to help avoid the same problems on other projects.

In essence, system development is a young discipline that continues to experience growing pains. As a result, it continues to represent a high-risk activity for most organizations. All stakeholders, including executive leadership, project owners, project managers, development team members, and users, need to fully appreciate the difficulty of systems, the tremendous odds of failure, and the need to identify and control risk to improve the likelihood of success. While the project manager and team bear most of the responsibility for managing risk, all stakeholders are likely to suffer unless risk is controlled and managed.

Given the importance of risk information and disclosure in the CPIC process, the remainder of this chapter presents an overview of the key requirements for risk management. The information provides a substantive basis for considering risk as part of the investment process.

Reasons for System Development Project Failures

Systems development is inherently risky because it involves an immature process. Even in those cases where more mature processes are applied, numerous other factors can cause significant problems. The most commonly cited risks are listed in Table 10-1.

Each of the factors in Table 10-1 is significant, but taken as a whole (and acknowledging that the list is incomplete), readers can appreciate the degree of risk involved in system development initiatives. Although the list is pretty much self-explanatory, a few of the items deserve specific attention.

Table 10-1    Risks of System Development

Early Project Risks

All the issues in Table 10-1 are significant, and the Clinger-Cohen Act was designed to address most of them. In the early stages of a project, however, three of the cited risks are most significant: lack of user involvement, lack of executive support, and poorly designed or inadequately defined project objectives. When these conditions are present, the risk increases exponentially, especially for a large project.

User involvement is essential because users are the consumers of system development initiatives. They must approach an initiative in the same way they would approach a sizeable purchase in their personal lives—buying a car, say—to ensure that agencies and taxpayers receive maximum return on investment. Users must articulate the problem that needs to be solved, discuss existing problems and limitations of automated approaches, and communicate their vision regarding how a new initiative might solve the problem. They must play a key role in defining and refining a project’s scope and functional requirements.

If requirements are clearly defined and if, once designed, additions and changes do not occur, system development efforts have a greater chance of success. Changes that have to be made later in the development process are generally more expensive and may have unintended consequences. Users must be engaged and accountable, serving as a critical member of the teams that prepare the business case and carry out the systems development.

One significant issue that hurts projects is lack of executive support. In the past, this lack of support was common for IT projects because executives were either too busy or lacked sufficient knowledge about how technology worked and how it was created. In more recent times, executives have become more technologically savvy, but sometimes still maintain an arms-length relationship to ensure that a risky initiative does not taint their career record.

Having full executive support is critical because technology initiatives often require strong leadership and political support to overcome barriers and obstacles that inhibit successful project implementation. Agency managers and staff pay more attention when executive leadership mandates that projects receive full support.

Executive support can be communicated in positive ways, such as recognition and rewards, or in a negative ways, such as fear and retribution. For example, the head of a large law enforcement agency communicated his commitment by explicitly setting expectations for the senior managers involved in a project. He wrote down the names of the key people he was going to hold accountable and told them that the fate of their careers depended on the success of the project: If the project was successful, they would be promoted and enjoy a fast path to further success, but if the project failed, they would be transferred to undesirable locations and not promoted. While this is an extreme and perhaps undesirable method of championing a project, it had the effect of getting everyone’s attention. Ultimately, the project was delivered within the anticipated timeframes, but it never really met user expectations.

The third critical factor in the early stages of a project is development of clear and comprehensive project objectives. This is essential for constraining a growing user wish list, tamping down scope creep, and preventing changes to requirements and expectations after the design has been finalized and approved. Developing systems requires compromise on cost, schedule, and features among the various project stakeholders. When expectations are not clear, developers might substitute their own preferences or make guesses regarding features and functionality—resulting in a system that might not accomplish what was originally promised or expected.

Design and Technology Risks

Many of the most intractable problems encountered by projects, and the ones that will stymie the most active ITIRBs, involve design flaws and poor technology choices. The technology field has evolved into specialties and, in some cases, talented individuals cross from one specialty to another without the appropriate background and experience for doing so. For example, a very successful network engineer might be promoted to project manager for an application development project without ever having managed the design of a complex database or the development of the associated application software, both of which are highly specialized disciplines. Any knowledge or experience gaps on the part of the project manager should be balanced by having an experienced team.

Many studies, including those published by GAO, emphasize poor management as the chief cause of project failure. Yet in his book Software Runaways, Robert Glass reviewed 16 failed projects and found that nearly half of them failed because of problems with technology.² The use of new technology and performance problems are highly correlated. Interestingly, Glass found that four of the failures were related to the use of the latest software engineering concepts.

Glass found that problems resulted from a lack of familiarity and experience with one or more technologies, poor choice of technology platform, and design flaws. In one agency, several dozen Java language programmers were hired, but they were idle for weeks because the agency could not get its Java compiler software to work properly. There have been numerous instances where agencies selected a particular technology platform only to be confronted with vendor bankruptcy, necessitating substantial rework.

Design flaws are common but difficult to detect until long after decisions have been made. For instance, one agency the authors are familiar with designed a comprehensive database using technical personnel who had not previously designed a database. They consulted textbooks and followed the academic guidance explicitly, separating repeating data elements into separate database tables. Once done, the design of a single entity included thousands of separate tables, making it massively complex and virtually impossible for users to see all of the information about a single entity at one time. The design eventually was scrapped, and a commercial off-the-shelf package was used in its place.

The use of new technology in itself creates risk, especially if the project team does not have the experience and expertise to evaluate the appropriateness of the technology to the business problem and to use it efficiently in the development process. In the mid-1980s the IRS rewrote its Assembler language tax-processing programs using the Common Business-Oriented Language (COBOL), causing the system to become so slow that it was unworkable. The Assembler language programmers found that they could not do things as efficiently using COBOL. Additionally, many of the development staff were essentially self-taught and did not have software engineering knowledge or expertise. Because IRS had no contingency plan, it struggled through the tax season and regrouped afterwards.³

Project Management Risks

The skill and experience of a project manager often determines whether a project is a success or failure. Project managers are responsible for many of the issues discussed on previous pages. They are also responsible for ensuring that project staff members have the right skills and development opportunities, that any knowledge gaps are identified and addressed, and that there is a sound project plan for guiding project activities and spending.

Key project management responsibilities are planning and designing work activities and developing cost and schedule estimates. Accurate estimation is not easy (as Chapter 11 discusses in detail). It requires extensive preliminary work, completion of a general system design, development of a work breakdown structure, and preparation of cost and schedule estimates for each unit of work (work package). If estimates are made prematurely, before sufficient design decisions have been made, or if the design changes after the estimates are made, project risk increases substantially.

Staffing issues are an often-overlooked source of project failure, even though the development team’s ability to perform is a critical aspect of success. The best management practices in the world cannot overcome lack of technical skills. The employment market for talented and experienced technology experts is tight, making it difficult to attract and retain talent. In the past ten years alone, employment in the U.S. computer and software industries has almost tripled. Workers who can create, apply, and use information technology are in demand not only in the United States but throughout the world. Despite its role in pioneering the use of technology, the United States is struggling to meet the demand for new information technology workers. A survey of midsize and large U.S. companies by the Information Technology Association of America (ITAA) concluded that there are about 190,000 unfilled IT jobs in the United States today.⁴

In another study, conducted by the accounting firm Coopers and Lybrand, nearly half the chief executive officers of America’s fastest growing companies reported that they had inadequate numbers of IT workers to staff their operations.⁵

The imbalance in supply and demand has also resulted in IT positions filled by less-experienced and less-qualified staff. Because software development is an opaque process, it is difficult to identify less talented individuals until it is too late, after they have done their damage and moved on.

Within the federal government, the staffing problem is compounded because of pressure from OMB to use commercial contractors for systems development. Many agencies no longer maintain a large internal staff of developers and instead use commercial contractors for systems development and support. Doing so has a number of advantages. It provides access to a wide talent pool that can be acquired incrementally as needed. It can also provide specialized expertise in emerging IT trends and technologies. However, contracting does not entirely resolve the talent issue because commercial companies are competing among themselves and the government for the same limited talent pool. The acquisition process for acquiring commercial services takes a long time, and once a contract is awarded agencies tend to develop a long-term relationship with that vendor. Because the agency is tied to the contractor, success then depends on the vendor’s ability to attract and keep talent.

Given all the things that can go wrong on a software development project, it is incumbent upon the project stakeholders to be vigilant and proactive about investment risk. The ITIRB and CPIC-SG should carefully scrutinize both in-process and proposed new investments to assess potential risk and ensure that the project team has carefully considered mitigation strategies. The following sections address risk management and mitigation approaches that should be incorporated in investment business cases and risk-management plans.

Risk Management

A formal and structured risk management approach has become recognized as the single most important factor in ensuring project success. Congress addressed the problem in the Clinger-Cohen Act, specifying that “the head of each executive agency shall design and implement in the executive agency a process for maximizing the value and assessing and managing the risks of the information technology acquisitions of the executive agency.”⁶

Building on the legislative mandate, OMB included risk management as a key element in the federal IT CPIC process. OMB Circular A-130 requires agencies to “prepare and update a strategy that identifies and mitigates risks associated with each information system.”⁷ Prior to 2006, the OMB Exhibit 300 business case required agencies to discuss the status of 19 specific risks for each investment. In 2006, Exhibit 300 was changed, and agencies were no longer required to address each individual risk but instead had to confirm the existence of an updated risk-management plan.

Within the federal sector, therefore, risk management is a legislated and regulatory requirement, connoting the importance that has been placed upon it. The CPIC process provides the logical place for meeting the requirements, and the ITIRB serves as its agency’s agent in terms of protecting the agency from unwarranted risk, especially in cases where most risks can be predicted and mitigated.

Defining Risk

Before discussing how to identify and control risk, it is useful to explore the various definitions of risk and to gain an understanding of just what needs to be identified and controlled. There are a number of definitions of risk. Some emphasize both the positive and negative eventualities; others focus just on what can go wrong. The Project Management Body of Knowledge (PMBOK^® Guide) defines risk as “an uncertain event or condition that if it occurs will have a negative or positive effect on one or more project outcomes.”⁸ Note that this definition includes events that could have a positive effect on the project. Paul Royer, author of Project Risk Management: A Proactive Approach, takes a narrower view of risk. He describes risk as “the potential events or circumstances that threaten the planned execution of the project.”⁹ Likewise, the Software Engineering Institute (SEI) focuses on the negative aspects of risk by adopting Webster’s definition: “The possibility of suffering loss.”¹⁰ While it is acceptable to include a broad definition that includes positive opportunities, most risk management activities focus on what can go wrong.

Defining Risk Management

Risk management also has a number of definitions. The SEI definition concisely sums up the key elements of risk management. SEI gives the following definition for risk management:

[A] software engineering practice with processes, methods, and tools for managing risks in a project. It provides a disciplined environment for proactive decision-making to: assess continuously what can go wrong (risks), determine what risks are important to deal with, and implement strategies to deal with those risks.¹¹ (Authors have added emphasis.)

This definition contains the most important aspects of risk management, including a formal process that creates a disciplined environment for addressing risks, including continuous assessment, prioritization, and action. Proactive risk management enables the organization to accomplish its mission by:

   Better securing the IT systems that store, process, or transmit organizational information

   Enabling management to make informed risk-management decisions to justify the expenditures that are part of an IT budget

   Assisting management in authorizing (or accrediting) the IT systems on the basis of the supporting documentation resulting from the performance of risk management

Risk-management planning begins early in the systems development life cycle. For example, the impact of risk on the project must be factored into the cost-benefit analysis, resulting in a risk-adjusted return on investment (ROI). The risk-adjusted life cycle costs consist of the total estimated investment cost for the investment’s entire useful life. The costs have been adjusted to account for any risks that have been identified. For example, if it is anticipated that a new technology may be challenging to implement, additional costs should be added to the project plan to accommodate working through those challenges. Schedules can also be risk-adjusted to accommodate additional time based on identified risks.

The risk-adjusted ROI is used to evaluate the proposed project during the CPIC selection phase. Risk-management planning continues into the planning, design, and development phases, during which the potential risks are identified and assessed. Mitigation strategies are then developed to address the risks. Risk monitoring continues throughout the project as new risks are identified and mitigation strategies assessed to determine if they are effective.

Several techniques, described below, can be used to identify, monitor, and control risks. Project managers should use them to prepare their risk-management plans, and the ITIRB and CPIC-SG should use them as part of the CPIC selection, control, and evaluation phases to ascertain how effectively risk will be managed, or has been managed, by the investment management and project management teams.

Identifying Risks

Numerous frameworks have been established to classify risks. OMB has set expectations that risk analysis will be comprehensive by providing 19 risk categories. For most investments, many of the categories will be classified as low risk, but using the OMB categories and any others that might be relevant provides a framework for considering various types of risks and identifying those that might surface during the project. OMB’s risk categories are:

   Schedule

   Initial cost

   Life cycle costs

   Technical obsolescence

   Feasibility

   System reliability

   Dependencies and interoperability between this investment and others

   Surety (asset protection) considerations

   Risk of creating a monopoly for future procurements

   Capability of agency to manage the investment

   Overall risk of investment failure

   Organizational and change management

   Business

   Data and information

   Technology

   Strategic

   Security

   Privacy

   Project resources

Agencies can add other risks to this list. Each risk should be carefully considered and rated as high, medium, or low risk on at least an annual basis. All medium and high risks should include a mitigation strategy, and they should be assessed in terms of risk-adjusting the investment budget and schedule. The OMB categories provide for consistent terminology and communication across all projects.

Depending on the status and nature of the investment, some risk categories will be more relevant than others. For example, if government personnel are developing the system, the risk of creating a monopoly for future procurements is low. A project that is implementing new technology would have a low risk of technical obsolescence, while a major modification or enhancement of a legacy system would have a higher risk of technical obsolescence.

Evaluating Risk

After risks are identified, they need to be assessed. Generally this means asking three questions:

   How severe is the risk?

   What is the probability of the risk event happening?

   What is the timeframe?

Most risk-management schemes apply some type of measuring scale to these three areas (e.g., high, medium, low, ratings of 1–5). As with risk categories, rating system definitions should be developed as part of the agency’s overall risk-management framework. In terms of severity, for example, a rating of “high” could mean that one or more of the project objectives will likely not be met without mitigation. The purpose of the evaluation process is to bring the most significant risks to management’s attention for action. Clearly, a severe risk with a high imminent probability would take precedence over other risks.

Processes can be developed to quantify the risks. For example, a matrix is often used to bring the risk factors together. Elaborate processes and software can also be used to help quantify and track risks. But these tools are useful only as long as staff does not get too entwined in an overcomplicated evaluation process.

Mitigation Strategies

The purpose of mitigation is to reduce the likelihood that a risk event will occur and, if a risk event does occur, to reduce its effect. Mitigation is the ultimate outcome of effective risk management. The following mitigation categories are commonly used to classify alternative actions:

   Accept the Risk: Do nothing and accept its consequences.

   Avoid the Risk: Take action to prevent the risk from occurring.

   Indemnify: Obtain insurance against the risk or otherwise transfer the risk to another party.

More elaborate schemes can be developed, but they will likely be refinements of these three overall courses of action. The most frequent strategy is to try to avoid the risk by taking some action that lessens the odds of its occurrence or lessen its impact if it does occur.

In evaluating specific mitigation options, project teams must identify mitigation alternatives, develop associated costs and benefits, and assess the likelihood of success. The mitigation strategies must be practical and workable within the project environment. Extreme risks to the project warrant the development of contingency plans that provide a course of action should the mitigation strategy fail.

After mitigation strategies are selected and approved by senior management, they must be implemented and monitored. Specific responsibilities should be assigned, as well as a timeframe for implementation. Most important, the success of the mitigation strategies should be evaluated and reassessed if they are not having the desired effect.

Risk Management throughout the Project Life Cycle

Risk management is a continuous and iterative process. The identification and evaluation of risks and implementation of mitigation strategies continue throughout the system planning, design, and development phases. Any assumptions also need to be monitored throughout the project and factored into the risk-assessment process. Formal reports on a project’s risk status should be provided to the investment owner, ITIRB, and other stakeholders.

Risk Management as a Paperwork Exercise

The overall risk management process outlined in this chapter represents standard industry practice for managing software development. It is used by certified IT project managers to identify and control risks. Some consulting firms have developed elaborate software tools to implement risk management, employing tracking systems and decision-support systems to analyze and quantify risk.

It is true that risk management activities were carried out on many projects that ultimately failed. Ostensibly, risks were identified, discussed, classified, and mitigated. All too often, however, risk management ends up being a paperwork exercise that does not achieve its intended result. This can happen because project managers, and especially software developers, tend to be eternal optimists and can ignore the possibility that things can go wrong. In other cases, risk management is a mere paperwork exercise because those involved lack experience, creativity, focus, or the courage to confront problems and challenges that often accompany system development initiatives.

Some risks are not addressed because the team lacks the experience or creativity to foresee what could go wrong. Because software development is an opaque process with many competing constraints, it is easy to ignore or overlook potential problems. In some cases, everyone on the team is occupied with narrowly focused project activities, and risk management is relegated as an ancillary duty and treated reactively rather than proactively.

One of the most common reasons that risks are not addressed is that no one wants to be the bearer of bad news. It often takes courage to point out potential problems, especially when they involve other team members or project leadership. As in ancient times, the bearer of bad news is the one punished. To counter organizational cultural norms and resistance, an ITIRB must communicate that projects teams have an obligation to identify and communicate risk exposures at the earliest possible time.

Another reason that risk-management has not always salvaged troubled projects is that the mitigation strategies are either ineffective or implemented too late to address the underlying cause of the problem. Finding and fixing significant problems as early as possible is a key to successful risk mitigation. Another key to success is ensuring that mitigation strategies target the underlying systemic or root cause of the identified problem or issue.

Risk Management as a Key Element of Portfolio Management

This chapter has dealt with risk as it pertains to individual investments. Within the overall CPIC process, however, risk management is also an important dynamic. Risk is a consideration during the selection process, when risks are anticipated in calculating the risk-adjusted ROI. Risk is also a consideration in portfolio management. When candidate investments are reviewed for funding, risk is part of the selection criteria. From an enterprise perspective, senior leadership typically desires to control the level of risk to which the agency is exposed at any one time. Funding numerous high-risk projects creates a highly volatile environment, and agencies have the capacity to manage only a given level of agency-wide risk at any one time. Also, in cases where high-risk projects are interdependent, problems can quickly compound.

The ITIRB must determine the agency’s level of risk tolerance and develop a risk profile that can be used, over time, for making selection decisions within those parameters. It must make informed decisions regarding risk when making critical resource-allocation decisions, and the risk-management process should be a critical component in ITIRB decision-making.

ENDNOTES

1. David Rubinstein, “Standish Group Report: There’s Less Development Chaos Today,” Software Development Times, March 1, 2007. Online at http://www.sdtimes.com/article/story-20070301-01.html (accessed December 2007).

2. Robert L. Glass, Software Runaways: Lessons Learned from Massive Software Project Failures, (Upper Saddle River, NJ: Prentice Hall Professional Books, 1998).

3. Tom Dworetzky, “Taxing times for the I.R.S.—the Internal Revenue Service’s new computer system results in delays,” Discover, May 1983. Online at http://findarticles.com/p/articles/mi_m1511/is_v7/ai_4227449/pg_4 (accessed December 2007).

4. Charlotte Thomas, “Hit the Hot Button for Jobs,” Graduating Engineer and Computer Careers Online. Online at http://www.graduatingengineer.com/futuredisc/compsci.html (accessed December 2007).

5. ASEE Prism, “America’s new deficit: The shortage of information technology workers,” ASEE Prism, February 1998. Online at http://findarticles.com/p/articles/mi_qa3797/is_199802/ai_n8804905 (December, 2007).

6. Clinger-Cohen Act, U.S. Public Law 104-208, September 30, 1996. Online at http://www.cio.gov/Documents/it_management_reform_act_Feb_1996.html (accessed December 2007).

7. Office of Management and Budget, Circular A-130 (Revised): Management of Federal Information Resources, November 28, 2000. Online at http://www.whitehouse.gov/omb/circulars/a130/a130trans4.html (accessed December 2007).

8. Project Management Institute, A Guide to the Project Management Body of Knowledge, Third Edition (Newtown Square, PA: Project Management Institute, Inc., 2004).

9. Paul S. Royer, Project Risk Management: A Proactive Approach (Vienna, VA: Management Concepts, Inc., 2001).

10. Merriam-Webster Online, Merriam-Webster Online Search. Online at http://www.merriam-webster.com (accessed December 2007).

11. Software Engineering Institute, Carnegie Mellon, Risk Management. Online at http://www.sei.cmu.edu/risk (accessed December 2007).