CHAPTER 9
Summary and Conclusion
Reaping the Rewards of Risk Management
Satirical cartoonist Walt Kelly was best known for his syndicated comic strip Pogo, which ran from the early 1940s until the early 1970s. The title character was Pogo Possum. A poster for the first Earth Day in 1970 features Pogo and his friend Porky Pine walking in a forest, the floor of which is covered with litter. This situation leads Pogo to proclaim, “We have met the enemy and he is us!” Like Pogo, projects are their own worst enemy when it comes to risk management.
When everything is going well, people ignore risk. Only after a significant bad event do people regain a fleeting appreciation for it. Between 2000 and 2020, a major event that reminds us of the dragons lurking in the tails of risk has occurred three times: the terrorist attack on September 11, 2001; the financial crisis of 2008; and the pandemic in 2020. All renewed a temporary interest in risk analysis. Enterprise risk management, which we mentioned in the last chapter, was a hot topic in the early 2000s. It gained some traction after the financial crisis of 2008. However, as of early 2020, the enthusiasm and appreciation for risk management had cooled again. Despite a growing emphasis on risk, in surveys between 2007 and 2017, only 20% to 25% of respondents said they conducted quantitative risk analysis on a regular basis.1
THE TWIN PROBLEMS OF COST AND SCHEDULE GROWTH
While extreme risks like black swans occur infrequently, projects regularly experience cost and schedule growth from the realization of risks. More than 70% of all types of projects, large and small, experience cost and schedule growth. Few projects spend less or take less time than planned. The average cost growth for most types of development projects exceeds 50%. More important, the severity of the increases is large. Approximately one in six projects experiences cost growth in excess of 100%. This problem has been around a long time and shows no signs of abating. The frequency of projects that more than double in cost is the biggest factor in project risk. It makes comprehending the magnitude of risk more difficult and makes traditional methods less meaningful in modeling risk.
Technical failures occur occasionally, but resource risks are more common. All the factors that impact technical performance also affect cost and schedule, but not all resource risks have an impact on technical performance. Much more can go wrong than can go right, so the risk of an event occurring that increases cost or schedule is much more likely than seizing an opportunity to reduce cost or schedule. A variety of sources of cost and schedule risk exist, both internal and external. A major internal reason is optimism. As a profession, project management tends to plan for the best-case scenario, which sets up cost and schedule growth from the outset. Significant mistakes are made during planning and execution. One planning issue is the use of immature technologies, which is a major driver of cost and schedule growth. Also, not every project manager or team is better than average. Errors are often made during execution, including unnecessary requirements changes. Mistakes are often made in the estimation of cost and schedule. These errors typically underestimate, leading to increases during execution.
The alignment of cost, schedule, and technical requirements is critical. Project decision makers do not always take the time to understand these relationships, but any misalignment among them leads to problems. Either cost and schedule will increase or the project will likely fail. An important source of external cost and schedule growth is the constantly changing world in which we live. This change is accelerating, as the rate of growth in technology is exponential. Projects take years to develop. Due to the rapid pace of change, leaders often face the issue of either updating the design multiple times during development or delivering a product that is already obsolete at completion. When projects produce multiple units, less production means higher cost per unit. Projects also have external dependencies. Any issues with these delay schedule and increase cost. These include the black swans we have mentioned, a recurring problem that we cannot predict and for which we cannot plan. Most projects experience the realization of multiple sources of risk. The problems of cost and schedule growth are even worse than they appear, as they do not account for technical failures, cancelled projects, or cuts in scope, features, or performance.
The frequency and severity of project cost and schedule growth provide compelling evidence of the need for better risk management. By ignoring the potential for large increases in cost and schedule, projects are unprepared to deal with these problems when they arise. Paradoxically, the attempt to achieve lower cost and shorter schedules leads to higher cost and longer schedules.
RISK BLIND: THE NEED TO APPRECIATE UNCERTAINTY
The first step in risk management is gaining an appreciation for uncertainty and risk. As a society, we tend to fixate on averages and not the variation around them. We are blind to risk. There is much more to the variation in the cost and schedule of projects than can be captured by an average. The use of averages results in a significant underestimation of risk. A lack of appreciation of risk led Nobel Prize–winning economists to drive a successful hedge fund into bankruptcy; bankrupted Orange County, California; and contributed to the 2008 financial crisis.
Risk and uncertainty should be assessed using rigorous quantitative methods. The use of qualitative methods and simple risk matrices is prevalent. However, they do not provide credible cost and schedule risk estimation. Fortunately, methods for analyzing risk the correct way are straightforward and can be implemented on any computer. Uncertainty is not a number but a shape. These shapes are called probability distributions. We have looked at four that are commonly used to model risk. The triangular distribution is a simple distribution that requires three inputs—a low value, a most likely value, and a high value. Triangle distributions are easy to use, especially when risk ranges are estimated with expert opinion. However, their practical use is limited due to their inability to model events outside the low and high bounds. The Gaussian, or normal, distribution is the most commonly encountered probability distribution. However, its use in modeling project risk is limited due to its lack of skew and its inability to model extreme variations in cost and schedule. The Pareto distribution can be used to model the wild risks exhibited in financial markets, motion picture profits, and natural disasters. Most projects will not experience these wild variations, as project managers can exert some control over an endeavor. Decision makers can develop and implement mitigation plans, cut scope, or even cancel an effort if things get too out of hand. A compromise between the mild variation of the Gaussian and the extreme swings of the Pareto is the lognormal distribution, which will be a good choice for modeling project risk in most situations.
Once risk is measured, it must be aggregated. This aggregation requires more than simple addition. Risks cannot be summed deterministically. A computer simulation, often called Monte Carlo simulation, is typically used to aggregate risk for both cost and schedule.
ISSUES IN RISK ANALYSIS
Even when risk is analyzed quantitatively, several issues with current practice hinder its effectiveness. One is that risk is significantly underestimated. This has several negative consequences. The belief is that, when an organization conducts multiple projects concurrently, low confidence levels can be aggregated to achieve a high confidence level for a portfolio. This is a myth. Current practice in risk analysis relies almost exclusively on the use of confidence levels to measure risk. Confidence levels provide useful information, but sole reliance upon them leads to logical inconsistencies in modeling risk because it ignores the right tail of cost and schedule risk distributions. Better risk measures should be used. Project managers do not conduct portfolio level risk analysis, which leads to the problem of trying to achieve too much with too few resources. Decision makers need to conduct portfolio level risk analysis, which provides guidance on setting cost reserves. Setting schedule reserves is not an automatic process and requires considering the timelines of activities for each specific endeavor.
Underestimation of Risk
The analysis of risk is adversely affected by optimism. The use of risk matrices also significantly underestimates cost and schedule risk. We saw that project cost risk analyses significantly underestimate risk. More recent trends, such as consideration of correlation and inclusion of model uncertainty in addition to the standard inclusion of variation in cost and schedule drivers, have helped make more recent analyses more credible. However, these estimates still have tended to under account for risk relative to the amount of cost and schedule growth experienced. Simply including correlation and model uncertainty is not a sure means of developing credible risk estimates. Incorporating tail dependency is an improvement that will help. More important, empirical cost growth data provides a means for understanding how much risk projects have experienced and can provide a means for comparing the amount of risk a project can expect to see versus what is predicted from project cost and schedule risk models. The empirical cost growth data has been shown to exhibit fat tails, although these tails are not as fat as found in some other industries, such as stock market prices and financial losses due to hurricanes. This is expected, based on differences between government projects and the denizens of Extremistan, such as stock markets and the whims of Mother Nature. For projects, leadership can have a significant influence in ameliorating cost and schedule growth through remedial measures, project rescoping, or outright cancellation. Schedule risk is not subject to fat tails, but as more can go wrong than go right, it should be modeled with an appropriate distribution, such as a lognormal.
We demonstrated the use of empirical cost growth data in calibrating project cost risk analyses to history. We compared this to a small sample measuring the performance of risk analyses. Empirical calibration measures historical growth well, unlike standard practice. This highlights that project risk often ignores major sources of uncertainty. Excluding some uncertainties from risk analysis is warranted, but decision makers need to realistically assess risk as well. Even if they don’t always have the budget to protect against some risks, they may be able to plan potential alternate courses of action, such as cutting scope. Understanding such risks may also help to enforce discipline and stress the impact of changing requirements or deviating from project plans.
The Mythical Portfolio Effect
When organizations conduct multiple projects, they tend to underfund individual projects in the hopes that the benefits of diversification will raise the confidence level of their entire organization to a high level. This is not realistic. Just as there is no such thing as a free lunch, there is also no such thing as a portfolio effect when funding to confidence levels. We also showed that the government funds to such low levels that the overall confidence level is lower than for individual programs. Organizations set themselves up for cost and schedule growth in such cases. The result is not only that lunch is not free, it is more expensive than the price quoted on the menu!
Confidence Level Funding Ignores the Dragons in the Tails
Current practice in quantitative risk management consists largely of setting reserves at fixed percentiles. This policy has much in common with the banking industry. However, it ignores the tails of the risk distributions, which is dangerous to the financial viability of the project. Funding to a percentile does not even provide a cushion for bad times. Exceeding a percentile funding level simply tells you that things are indeed bad. Confidence level funding will not cure the problem of cost and schedule growth. Empirical evidence suggests that an 80% confidence level funding policy will result, on average, in a significant amount of cost growth. Funding resources to percentiles is not a risk management policy but rather reflects a lack of thought in not implementing sophisticated and meaningful risk measures. Worst of all, percentile funding can result in a reverse portfolio effect, which means that funding an agency as a whole could be riskier than funding any single project! A better policy would be to use a risk measure such as expected shortfall, since it takes into account the right tail of the distribution. Such a policy will offer both a signal of a bad event (a specified confidence level is exceeded), as well as a cushion for the expected amount of money to guard against this event. Expected shortfall is a simple measure, represented by a single number just like percentile funding. It too can be easily explained to senior management and project managers, since it is simply the additional amount of money required to fund a project in case a specified confidence level is breached. It need not be significantly more expensive for the agency than current confidence level funding policies. Since it considers the full right tail of the distribution, a lower-level threshold such as the 50% confidence level could be chosen for the trigger. A reserves strategy should not consist of solely using confidence levels. Without a change in budgeting policy, projects will continue to incur cost increases and schedule delays.
Using Mountains to Produce Molehills
Portfolio management is critical to containing cost growth. A lack of planning in the portfolio management process leads to schedule delays and cost growth. Properly introducing new projects in a timely fashion can lead to getting more done in the long run. Other issues at the portfolio management level are a lack of risk analysis at the portfolio level and under accounting for risk at the project level. Relying upon a chimerical portfolio effect is not a substitute for calculating risk at the portfolio level. Accurately estimating risk is critical for project realism. Addressing these issues will go a long way toward addressing endemic cost growth in government projects and programs, which will result in accomplishing more.
Cost, schedule, and the phasing of cost over schedule are intrinsically linked. Changes in schedule for an established program result in cost growth. Reduction in annual funding also leads to schedule growth, which in turn leads to cost growth. Indeed, there is ample empirical evidence that schedule delays and funding constraints are strongly correlated with cost overruns.
Quantitative Risk Management: Guarding the Cookie Jar
Organizations that control multiple projects need to do quantitative portfolio management. This involves aggregating project cost risks to the portfolio level. Reserves need to be set. This must be carefully done to avoid running into the “money allocated is money spent” problem. A smart way to do this involves holding reserves at a variety of levels and providing incentives to keep costs down and schedules short. Schedule reserve setting requires careful deliberation to avoid Parkinson’s Law. This involves looking at the riskiest schedule activities and providing more time to those events likely to delay the entire schedule, such as those on the critical path.
Risk management is not just a set of individual activities. Rather, effective management of risk is a process. The steps we have described work in series. It begins with setting the scope and context. Risks are then identified and measured. Once risks are measured, they need to be aggregated to the project level. If the project is part of a larger portfolio, the risk of all projects must be combined to the portfolio level. Once aggregated, the risks need to be allocated to individual projects. Management of risks is both active and passive. The active part involves mitigating and eliminating critical risks a project manager can efficiently act on. The passive part involves setting reserves to guard against the risks the project either chooses to accept or cannot control. The process is iterative and must be repeated multiple times as a project progresses toward completion. The process encapsulates the discipline of enterprise risk management, which began in the 1990s and has been adopted to a greater or lesser degree in a variety of companies. However, surveys find that a small minority of companies conduct quantitative risk analysis. The discipline of risk management is relatively new with a significant amount of room for improvement.
Incentives can and should be used to achieve better outcomes. They have been successfully used in a variety of projects. Their widespread adoption will help projects achieve better value.
RECOMMENDATIONS AND LAST THOUGHTS
In summary, mature processes for quantitative risk measurement and risk management are sorely lacking. We have discussed several issues and ways these can be addressed. Optimism should be avoided. This can be accomplished with independent estimates or cross-checks. Also, quantitative risk analysis should be a standard practice and needs to be done well. Developing a cost or schedule risk analysis that vastly understates the amount of risk in a project is all too common. Care must be taken to avoid underestimating risk, particularly in the early stages of a project. Calibration of risk estimates to historical cost and schedule growth data should be done to ensure they are realistic.
S-curves provide useful information but are not good risk measures when used alone. Coherent measures of risk should be used, ones that take the full tail risk into account. These include expected shortfall and positive semi-deviation. Both are easy to calculate from Monte Carlo simulations.
Portfolio level risk analysis needs to be conducted, at least on an annual basis, before budgets are submitted. No shortcuts, such as the purported portfolio effect, can be used to circumvent a full-fledged quantitative portfolio risk analysis. Risk analysis also needs to be updated before a decision is made to start new projects, which can begin small but can soon take up a large amount of total budget at the peak of development.
A great deal of care needs to be taken to implement coherent risk measures and quantitative risk management. Careful setting of reserves for both cost and schedule must be done to ensure neither time nor money is wasted. Risk management should follow a consistent, iterative process that takes all necessary steps.
My purpose in writing this book is not just to be a critic. The flaws are evident, and pointing out the many shortcomings is the easy part. The harder task is to provide fixes. My intent is to provide helpful advice to enhance the odds of success. If decision makers want to perform better, spend less, and take less time, they need to pay more attention to risk management. Ignoring the possibility that something bad will occur ensures that projects are not prepared when these adverse events occur, and they will. Not all risks can be controlled. The impacts of these should be mitigated by setting aside reserves.
Two adages spring to mind. One is, “If you always do what you’ve always done, you’ll always get what you’ve always got.” Another is that the definition of insanity is doing the same thing over and over but expecting different results. The psychologist Daniel Kahneman discusses two primary systems of thought. One is intuitive and makes judgments quickly. The other is more thoughtful and deliberate. The intuitive system is good at some tasks, but it does a poor job at others, such as those that require careful planning.2 As a whole, project results indicate that the planning process uses the more intuitive system of thought. However, what is needed is the more deliberate one. Project managers need to stop the insanity and be serious about planning for risks in an intelligent way. The project management profession needs to incorporate the lessons in this book into their projects. Otherwise, the dismal track record with regard to risk will not improve. In the words of the philosopher George Santayana, “Those who cannot remember the past are condemned to repeat it.”3
We have mentioned that risk and opportunity are the two different faces of uncertainty. However, there is opportunity in risk. If you want to set yourself apart from the competition and provide exceptional support and leadership to projects, start incorporating proper risk management. Even small steps can help set you apart, as illustrated by the story of the two hikers in the woods who spot a brown bear headed their way. One frantically digs out a pair of running shoes from her pack. The other hiker tells her that she cannot outrun a bear. She responds that she does not have to outrun the bear, she only must outrun her hiking partner. Similarly, to succeed, you only have to be better than the competition. Risk management can make that difference.
This book has been written for a general project management audience. For readers interested in more technical details behind these concepts, visit https://www.iceaaonline.com/solvingprm for a collection of my technical conference papers and presentations. For interactive examples that provide a hands-on exploration of quantitative risk, visit my website https://christianbsmart.com.