We incorporate potential effects of uncertainty in our analysis to account for the fact that we don’t know what will happen in the future. Monte Carlo simulation is the method that allows us to include these considerations of uncertainty analytically.
Why and How Do We Represent Uncertainty?
Looking forward to future outcomes that are important to consider in planning exercises , we realize rather quickly that we rarely know the exact representative values of those outcomes. We don’t know exactly how much something will cost, or how long it will take to accomplish a set of tasks, or what the actual adoption rate will be for a given product. The list goes on. Just because we don’t know what the actual outcomes will be, however, doesn’t give us license to avoid considering them in our planning. On the other hand, when we consider all of the multiple ways in which the various uncertain outcomes can be combined, the number of calculations required becomes intractably large. Most people recognize this, and they try to overcome the limitations with the tools they use for analysis (i.e., spreadsheets), or through their knowledge of how to treat the uncertainty via workarounds, usually in at least two steps.
The first step many people take to handle the problem of uncertainty is to try estimating most likely cases for the appropriate assumptions in their planning, or to consider worst or best case scenarios in an effort to consider conservative positions. What they usually fail to consider are the numerous ways in which bias creeps into their estimates of most likely outcomes (indeed, they might not even be aware of the subtle ways it happens), or the number of improbable conditions that are required to justify the particular caveats that define best or worst case scenarios.
The second step people take attempts to account for the effect of ranges around the assumptions on the objective function. Here again, bias creeps in because the range of sensitivity chosen is not usually tied to a likely range of variation. Instead, people use arbitrary ranges of variation, like ±20%, across all their assumptions as a way to accommodate what they think is a significant range.
Of course, there are also those analysts with a masochistic bent who attempt a brute force approach of testing dozens of scenarios sequentially, convincing themselves they are gaining a comprehensive and detailed understanding of their problem. All they are really doing is getting mired in analysis paralysis. The temptation to test “just one more scenario” never ends. The worst case of this mindset I have ever observed was a client who had produced 216 scenarios of a capital expansion plan over the course of two years, and they were no closer to making meaningful decisions than the day they began. Such wasted time only leads to value attrition and missed opportunity.
The net result of the first two steps is that people accommodate uncertainty with biased rationalizations, falsely communicating an unwarranted degree of precision by the single point numbers chosen in their analysis. Then they “stress test” their analysis with arbitrarily chosen relative ranges on assumptions. Bias, false precision, and arbitrariness don’t seem like the basis for clear and valid analysis to me.
The answer to this problem is actually similar to the brute force approach ; that is, we consider a large number of scenarios across the range of likely outcomes of the assumptions. However, instead of performing our calculations sequentially and only on the scenario combinations we tend to think matter most, we perform the calculations in parallel and according to the coherent and consistent rules of probability. This means that we must get used to thinking in ranges of outcomes (not single points) and corresponding degrees of likelihood across each assumption’s range. In short, we should treat uncertain assumptions as probability distributions. That’s the notional side of the effort. The functional side incorporates a process for conducting the calculations called Monte Carlo simulation .
Peter MacNamee and John Celona, Decision Analysis for the Professional (San Jose, CA: SmartOrg, 2007).
Gregory S. Parnell, Handbook of Decision Analysis (Hoboken, NJ: Wiley, 2013).
Patrick Leach, Why Can't You Just Give Me the Number?: An Executive's Guide to Using Probabilistic Thinking to Manage Risk and Make Better Decisions (Sugar Land, TX: Probabilistic Publishing, 2014).
What Is Monte Carlo Simulation?
Before I delve into the technical details of Monte Carlo simulation , I provide a little analogy to help clarify the essential mechanism by which it operates. Imagine a BB gun (the Red Ryder Carbine Action 200-shot Range Model air rifle, to be exact) mounted to a slider on a vertical rail with a fixed height. The BB gun always points horizontally at a 90° angle to the vertical rail. When it fires a BB, the trajectory of the BB is perfectly flat until it hits a special obstacle that has been erected at a fixed distance from the vertical rail. The obstacle is like a wall, but it leans over at an angle such that its vertical height is exactly the same as the vertical rail that holds the BB gun, and it’s shaped a bit like a French curve.
Monte Carlo simulation is like a gremlin shooting BBs at a wall from a random vertical postion. The wall records the horizontal position of where it was struck by dropping the BB into a cup underneath.
The process describing how our gremlin randomly shoots BBs at a wall is almost exactly how Monte Carlo simulation of a random variable works, except that instead of BBs being fired from a random vertical position, a computer selects random numbers strictly in the interval from 0 to 1. Each of these random numbers gets mapped to another value with a function that can produce numbers in any other arbitrary domain. For example, suppose we want to re-create a possible sample distribution of adult humans in a situation in which their heights are of interest to us. If we had a function that describes the distribution of the heights of the population of interest, we could randomly create a test population of their heights without any prejudice toward one side of the range or another.
Let’s take our analogy a little further. Imagine now that we play a game with of these shooting ranges, each with its own gremlin and uniquely shaped function wall. Let’s denote the function walls with the names A, B, and C. Our gremlins are coordinated such that they each fire their BB guns at the same time, but they independently choose the vertical position from which they fire. As it turns out, we are given a numerical relationship between A, B, and C, which is Y = A * (B + C). On each round of firing, we note the value of the cups the BBs fall into from their respective ranges, we perform the calculation A * (B + C), and we record the value of Y. The game we are playing is to try to predict where most of the values of all possible values of Y will occur after we’ve observed the gremlins shoot a sufficiently large—but not exhaustively large—number of BBs.
This situation is not as silly as it sounds. Suppose we want to estimate the range and prevalence of most of our costs for manufacturing berets, raspberry and lemon colored, and the manufacturing cost could vary over a range during the time we receive orders for berets, the actual number of which we won’t know until it’s time to close the books. Our formula for the exact cost of manufacturing in our period of concern will be mfg.cost = unit.mfg.cost * (number.raspberry + number.lemon).
The potential range and prevalence of the final manufacturing cost could be simulated using the gremlin method or we could use Monte Carlo simulation to simulate the range of possible values of the terms of our cost formula as long as we keep track of the outcomes of each of the terms. Once we’ve done such a thing, we would be in a better position to more accurately budget our operational costs for the year or write favorable contingent contracts with our suppliers if unit costs exceeded a certain value. The Monte Carlo simulation process gives us the ability to gain better clarity about how the world might turn out so that we can plan and design more desirable situations for ourselves and others.
A probability distribution for an uncertain assumption is defined within some real number domain and over a specific interval within that domain. Within the specific interval, probabilities are assigned to nonoverlapping subintervals that express our degrees of belief that the subintervals can occur or long-run frequencies that the subintervals will occur. We must conform to the inviolate principle that the sum of the probabilities equals exactly 1 within the interval.
For example, a cost for a project task might be defined in the domain from $0 to infinity, but the potential interval denoted by the context of the problem at hand might only cover $100,000 to $500,000. Within that interval, probabilities are assigned either in discrete buckets (e.g., Prob($100,000 < X <= $200,000) = 0.1; Prob($200,000 < X <= $300,000) = 0.25; Prob($300,000 < X <= $400,000) = 0.5; Prob($400,000 < X <= $500,000) = 0.15) or according to some mathematical function that assigns probabilities in a continuous form.1
Once these parameters are defined, the Monte Carlo algorithm randomly selects values from within an assumption’s interval according to the frequency implied by its internal probabilities over hundreds to thousands of iterations. This process is repeated in parallel for all the assumptions, some of them independently and others in a coordinated, or conditional, manner as required by the business logic. On each iteration, the intermediate calculations in the business case model are then performed as if only single point values were considered. Each iteration of the simulation represents a parallel universe to the other iterations. The outcomes on each iteration are stored, and the net results are examined in bulk. Consequently, through the combination of the business logic and the probabilities defined on the assumptions’ intervals, we can get a much less biased view of what the most likely range of conditions will be for our objective function. Of course, our assumptions’ definitions might be biased from the beginning. The books mentioned earlier describe methods for controlling for that condition, as do later sections of this book.
The Matrix Structure of Business Case Simulation in R
With the exception of the depreciation array, we based the deterministic model framework in one-dimensional vectors, as all the calculations produced results along a single index, year. To add the risk layer of the model, we need to add another dimension that handles the parallel scenarios. This dimension, run, will exist orthogonally to the year index. Its length, the number of iterations we choose to run (kSampsize), will be the same consistently for every calculation.
Notice that the year index elements produce the columns of the matrix, and the run index elements produce the rows. Each row represents a potential manifestation of the world in which a specific set of outcomes are realized for the Phase 1 and Phase 2 durations.
Before I explain how to set up this kind of matrix, I discuss some ways in which we can generate subjective distributions for each of our assumptions that represent the uncertainty we might face .
Useful Distributions for Expert Elicited Assumptions
Business case analysis often depends on making assumptions for which we possess little or no documented data from which we can develop appropriate empirical distributions . As I mentioned earlier, though, not having so-called hard data doesn’t excuse us from using a distribution. In these cases, we need to extract the information we need from SMEs and encode their knowledge, expertise, and degrees of uncertainty into a procedural function. I describe here four useful approaches for using expert elicited values from SMEs to produce distributions for uncertainties.
Discrete Distributions: McNamee-Celona and Swanson-Megill
In the early days of the decision analysis program, tractable ways to simulate uncertainty were handled by the Gaussian quadrature, McNamee-Celona,2 and later, the extended Swanson-Megill3 methods. These methods provide discrete approximations of single-mode continuous distributions in some domain of concern. Specifically, they derive three probability weightings that specify the frequency of discrete outcomes representing an expert’s 80th percentile prediction interval values that preserves the mean and variance of the continuous distributions implied by the expert elicitation. Gaussian quadrature or McNamee-Celona were used for symmetric distributions; extended Swanson-McGill was used for skewed distributions. The three weights are associated with the 10th, 50th, and 90th cumulative probability quantiles.
This simplification of continuous variable outcomes permitted a cheaper means of running a simulation when numerical processing was much more expensive than it is today. In fact, the simplification could be conveniently performed by calculator and hand-drawn decision trees .
Gaussian quadrature, McNamee-Celona: Prob(X10, X50, X90) = (0.25, 0.5, 0.25)
Extended Swanson-Megill: Prob(X10, X50, X90) = (0.3, 0.4, 0.3)4
ProductionCosts10 = $2.75 million
ProductionCosts50 = $3.00 million
ProductionCosts90 = $3.75 million
These values reveal that, given the best information available to our SME, he believes that there is a 10% chance the outcome will be less than $2.75 million, and a 90% chance the outcome will be less than $3.75 million. He places even odds that the outcome could be higher or lower than $3.00 million. When we develop a simulation of values in this range, our algorithm should select the ProductionCosts10 = $2.75 million approximately 30% of the time; ProductionCosts50 = $3.00 million 40% of the time; and ProductionCosts90 = $3.75 million 30% of the time. The extended Swanson-Megill weightings are chosen because the range is skewed around ProductionCosts50.
Be aware that using a discrete method poses some problems, namely that it sacrifices accuracy for speed and ease of use. When the discrete method is used for many independent variables in a simulation, the loss of accuracy is minimized somewhat; however, if a simulation relies on only a few distributions represented by these discrete approximations, the loss of accuracy can be pronounced.
The following code provides the McNamee-Celona and Swanson-Megill approach.
The cumulative probability distribution of the discrete approximation of the continuously distributed production costs using the extended Swanson-Megill generator
Continuous Distributions
Triangular
The triangular distribution is a distribution frequently used for the simulation of uncertainties that preserves the continuous nature of the represented event. In this regard, the triangular is an incremental improvement over the discrete methods. Like the discrete methods, the distribution requires three parameters: a low estimate, a mode estimate, and a high estimate.
I rarely use this distribution (or its smoother close cousin, the Beta-PERT) unless the SME I consult knows specifically what the two extreme values are, usually defined by some constraint in the underlying system, and the mode (the most likely outcome across the distribution) is known with strong empirical evidence. I follow this guideline because my own experience has shown me that SMEs are not very good at estimating most likely outcomes (regardless of the distribution being considered) without importing a number of extremely powerful biases; we explore ways to deal with this in later sections of this book.
Availability: Recalling values that are memorable or easily accessible.
Anchoring: Using the first best guess as a starting point for subsequent estimating.
Expert overconfidence: Failure of creativity or hubris (e.g., “I know this information and can’t be wrong because I’m the expert”).
Incentives: The SME experiences some benefit or cost in relationship to the outcome of the term being measured, adjusting the estimate in the direction of the preferred outcome.
Entitlement: The SME provides an estimate that reinforces his or her sense of personal value.
Therefore, if you must use a triangular distribution (or the Beta-PERT), SMEs should be aware that when they think about the low and high values, they believe the chance that any other outcome beyond these limits is nearly zero. Furthermore, they clearly understand that the most likely case is the mode, and not an average of the range of the interval or the median case (the point at which they believe there are equal odds that the outcome could either be higher or lower). The Beta-PERT distribution also suffers from some other limitations; namely, it often underestimates the uncertainty in the natural distribution (i.e., task durations) it is frequently employed to represent. For this reason, I never use the Beta-PERT, as I think it performs even less well than the triangular distribution to communicate the full effect of uncertainty on the final value function .
The cumulative probability distribution of the production costs using the triangular distribution generator
However, you should not think of p10, p50, and p90 estimates as corresponding to low, mode, and high parameters for a triangular distribution. I did that here merely to demonstrate the shape of the resultant curve.
BrownJohnson Distribution
Some years ago I became frustrated with the limitations of the discrete and triangular parameterizations, the tedious process of assessing a full distribution from an expert when dozens of assumptions had to be assessed, or finding an approximate known distribution to fit a few data points gathered from an SME. To this latter point, it just wasn’t intellectually satisfying to use a distribution just because I could find a similar geometry to the data I had, regardless of whether the distribution actually matched the underlying process being assessed. Furthermore, in some sense, I am indifferent to the use of any known distributions for business case simulations (unless I have good empirical evidence and a clear rationale that convinces me otherwise), as all I really care about is representing the uncertainty as reported by an SME in a logically consistent and probabilistically coherent manner. My belief is that if the SME is properly calibrated, his or her report will conform to any underlying process as understood by his or her mind.
p0 = 2.5 * p10 - 1.5 * p50
p100 = 2.5 * p90 - 1.5 * p50
The p0 and p100 are not intended to represent the actual values beyond which no other outcome can occur. Rather, they are virtual endpoints that capture approximately 99.9% of the range of uncertainty implied by the p10, p50, and p90 with Gaussian-like tails.
ProductionCosts10 = $2.75 million
ProductionCosts50 = $3.00 million
ProductionCosts90 = $3.75 million
p0 = 2.5 * $2.75 million - 1.5 * $3.00 million = $2.38 million
p100 = 2.5 * $3.75 million - 1.5 * $3.00 million = $4.88 million
If the p0 or p100 values fall outside allowable intervals (e.g., below 0), we can truncate the distribution at the desired boundaries using something like max(low.constraint, p0) or min(high.constraint, p100) .
My colleague, Eric Johnson, later replaced the piecewise quadratic spline with a piecewise linear because it was just easier to handle in this form. He also describes this approach in the book he recently coauthored, Handbook of Decision Analysis . 5
The following code provides the linear spline approach .
The cumulative probability distribution of the production costs using the piecewise linear BrownJohnson generator
Lognormal Distribution
Potential outcomes follow an asymmetric distribution.
The upper tail theoretically extends indefinitely.
The lower tail never extends below 0.
The effects that lead to the outcome are multiplicative in nature and not merely additive (i.e., not mean-reverting).
The lognormal distribution possesses the interesting characteristic that if two estimated quantile points are available that are symmetric around a p50 (e.g., the p10 and p90), then we can simulate the distribution easily from readily derived parameters that R recognizes for use in either the rnorm() or rlnorm() functions .
The following uses p10 and p90 estimates to generate a vector of normally distributed values in the log scale before returning the values in the measured domain. It uses the rnorm() function to do this.
This approach simply uses the rlnorm() function.
Which one you choose to use is entirely up to your discretion.
The cumulative probability distributions of the production costs using two slightly different approaches to implement a lognormal distribution generator
Note the high degree of coherence between them. We can attribute the small observable difference to simulation error.
When petroleum reservoir engineers use this type of function to estimate the amount of reserves in a field, they typically truncate the upper tail at the 99th percentile to accommodate the fact that a reservoir cannot be infinite in size, which the lognormal distribution would theoretically predict.
Comparison of the four example distribution generators discussed here
Modify the Influence Diagram to Reflect the Risk Layer
Now we need to modify the influence diagram to account for the competitive risk considerations.
To simplify the visual complexity of the diagram, we can collapse the operational expenses into a single node called Annual OPEX Module. Of course we can refer to the prior diagram to explain what the underlying components contain.
The influence diagram modified to account for competitive risk with operating expenses aggregated into one intermediate node called the Annual OPEX Module
Include the Run Index
The data table used for the deterministic business case in .csv format rendered in a spreadsheet
The header lines of our new uncertainty-enabled R model ( Risk_Model.R ) look like this now:
Next we define a probability distribution for the variables that will change with each iteration of the simulation. For this example, I used the BrownJohnson distribution described earlier. Notice, too, that I limited the range of the distributions to truncate at natural boundaries using the pmin() and pmax() functions as necessary. For example, because the Phase 1 capital cannot be negative, I surrounded the p1.capex definition with pmax(..., 0) to keep any stray simulation samples from going below 0. For definitions that need to be constrained between 0 and 1, I nested the pmin() and pmax() functions, as in
The first few samples from p1.capex look like this:
After you have set the variables in your own code file, load the source into the R console and run a few of the variables to get a feel for how their results appear and make sure that all results conform to the distributions’ definitions. Anomalies will indicate that you have possibly misdefined a variable. Of course, they can also indicate that your definition, although working just fine, is producing results that do not conform to the context of the business case. Instead of repairing the definition, you will likely need to rethink the definition employed or reassess the inputs.
We need to make one exception to using the BrownJohnson distribution for rr.comes.to.market , which represents the probability that RoadRunner comes to market. Although a broad discussion about the nature of probabilities is beyond the scope of this document, I maintain here without discussion that probabilities are not expressed as distributions. Only the uncertainty about event outcomes or measurable conditions are expressed as distributions. Because an event either occurs or not, it makes sense to describe a binary event with a single probability.6 The appropriate distribution for this is the Bernoulli distribution, which is a special limiting case of the binomial distribution. So, for rr.comes.to.market , we can define it as rbinom(kSampsize, 1, d.vals[14, 2]), where the prob parameter is satisfied with the corresponding probability value in the CSV data file.
You will notice, however, that I placed a value of 0 for the rr.comes.to.market p10, and 1 for the p90. This is not to indicate an uncertainty about the probability in question. Rather, I use these values here to serve as placeholders to compare the effects in later sensitivity analysis of the value of no competition (rr.comes.to.market = 0) to being certain that competition will occur (rr.comes.to.market = 1).
Now, each variable defined earlier will simulate a series of outcomes. The length of each series is kSampsize . You can think of these series as existing in the run index. To run our simulation model, we want our R code to select a single value from each variable starting at run = 1, step through all the calculations in the same order as the deterministic model in the last chapter, and then repeat the process through each step in the run index until our simulation reaches run = kSampsize. Essentially, we want the sample values in the run index to operate orthogonally to the year index .
This now presents a little problem, though, because of R’s vectorizing and recycling nature, which is usually a very powerful characteristic of the language. We naturally want to think of the run index as being orthogonal to the year index, but unless we take some steps to force that condition, R will recycle the values in the run index of each variable through the year index, completely messing up the indexing of the resulting calculations. As you might have guessed, we can use the sapply() function to address this issue.
To run the simulation, we wrap each variable calculation from our original deterministic model with the sapply() function, starting at the CAPEX module to the NPV, in which arrays of different shapes are applied to each other. Also, because sapply() wants to reorient the resulting calculations such that the year index becomes rows and the run index becomes columns, we transpose each calculation with the t() function to maintain the calculations in the run x year orientation so that we can easily read results of intermediate calculations in the R console when the run index is longer than the year index , as will most likely always be the case.
As we discussed earlier in the development of the depreciation matrix in Chapter 2, the sapply() function effectively iterates across an index just like a for loop, except it does so in one line of code. For example with phase, as R steps through the run index, a sample value from each of p1.dur and p2.dur is selected out of their respective distribution arrays, and the logic performed against the year index is as if the deterministic model were in play. When the calculation is complete, the resultant value is placed in the rth row of the phase array . The logic is similar for capex and all the other intermediate calculations through mean.npv . You can follow the entire R code for the risk model in Appendix B.
Once we reach NPV simulation, we can easily find the mean NPV:
No other function is required but the mean() function because all indexes but the run index have been reduced out, leaving only the samples in run. The mean() function simply finds the average of all the values in a given vector. The result gives us a depressing -$5,793,909. If we made decisions based on average values alone, we would stop here and go no further with our analysis. However, Chapter 4 will show us that there might yet be a reason to consider how to improve the value of this opportunity to our economic benefit.
When I calculate the mean NPV, I get -$5,793,909 on this particular run of the model. Running the model again, I get -$5,451,535. You will notice this same effect as well because of the simulation error inherent in simple Monte Carlo routines. One way to overcome this error, other than employing more advanced simulation routines (e.g., median Latin hypercube sampling), is to use more samples. We defined the sample size in our global_assumptions.R file as kSampsize <- 1000. Just change this value to a larger number to observe that the predicted mean NPV converges to a more stable position over several runs. On my computer, 1,000 iterations requires 1.542 seconds. I determined this value using the system.time() function.
Changing kSampsize to 5,000 requires 7.307 seconds. The desire for improved precision comes at the price of a slightly longer runtime.
Just to illustrate how powerful the vectorizing sapply() function is compared to running the risk model in one big for loop, a similar model constructed in a for loop requires 2.701 seconds using a sample size of 1,000—an increase of 1.75-fold. The simulation using 5,000 samples requires 31.237 seconds, a greater than fourfold increase over the model using the sapply() function , using the same number of samples. Of course, the exact numbers you get will vary by your computing configuration (i.e., operating system, processor speed, RAM), but the relative differences between the vector versus the looping approach should be similar.
Moving along, we can follow the same sapply() logic in the construction of the pro forma statement based on the mean values of the pro forma elements.
We will discuss the preparation for graphing of the cash flow and cumulative cash flow with confidence bands using the sapply() function in the next chapter.