Define the Uncertain Event

Defining the uncertain event simply means that you will produce a statement that allows you or others to determine, without contention, that an event occurred or that it occurred in a certain way pertinent to the analysis at hand when the facts of the event are examined a posteriori. Determine if the event will occur in discrete values (e.g., [True or False], or [A, B, C]) or along a continuous scale (e.g., costs, revenue, temperature, mass). Explore how the SME thinks about the event and in which units (if any).

The goal is to produce a definition that would pass the so-called clarity test. The clarity test requires that a clairvoyant, who has access to all possible information, be given the definition of the uncertain event. The definition should permit the clairvoyant to make a determination about the event without requiring the clairvoyant to make any special judgments to determine that his or her data satisfies the definition.

Identify the Sources of Bias

Common thinking traps or errors that often plague business people are entitlement bias and wishful thinking. Entitlement bias is related to thinking that just because one has been successful in the past with important business initiatives that future success is not only possible, but virtually certain. This behavior is also related to expert bias , a state of mind in which a person with extensive and often publicly recognized experience in a field of practice regards his or her predictions about future outcomes as incontrovertible. Wishful thinking occurs when people believe that their plans will work as designed simply because they planned them and visualized the outcome (“If we build it, they will come”), or because they have strong emotional investment in the outcome that they desire.

Some of the common cognitive illusions that show up in uncertainty assessments are availability bias and anchoring. Availability bias is the tendency to use recently observed, emotionally impactful, or easily accessible information to lend belief that a certain state of affairs is the case or that an eventual outcome will be the case. Availability bias has a close cousin in confirmation bias (sometimes called “cherry picking”), which is the selective observation of evidence used to justify or buttress preconceived notions or closely held beliefs. To make a playful inversion on the warning commonly printed on rearview mirrors, availability bias drives the perception that images in the mirror are actually closer than they really are. Anchoring is the tendency of the mind to work from the first value that it conceives as a reference point for all other variation around it. Both availability bias and anchoring drive a tendency to believe that initial impressions serve as best guesses for most likely future outcomes.

To begin addressing these biases, ask the SME to describe any recent events or memorable events that come to mind associated with the event in question. Has something related to the event in question appeared in the news? Has there been a recent exposure to the event with a given value associated with the outcome?

If it is not obvious, ask the SME if there is a desirable outcome associated with the event. Would the SME, or those he or she supports, desire a higher or lower outcome or a targeted outcome?

Finally, explore how the outcome might affect the SME personally. Are there preferences, motivations, or incentives that might be associated with the SME and the event? Is the SME compensated in any way for the event occurring at some degree or level of outcome?

This line of questioning should help to reveal confirmation and availability bias, motivational bias, and wishful thinking, among others, at work in the mind of the SME. As a friend of mine says, “Confirmation bias is the root of all evil. And wishful thinking is root fertilizer.” These must be rooted out and exposed.

Postulate and Document Causes of Extrema

Ask the SME to imagine the extreme case representing the opposite direction of the desired outcome. Explore what might have caused this outcome to occur. What had to occur for the future to be in the extreme outcome described? What are those factors? Repeat this process for the opposite extreme. Record the factors for each extreme in separate columns. When you finish tabulating the causative factors, place a quick rank of importance on the top three or four causes on the low and high side. Later, when you go back to consider risk mitigation on the uncertainty, if it’s required, you might want to consider these higher ranked factors first for risk management or risk transfer.

Be sure to emphasize that at this point you are not seeking a number or a value. Resist the tendency to get a value as a starting point because that introduces anchoring. The objective is to counteract anchoring and availability biases by getting the expert away from their preconceived notions or influenced inclinations.

Measure the Range of Uncertain Events with Probabilities

Before proceeding with quantifying the uncertain event with the SME, assure the SME that the purpose of the interview is not to acquire a commitment or a firm projection about the outcome. You are merely trying to measure the overall uncertainty you face with the event. You should also assure the SME that you are not asking the SME to know the probabilities you are trying to assess as if probabilities are a thing or an intrinsic property of an event. Rather, you want to determine, by comparison to a defined lottery or game, the odds at which the SME is indifferent to choosing to play the lottery to win a prize or calling the outcome of the event in question to win a prize. The point of indifference for them is their subjective degree of belief—the probability—that the event will occur as defined.

There are generally two kinds of uncertain events that an SME will be called on to assess: discrete binary events and continuous events.

Discrete binary events are those that produce either a TRUE or FALSE state to describe the outcome of the event, that it either occurred or did not. Examples of this kind of event could be “win the deal,” or “Phase 1 succeeds,” or “Plaintiff claim is dismissed.” The SME will assess the probability of one of the states. The probability of the complementary state must be 1 - Pr(state) by definition of the laws of probability. Under no circumstances can an SME assess the probability of one state, say TRUE, to be Pr(TRUE) = 0.25, and then assert that Pr(FALSE) = 0.55 or Pr(FALSE) = 0.95 without revealing a mental incoherence about the definition of the event. If this does occur, explore what is missing from the definition that might have led the SME to such incoherence.

Continuous events are assessed across an effectively infinitely divisible range with several probabilities, often at the 10th, 50th, and 90th percentile intervals. Examples of these kinds of events might be “capital cost of the reactor,” or “duration of activity 25,” or “sales in quarter 1.” The p10 value is the value of the event at the 10th percentile for which the SME believes there is yet a 1/10 chance that the event outcome could still be lower. Conversely, the p90 value is the value of the event at the 90th percentile for which the SME believes there is yet a 1/10 chance that the event outcome could still be higher. The p50 value (or the median) is the value at which the SME considers that the outcome faces even odds or a 1/2 chance that the outcome could be either higher or lower.

Discrete Binary Uncertainties

Tell the SME he or she will have the hypothetical opportunity to win a prize by opting to play one of two games: Either the SME correctly calls the event occurring, or he or she can play a lottery by pulling a blue token from an urn or spinning an arrow on a wheel of fortune such that it lands on a blue sector for success. The choice to accept either the lottery or to call the outcome of the event is based on the odds presented for the lottery. If the SME thinks the odds of winning the lottery exceed the perceived odds of correctly calling the outcome of the uncertain event, then he or she should play the lottery. On the other hand, if the odds of winning the lottery are lower than the perceived odds of calling the event, then the SME should call the event. After a choice is made, the odds on the lottery are updated such that it should be more difficult to make a clear choice. This process is repeated until the SME is indifferent to choosing either game to win the prize.

The following steps demonstrate the session with the previously described sales manager who assessed the discrete binary probability of winning a competitive bid for a government contract. Each step is illustrated by a result from this online tool ¹ to facilitate the visualization of lottery probabilities on each iteration as the SME converges on an indifference point. Recall that the bid team initially expressed a high degree of confidence that their company would win the bid. After going through the rationale eliciting process, however, they realized that there might be more strong reasons for why they might lose the bid rather than win it; therefore, I switched the probability for them to assess to the probability of losing the bid to continue to counteract their initial overconfidence.

In the first iteration (Figure 11-3), a random value between 0 and 1 was selected, and the “win” state of the urn and the wheel of fortune are initialized with a proportion that matches the probability. So, the question now for the SME was this: Would you rather play the depicted lottery to win a prize, or would you rather call the outcome of the bid to win a prize (not win the bid)? Based on her internal sense of the weight of the rationales, the SME believed that the first iteration represented too low a value to play the lottery, so she chose to call the event, meaning that given her current information she believed there was a higher chance of losing the bid. Through the method described previously, the probability was updated to Pr_2 = 0.86 (Figure 11-4). Again, the SME believed that the second iteration represented too low a value to chance the lottery, so she chose to call the event outcome. The probability was updated to Pr_3 = 0.93 (Figure 11-5). On the fourth iteration, the SME believed the last update overshot the probability, so she chose to switch the game to play the lottery. The probability was then updated to Pr_4 = 0.89 (Figure 11-6). At this point, the SME felt indifferent to choosing a game or calling the event outcome to win a prize, so she stayed with Pr_4 = 0.89 as the probability of the event. Now, we knew the probability of winning the bid was 1 – Pr_4 = 1 – 0.89 = 0.11. Now the team had to carefully consider whether pursuing the bid was worth the effort at all; but if they did pursue it, they needed to plan how they might significantly improve their odds.

Sometimes you will need to assess discrete events of more than two possible outcomes. These are the [A, B, C, …] kinds of events. This is the general case of the binary discrete event, and it can be assessed in a similar manner as the binary one. Simply treat the A case as True, and assess its probability. The complementary probability is the residual probability that categories [B, C, …] could occur. Now, assess the probability of B occurring (excluding A from the list). When the SME prefers to call the outcome, use this formula:

instead of

Keep repeating this process until only one discrete category remains. The probability associated with this category will be 1 – (sum of all prior residual probabilities).

Continuous Uncertainties

As I have already alluded , the process of assessing the continuous uncertainty is very similar to that of assessing the discrete uncertainty, except that it usually includes three segments. However, instead of assessing a probability that an event will occur, the SME will be called on to provide an assessment for an 80th percentile prediction interval; that is, the range of values for which the SME posits an 80% degree of belief that will bracket the actual outcome.

If the SME wants to adjust the shape, he or she can do so either by moving the P10, P50, or P90, or by truncating the P0 to a position closer to the P10 or the P100 to a position closer to the P90. Typically, I do not recommend adjusting the P0 or P100 unless there is a systematic reason to impose a constraint, such as the actual value cannot go below zero or it is restricted from going above some number by policy, physical, or logical constraint . Otherwise, let the symmetry in the relative positions of the P10, P50, P90 determine the P0 and P100.

In the case presented earlier in which the construction engineer was called on to assess the duration of the structural steel construction for a chemical reactor, the engineer, reviewing the list of causative factors to the event duration, revealed his first indifference point for the 80th percentile prediction interval at P10 = 75 days. Then he revealed his P90 = 150 days. Finally, he was indifferent to betting whether the outcome would either be higher or lower than P50 = 107 days. Once we obtained these values, we used them to simulate distributions that allowed him to visually understand the implications of his assessments (Figure 11-7). Looking at the graph, though, made him realize that there might still be room for more upper tail. He then adjusted the P90 to 170 days. By taking this approach of exploring the edges of his belief about potential outcomes, we avoided an anchoring and adjustment bias. After the assessment ended, I asked the engineer what his original thoughts were about the duration distribution, and he conveyed to me that he actually had been concerned about the recent delay on a similar project. His original inclination was to suggest a “most likely” duration of 130 days with a ±20% variation (i.e., 104 days – 130 days – 156 days). The thinking process here helped him realize that not only might he have been too pessimistic, but that he was overly precise in his pessimism. The final assessment was much more accurate to the overall range of realistic durations, yet it still retained the possibility for his justified pessimism. You can use the online tool found at http://www.incitedecisiontech.com/distcal.shtml to help SMEs visualize the full range of their uncertainty assessments, too.

It is absolutely imperative that you follow this sequence of events of assessing the outer values first and the interior value last, with no exceptions. The reason for this is that by assessing the outer values first, you will be helping SMEs think about events beyond their initial inclination to consider ; thus, you will be avoiding availability bias. By assessing the inner value last, you will be helping SMEs avoid anchoring. Unfortunately, the all too common practice for producing ranged estimates starts with a “best guess” that is then padded with a ±X% that the SME believes represents a reasonable range of variation without thinking about how probable this range might actually be (it could be either grossly overstated or understated). The best guess is usually anchored in a biased direction, and the padding is just an arbitrary rule of thumb. Biased and arbitrary thinking are not mental characteristics that lead to accuracy in measurements.

Remember, although you are seeking accuracy with the best information you have at hand, you should not try to achieve a false sense of confidence through unwarranted precision by coaxing the SME for narrower ranges rather than wider ones. In fact, the wider the range you assess, the more likely you will take into account events outside the original biased inclination. Furthermore, you should avoid unnecessarily overworking the assessments to get the numbers perfect. The goal is to set bookends to the potential range of outcomes given the quality and limits of the SME’s knowledge at the time of assessment.

Document the SME Interview

You will most likely find it helpful to document the content of the SME interview in a table like that shown in Figure 11-8.