Probability versus Statistics

Sometimes people use the terms probability and statistics interchangeably, and while this is understandable to conflate the two disciplines they do have distinctions. Probability is the study of how likely an event is to happen. Statistics utilizes data to discover probability and provides tools to decribe data. You can have probability without statistics, but statistics is pretty limited without probability.

Think of predicting the outcome of rolling a “4” on a die (that’s the singular of “dice”). Approaching the problem with a pure probability mindset, one simply says there are 6 sides on a die. We assume each side is equally likely, so the probability of getting a “4” is $\frac{1}{6}$, or 16.666%.

However, a zealous statistician might say “No! We need to roll the die to get data. If we can get 30 rolls or more, and the more rolls we do the better, only then will we have data to determine the probability of getting a 4.” This approach may seem silly if we assume the die is fair, but what if it’s not? If that’s the case, collecting data is the only way to discover the probability of rolling a “4”. We will talk about hypothesis testing in the next chapter.

Frequentist versus Bayesian Probability

While we are discussing statistics’ relationship to probability, let’s talk about two schools of thought on how we use probability. Let’s dissect this statement: “There is a 60% chance my flight will be late.” What exactly does that mean? Stop and think about that for a moment. Where could that “60%” number have come from?

One school of thought will say “Well out of 100 flights in the past similar to this one, 60 of them were late.” This is what we call a Frequentism mindset, where we derive probabilities off historical data and count the frequency of an event. This is a somewhat simplistic definition as I would use confidence intervals and other tools, but that is the essence of Frequentism.

But here comes the Bayesian who says differently. Bayesianism measures probability based on belief. She says “well given the poor weather forecast, a rival college game happening tonight, and the fact a terminal is closed for construction all causing congestion, my gut feel says there is an 80% chance this flight will be late.”

“Wait! You can’t just make up numbers on your gut feel!” says the Frequentist.

The Bayesian retorts “Why not? I fly frequently and have seen bad weather, college games, and terminal construction congest airports and make flights late.”

“Data is truth! And the data says 60% of flights at this time of day are late!”

“I don’t doubt that. As a matter of fact I started my belief with that 60% number as my baseline. I then accounted for the bad weather, the college game, and terminal construction and therefore increased my belief to 80%”

“I still don’t like you made up numbers you never measured” says the Frequentist. “Your conclusions are anecdotal at best!”

So who is right here? Is it the Frequentist or the Bayesian? Before reading on, think about this carefully and what side you should take.

Is it possible both the Frequentist and Bayesian are correct? Absolutely! Consider that Frequentism and Bayesianism both have merits depending on the situation. Frequentism works best when there is enough high quality, consistent data you can reliably draw conclusions from. But when you lack data, the data is unreliable, the data is always changing, or there are just too many chaotic variables you cannot control, Bayesianism can be a great tool to account for uncertainty. You can even merge data and beliefs together to get the best of both worlds.

Even when you do have lots of data, Bayesianism can still be a helpful tool to entertain a span of possibilities. We will see this in action with the Beta distribution where we estimate parameters as distributions rather than a single value.

Regarding the Frequentist accusing the Bayesian of being anecdotal, that is certainly a bias to watch out for. While her experiences should not be generalized for everyone else, she did establish herself as an “expert” because she frequently flies. Bayesianism realizes that beliefs can be just as important in probability, as we may not always have sufficient data on hand and our “experts” and “common sense” are the best tools we have.

We will explore a few more Bayesian ideas in this chapter, and we will talk about statistics and hypothesis testing in Chapter 4.

Joint Probabilities

Let’s say you have a fair coin and a fair 6-sided die. You want to find the probability of flipping a “Heads” and rolling a “6” on the coin and die respectively. These are two separate probabilities of two separate events, but we want to find the probability that both events will occur together. This is known as a joint probability.

Think of a joint probability as an AND operator. I “want to find the probability of flipping a heads AND rolling a 6.” We want both events to happen together, so how do we calculate this probability?

There are 2 sides on a coin and 6 sides on the die, so the probability of heads is $\frac{1}{2}$ and probability of “6” is $\frac{1}{6}$. The probability of both events occuring (assuming they are independent, more on this later!) is simply multiplying the two together:

Easy enough, but why is this the case? A lot of probability rules can be discovered by generating all possible combinations of events, which comes from an area of math known as permutations and combinations. For this case, generate every possible outcome between the coin and die, pairing heads (H) and tails (T) with the numbers 1 through 6:

H1  H2  H3  H4  H5  *H6*  T1  T2  T3  T4  T5  T6

Notice there are 12 possible outcomes when flipping our coin and rolling our die. The only one that is of interest to us is “H6,” getting a heads and a six. So because there is only 1 outcome that satisfies our condition, and there are 12 possible outcomes, the probability of getting a heads and a six is $\frac{1}{12}$.

Rather than generate all possible combinations and counting the ones of interest to us, we can use multiplication as a shortcut to find the joint probability. This is known as the product rule.

But what if we are interested in 3 joint events? 10 events? 1000 events? Does the product rule still apply? Yes it does! We just multiply the probability of each event together to find the probability of all events of interest occurring.

If I roll a die three times, what is the probability P(X) of getting a “1” on the first role? A “2” or “4” on the second roll? And a “3”, “4”, or “6” on the third roll?

So the probability of getting that outcome for those 3 rolls is $2.\overline{777}\%$.

Floating Point Underflow

 — Note to self, move this to later chapter based on reviewer feedback

A problem you can come across when doing joint probabilities on computers is something called floating point underflow, which happens when you multiply a lot of small decimals together. Computers can only keep track of so many decimal places, so when the decimals start getting small your computer gives up and just lets your decimals approach 0. A clever hack you can do (as presented in Joel Grus’ book Data Science from Scratch by O’Reilly) is to take the log() of each probability, add them together, and then call exp() to convert the result back!

In Example 2-1 we use logarithmic addition of probabilities (rather than multiplication) to calculate the probability of 3 consecutive engine failures if probability of each failure is 10%. It really does not matter what base you use for the logarithm, so we can just use the default Euler’s number e.

from math import log, exp

prob_engine_failure = .10
three_fails_prob = 0.0

for i in range(0,3):
    # Perform logarithmic addition instead of multiplication
    three_fails_prob += log(prob_engine_failure)

# Use exp() to convert back!
three_fails_prob = exp(three_fails_prob)

# 0.0010000000000000002
print("Probability of 3 consecutive failures: {}".format(three_fails_prob))

Union Probabilities

We discussed joint probabilities, which is the probability of two or more events occurring simultaneously. But what about the probability of getting event A or B? When we deal with “OR” operations with probabilities, this is known as a union probability.

Let’s start with mutually exclusive events, which are events that cannot occur simultaneously. For example, if I roll a die I cannot simultaneously get a “4” and a “6”. I can only get one outcome. Getting the union probability for these cases are easy. I just simply add them together. So if I want to find the probability of getting a “4” or “6” on a die roll:

Just like the joint operation, the union operation extends to more than two events. The probability of getting a “2”, “4” or “6” on a single die roll is $\frac{1}{6}+\frac{1}{6}+\frac{1}{6}=\frac{1}{2}$

But what about non-mutally exclusive events, which are events that can occur simulataneously? Let’s go back to the coin flip and die roll example. What is the probability of getting a heads OR a “6”? Before you are tempted to add those probabilities, let’s generate all possible outcomes again and highlight the ones we are interested in:

*H1*  *H2*  *H3*  *H4*  *H5*  *H6*  T1  T2  T3  T4  T5  *T6*

Above we are interested in all the “heads” outcome as well as the “6” outcomes. If we proportion the 7 out of 12 outcomes we are interested in, $\frac{7}{12}$, we get a correct probability of $.58\overline{333}$.

But what happens if we add the two probabilities of heads and “6” together? We get a different (and wrong!) answer of $.\overline{666}$.

Why is that? Study the combinations of coin flip and die outcomes again and see if you can find something fishy. Notice when we add the probabilities, we double-count the probability of getting a “6” for “H6” and “T6”! If this is not clear, try finding the probability of getting “Heads” and a die roll of 1 through 5.

We get a probability of 133.333% which is definitely not correct, as a probability must be no more than 100% or 1.0. The problem again is we are double-counting outcomes.

If you ponder long enough, you may realize the logical way to remove double-counting in a union probability is to subract the joint probability. This is known as the sum rule of probability and ensures every joint event is only counted once.

So going back to our example of calculating the probability of a Heads or a “6”, we need to subtact the joint probability of getting a Heads or a “6” from the union probability:

Note that this formula $P\left(\text{A}\text{OR}\text{B}\right)=P\left(A\right)+P\left(B\right)-P\left(\text{A}\text{AND}\text{B}\right)$ also applies to mutually exclusive events. Stop and wonder why. If the events are mutually exclusive (like outcomes of a single die roll), then the joint probability $P\left(\text{A}\text{AND}\text{B}\right)$ is going to be 0, and therefore remove itself from the formula. You are then left with simply summing the events as we did earlier.

In summary, when you have a union probability between two or more events that are not mutually exclusive, be sure to subtract the joint probability so no probabilities are double-counted.

Conditional Probability and Bayes Theorem

A probability topic that people easily get confused by is the concept of conditional probability, which is the probability of an event A occuring given event B has occurred. It is typically expressed as $P\left(\text{A}\text{GIVEN}\text{B}\right)$ or $P\left(\text{A|B}\right)$.

Let’s say some study makes a claim that 85% of cancer patients drank coffee. How do you react to this claim? Does this alarm you and make you want to abandon your favorite morning drink? Let’s first define this as a conditonal probability $P\left(\text{COFFEE}\text{given}\text{CANCER}\right)$ or $P\left(\text{COFFEE|CANCER}\right)$. This represents a proportion of people who drink coffee given they have cancer.

Within the United States, let’s compare this to the percentage of people diagnosed with cancer (0.5% according to cancer.gov) and the percentage of people who drink coffee (65% according to statista.com):

Hmmmm… study these numbers for a moment and ask whether coffee is really contributing to cancer here. Notice again that only 0.5% of the population has cancer at any given time. However 65% of the population drinks coffee regularly. If coffee contributes to cancer, should we not have much higher cancer numbers than 0.5%? Would it not be closer to 65%?

This is the sneaky thing about proportional numbers. They may seem significant without any given context, and media headlines can certainly exploit this for clicks: “New Study Reveals 85% of Cancer Patients Drink Coffee” it might read. Of course, this is silly because we have taken a common attribute (drinking coffee) and associated it with an uncommon one (having cancer).

The reason people can be so easily confused by conditional probabilities is because the direction of the condition matters, and the two conditions are conflated as somehow being equal. The “probability of having cancer given you are a coffee drinker” is different than the “probability of being a coffee drinker given you have cancer.” To put it simply: few coffee drinkers have cancer, but many cancer patients drink coffee!

If we are interested in studying whether coffee contributes to cancer, we really are interested in the first conditional probability: the probability of somone having cancer given they are a coffee drinker.

How do we flip the condition? There’s a powerful little formula called Bayes Theorem and we can use it to flip conditional probabilities.

If we plug the information we already have into this formula, we can solve for the probability someone has cancer given they drink coffee:

If you want to calculate this in Python, check out Example 2-2.

p_coffee_drinker = .65
p_cancer = .005
p_coffee_drinker_given_cancer = .85

p_cancer_given_coffee_drinker = p_coffee_drinker_given_cancer *
    p_cancer / p_coffee_drinker

# prints 0.006538461538461539
print(p_cancer_given_coffee_drinker)

Wow! So the probability someone has cancer given they are a coffee drinker is only 0.65%! This number is very different than the probability someone is a coffee drinker given they have cancer, which is 85%. Now do you see why the direction of the condition matters? Bayes Theorem is helpful for this reason, and it can be used to chain several conditional probabilities together to keep updating our beliefs based on new information.

We will talk about the intuition behind Bayes Theorem in a moment. For now just know it helps us flip a conditional probability. Let’s talk about how conditional probabilities interact with joint and union operations next.

population = 100_000

p_coffee_drinker = .65
p_cancer = .005

coffee_drinker_ct = population * p_coffee_drinker
cancer_ct = population * p_cancer

# COFFEE DRINKERS: 65000.0
# CANCER PATIENTS: 500.0
print("# COFFEE DRINKERS: {}".format(coffee_drinker_ct))
print("# CANCER PATIENTS: {}".format(cancer_ct))

p_coffee_drinker = .65
p_cancer = .005
p_coffee_drinker_given_cancer = .85

coffee_drinker_ct = population * p_coffee_drinker
cancer_ct = population * p_cancer

# Count of people who are both coffee drinkers and have cancer
coffee_and_cancer_ct = cancer_ct * p_coffee_drinker_given_cancer

# DRINK COFFEE AND HAVE CANCER: 425.0
print("# DRINK COFFEE AND HAVE CANCER: {}".format(coffee_and_cancer_ct)

population = 100_000

p_coffee_drinker = .65
p_cancer = .005
p_coffee_drinker_given_cancer = .85

coffee_drinker_ct = population * p_coffee_drinker
cancer_ct = population * p_cancer

# Count of people who are both coffee drinkers and have cancer
coffee_and_cancer_ct = cancer_ct * p_coffee_drinker_given_cancer

# Percentage of coffee drinkers with cancer
p_cancer_given_coffee_drinker = coffee_and_cancer_ct / coffee_drinker_ct

# Probability of cancer given coffee drinker: 0.006538461538461538
print("Probability of cancer given coffee drinker: {}".format(p_cancer_given_coffee_drinker))

population = 100_000

p_coffee_drinker = .65
p_cancer = .005
p_coffee_drinker_given_cancer = .85

coffee_drinker_ct = population * p_coffee_drinker
cancer_ct = population * p_cancer

# coffee_and_cancer_ct = cancer_ct * p_coffee_drinker_given_cancer
coffee_and_cancer_ct = population * p_cancer * p_coffee_drinker_given_cancer

# p_cancer_given_coffee_drinker = coffee_and_cancer_ct / coffee_drinker_ct
p_cancer_given_coffee_drinker = (population * p_cancer * p_coffee_drinker_given_cancer) / (population * p_coffee_drinker)

print("Probability of cancer given coffee drinker: {}".format(p_cancer_given_coffee_drinker))

p_cancer_given_coffee_drinker = (population * p_cancer * p_coffee_drinker_given_cancer) / (population * p_coffee_drinker)

p_cancer_given_coffee_drinker = (p_cancer * p_coffee_drinker_given_cancer) /  p_coffee_drinker

Binomial Distribution from Scratch

If you want to implement a binomial distribution from scratch, here are all the parts you need in Example 2-10.

# Factorials multiply consecutive descending integers down to 1
# EXAMPLE: 5! = 5 * 4 * 3 * 2 * 1
def factorial(n: int):
    f = 1
    for i in range(n):
        f *= (i + 1)
    return f

# Generates the coefficient needed for the binomial distribution
def binomial_coefficient(n: int, k: int):
    return factorial(n) / (factorial(k) * factorial(n - k))


# Binomial distribution calculates the probability of k events out of n trials
# given the p probability of k occurring
def binomial_distribution(k: int, n: int, p: float):
    return binomial_coefficient(n, k) * (p ** k) * (1.0 - p) ** (n - k)

# 10 trials where each has 90% success probability
n = 10
p = 0.9

for k in range(n + 1):
    probability = binomial_distribution(k, n, p)
    print("{0} - {1}".format(k, probability))

Using the factorial() and the binomial_coefficient(), we can build a binomial distribution function from scratch. The factorial function multiplies a consecutive range of integers from 1 to n. For example, a factorial of 5! would be $1*2*3*4*5=120$.

The binomial coefficient function allows us to select k outcomes from n possibilities with no regard for ordering. If you have k = 2 and n = 3, that would yield sets (1,2) and (1,2,3), respectively. Between those two sets, the possible distinct combinations would be (1,3), (1,2), and (2,3). That is 3 combinations so that would be a binomial coefficient of 3. Of course, using the binomial_coefficient() function we can avoid all that permutation work by using factorials and multiplication instead.

When implementing binomial_distribution(), notice how we take the binomial coefficient and multiply it by the probability of success p occuring k times (hence the exponent). We then multiply it by the opposite case: the probability of failure 1.0 - p occurring n - k times. This allows us to account for the probability p of an event occuring versus not occuring across several trials.

Beta Distribution from Scratch

If you are curious how to build a beta distribution from scratch, you will need to re-use the factorial() function we used for the binomial distribution as well as the approximate_integral() function we built in the last chapter.

Just like we did in the previous chapter, we pack rectangles under the curve for the range we are interested in as shown in Figure 2-10?

Figure 2-10. Packing rectangles under the curve to find the area/probability

This is just using 6 rectangles and we will get better accuracy if we were to use more rectangles. Let’s implement the beta_distribution() from scratch and integrate 1000 rectangles between .9 and 1.0 as shown in Example 2-15.

# Factorials multiply consecutive descending integers down to 1
# EXAMPLE: 5! = 5 * 4 * 3 * 2 * 1
def factorial(n: int):
    f = 1
    for i in range(n):
        f *= (i + 1)
    return f


def approximate_integral(a, b, n, f):
    delta_x = (b - a) / n
    total_sum = 0

    for i in range(1, n + 1):
        midpoint = 0.5 * (2 * a + delta_x * (2 * i - 1))
        total_sum += f(midpoint)

    return total_sum * delta_x

def beta_distribution(x: float, alpha: float, beta: float) -> float:
    if x < 0.0 or x > 1.0:
        raise ValueError("x must be between 0.0 and 1.0")

    numerator = x ** (alpha - 1.0) * (1.0 - x) ** (beta - 1.0)
    denominator = (1.0 * factorial(alpha - 1) * factorial(beta - 1)) / (1.0 * factorial(alpha + beta - 1))

    return numerator / denominator


greater_than_90 = approximate_integral(a=.90, b=1.0, n=1000, f=lambda x: beta_distribution(x, 8, 2))
less_than_90 = 1.0 - greater_than_90

print("GREATER THAN 90%: {}, LESS THAN 90%: {}".format(greater_than_90, less_than_90))

You will notice with the beta_distribution() function, we provide a given probability x, an alpha value quantifying successes, and a beta value quantifying failures. The function will return how likely we are to observe a given likelihood x. But again, to get a probability of observing probability x we need to find an area within a range of x values. Thankfully, we have our approximate_integral() function defined and ready to go from the last chapter. We can calculate the probability that the success rate is greater than 90%, as well as less than 90% as shown in the last few lines above.