The field of Bayesian statistics is built on the work of Reverend Thomas Bayes, an 18th century statistician, philosopher, and Presbyterian minister. His famous Bayes' theorem, which forms the theoretical underpinnings for Bayesian statistics, was published posthumously in 1763 as a solution to the problem of inverse probability. For more details on this topic, refer to http://en.wikipedia.org/wiki/Thomas_Bayes.

Inverse probability problems were all the rage in the early 18th century and were often formulated as follows:

Suppose you play a game with a friend. There are 10 green balls and 7 red balls in bag 1 and 4 green and 7 red balls in bag 2. Your friend turns away from your view, tosses a coin and picks a ball from one of the bags at random, and shows it to you. The ball is red. What is the probability that the ball was drawn from bag 1?

These problems are termed inverse probability problems because we are trying to estimate the probability of an event that has already occurred (which bag the ball was drawn from) in light of the subsequent event (that the ball is red).

Bayesian_balls_illustration

Let us quickly illustrate how one would go about solving the inverse probability problem illustrated earlier. We wish to calculate the probability that the ball was drawn from bag 1, given that it is red. This can be denoted as .

Let us start by calculating the probability of selecting a red ball. This can be calculated by following the two paths in red as shown in the preceding figure. Hence, we have .

Now, the probability of choosing a red ball from bag 1 is via the upper path only and is given as follows:

And, the probability of choosing a red ball from bag 2 is given as follows:

Note that this probability can be written as follows:

By inspection we can see that , and the final branch of the tree is only traversed if the ball is firstly in bag 1 and is a red ball. Hence, intuitively we'll get the following outcome: