In statistics today, there are two schools of thought as to how we interpret data and make statistical inferences. The classic and more dominant approach to date has been what is termed the Frequentist approach (refer to Chapter 7, A Tour of Statistics – The Classical Approach), while we are looking at the Bayesian approach in this chapter.
At the heart of the debate between the Bayesian and Frequentist worldview is the question—how do we define probability?
In the Frequentist worldview, probability is a notion that is derived from the frequencies of repeated events. For example, when we define the probability of getting heads when a fair coin is tossed as being equal to half. This is because when we repeatedly toss a fair coin, the number of heads divided by the total number of coin tosses approaches 0.5 when the number of coin tosses is sufficiently large.
The Bayesian worldview is different, and the notion of probability is that it is related to one's degree of belief in the event happening. Thus, for a Bayesian statistician, having a belief that the probability of a fair die turning up 5 is relates to our belief in the chances of that event occurring.
From the model definition point of view Frequentists analyze how data and calculated metrics vary by making use of repeated experiments while keeping the model parameters fixed. Bayesians, on the other hand, utilize fixed experimental data but vary their degrees of belief in the model parameters, this is explained as follows:
The Frequentist approach uses what is known as the maximum likelihood method to estimate model parameters. It involves generating data from a set of independent and identically distributed observations and fitting the observed data to the model. The value of the model parameter that best fits the data is the maximum likelihood estimator (MLE), which can sometimes be a function of the observed data.
Bayesianism approaches the problem differently from a probabilistic framework. A probability distribution is used to describe the uncertainty in the values. Bayesian practitioners estimate probabilities using observed data. In order to compute these probabilities, they make use of a single estimator, which is the Bayes formula. This produces as distribution rather than just a point estimate, as in the case of the Frequentist approach.
Let us compare what is meant by a 95 percent confidence interval, a term used by Frequentists with a 95 percent credible interval, used by Bayesian practitioners.
In a Frequentist framework, a 95 percent confidence interval means that if you repeat your experiment an infinite number of times, generating intervals in the process, 95 percent of these intervals would contain the parameter we're trying to estimate, which is often referred to as θ. In this case, the interval is the random variable and not the parameter estimate θ, which is fixed in the Frequentist worldview.
In the case of the Bayesian credible interval, we have an interpretation that is more in-line with the conventional interpretation ascribed to that of a Frequentist confidence interval. Thus, we have that . In this case, we can properly conclude that there is a 95 percent chance that θ lies within the interval.
For more information, refer to Frequentism and Bayesianism: What's the Big Deal? | SciPy 2014 | Jake VanderPlas at https://www.youtube.com/watch?v=KhAUfqhLakw.