The goal of ensemble methods is to combine the predictions of several individual estimators built with a given learning algorithm in order to solve a shared problem. Typically, an ensemble consists of two major components:
- A set of models
- A set of decision rules that govern how the results of these models are combined into a single output
A consequence of this procedure is that we get a multitude of opinions about any given problem. So, how do we know which classifier is right?
This is why we need a decision rule. Perhaps we consider everybody's opinion of equal importance, or perhaps we would want to weigh somebody's opinion based on their expert status. Depending on the nature of our decision rule, ensemble methods can be categorized as follows:
- Averaging methods: These develop models in parallel and then use averaging or voting techniques to come up with a combined estimator. This is as close to democracy as ensemble methods can get.
- Boosting methods: These involve building models in sequence, where each added model aims to improve the score of the combined estimator. This is akin to debugging the code of your intern or reading the report of your undergraduate student: they are all bound to make errors, and the job of every subsequent expert laying eyes on the topic is to figure out the special cases where the preceding expert got it wrong.
- Stacking methods: Also known as blending methods, these use the weighted output of multiple classifiers as inputs to the next layer in the model. This is akin to having expert groups who pass on their decision to the next expert group.
The same three categories of ensemble methods are illustrated in the following diagram:
Ensemble classifiers afford us the opportunity to choose the best elements of each of the proposed solutions in order to come up with a single, final result. As a result, we typically bet much more accurate and robust results.
Let's briefly discuss the three different types of ensemble methods and look at how they can be used to classify data.