In this chapter, we are going to discuss some important algorithms that exploit different estimators to improve the overall performance of an ensemble or committee. These techniques work either by introducing a medium level of randomness in every estimator belonging to a predefined set or by creating a sequence of estimators where, each new model is forced to improve the performance of the previous ones. These techniques allow us to reduce both the bias and the variance (thereby increasing validation accuracy) when employing models with a limited capacity or more prone to overfit the training set.

In particular, the topics covered in the chapter are as follows:

• Introduction to ensemble learning
• A brief introduction to decision trees
• Random forest and extra randomized forests
• AdaBoost (algorithms M1, SAMME, SAMME.R, and R2)
• Gradient boosting
• Ensembles of voting classifiers, stacking, and bucketing