In this chapter, we will learn about two new classes of machine learning models: decision trees and random forests. We will see how decision trees learn rules from data that encodes non-linear relationships between the input and the output variables. We will illustrate how to train a decision tree and use it for prediction for regression and classification problems, visualize and interpret the rules learned by the model, and tune the model's hyperparameters to optimize the bias-variance tradeoff and prevent overfitting. Decision trees are not only important standalone models but are also frequently used as components in other models.

In the second part of this chapter, we will introduce ensemble models that combine multiple individual models to produce a single aggregate prediction with lower prediction-error variance. We will illustrate bootstrap aggregation, often called bagging, as one of several methods to randomize the construction of individual models and reduce the correlation of the prediction errors made by an ensemble's components.

Boosting is a very powerful alternative method that merits its own chapter to address a range of recent developments. We will illustrate how bagging effectively reduces the variance, and learn how to configure, train, and tune random forests. We will see how random forests as an ensemble of a large number of decision trees, can dramatically reduce prediction errors, at the expense of some loss in interpretation. 

In short, in this chapter, we will cover the following:

• How to use decision trees for regression and classification
• How to gain insights from decision trees and visualize the decision rules learned from the data
• Why ensemble models tend to deliver superior results
• How bootstrap aggregation addresses the overfitting challenges of decision trees
• How to train, tune, and interpret random forests