Now that we have explored a wide variety of machine learning algorithms, I am sure you have realized that most of them come with a great number of settings to choose from. These settings or tuning knobs, the so-called hyperparameters, help us to control the behavior of the algorithm when we try to maximize performance.

For example, we might want to choose the depth or split criterion in a decision tree or tune the number of neurons in a neural network. Finding the values of important parameters of a model is a tricky task but necessary for almost all models and datasets.

In this chapter, we will dive deeper into model evaluation and hyperparameter tuning. Assume that we have two different models that might apply to our task, how can we know which one is better? Answering this question often involves repeatedly fitting different versions of our model to different subsets of the data, such as in cross-validation and bootstrapping. In combination with different scoring functions, we can obtain reliable estimates of the generalization performance of our models.

But what if two different models give similar results? Can we be sure that the two models are equivalent, or is it possible that one of them just got lucky? How can we know whether one of them is significantly better than the other? Answering these questions will lead us to discuss some useful statistical tests such as Students t-test and McNemar's test.

In this chapter, we will cover the following topics:

• Evaluating a model
• Understanding cross-validation
• Estimating robustness using bootstrapping
• Tuning hyperparameters with grid search
• Scoring models using different evaluation metrics
• Chaining algorithms together to form a pipeline