This chapter describes the bias and variance effects and their pathological cases, which usually appear when training machine learning (ML) models. For example, the high variance effect, also known as overfitting, is a phenomenon in ML where the constructed model explains the examples from the training set but works relatively poorly on the examples that did not participate in the training process. This occurs because while training a model, random patterns will start appearing that are normally absent from the general population. The opposite of overfitting is known as underfitting. This happens when the trained model becomes unable to predict patterns in new data or even in the training data. Such an effect can be the result of a limited training dataset or weak model design.

In this chapter, we will learn how to deal with overfitting by using regularization and discuss the different techniques we can use. We shall also consider the different model performance estimation metrics and how they can be used to detect training problems. Toward the end of this chapter, we shall look at how to find the best hyperparameters for a model by introducing the grid search technique and its implementation in C++.

The following topics will be covered in this chapter:

• Performance metrics for ML models
• Understanding the bias and variance characteristics
• Model selection with the grid search technique