In this chapter, we studied linear regression, a method that allows us to fit a linear model in a supervised learning setting where we have a number of input features and a single numeric output. Simple linear regression is the name given to the scenario where we have only one input feature, and multiple linear regression describes the case where we have multiple input features. Linear regression is very commonly used as a first approach to solving a regression problem. It assumes that the output is a linear weighted combination of the input features in the presence of an irreducible error component that is normally distributed and has zero mean and constant variance. The model also assumes that the features are independent. The performance of linear regression can be assessed by a number of different metrics from the more standard MSE to others, such as the R² statistic. We explored several model diagnostics and significance tests designed to detect problems from violated assumptions to outliers. Finally, we also discussed how to perform feature selection with stepwise regression and perform regularization using ridge regression and lasso.

Linear regression is a model with several advantages, which include fast and cheap parameter computation and a model that, by virtue of its simple form, is very easy to interpret and draw inferences from. There is a plethora of tests available to diagnose problems with the model fit and perform hypothesis testing to check the significance of the coefficients. In general, as a method, it is considered to be low variance because it is robust to small errors in the data. On the negative side, because it makes very strict assumptions, notably that the output function must be linear in the model parameters, it introduces a high degree of bias, and for general functions that are complex or highly nonlinear this approach tends to fare poorly. In addition, we saw that we cannot really rely on significance testing for coefficients when we move to a high number of input features. This fact, coupled with the independence assumption between features, renders linear regression a relatively poor choice to make when working in a higher dimensional feature space.

In the next chapter, we will study logistic regression, which is an important method used in classification problems.