In simple terms, linear regression tries to fit as many of the data points as possible with a line in two-dimensional space or a plane in three-dimensional space, and so on. It explores the linear relationship between observations and targets and the relationship is represented in a linear equation or weighted sum function. Given a data sample x with n features, x₁, x₂, …, x_n (x represents a feature vector and x = (x₁, x₂, …, x_n)), and weights (also called coefficients) of the linear regression model w (w represents a vector (w₁, w₂, …, w_n)), the target y is expressed as follows:

Also, sometimes, the linear regression model comes with an intercept (also called bias) w₀, so the preceding linear relationship becomes as follows:

Doesn't it look familiar? The logistic regression algorithm we learned in Chapter 7, Predicting Online Ads Click-through with Logistic Regression, is just an addition of logistic transformation on top of the linear regression, which maps the continuous weighted sum to 0 (negative) or 1 (positive) class. Similarly, a linear regression model, or specifically, its weight vector w is learned from the training data, with the goal of minimizing the estimation error defined as mean squared error (MSE), which measures the average of squares of difference between the truth and prediction. Give m training samples, (x⁽¹⁾,y⁽¹⁾), (x⁽²⁾,y⁽²⁾),… (x⁽ⁱ⁾,y⁽ⁱ⁾)…, (x^(m),y^(m)), the cost function J(w) regarding the weights to be optimized is expressed as follows:

Here,  is the prediction.

Again, we can obtain the optimal w so that J(w) is minimized using gradient descent. The first-order derivative, the gradient ∆w, is derived as follows:

Combined with the gradient and learning rate η, the weight vector w can be updated in each step as follows:

After a substantial number of iterations, the learned w is then used to predict a new sample x' as follows: