Technical Toolkit Required

We are going to use Python 3.5 or above in this book. You are advised to get Python installed on your machine. We will be using Jupyter notebook; installing Anaconda-Navigator is required for executing the codes. All the datasets and codes have been uploaded to the Github repository at https://github.com/Apress/supervised-learning-w-python/tree/master/Chapter2 for easy download and execution.

The major libraries used are numpy, pandas, matplotlib, seaborn, scikit learn, and so on. You are advised to install these libraries in your Python environment.

Let us go into the regression analysis and examine the concepts in detail!

Regression analysis and Use Cases

Regression analysis is used to estimate the value of a continuous variable. Recall that a continuous variable is a variable which can take any numerical value; for example, sales, amount of rainfall, number of customers, number of transactions, and so on. If we want to estimate the sales for the next month or the number of passengers expected to visit the terminal in the next week or the number of customers expected to make bank transactions, we use regression analysis.

Simply put, if we want to predict the value of a continuous variable using supervised learning algorithms, we use regression methods. Regression analysis is quite central to business solving and decision making. The predicted values help the stakeholders allocate resources accordingly and plan for the expected increase/decrease in the business.

We will study the first three algorithms in this chapter and the rest in Chapter 4. We are starting with linear regression in the next section.

What Is Linear Regression

Recall from the section in Chapter 1 where we discussed the house price prediction problem using area, number of bedrooms, balconies, location, and so on. Figure 2-1(i) represents the representation of data in a vector-space diagram, while on the right in Figure 2-1(ii) we have suggested an ML equation termed as the line of best fit to explain the randomness in the data and predict the prices.

The objective of the linear regression analysis is to measure this relationship and arrive at a mathematical equation for the relationship. The relationship can be used to predict the values for unseen data points. For example, in the case of the house price problem, predicting the price of a house will be the objective of the analysis.

Formally put, linear regression is used to predict the value of a dependent variable based on the values of at least one independent variable. It also explains the impact of changes in an independent variable on the dependent variable. The dependent variable is also called the target variable or endogenous variable. The independent variable is also termed the explanatory variable or exogenous variable.

Linear regression has been in existence for a long time. Though there are quite a few advanced algorithms (some of which we are discussing in later chapters), still linear regression is widely used. It serves as a benchmark model and generally is the first step to study supervised learning. You are advised to understand and examine linear regression closely before you graduate to higher-degree algorithms.

Let us say we have a set of observations of x and Y where x is the independent variable and Y is the dependent variable. Equation 2-1 describes the linear relation between x and Y:

(Equation 2-1)

where

Y_i = Dependent variable or the target variable which we want to predict

x_i = Independent variable or the predicting variables used to predict the value of Y

β₀ = Population Y intercept. It represents the value of Y when the value of x_i is zero

β₁ = Population slope coefficient. It represents the expected change in the value of Y by a unit change in x_i

ε = random error term in the model

Sometimes, β₀ and β₁ are called the population model coefficients.

In the preceding equation, we are suggesting that the changes in Y are assumed to be caused by changes in x. And hence, we can use the equation to predict the values of Y. The representation of the data and Equation 2-1 will look like Figure 2-2.

These model coefficients (β₀ and β₁) have an important role to play. Y intercept (β₀) is the value of dependent variable when the value of independent variable is 0, that is, it is the default value of dependent variable. Slope (β₁) is the slope of the equation. It is the change expected in the value of Y with unit change in the value of x. It measures the impact of x on the value of Y. The higher the absolute value of (β₁), the higher is the final impact.

Figure 2-2 also shows the predicted values. We can understand that for the value of x for observation i, the actual value of dependent variable is Y_i and the predicted value or estimated value is .

There is one more term here, random error , which is represented by ε. After we have made the estimates, we would like to know how we have done, that is, how far the predicted value is from the actual value. It is represented by random error, which is the difference between the predicted and actual value of Y and is given by . It is important to note that the smaller the value of this error, the better is the prediction.

There is one more important point to ponder upon here. There can be multiple lines which can be said to represent the relationship. For example, in the house price prediction problem, we can use many equations to determine the relationship as shown in Figure 2-3(ii) in different colors.

Hence, it turns out that we have to find out the best mathematical equation which can minimize the random error and hence can be used for making the predictions. This is achieved during training of the regression algorithm. During the process of training the linear regression model, we get the values of β₀ and β₁, which will minimize the error and can be used to generate the predictions for us.

The linear regression model makes some assumptions about the data that is being used. These conditions are a litmus test to check if the data we are analyzing follows the requirements or not. More often, data is not conforming to the assumptions and hence we have to take corrective measures to make it better. These assumptions will be discussed now.

Assumptions of Linear Regression

The linear regression has a few assumptions which need to be fulfilled. We check these conditions and depending on the results, decide on next steps:

1. 1.
   Linear regression needs the relationship between dependent and independent variables to be linear. It is also important to check for outliers since linear regression is sensitive to outliers. The linearity assumption can best be tested with scatter plots. In Figure 2-4, we can see the meaning of linearity vs. nonlinearity. The figure on the left has a somewhat linear relationship where the one on the right does not have a linear relationship.

   

    
2. 2.

   Linear regression requires all the independent variables to be multivariate normal. This assumption can be tested using a histogram or a Q-Q plot. The normality can be checked with a goodness-of-fit test like the Kolmogorov-Smirnov test. If the data is not normally distributed, a nonlinear transformation like log transformation can help fix the issue.

    
3. 3.
   The third assumption is that there is little or no multicollinearity in the data. When the independent variables are highly correlated with each other, it gives rise to multicollinearity. Multicollinearity can be tested using three methods:

   1. a.

      Correlation matrix: In this method, we measure Pearson’s bivariate correlation coefficient between all the independent variables. The closer the value to 1 or –1, the higher is the correlation.

       
   2. b.

      Tolerance: Tolerance is derived during the regression analysis only and measures how one independent variable is impacting other independent variables. It is represented as (1 – R²). We are discussing R² in the next section If the value of tolerance is < 0.1, there are chances of multicollinearity present in the data. If it is less than 0.01, then we are sure that multicollinearity is indeed present.

       
   3. c.

      Variance Inflation Factor (VIF) : VIF can be defined as the inverse of tolerance (1/T). If the value of VIF is greater than 10, there are chances of multicollinearity present in the data. If it is greater than 100, then we are sure that multicollinearity is indeed present.

      Note Centering the data (deducting mean from each score) helps to solve the problem of multicollinearity. We will examine this in detail in Chapter 5!

       
    
4. 4.

   Linear regression assumes that there is little or no autocorrelation present in the data. Autocorrelation means that the residuals are not independent of each other. The best example for autocorrelated data is in a time series data like stock prices. The prices of stock on T_n+1 are dependent on T_n. While we use a scatter plot to check for the autocorrelation, we can use Durbin-Watson’s “d-test” to check for the autocorrelation. Its null hypothesis is that the residuals are not autocorrelated. If the value of d is around 2, it means there is no autocorrelation. Generally d can take any value between 0 and 4, but as a rule of thumb 1.5 < d < 2.5 means there is no autocorrelation present in the data. But there is a catch in the Durbin-Watson test. Since it analyzes only linear autocorrelation and between direct neighbors only, many times, scatter plot serves the purpose for us.

    
5. 5.

   The last assumption in a linear regression problem is homoscedasticity, which means that the residuals are equal over the regression line. While a scatter plot can be used to check for homoscedasticity, the Goldfeld-Quandt test can also be used for heteroscedasticity. This test divides the data into two groups. Then it tests if the variance of residuals is similar across the two groups. Figure 2-5 shows an example of residuals which are not homoscedastic. In such a case, we can do a nonlinear correction to fix the problem. As we can see in Figure 2-5, heteroscedasticity results in a distinctive cone shape in the residual plots.

    

These assumptions are vital to be tested and more often we transform our data and clean it. It is imperative since we want our solution to work very well and have a good prediction accuracy. An accurate ML model will have a low loss. This accuracy measurement process for an ML model is based on the target variable. It is different for classification and regression problems. In this chapter, we are discussing regression base models while in the next chapter we will examine the classification model’s accuracy measurement parameters. That brings us to the important topic of measuring the accuracy of a regression problem, which is the next topic of discussion.

Measuring the Efficacy of Regression Problem

There are different ways to measure the robustness of a regression problem. The Ordinary least-squares (OLS) method is one of the most used and quoted ones. In this method, β₀ and β₁ are obtained by finding their respective values which minimize the sum of the squared distance between Y and , which are nothing but the actual and predicted values of the dependent variable as shown in Figure 2-6. This is referred to as the loss function, which means the loss incurred in making a prediction. Needless to say, we want to minimize this loss to get the best model. The errors are referred as residuals too.

A very important point: why do we take a square of the errors? The reason is that if we do not take the squares of the errors, the positive and negative terms can cancel each other. For example, if the error₁ is +2 and error₂ is -2, then net error will be 0!

From Figure 2-6, we can say that the

(Equation 2-2)

The estimated slope coefficient is

(Equation 2-3)

The estimated intercept coefficient is

(Equation 2-4)

A point to be noted is that the regression line always passes through the mean and .

Based on the preceding discussion, we can study measures of variation. The total variation is made of two parts, which can be represented by the following equation:

Where : Average value of dependent variable

Y_i: Observed value of the dependent variable

: Predicted value of y for the given value of xi

In the preceding equation, SST is the total sum of squares. It measures the variation of y_i around their mean . SSR is a regression sum of squares, which is the explained variation attributable to the linear relationship between x and y. SSE is the error sum of squares, which is the variation attributable to factors other than the linear relationship between x and y. The best way to understand this concept is by means of Figure 2-7.

(Equation 2-5)

(Equation 2-6)

1. 3.

   Root MSE: Root MSE is the square root of the average squared error and is represented by Equation 2-7.

    

(Equation 2-7)

1. 4.

   R square (R²): It represents how much randomness in the data is being explained by our model. In other words, out of the total variation in our data how much is our model able to decipher.

   R² = SSR/SST = Regression sum of squares/Total sum or squares

   R² will always be between 0 and 1 or 0% and 100%. The higher the value of R², the better it is. The way to visualize R² is depicted in the following figures. In Figure 2-8, the value of R² is equal to 1, which means 100% of the variation in the value of Y is explainable by x. In Figure 2-9, the value of R² is between 0 and 1, depicting that some of the variation is understood by the model. In the last case, Figure 2-9, R² = 0, which shows that no variation is understood by the model. In a normal business scenario, we get R² between 0 and 1, that is, a portion of the variation is explained by the model.

   

   

    
2. 5.

   Pseudo R square : It extends the concept of R square. It penalizes the value if we include insignificant variables in the model. We can calculate pseudo R² as in Equation 2-8.

    

(Equation 2-8)

where n is the sample size and k is the number of independent variables

Using R square, we measure all the randomness explained in the model. But if we include all the independent variables including insignificant ones, we cannot claim that the model is robust. Hence, we can say pseudo R square is a better representation and measure to measure the robustness of a model.

Now we are clear on the assumptions of a regression model and how we can measure the performance of a regression model. Now let us study types of linear regression and then develop a Python solution.

Linear regression can be studied in two formats: simple linear regression and multiple linear regression.

Simple linear regression is, as its name implies, simple to understand and easy to use. It is used to predict the value of the target variable using only one independent variable. As in the preceding example, if we have only the house area as an independent variable to predict the house prices, it is an example of simple linear regression. The sample data have only the area and the price is shown in the following. In Figure 2-10, we have the scatter plot of the same data.

Square feet (x)	Price (in 1000 $)
1200	100
1500	120
1600	125
1100	95
1500	125
2200	200
2100	195
1410	110

We can represent the data in Equation 2-9.

(Equation 2-9)

Here price is Y (dependent variable) and area is the x variable (independent variable). The goal of the simple linear regression will be to estimate the values of β₀ and β₁ so that we can predict the values of price.

There is an important limitation to linear regression. We cannot use it to make predictions beyond the limit of the variables we have used in training. For example, in the preceding data set we cannot model to predict the house prices for areas above 2200 square feet and below 1100 square feet. This is because we have not trained the model for such values. Hence, the model is good enough for the values between the minimum and maximum limits only; in the preceding example the valid range is 1100 square feet to 2200 square feet.

We will be creating two examples for simple linear regression. In the first example, we are going to generate a simple linear regression problem. Then in example 2, simple linear regression problem will be solved. Let us proceed to example 1 now.

Example 1: Creating a Simple Linear Regression

We are going to create a simple linear regression by generating a dataset. This code snippet is only a warm-up exercise and will familiarize you with the simplest form of simple linear regression.

Step 1: Import the necessary libraries in the first step. We are going to import numpy, pandas, matplotlib, and scikit learn.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

from sklearn.linear_model import LinearRegression

from sklearn.datasets import make_regression

Step 2: In the next step, we are going to create a sample dataset which can be used for regression.

X,Y = make_regression(n_features=1, noise=5, n_samples=5000)

In the preceding code, n_features is the number of features we want to have in the dataset. n_samples is the number of samples we want to generate. Noise is the standard deviation of the gaussian noise which gets applied to the output.

Step 3: Plot the data using matplotlib library. Xlabel and ylabel will give the labels to the figure.

plt.xlabel('Feature - X')

plt.ylabel('Target - Y')

plt.scatter(X,Y,s=5)

Step 4: Initialize an instance of linear regression now. The name of the variable is linear_model

linear_model = LinearRegression()

Step 5: Fit the linear regression now. The input independent variable is X and target variable is Y

linear_model.fit(X,Y)

Step 6: The model is trained now. Let us have a look at the coefficient values for both the intercept and the slope of the linear regression model

linear_model.coef_

linear_model.intercept_

The value for the intercept is 0.6759344 and slope is 33.1810

Step 7: We use this trained model to predict the value using X and then plot it

pred = linear_model.predict(X)

plt.scatter(X,Y,s=25, label="training")

plt.scatter(X,pred,s=25, label="prediction")

plt.xlabel('Feature - X')

plt.ylabel('Target - Y')

plt.legend()

plt.show()

Blue dots represent maps to actual target data while orange dots represent the predicted values.

We can see that how close the training and actual predicted values are. Now, we will create a solution for simple linear regression using a dataset.

Example 2: Simple Linear Regression for Housing Dataset

We have a dataset which has one independent variable (area in square feet), and we have to predict the prices. It is again an example of simple linear regression, that is, we have one input variable. The code and dataset are uploaded to the Github link shared at the start of this chapter.

Step 1: Import all the required libraries here:

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

import warnings

warnings.filterwarnings(action="ignore", module="scipy", message="^internal gelsd")

Step 2: Load the data set using pandas function:

house_df= pd.read_csv('House_data_LR.csv')

Note

It is a possibility that data is present in the form of .xls or .txt file. Sometimes, the data is to be loaded from the database directly by making a connection to the database.

house_df.head()

Step 3: Check if there is any null value present in the dataset:

house_df.isnull().any()

Step 4: There is a variable which does not make sense. We are dropping the variable “Unnamed”:

house_df.drop('Unnamed: 0', axis = 1, inplace = True)

Step 5: After dropping the variable, let’s have a look at the first few rows in the dataset:

house_df.head()

Step 6: We will now prepare the dataset for model building by separating the independent variable and target variable.

X = house_df.iloc[:, :1].values

y = house_df.iloc[:, -1].values

Step 7: The data is split into train and test now.

Train/Test Split: Creating a train and test dataset involves splitting the dataset into training and testing sets respectively, which are mutually exclusive. After which, you train with the training set and test with the testing set. This will provide a more accurate evaluation of out-of-sample accuracy because the testing dataset is not part of the dataset that has been used to train the data. It is more realistic for real-world problems.

This means that we know the outcome of each data point in this dataset, making it great to test with! And since this data has not been used to train the model, the model has no knowledge of the outcome of these data points. So, in essence, it is truly an out-of-sample testing. Here test data is 25% or 0.25.

Random state, as the name suggests, is for initializing the internal random number generator, which in turn decides the splitting the train/test indices. Keeping it fixed allows us to replicate the same train/test split and hence in verification of the output.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 5)

Step 8: Fit the data now using the linear regression model:

from sklearn.linear_model import LinearRegression

simple_lr= LinearRegression()

simple_lr.fit(X_train, y_train)

Step 9: The model is trained now. Let us use it to predict on the test data

y_pred = simple_lr.predict(X_test)

Step 10: We will first test the model on the training data. We will try to predict on training data and visualize the results on it.

plt.scatter(X_train, y_train, color = 'r')

plt.plot(X_train, simple_lr.predict(X_train), color = 'b')

plt.title('Sqft Living vs Price for Training')

plt.xlabel('Square feet')

plt.ylabel('House Price')

plt.show()

Step 11: Now let us test the model on the testing data. It is the correct measurement to check how robust the model is.

plt.scatter(X_test, y_test, color = 'r')

plt.plot(X_train, simple_lr.predict(X_train), color = 'b')

plt.title('Sqft Living vs Price for Test')

plt.xlabel('Square feet')

plt.ylabel('House Price')

Step 12: Now let’s figure out how good or how bad we are doing in the predictions. We will calculate the MSE and R².

from sklearn.metrics import mean_squared_error

from math import sqrt

rmse = sqrt(mean_squared_error(y_test, y_pred))

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_pred)

adj_r2 = 1 - float(len(y)-1)/(len(y)-len(simple_lr.coef_)-1)*(1 - r2)

rmse, r2, adj_r2, simple_lr.coef_, simple_lr.intercept_

Step 13: We will now make a prediction on unseen value of x:

import numpy as np

x_unseen=np.array([1500]).reshape(1,1)

simple_lr.predict(x_unseen)

The prediction is 376666.84

In the preceding two examples, we saw how we can use simple linear regression to train a model and make a prediction. In real-world problems, only one independent variable will almost never happen. Most business world problems have more than one variable, and such problems are solved using multiple linear regression algorithms which we are discussing now.

Multiple linear regression or multiple regression can be said as an extension of simple linear regression where instead of one independent variable we have more than one independent variable. For example, Figure 2-11 shows the representation of more than one variable of a similar dataset in the vector-space diagram:

Square Feet	No of bedrooms	Price (in 1000 $)
1200	2	100
1500	3	120
1600	3	125
1100	2	95
1500	2	125
2200	4	200
2100	4	195
1410	2	110

Hence the equation for multiple linear regression is shown in Equation 2-10.

(Equation 2-10)

Hence in the case of a multiple training of the simple linear regression, we will get an estimated value for β₀ and β₁ and so on.

The residuals in the case of a multiple regression model are shown in Figure 2-12. The example is for a two-variable model, and we can clearly visualize the value of residual, which is the difference between actual and predicted values.

We will now create two example cases using multiple linear regression models. During the model development, we are going to do EDA, which is the first step and will also resolve the problem of null values in the data and how to handle categorical variables too.

Example 3: Multiple Linear Regression for Housing Dataset

We are working on the house price dataset. The target variable is the prediction of house prices and there are some independent variables. The dataset and the codes are uploaded to the Github link shared at the start of this chapter.

Step 1: Import all the required libraries first.

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline

import warnings

warnings.filterwarnings(action="ignore", module="scipy", message="^internal gelsd")

Step 2: Import the data which is in the form of .csv file. Then check the first few rows.

 house_mlr = pd.read_csv('House_data.csv')

house_mlr.head()

We have 21 variables in this dataset.

Step 3: Next let’s explore the dataset we have. This is done using the house_mlr.info() command.

By analyzing the output we can see that out of the 21 variables, there are a few float variables, some object and some integers. We will be treating these categorical variables to integer variables.

Step 4: house_mlr.describe() command will give the details about all the numeric variables.

Here we can see the range for all the numeric variables: the mean, standard deviation, the values at the 25th percentile, 50th percentile, and 75th percentile. The minimum and maximum values are also shown.

A very good way to visualize the variables is using a box-and-whisker plot using the following code.

fig = plt.figure(1, figsize=(9, 6))

ax = fig.add_subplot(111)

ax.boxplot(house_mlr['sqft_living15'])

The plot shows that there are a few outliers. In this case, we are not treating the outliers. In later chapters, we shall examine the best practices to deal with outliers.

Step 5: Now we are going to check for the correlations between the variables. This will be done using a correlation matrix which is developed using the following code:

house_mlr.drop(['id', 'date'], axis = 1, inplace = True)

fig, ax = plt.subplots(figsize = (12,12))

ax = sns.heatmap(house_mlr.corr(),annot = True)

The analysis of the correlation matrix shows that there is some correlation between a few variables. For example, between sqft_above and sqft_living there is a correlation of 0.88. And that is quite expected. For this first simple example, we are not treating the correlated variables.

Step 6: Now we will clean the data a little. There are a few null values present in the dataset. We are dropping those null values now. We are examining the concepts of missing value treatment in Chapter 5.

house_mlr.isnull().any()

house_mlr ['basement'] = (house_mlr ['sqft_basement'] > 0).astype(int)

house_mlr ['renovated'] = (house_mlr ['yr_renovated'] > 0).astype(int)

to_drop = ['sqft_basement', 'yr_renovated']

house_mlr.drop(to_drop, axis = 1, inplace = True)

house_mlr.head()

Step 7: The categorical variables are converted to numeric ones using one-hot encoding.

One-hot encoding converts categorical variables to numeric ones. Simply put, it adds new columns to the dataset with 0 or assigned depending on the value of the categorical variable, as shown in the following:

categorical_variables = ['waterfront', 'view', 'condition', 'grade', 'floors','zipcode']

house_mlr = pd.get_dummies(house_mlr, columns = categorical_variables, drop_first=True)

house_mlr.head()

Step 8: We will now split the data into train and test and then fit the model. Test size is 25% of the data.

X = house_mlr.iloc[:, 1:].values

y = house_mlr.iloc[:, 0].values

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 5)

from sklearn.linear_model import LinearRegression

multiple_regression = LinearRegression()

multiple_regression.fit(X_train, y_train)

Step 9: Predict the test set results.

y_pred = multiple_regression.predict(X_test)

Step 10: We will now check the accuracy of our model.

from sklearn.metrics import mean_squared_error

from math import sqrt

rmse = sqrt(mean_squared_error(y_test, y_pred))

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_pred)

adj_r2 = 1 - float(len(y)-1)/(len(y)-len(multiple_regression.coef_)-1)*(1 - r2)

rmse, r2, adj_r2

The output we receive is (147274.98522602883, 0.8366403242154088, 0.8358351477235848).

The steps used in this example can be extended to any example where we want to predict a continuous variable. In this problem, we have predicted the value of a continuous variable but we have not selected the significant variables from the list of available variables. Significant variables are the ones which are more important than other independent variables in making the predictions. There are multiple ways to shortlist the variables. We will discuss one of them using the ensemble technique in the next section. The popular methodology of using p-value is discussed in Chapter 3.

With this, we have discussed the concepts and implementation using linear regression. So far, we have assumed that the relationship between dependent and independent variables is linear. But what if this relation is not linear? That is our next topic: nonlinear regression.

Nonlinear Regression Analysis

Consider this. In physics, we have laws of motion to describe the relationship between a body and all the forces acting upon it and how the motion responds to those forces. In one of the laws, the relationship of initial and final velocity is given by the following equation:

Final velocity = initial velocity + ½ acceleration*time² OR v = u + ½ at²

If we analyze this equation, we can see that the relation between final velocity and time is not linear but quadratic in nature. Nonlinear regression is used to model such relationships. We can review the scatter plot to identify if the relationship is nonlinear as shown in Figure 2-13.

Formally put, if the relationship between target variable and independent variables is not linear in nature, then nonlinear regression is used to fit the data.

The model form for nonlinear regression is a mathematical equation, which can be represented as Equation 2-11 and as the curve in Figure 2-13. The shape of the curve will depend on the value of “n” and the respective values of β₀, β₁…, and so on.

(Equation 2-11)

Where β₀: Y intercept

β₁: regression coefficient for linear effect of X on Y

β₂ = regression coefficient for quadratic effect on Y and so on

ε_i = random error in Y for observation i

Let’s dive deeper and understand nonlinear relationships by taking quadratic equations as an example. If we have a quadratic relationship between the dependent variables, it can be represented by Equation 2-12.

(Equation 2-12)

Where β₀: Y intercept

β₁: regression coefficient for linear effect of X on Y

β₂ = regression coefficient for quadratic effect on Y

ε_i = random error in Y for observation i

The quadratic mode will take the following shapes depending on the values of β₁ and β₂ as shown in Figure 2-14, in which β₁ is the coefficient of the linear term and β₂ is the coefficient of the squared term.

Depending on the values of the coefficients, the shape of the curve will change its shape. An example of such a case is shown in the following data. Figure 2-15 on the right side shows the relationship between two variables: velocity and distance:

Velocity	Distance
3	9
4	15
5	28
6	38
7	45
8	69
10	96
12	155
18	260
20	410
25	650

The data and the respective plot of the data indicate nonlinear relationships. Still, we should know how to detect if a relationship between target and independent variables is nonlinear, which we are discussing next.

Identifying a Nonlinear Relationship

Similar to linear regression, nonlinear models too have some assumptions, which we will discuss now.

Assumptions for a Nonlinear Regression

1. 1.

   The errors in a nonlinear regression follow a normal distribution. Error is the difference between the actual and predicted values and nonlinear requires the variables to follow a normal distribution.

    
2. 2.

   All the errors must have a constant variance.

    
3. 3.

   The errors are independent of each other and do not follow a pattern. This is a very important assumption since if the errors are not independent of each other, it means that there is still some information in the model equation which we have not extracted.

    

We can also use a log transformation to solve some of the nonlinear models. The equation of a log transformation can be put forward as Equation 2-13 and Equation 2-14.

(Equation 2-13)

and by taking a log transformation on both the sides:

(Equation 2-14)

The coefficient of the independent variable can be interpreted as follows: a 1% change in the independent variable X₁ leads to an estimated β₁ percentage change in the average value of Y.

Note

Sometimes, β₁ can also refer to elasticity of Y with respect to change in X₁.

We have studied different types of regression models. It is one of the most stable solutions, but like any other tool or solution, there are a few pitfalls with regression models too. Some of those in the form of important insights will be uncovered while doing EDA. These challenges have to be dealt with while we are doing the data preparation for our statistical modeling . We will now discuss these challenges.

Challenges with a Regression Model

Linear regression is one of widely used techniques to predict the continuous variables. The usages are vast and many and across multiple domains. It is generally the first few methods we use to model a continuous variable and it acts as a benchmark for the other ML models.

This brings us to the end of our discussion on linear regression models. Next we will discuss a quite popular family of ML models called tree models or tree-based models. Tree-based models can be used for both regression and classification problems. In this chapter we will be studying only the regression problems and in the next chapter we will be working on classification problems.

Tree-Based Methods for Regression

The next type of algorithms used to solve ML problems are tree-based algorithms . Tree-based algorithms are very simple to comprehend and develop. They can be used for both regression and classification problems. A decision tree is a supervised learning algorithm, hence we have a target variable and depending on the problem it will be either a classification or a regression problem.

A decision tree looks like Figure 2-19. As you can see, the entire population is continuously split into groups and subgroups based on a criterion. We start with the entire population at the beginning of the tree and subsequently the population is divided into smaller subsets; at the same time, an associated decision tree is incrementally developed. Like we do in real life, we consider the most important factor first and then divide the possibilities around it. In a similar way, decision tree building starts by finding the features for the best splitting condition.

Decision trees can be used to predict both continuous variables and categorical variables. In the case of a regression tree, the value achieved by the terminal node is the mean of the values falling in that region. While in the case of classification trees, it is the mode of the observations. We will be discussing both the methods in this book. In this chapter, we are examining regression problems; in the next chapter classification problems are solved using decision trees.

Before moving ahead , it is imperative we study the building blocks for a decision tree. As you can see in Figure 2-20, a decision tree is represented using the root node, decision node, terminal node, and a branch.

Let us now understand the decision tree using the house price prediction problem. For the purpose of understanding, let us assume the first criteria of splitting is area. If the area is less than 100 sq km. then the entire population is split in two nodes as shown in Figure 2-21(i). On the right hand, we can see the next criteria is number of bedrooms. If the number of bedrooms is less than four, the predicted price is 100; otherwise, the predicted price is 150. For the left side the criteria for splitting is distance. And this process will continue to predict the price values.

A decision tree can also be thought as a group of nested IF-ELSE conditions (as shown in Figure 2-21(ii)) which can be modeled as a tree wherein the decisions are made in the internal node and the output is obtained in the leaf node. A decision tree divides the independent variables into non-overlapping spaces. These spaces are distinct to each other too. Geometrically, a decision tree can also be thought of as a set of parallel hyperplanes which divide the space into a number of hypercuboids as shown in Figure 2-21(iii). We predict the value for the unseen data based on which hypercuboid it falls into.

Now you have understood what a decision tree is. Let us also examine the criteria of splitting a node.

A decision tree utilizes a top-down greedy approach . As we know, a decision tree starts with the entire population and then recursively splits the data; hence it is called top-down. It is called a greedy approach, as the algorithm at the time of decision of split takes the decision for the current split only based on the best available criteria, that is, variable and not based on the future splits, which may result in a better model. In other words, for greedy approaches the focus is on the current split only and not the future splits. This splitting takes place recursively unless the tree is fully grown and the stopping criteria is reached. In the case of a classification tree, there are three methods of splitting: Gini index, entropy loss, and classification error. Since they deal with classification problems, we will study these criteria in the next chapter.

For a regression tree, variance reduction is the criteria for splitting. In variance splitting, variance at each node is calculated using the following formula.

(Equation 2-15)

We calculate variance for each split as the weighted average of variance for each node. And then the split with a lower variance is selected for the purpose of splitting.

There are quite a few decision tree algorithms available, like ID3, CART, C4.5, CHAID, and so on. These algorithms are explored in more detail in Chapter 3 after we have discussed concepts of classification using a decision tree.

Case study: Petrol consumption using Decision tree

It is time to develop a Python solution using a decision tree. The code and dataset are uploaded to the Github repository. You are advised to download the dataset from the Github link shared at the start of the chapter.

The MAE for our algorithm is 50.9, which is less than 10% of the mean of all the values in the 'Petrol_Consumption' column, indicating that our algorithm is doing a good job.

The visualization of the preceding solution is the code is checked in at GitHub link.

This is the Python implementation of a decision tree. The code can be replicated for any problem we want to solve using a decision tree. We will now explore the pros and cons of decision tree algorithms.

We have discussed the concepts of decision tree and developed a case study using Python. It is very easy to comprehend, visualize, and explain. Everyone is able to relate to a decision tree, as it works in the way we make our decisions. We choose the best parameter and direction, and then make a decision on the next step. Quite an intuitive approach!

This brings us towards the end of decision tree–based solutions. So far we have discussed simple linear regression, multinomial regression, nonlinear regression, and decision tree. We understood the concepts, the pros and cons, and the assumptions, and we developed respective solutions using Python. It is a very vital and relevant step towards ML.

But all of these algorithms work individually and one at a time. It allows us to bring forward the next generation of solutions called ensemble methods, which we will examine now.

Ensemble Methods for Regression

“United we stand” is the motto for ensemble methods. They use multiple predictors and then “unite” or collate the information to make a final decision.

Formally put, ensemble methods train multiple predictors on a dataset. These predictor models might or might not be weak predictors themselves individually. They are selected and trained in such a way that each has a slightly different training dataset and may get slightly different results. These individual predictors might learn a different pattern from each other. Then finally, their individual predictions are combined and a final decision is made. Sometimes, this combined group of learners is referred to as meta model.

In ensemble methods, we ensure that each predictor is getting a slightly different data set for training. This is usually achieved at random with replacement or bootstrapping. In a different method, we can adjust the weights assigned to each of the data points. This increases the weights, that is, the focus on those data points.

To visualize how ensemble methods make a prediction, we can consider Figure 2-22 where a random forest is shown and predictor models are trained on slightly different datasets.

It is time for us to develop a solution using random forest. We will be exploring more on boosting in Chapter 4, where we study supervised classification algorithms.

Case study: Petrol consumption using Random Forest

For a random forest regression problem, we will be using the same case study we used for decision tree. In the interest of space, we are progressing after creating the training and testing dataset

As we can observe, after selecting the significant variables the error has reduced for random forest.

Ensemble learning allows us to collate the power of multiple models and then make a prediction. These models individually are weak but together act as a strong model for prediction. And that is the beauty of ensemble learning. We will now discuss pros and cons of ensemble learning.

We have covered the concept of ensemble learning and developed a regression solution using random forest. Ensemble learning has been popular for a long time and has won quite a few competitions in Kaggle. You are advised to understand the concepts and replicate the code implementation.

Before we close the discussion on ensemble learning, we have to study an additional concept of feature selection using decision tree. Recall from the last section where we developed a multiple regression problem; we will be continuing with the same problem to select the significant variables.

Feature Selection Using Tree-Based Methods

We are continuing using the dataset we used in the last section where we developed a multiple regression solution using house data. We are using ensemble-based ExtraTreeClassifier to select the most significant features.

The initial steps of importing the libraries and dataset remain the same.

The output shows that the total number of variables was 30; from that list, 11 variables have been found to be significant.

Using ensemble learning–based ExtraTreeClassifier is one of the techniques to shortlist significant variables. We can look at the respective p-values and shortlist the variables.

Ensemble learning is a very powerful method to combine the power of weak predictors and make them strong enough to make better predictions. It offers a fast, easy, and flexible solution which is applicable for both classification and regression problems. Owing to their flexibility sometimes we encounter the problem of overfitting, but bagging solutions like random forest tend to overcome the problem of overfitting. Since ensemble techniques promote diversity in the modeling approach and use a variety of predictors to make the final decision, many times they have outperformed classical algorithms and hence have gained a lot of popularity.

This brings us to the end of ensemble learning methods. We are going to revisit them in Chapter 3 and Chapter 4.

So far, we have covered simple linear regression, multiple linear regression, nonlinear regression, decision tree, and random forest and have developed Python codes for them too. It is time for us to move to the summary of the chapter.

Summary

We are harnessing the power of data in more innovative ways. Be it through reports and dashboards, visualizations, data analysis, or statistical modeling, data is powering the decisions of our businesses and processes. Supervised regression learning is quickly and quietly impacting the decision-making process. We are able to predict the various indicators of our business and take proactive measures for them. The use cases are across pricing, marketing, operations, quality, Customer Relationship Management (CRM), and in fact almost all business functions.

And regression solutions are a family of such powerful solutions. Regression solutions are very profound, divergent, robust, and convenient. Though there are fewer use cases of regression problems as compared to classification problems, they still serve as the foundation of supervised learning models. Regression solutions are quite sensitive to outliers and changes in the values of target variables. The major reason is that the target variable is continuous in nature.

Regression solutions help in modeling the trends and patterns, deciphering the anomalies, and predicting for the unseen future. Business decisions can be more insightful in light of regression solutions. At the same time, we should be cautious and aware that the regression will not cater to unseen events and values which are not trained for. Events such as war, natural calamities, government policy changes, macro/micro economic factors, and so on which are not planned will not be captured in the model. We should be cognizant of the fact that any ML model is dependent on the quality of the data. And for having an access to clean and robust dataset, an effective data-capturing process and mature data engineering is a prerequisite. Then only can the real power of data be harnessed.

In the first chapter, we learned about ML, data and attributes of data quality, and various ML processes. In this second chapter, we have studied regression models in detail. We examined how a model is created, how we can assess the model’s accuracy, pros and cons of the model, and implementation in Python too. In the next chapter, we are going to work on supervised learning classification algorithms.

You should be able to answer the following questions.