Technical Toolkit Required

We are using Python 3.5+ in this book. You are advised to get Python installed on your machine. We are using Jupyter notebook; installing Anaconda-Navigator is required for executing the codes.

The major libraries used are numpy, pandas, matplotlib, seaborn, scikitlearn, and so on. You are advised to install these libraries in your Python environment. All the codes and datasets have been uploaded at the Github repository at the following link: https://github.com/Apress/supervised-learning-w-python/tree/master/Chapter%203

Before starting with classification “machine learning,” it is imperative to examine the statistical concept of critical region and p-value, which we are studying now. They are useful to judge the significance for a variable out of all the independent variables. A very strong and critical concept indeed!

Hypothesis Testing and p-Value

Imagine a new drug X is launched in the market, which claims to cure diabetes in 90% of patients in 5 weeks. The company tested on 100 patients and 90 of them got cured within 5 weeks. How can we ensure that the drug is indeed effective or if the company is making false claims or the sampling technique is biased?

Hypothesis testing is precisely helpful in answering the questions asked.

In hypothesis testing, we decide on a hypothesis first. In the preceding example of a new drug, our hypothesis is that drug X cures diabetes in 90% of patients in 5 weeks. That is called the null hypothesis and is represented by H₀. In this case, H₀ is 0.9. If the null hypothesis is rejected based on evidence, an alternate hypothesis H₁ needs to be accepted. In this case, H₁ < 0.9. We always start with assuming that the null hypothesis is true.

Then we define our significance level, ɑ. It is a measure for how unlikely we want the results of the sample to be, before we reject the null hypothesis H₀. Refer to Figure 3-1(i) and you will be able to relate to the understanding.

We then define the critical region as “c” as shown in Figure 3-1(ii).

If X represents the number of diabetic patients cured, the critical region is defined as P(X<c) < α where α = 5%. In a 95% confidence interval, there is a 5% chance that the sample will not have the population mean. It is interpreted as, if a sample we are interested in falls in a critical region then we can safely reject the null hypothesis.

This is precisely the reason for 5% or 0.05 being referred to as the significance level. If we want a confidence interval of 99%, then 0.01 is the significance level.

Then the next step is to get the p-value. Refer to Figure 3-1(iii) for better understanding.

Formally out, p-value is the probability of getting a value up to and including the one in the sample in the direction of the critical region. It is a method to check if the results of a sample fall in the critical region of the hypothesis test. So, based on the p-value we take a call to reject or accept the null hypothesis.

Once the p-value is calculated, we analyze if the value falls in the critical region. If the answer we get is yes, we can reject the null hypothesis.

This knowledge of hypothesis testing and p-value is critical as it paves the way for identifying the significant variables. Whenever we train our ML algorithms, along with other results we get a p-value for each of the variables. As a rule of thumb, if the p-value is less than or equal to 0.05, the variable is considered as significant.

In this case, the null hypothesis is that the independent variable is not significant and does not impact the target variable. If the p-value is less than or equal to 0.05, we can reject the null hypothesis and hence can conclude that the variable is indeed significant. In the language of statistics, if the p-value for an independent variable x is less than or equal to 0.05, it suggests strong evidence against the null hypothesis as there is less than a 5% chance of the null hypothesis being correct. Or in other words, the variable x is a significant variable to make a prediction for the target variable y. But it does not mean there is 95% probability for the alternate hypothesis to be correct. To be noted is that p-value is a condition upon the null hypothesis and is unrelated to the alternate hypothesis.

We use the p-value to shortlist the significant variables and compare their respective importance. It is a universally accepted metric to choose significant variables.

We will now proceed to the concepts of classification algorithms in the next section.

Classification Algorithms

In our day-to-day business, we make decisions to either invest in stock or not, send a communication to a customer or not, accept a product or reject it, accept an application or ignore it. The basis of these decisions is some sort of insight we have about our business, our processes, our goals, and the factors which enter into our decision making. At the same time, we do expect a favorable output from this decision of ours.

Supervised classified algorithms are used to generate such insights and help us take that decision. They predict the probability for an event to happen or not. At the same time, depending on the choice of supervised learning algorithm, we get to know the factors which impact the occurrences of such an event.

Formally put, classification algorithms are a branch of supervised ML algorithms which are used to model the probability for a certain class. For example, if we want to perform binary classification, we will be modeling for two classes like pass or fail, healthy or sick, fraudulent or genuine, yes or no, and so on. It can be extended to multiclass classification problems—for example, good/bad/neutral, red/yellow/blue/green, cat/dog/horse/car, and so on.

Like a regression model, there is a target variable and independent variables. The only difference is that the target variable is categorical in nature. The independent variables used to make the predictions can be continuous or categorical, and it depends on the algorithm used for modeling.

The preceding use cases are some of the pragmatic implementations in the industry. A classification algorithm generates a probability score for an event and accordingly the sales/marketing/operations/quality/risk teams can take a business call. Quite a powerful usage and very handy too!

There are quite a few algorithms which serve the purpose and we will be discussing a few in this chapter and rest in Chapter 4.

We are discussing the first five algorithms in this chapter and rest in the next chapter. Let us start the discussion with the logistic regression algorithm in the next section.

Logistic Regression for Classification

In Chapter 2 we learned how to predict the value for a continuous variable like number of customers, sales, rainfall, and so on using linear regression. Now we have to predict whether a customer will visit or not, whether the sale will go up or not, and so on. Using logistic regression, we can model and solve the preceding problems.

Formally put, logistic regression is a statistical model that utilizes logit function to model classification problems, that is, a categorical dependent variable. In the basic form, we model a binary classification problem and refer to it as binary logistic regression. In complex problems, where more than one categories have to be modeled, we use multinomial logistic regression.

Let us understand logistic regression by means of an example.

Consider we have to make a decision whether a credit card transaction is fraudulent or not. The response is binary (Yes or No); if the transaction seems promising it will be accepted, otherwise not. We will have incoming transaction attributes like amount, time of transaction, payment mode, and so on, and we have to make a decision based on them.

For such a problem, logistic regression models the probability of fraud. In the preceding case, the probability of fraud can be

Probability (fraud = Yes | amount)

The value for this probability will lie between 0 and 1. It can be interpreted as follows: given a value of “fraud amount” we can make a prediction for the genuineness of a credit card transaction.

The question arises of how we model such a relationship. If we use a linear equation, we can simply write it as

(Equation 3-1)

If we want to predict for “success” using “amount” by fitting the preceding formula, it can be represented in the form of the following graph. For the smaller values of the fraud amount, we can see that probability is less than zero while for the large values, fraud probability is greater than one. And both of these situations are not possible.

Hence, logistic regression is used to tackle this problem. Logistic regression uses the sigmoid function, which takes input as any real value and gives an output between 0 and 1. The standard logistic regression equation can be represented as in Equation 3-2 and shown in Figure 3-2.

(Equation 3-2)

which can be rewritten as

(Equation 3-3)

The next question that comes to mind is how to fit this equation. Recall in the case of linear regression, we want to make a prediction of the target variable as close to the actual values. A similar approach is followed here too. The fitting of the logistic regression is done using the maximum likelihood function. The likelihood function measures the goodness of fit of a statistical model and for logistic regression sometimes referred to as log-likelihood function. The mathematical proof is beyond the scope of the book.

If we manipulate Equation 3-3 we will get

(Equation 3-4)

If we take a natural log of both the sides,

(Equation 3-5)

In Equation 3-5, the quantity is referred to as odds. It iscalled odds because it is more intuitive as compared to probability, as odds are a common jargon in betting.

The term log is called logit. If we compare with a linear regression equation, we can easily make out that with each unit increase in x, the logit (or sometimes called log-odds) changes by β₁. This value can take any value between 0 and infinity. We can visualize the function in Figure 3-2(ii).

In most of the business problems, we have more than one independent variable and hence we use multinomial logistic regression to solve it. Mathematically, it can be represented as follows:

(Equation 3-6)

But there is a question which still remains unanswered: why do we need logistic regression when we have linear regression with us?

Let’s say a bank is making an assessment of its service quality, based on the customer’s historical transactions and service details. In such a case, the predicted response is going to be positive, negative, or neutral. We code these responses as

If we treat the target variable as a continuous variable, it means we have to predict the actual value of y. But this implies that positive is one less than negative and negative is one less than neutral. And the difference between positive and negative is the same as the difference between negative and neutral. This argument is intrinsically wrong and does not make any practical sense.

Moreover, even if we reduce the number of responses from three to two, let’s say positive and negative only, then too the linear regression might give us some probability score beyond 1 or less than 0, which is mathematically not possible. If we fit the best-found regression line, it still won’t be enough to decide any point by which we can differentiate between the two classes. It will classify some positive as negative and vice versa. Moreover, if we get a score of 0.5 from linear regression, should it be classified as positive or negative? And an outlier can completely mess the outputs for us. Hence it is practically more sensible to use a classification algorithm like logistic regression instead of a linear regression model to solve the problem.

Like linear regression, logistic regression has a few assumptions:

1. 1)

   Being a classification algorithm, the outcome of the logistic regression model is a binary or dichotomous variable like success/fail, yes/no, or zero/one.

    
2. 2)

   There exists a linear relationship between the logit of the outcome and each of the independent variables.

    
3. 3)

   Outliers do not exist or at least there are no significant outliers for continuous variables.

    
4. 4)

   There exists no correlation between the independent variables.

    

An important point to be noted is that the accuracy of the algorithm depends on the training data which has been used to train the algorithm. If the training data is not representative, then the resultant model will not be a robust one. The training data should conform to the data quality standards we discussed in Chapter 1. We will be revisiting this concept in detail in Chapter 5.

Tip

To be able to have a representative dataset, it is advisable to have a minimum of 10 data points for each of the independent variables with reference to their least frequent value. For example, for 20 independent variables and a least frequent outcome of 0.2, we should have (20*10)/0.2 = 1000 data points.

Key points to note about logistic regression:

1. 1)

   The output of a logistic classification model generally is a probability score for an event. It can be used for both binary classification and multi classification problems.

    
2. 2)

   Since the output is probability, it cannot go beyond 1 and cannot be less than 1. And hence the shape of the logistic curve is “S”.

    
3. 3)

   It can handle any number of classes as the target variable as well as both categorical and continuous independent variables.

    
4. 4)

   The maximum likelihood algorithm helps to determine the respective coefficients for the equation. It is not required for the independent variables to be normally distributed or have an equal variance in each group.

    
5. 5)

   is the odds ratio and whenever this value is positive, the chances of success are above 50%.

    
6. 6)

   The interpretation of coefficients is difficult in logistic regression as the relation is not straightforward as in the case of linear regression.

    

Before moving further, it is imperative to carefully examine the accuracy measurement methods. You are advised to be thorough with each of them. A vital component in supervised learning, indeed!

Assessing the Accuracy of the Solution

The objective to create an ML solution is to predict for future events. But before deploying the model to a production environment, it is imperative we measure the performance of the model. Moreover, we generally train multiple algorithms with multiple iterations. We have to choose the best algorithm based on the variously accurate KPIs. In this section, we are studying the most important accuracy assessment criteria.

The most important measures to measure the efficacy of a classification problem are as follows:

1. 1.

   Confusion matrix : One of the most popular methods is confusion matrix. It can be used for both binary and multiclass problems. In its simplest form it is represented as a 2×2 matrix in Figure 3-3.

    

We will now learn about each of the parameters separately:

These are the various measures which are used to check the model’s accuracy. We generally create more than one model using multiple algorithms. And for each algorithm, there are multiple iterations done. Hence, these measures are also used to compare the models and pick and choose the best one.

There is one more point we should be cognizant of. We would always want our systems to be accurate. We want to predict better if the share prices are going to increase or decrease or whether it will rain tomorrow or not. But sometimes, accuracy can be dubious. We will understand it with an example.

For example, while developing a credit card fraud transaction system our business goal is to detect transactions which are fraudulent. Now generally most (more than 99%) of the transactions are not fraudulent. This means that if a model predicts each incoming transaction as genuine, still the model will be 99% accurate! But the model is not meeting its business objective of spotting fraud transactions. In such a business case, recall is the important parameter we should target.

With this, we conclude our discussion of accuracy assessment. Generally, for a classification problem, logistic regression is the very first algorithm we use to baseline. Let us now solve an example of a logistic regression problem.

Case Study: Credit Risk

Business Context: Credit risk is nothing but the default in payment of any loan by the borrower. In the banking sector, this is an important factor to be considered before approving the loan of an applicant. Dream Housing Finance company deals in all home loans. They have presence across all urban, semiurban, and rural areas. Customers first apply for a home loan; after that company validates the customers’ eligibility for the loan.

Business Objective : The company wants to automate the loan eligibility process (real time) based on customer detail provided while filling out the online application form. These details are gender, marital status, education, number of dependents, income, loan amount, credit history, and others. To automate this process, they have given a problem to identify the customer segments that are eligible for loan amounts so that they can specifically target these customers. Here they have provided a partial dataset.

Dataset: The dataset and the code is available at the Github link for the book shared at the start of the chapter. A description of the variables is given in the following:

Let’s start the coding part using logistic regression. We will explore the dataset, clean and transform it, fit a model, and then measure the accuracy of the solution.

Step 2: Load the dataset using read_csv command. The output is shown in the following.

loan_df = pd.read_csv('CreditRisk.csv')

loan_df.head()

Step 4: credit_df = loan_df.drop('Loan_ID', axis =1 ) # dropping this column as it will be 1-1 mapping anyways:

credit_df.head()

Step 5: Next normalize the values of the Loan Value and visualize it too.

credit_df['Loan_Amount_Term'].value_counts(normalize=True)

plt.hist(credit_df['Loan_Amount_Term'], 50)

Step 6: Visualize the data next like a line chart.

plt.plot(credit_df.LoanAmount)

plt.xlabel('Loan Amount')

plt.ylabel('Frequency')

plt.title("Plot of the Loan Amount")

Step 9: Next we will analyze how our variables are distributed.

credit_df.describe().transpose()

You are advised to create box-plot diagrams as we have discussed in Chapter 2.

Step 10: Let us look at the target column, ‘Loan_Status’, to understand how the data is distributed among the various values.

credit_df.groupby(["Loan_Status"]).mean()

Step 12: Check the data types present in the data we have now as shown in the output:

Step 13: Check how the data is balanced. We will get the following output.

prop_Y = credit_df['Loan_Status'].value_counts(normalize=True)

print(prop_Y)

There seems to be a slight imbalance in the dataset as one class is 31.28% and the other is 68.72%.

Step 18: We will now check the summary of the model. The results are as follows:

from scipy import stats

stats.chisqprob = lambda chisq, df: stats.chi2.sf(chisq, df)

print(lg.summary())

Let us interpret the results. The pseudo r-square shows that only 24% of the entire variation in the data is explained by the model. It is really not a good model!

Step 20: We will now filter all the independent variables by significant p-value (p value <0.1) and sort descending by odds ratio. We will get the following output:

log_coef = log_coef.sort_values(by="Odds_ratio", ascending=False)

pval_filter = log_coef['pval']<=0.1

log_coef[pval_filter]

If we analyze the data, we can see that the customers who have credit history 1 have a 97% probability of defaulting the loan while the ones having history of 0 have 98% probability of defaulting.

Similarly, the customers in semiurban areas have odds of 2.50 times to default as compared to others.

Step 22: Once the model is ready and fit, we can use it to make a prediction. But first we have to check the accuracy of the model on the training data using confusion matrix; the output is as follows.

pred_train = log_reg.predict(X_train)

from sklearn.metrics import classification_report,confusion_matrix

mat_train = confusion_matrix(y_train,pred_train)

print("confusion matrix = ",mat_train)

Step 23: Next we will make the prediction for test set and visualize it, and we will get the following output.

pred_test = log_reg.predict(X_test)

mat_test = confusion_matrix(y_test,pred_test)

print("confusion matrix = ",mat_test)

ax= plt.subplot()

ax.set_ylim(2.0, 0)

annot_kws = {"ha": 'left',"va": 'top'}

sns.heatmap(mat_test, annot=True, ax = ax, fmt= 'g',annot_kws=annot_kws); #annot=True to annotate cells

ax.set_xlabel('Predicted labels');

ax.set_ylabel('True labels');

ax.set_title('Confusion Matrix');

ax.xaxis.set_ticklabels(['Not Approved', 'Approved']);

ax.yaxis.set_ticklabels(['Not Approved', 'Approved']);

Step 24: Let us now create the AUC ROC curve and get the AUC score. We will get the following output.

from sklearn.metrics import roc_auc_score

from sklearn.metrics import roc_curve

logit_roc_auc = roc_auc_score(y_test, log_reg.predict(X_test))

fpr, tpr, thresholds = roc_curve(y_test, log_reg.predict_proba(X_test)[:,1])

plt.figure()

plt.plot(fpr, tpr, label='Logistic Regression (area = %0.2f)' % logit_roc_auc)

plt.plot([0, 1], [0, 1],'r--')

plt.xlim([0.0, 1.0])

plt.ylim([0.0, 1.05])

plt.xlabel('False Positive Rate')

plt.ylabel('True Positive Rate')

plt.title('Receiver operating characteristic')

plt.legend(loc="lower right")

plt.savefig('Log_ROC')

plt.show()

Interpretations of the Results: By comparing the training confusion matrix and testing confusion matrix, we can determine the efficacy of the solution as shown in the confusion matrix.

On testing data, the model’s overall accuracy is 85%. Sensitivity or recall is 97% and precision is 84%. The model has a good overall accuracy. However, the model can be improved as we can see that 24 applications were predicted as approved while they were actually not approved.

Additional Notes

You can do a quick visualization for all the variables in using only one command using the following code and as shown in the following graph. It depicts the relationship of loan status with all the variables.

import seaborn as sns

sns.pairplot(credit_df, hue="Loan_Status", palette="husl")

Note

If the testing accuracy is not similar to training accuracy and is significantly lower, it means the model is overfitting. We will study how to tackle this problem in Chapter 5.

Logistic regression is generally the first few algorithms which are used whenever we approach a classification problem. It is fast, easy to comprehend, and compact to handle categorical and continuous data points alike. Hence, it is quite popular. It can also be used to get the significant variables for a problem.

Now that we have examined logistic regression in detail, let us move to the second very important classifier used: naïve Bayes. Don’t go by the word “naïve”; this algorithm is quite robust in making classifications!

Naïve Bayes for Classification

There are many events which are mutually dependent on each other and hence it becomes imperative to understand the relationship: P(A|B) and P(B|A). This is true in the case of business activities too where the sales are dependent on the number of customers visiting the store, a customer will come back for shopping or not will depend on the previous experiences, and so on. Bayes’ theorem helps to model for such factors and make a prediction.

As per Bayes’s rule, if we have two events A and B. Then the conditional probability of A given B can be represented as

(Equation 3-11)

where P(A) and P(B): probability of A and B respectively, P(A|B): probability of A given B and P(B|A): probability of B given A.

For example, if we want to know in finding out a patient’s probability to have a heart disease if they have diabetes. The data we have is as follows: 10% of patients entering the clinic have heart disease while 5% of the patients have diabetes. Among the patients diagnosed with heart disease, 8% are diabetic. Then P(A) = 0.10, P(B) = 0.05, and P(B|A) = 0.08. And using Bayes’ rule, P(A|B) = (0.08×0.1)/0.05 = 0.16.

If we generalize the rule, let A₁ through A_n be a set of mutually exclusive outcomes. The probabilities of the events A are P(A₁) through P(A_n). These are called prior probabilities. Because an information outcome might influence our thinking about the probabilities of any A_i, we need to find the conditional probability P(A_i|B) for each outcome A_i. This is called the posterior probability of A_i.

Using Bayes’ rule, we can say that

(Equation 3-12)

Bayes’ rule says that the posterior is the likelihood times the prior, divided by a sum of likelihood times priors. The denominator in Bayes’ rule is the probability P(B).

(Equation 3-13)

Bayes’ theorem is used for making classification (binary or multiclass), and it is referred to as naïve Bayes. It is called naïve due to a very strong assumption that the variables and features are independent of each other which is generally not true in the real world. Often this assumption is violated and still naïve Bayes tends to perform well. The idea is to factor all available evidence in the form of predictors into the naïve Bayes rule to obtain more accurate probability for class prediction.

As per Bayes’ rule, the naïve Bayes estimates conditional probability (i.e., the probability that something will happen, given that something else has already occurred). For example, if we want to design an email spam filter and find a given mail is spam if there is an appearance of the word “discount.” It is easy to implement, fast, robust, and quite accurate. Because of its ease of use, it is quite a popular technique.

But when some of our independent variables are continuous, we cannot calculate conditional probabilities! And hence in real-life variables, naïve Bayes is extended to Gaussian naïve Bayes.

In Gaussian naïve Bayes, continuous values associated with each attribute or independent variable are assumed to be following a Gaussian distribution. They are also easier to work with, as for the training we would only have to estimate the mean and standard deviation of the continuous variable.

Let us move to the case study to develop the naïve Bayes solution.

Case Study: Income Prediction on Census Data

Business Objective: We have census data and the objective is to predict whether income exceeds 50K/yr for an individual based on the value of other attributes.

Dataset: The dataset and code is available at the Github link shared at the start of this chapter.

Step 5: Check the presence of null values in our dataset, as follows:

census_df.isnull().sum()

The preceding output shows that there is no “null” value in our dataset.

There can be some categorical variables having missing values. We will check that, sometimes they have “?” in place of missing values.

for value in ['workclass','education','marital_status','occupation','relationship','race','sex','native_country','income']:

    print(value,":", sum(census_df[value] == '?'))

The output of the preceding code snippet shows that there are 1836 missing values in the workclass attribute, 1843 missing values in the occupation attribute, and 583 values in the native_country attribute.

Step 7: Before doing missing value handling tasks, we need some summary statistics of our data frame. For this, we can use the describe() method. It can be used to generate various summary statistics, excluding NaN values.

census_df_rev.describe(), as follows:

If all is passed, it means we want to check the summary of all the attributes as follows:

Step 11: Look at the first few lines of our data:

census_df_rev.head()

Step 13: We will have to reindex all the columns and for that we will use the reindex_axis method.

census_df_rev = census_df_rev.reindex_axis(['age', 'workclass_category', 'fnlwgt', 'education_category', 'education_num', 'marital_category', 'occupation_category', 'relationship_category', 'race_category', 'sex_category', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country_category', 'income'], axis= 1) census_df_rev.head(5)

The method has deprecated and hence we will receive this error, as shown in the preceding illustration. Hence, let’s use a newer method and we will not get the error shown.

census_df_rev = census_df_rev.reindex(['age', 'workclass_category', 'fnlwgt', 'education_category', 'education_num', 'marital_category', 'occupation_category',  'relationship_category', 'race_category', 'sex_category', 'capital_gain',  'capital_loss', 'hours_per_week', 'native_country_category', 'income'], axis= 1)

census_df_rev.head(5)

With it we have implemented naïve Bayes using a live dataset. You are advised to understand each of the steps and practice the solution by replicating it.

Naïve Bayes is a fantastic algorithm to practice and use. Bayesian statistics is gathering a lot of attention and its power is being harnessed in research areas a lot. You might have heard the term Bayesian optimization . The beauty of the Bayes’ theorem lies in its simplicity, which is very much visible in our day-to-day life. A straightforward method indeed!

We have covered logistic regression and naïve Bayes so far. Now we will examine one more very widely used classifier called k-nearest neighbor. One of the popular methods, easy to understand and implement—let’s study knn next.

k-Nearest Neighbors for Classification

“Birds of the same feather flock together .” This old adage is perfect for k-nearest neighbors. It is one of the most popular ML techniques where the learning is based on the similarity of data points with each other. knn is a nonparametric model; it does not construct a “model” and the classification is based on a simple majority vote from the neighbors. It can be used for classification where the relationship between attributes and target classes is complex and difficult to understand, and yet items in a class tend to be fairly homogenous on the values of attributes. But it might not be the best choice for an unclean dataset or where the target classes are not distinctively clear. If the target classes are not clearly demarcated, then it leads to an obvious confusion while taking the majority vote. knn can also be used for regression problems to make a prediction for a continuous variable. In the case of regression, the final output will be the average of the values of the neighbors and that average will be assigned to the target variable. Let us examine this visually:

For example, we have some data points represented by circles and plus signs in a vector space diagram as shown in Figure 3-5. There are clearly two classes in this case. The objective is to classify a new data point (marked in yellow) shown in Figure 3-5(ii) and identify which class it belongs to.

This yellow point can be a circle or a plus sign and nothing else. knn will help in this classification by taking a vote of majority from the other data points in the vicinity. And the value of “k” will guide us on how many data points to be considered for voting. Let’s assume we took k = 4. Hence, we will now make a circle with a yellow point as the center. And the circle should be just as big as to enclose only four data points. It is represented in Figure 3-6(i).

The four closest points to the yellow point all belong to circles or we can say that all the neighbors of the unknown yellow point are circles. Hence, with a good confidence level we can predict that the yellow point should belong to the circle. Here the choice was comparatively easy and straightforward. Refer to Figure 3-6(ii), where it is not that simple. Hence, the choice of k plays a very crucial role.

From the steps discussed for k-nearest neighbor, we can clearly understand that the final accuracy depends on the distance matrix used and the value of “k”.

The various distances can be viewed as shown in Figure 3-7.

There are other forms of knn too, which are as follows:

It is now time to create a Python solution using knn, which we are executing next.

Case Study: k-Nearest Neighbor

The dataset to be audited, which consists of a wide variety of intrusions simulated in a military network environment, was provided. It created an environment to acquire raw TCP/IP dump data for a network by simulating a typical US Air Force LAN. The LAN was focused like a real environment and blasted with multiple attacks. A connection is a sequence of TCP packets starting and ending at some time duration between which data flows to and from a source IP address to a target IP address under some well-defined protocol. Also, each connection is labeled either as normal or as an attack with exactly one specific attack type. Each connection record consists of about 100 bytes. For each TCP/IP connection, 41 quantitative and qualitative features are obtained from normal and attack data (3 qualitative and 38 quantitative features).

The class variable has two categories: normal and anomalous.

Business Objective

We have to fit a k-nearest neighbor algorithm to detect network intrusion.

Step 1: Import all the required libraries. We are importing pandas, numpy, matplotlib, seaborn.

import pandas as pd

import numpy as np

import seaborn as sns

import matplotlib.pyplot as plt

%matplotlib inline

Step 2: Import the dataset using read.csv method from pandas. Let’s have a look at the top five rows of the data first as follows:

network_data= pd.read_csv('Network_Intrusion.csv')

network_data.head()

Step 3: Now we will do the regular checkup of the data using info() and describe command.

network_data.info()

network_data.describe().transpose()

Step 4: Now check for the null values. In our dataset there are no null values fortunately.

network_data.isnull().sum()

Note

This dataset does not have any null values; we will study in detail on how to deal with null values, NA, NaN, and so on in Chapter 5.

Step 5: Let’s have a look at the class distribution. And we will visualize it too.

network_data["class"].value_counts(normalize=True)

pd.value_counts(network_data["class"]).plot(kind="bar")

Step 6: There are a few categorical variables in our dataset. We have to convert them to numerical variables using one-hot encoding .

from sklearn.preprocessing import LabelEncoder

label_encoder = LabelEncoder()

network_data['class'] = label_encoder.fit_transform(dataset['class'])

network_data['protocol_type'] = label_encoder.fit_transform(dataset['protocol_type'])

network_data['service'] = label_encoder.fit_transform(dataset['service'])

network_data['flag'] = label_encoder.fit_transform(dataset['flag'])

Step 7: One-hot encoding increases the number of variables in the dataset. Let’s see the number of columns in the dataset with added variables:

network_data.columns

Step 8: Next we will standardize our dataset by using StandardScaler in scikit learn.

from sklearn import preprocessing

from sklearn.preprocessing import StandardScaler

X_std = pd.DataFrame(StandardScaler().fit_transform(network_data))

X_std.columns = network_data.columns

Step 9: Now it is the time to split the data in train and test. We are dividing the data in an 80:20 ratio.

import numpy as np

from sklearn.cross_validation import train_test_split

X = np.array(network_data.ix[:, 1:5]) #Transform data into features

y = np.array(network_data['class']) #Transform data into targets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)

Step 10: The sklearn.cross_validation has deprecated and hence we received this error. Again, try to split in train and test using sklearn.model_selection.

from sklearn.model_selection import train_test_split

# Transform data into features and target

X = np.array(network_data.ix[:, 1:5])

y = np.array(network_data['class'])

# split into train and test

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)

Step 11: Print the shape of the data by print(X_train.shape).

print(y_train.shape)

Step 12: Print the shape of the test data by print(X_test.shape).

print(y_test.shape)

Step 13: We will now train the model using training data and iterate with different values of k=3,5,9.

from sklearn.neighbors import KNeighborsClassifier

from sklearn.metrics import accuracy_score

from sklearn.metrics import recall_score

# instantiate learning model (k = 3)

knn_model = KNeighborsClassifier(n_neighbors = 3)

Fitting the model

knn_model.fit(X_train, y_train)

y_pred = knn_model.predict(X_test) # predict the response

print(accuracy_score(y_test, y_pred)) # Evaluate accuracy

The answer is 0.9902758483826156

knn_model = KNeighborsClassifier(n_neighbors=5) # With k = 5

knn_model.fit(X_train, y_train) # Fitting the model

y_pred = knn_model.predict(X_test) # Predict the response

print(accuracy_score(y_test, y_pred)) # Evaluate accuracy

The answer is 0.9882913276443739

With k = 9

knn_model = KNeighborsClassifier(n_neighbors=9)

knn_model.fit(X_train, y_train) # Fitting the model

y_pred = knn_model.predict(X_test) # Predict the response

print(accuracy_score(y_test, y_pred)) # Evaluate accuracy

The answer is 0.9867037110537805

Step 14: We have tested with three values of k. We will now iterate on multiple values of k. We will run the knn with the no. of neighbors to be 1,3,5…19 and then find the optimal number of neighbors based on the lowest misclassification error.

k_list = list(range(1,20)) # creating odd list of K for KNN

k_neighbors = list(filter(lambda x: x % 2 != 0, k_list)) # subsetting just the odd ones

ac_scores = [] # empty list that will hold accuracy scores

# perform accuracy metrics for values from 1,3,5....19

for k in k_neighbors:

    knn_model = KNeighborsClassifier(n_neighbors=k)

    knn_model.fit(X_train, y_train)

    y_pred = knn_model.predict(X_test)   # predict the response

    scores = accuracy_score(y_test, y_pred)  # evaluate accuracy

    ac_scores.append(scores)

# changing to misclassification error

MSE = [1 - x for x in ac_scores]

# determining best k

optimal_k = k_neighbors[MSE.index(min(MSE))]

print("The optimal number of neighbors is %d" % optimal_k)

Step 15: Let’s print the impact of different values of k on the misclassification error.

import matplotlib.pyplot as plt

# plot misclassification error vs k

plt.plot(k_neighbors, MSE)

plt.xlabel('Number of Neighbors K')

plt.ylabel('Misclassification Error')

plt.show()

Step 16: It turns out that k=3 gives us the best result. Let’s implement it:

#Use k=3 as the final model for prediction

knn = KNeighborsClassifier(n_neighbors = 3)

# fitting the model

knn.fit(X_train, y_train)

# predict the response

y_pred = knn.predict(X_test)

# evaluate accuracy

print(accuracy_score(y_test, y_pred))

print(recall_score(y_test, y_pred))

The accuracy and recall are 0.9902758483826156 and 0.9911944869831547 respectively.

With k = 3, we are getting a very good accuracy and a great recall value too. This model can be considered as the final one to be used.

We developed a solution using knn and are getting good accuracies. k-nearest neighbor is very easy to explain and visualize. Not being very statistics- or mathematics-heavy, even for non–data science users, the method is not very tedious to understand. And this is one of the most important properties of this method. Being a nonparametric method, there is no assumption about the data and hence it requires less data preparation.

We have covered logistic regression, naïve Bayes, and k-nearest neighbor. Generally, when we start any classification solution, we start with these three algorithms to check their accuracy. The next in the series are tree-based algorithms: decision tree and random forest. We have discussed the theoretical concepts about both in Chapter 2. In the next section, we will cover the differences and then implement them on a dataset.

Tree-Based Algorithms for Classification

Recall in Chapter 2 that we studied decision trees to predict the values of a continuous variable. Since decision trees can be used for both classification and regression problems, in this chapter we will study classification solutions using decision tree. The building blocks for a decision tree remain the same as shown in Figure 3-8.

The difference is the process of splitting followed by a classification algorithm which is discussed now.

(Equation 3-17)

• where p and q are the probability of success and failure, respectively, in that node. The logarithm is to the base of 2 here.

• In the preceding example, entropy = –0.4 (log 0.4) – 0.3(log 0.3) – 0.3(log 0.3) = 1.571

• Entropy of a pure node is 0 and can be represented as Figure 3-9.

• Gini coefficient : Gini index can also be used to measure the impurity. The formula to be used is as follows:

  

  (Equation 3-18)

• In the preceding example, Gini index = 1 – (0.4^2 + 0.3^2 + 0.3^2) = 0.660

• Gini of a node containing a single class is 0 because the probability is 1. Like entropy, it also takes maximum value when all the classes in the node have the same probability. The movement of Gini can be represented as in Figure 3-10.

(Equation 3-19)

• where i is the number of classes

• Similar to the other two, its value lies between 0 and 1.

• For the preceding example, classification error index = 1 – max(0.4,0.3,0.3) = 0.6

We can use any of these three splitting methods for classification. There are some common decision tree algorithms which are used for regression and classification problems. Since we have studied both regression and classification concepts, it is a good time to examine different types of decision tree algorithms, which is the next section.

Types of Decision Tree Algorithms

There are some significant decision tree algorithms which are used in the industry. Some of them are suitable for classification problems while some are a better choice for regression solutions. We are discussing all the aspects of the algorithms.

The tree-based algorithms discussed previously are unique in their own way. Some of them are more suitable for classification problems, while some are a better choice for regression problems. CART can be used for both classification and regression problems.

Now it is time to develop a case study using decision tree.

The code and the dataset are available at the Github link shared at the start of this chapter.

We can use the same dataset we have used for the logistic regression problem. The implementation follows after we have created the training and testing data.

Step 4: Get the confusion matrix. To visualize it, you are advised to use the method used in the logistic regression method.

print(confusion_matrix(y_test, y_pred))

ax= plt.subplot()

ax.set_ylim(2.0, 0)

annot_kws = {"ha": 'left',"va": 'top'}

sns.heatmap(mat_test, annot=True, ax = ax, fmt= 'g',annot_kws=annot_kws); #annot=True to annotate cells

ax.set_xlabel('Predicted labels');

ax.set_ylabel('True labels');

ax.set_title('Confusion Matrix');

ax.xaxis.set_ticklabels(['Not Approved', 'Approved']);

ax.yaxis.set_ticklabels(['Not Approved', 'Approved']);

We will now model the same problem using a random forest model. Recall that random forest is an ensemble-based technique where it creates multiple smaller trees using a subset of data. The final decision is based on the voting mechanism by each of the trees. In the last chapter, we have used random forest for a regression problem; here we are using random forest for a classification problem.

Step 2: Predict on test data and plot the confusion matrix.

y_pred = rf_model.predict(X_test)

print(confusion_matrix(y_test, y_pred))

ax= plt.subplot()

ax.set_ylim(2.0, 0)

annot_kws = {"ha": 'left',"va": 'top'}

sns.heatmap(mat_test, annot=True, ax = ax, fmt= 'g',annot_kws=annot_kws); #annot=True to annotate cells

ax.set_xlabel('Predicted labels');

ax.set_ylabel('True labels');

ax.set_title('Confusion Matrix');

ax.xaxis.set_ticklabels(['Not Approved', 'Approved']);

ax.yaxis.set_ticklabels(['Not Approved', 'Approved']);

This is the implementation of a decision tree algorithm and ensemble-based random forest algorithm. Tree-based algorithms are very easy to comprehend and implement. They are generally the first few algorithms which we implement and test the accuracy of the system. Tree-based solutions are recommended if we want to create a quick solution, but they are prone to overfitting. We can use tree pruning or setting a constraint on tree size to overcome overfitting. We will again visit this concept in Chapter 5, in which we will discuss all the techniques to overcome the problem of overfitting in our ML model.

With this, we come to the end of our discussion on tree-based algorithms. In this chapter, we have studied the classification algorithms and implemented them too. These algorithms are quite popular in the industry and powerful enough to help us make a robust ML model. Generally, we test the data on these algorithms at the start and then choose the one which is giving us the best results. And then we tune it further till we achieve the most desirable output. The desirable output may not be the most complex solution but surely will be one which will deliver the desired level of measurement parameters, reproducibility, robustness, flexibility, and ease of deployment. Remember, complexity is not proportional to accuracy. A more complex model does not mean a higher degree of performance!

Summary

Prediction is a powerful tool in our hands. Using these ML algorithms, we can not only take a confident decision we can also ascertain the factors which affect that decision. These algorithms are heavily used across sectors like banking, retail, manufacturing, insurance, aviation, and so on. The uses include fraud detection, quality inspection, churn prediction, loan default prediction, and so on.

You should note that these algorithms are not the only sources of knowledge. A sound exploratory analysis is a prerequisite for a good ML algorithm. And the most important resource is “data” itself. A good-quality and representative dataset is of paramount importance. We have discussed the qualities of a good dataset in Chapter 1.

It is also imperative that a sound business problem has been created from the start. The choice of the target variable should align with the business problem at hand. The training data used to train the algorithm plays a very crucial role, as on it depends the patterns learned by the algorithm. It is important to note that we do measure the performance of the algorithms using the various parameters like precision, recall, AUC, F1 score, and so on. An algorithm which is performing well on training, testing, and validation datasets will be the best algorithm. But still there are a few other parameters based on which we choose the final algorithm which can be deployed into production, which we discuss in Chapter 5.

In Chapter 1, we examined ML, various types, data and attributes of data quality and ML process. In Chapter 2, we studied ML algorithms to model a continuous variable. In this third chapter, we complemented the knowledge with classification algorithms. These first chapters have created a firm base for you to solve most of the business problems in the data science world. Also, you are now ready to take the next step in Chapter 4.

In the first three chapters, we have discussed basic and intermediate algorithms. In the next chapter, we are going to cover much more complex algorithms like SVM, gradient boosting, and neural networks for regression and classification problems. So stay focused!