Technical Toolkit Required

We are going to use Python 3.5 or above in this chapter. We will be using Jupyter notebook; installing Anaconda-Navigator is required for executing the codes. All the datasets and codes have been uploaded to the Github library at https://github.com/Apress/supervised-learning-w-python/tree/master/Chapter%205 for easy download and execution.

ML Model Development

Recall that in Chapter 1 we briefly discussed the end-to-end process of model development. We will discuss each of the steps in detail; the most common issues we face with each of them and how we can tackle them are going to be examined in detail now. It will culminate in the model deployment phase. Figure 5-1 is the model development process we follow.

We will be discussing all of the steps in detail, the common issues we face in them, and how to tackle them. The entire process will be complemented by actual Python code.

Step 1: Define the Business Problem

It all starts with the business problem. A business problem has to be concise, clear, measurable, and achievable. It cannot be vague such as “increase profits.” It has to be precisely defined with a clean business objective and along with a KPI which can be measured.

A good business problem is one which is precise, achievable, measurable, and repeatable, as shown in Figure 5-2.

In a nutshell, having a tight scope and a robustly defined business problem paves the way for a successful ML model creation.

We now move to the next step, which is the data discovery phase. This step is a very significant one, which will decide if we can go ahead with the ML model or not!

Step 2: Data Discovery Phase

Data discovery will result in having tables and datasets which are finalized to be used. If there is really a situation where the data is not at all complete, we might decide to not go ahead and complete the dataset instead.

Now it is time to discuss perhaps the most time-consuming of all the steps, which is data cleaning and data preparation.

Step 3: Data Cleaning and Preparation

Perhaps the most-time consuming of all the steps is data cleaning and preparation. It typically takes 60% to 70% of the time along with Step 2.

The data discovery phase will result in creation of datasets which can be used for further analysis. But this does not mean that these datasets are clean and ready to be used. We address them by feature engineering and preprocessing.

Let’s start with the first issue, which is duplicate values in the dataset.

Duplicates in the Dataset

If we have rows in our dataset which are complete copies of each other, then they are duplicates. This means that each value in each of the columns is exactly the same and in the same order too. Such a problem occurs during the data-capturing phase or the data-loading phase. For example, if a survey is getting filled and the same person fills the survey once and then provides exactly the same details again.

An example of duplicate rows is shown in the following Table 5-4. Note that rows 1 and 5 are duplicates.

Table 5-4

Sample Data with Duplicate Rows That Should Be Removed

-

If we have duplicates in our dataset, we cannot trust the performance of the model. If the duplicated records become a part of both the training and testing datasets, the model’s performance will be erroneously elevated and biased. We will conclude that the model is performing well when it is not.

We can identify the presence of duplicates by a few simple commands and drop them from our datasets then.

An example of duplicate rows (row 1 and row 5) is shown in Table 5-4. We will now load a dataset in Python having duplicates and then will treat them. The dataset used is the Iris dataset, which we have used in the exercise sections of Chapters 2 and 3. The code is available at the Github link at https://github.com/Apress/supervised-learning-w-python/tree/master/Chapter%205.

Step 1: Import the library.

from pandas import read_csv

Step 2: Now load the dataset using read_csv command.

data_frame = read_csv("IRIS.csv", header=None)

Step 3: We will now look at the number of rows and columns in the dataset.

print(data_frame.shape)

(151,5) there are 151 rows present in the dataset

Step 4: We will now check if there are duplicates present in the dataset.

duplicates = data_frame.duplicated()

Step 5: We will now print the duplicated rows.

print(duplicates.any())

print(data_frame[duplicates])

From the output we can see that there are three rows which are duplicates and have to be removed.

Step 6: Drop the duplicate rows using the drop_duplicates command and check the shape again.

data_frame.drop_duplicates(inplace=True)

print(data_frame.shape)

The shape is now reduced to 148 rows.

This is how we can check for the duplicates and treat them by dropping from the dataset set.

Duplicates may increase the perceived accuracy of the ML model and hence have to be dealt with. We now discuss the next common concern: categorical variables.

Imbalance in the Dataset

Consider this. A bank offers credit cards to the customers. Out of all the millions of transactions in a day, there are a few fraudulent transactions too. The bank wishes to create an ML model to be able to detect these fraudulent transactions among the genuine ones.

Now, the number of fraudulent transactions will be very few. Maybe less than 1% of all the transactions will be fraudulent. Hence, the training dataset available to us will not have enough representation from the fraudulent transactions. This is called an imbalanced dataset.

The imbalanced dataset can cause serious issues in the prediction model. The ML model will not be able to learn the features of a fraudulent transaction. Moreover it can lead to an accuracy paradox. In the preceding case, if our ML models predict all the incoming transactions as genuine, then the overall accuracy of our model will be 99%! Astonishing, right? But the model is predicting all the incoming transactions as genuine, which is incorrect and defeats the purpose of the ML.

There are a few solutions to tackle the imbalance problem:

1. 1.

   Collect more data and correct the imbalance. This is the best solution which can be implemented and used. When we have a greater number of data points, we can take a data sample having a balanced mix of both the classes in case of a binary model or all the classes in case of a multiclass model.

    
2. 2.
   Oversampling and undersampling methods: we can oversample the class which is under-represented OR undersample the class which is over-represented. These methods are very easy to implement and fast to execute. But there is an inherent challenge in both. When we use undersampling or oversampling, the training data might become biased. We might miss some of the important features or data points while we are trying to sample. There are a few rules which we can follow while we oversample or undersample:

   1. a.

      We prefer oversample if we have a smaller number of data points.

       
   2. b.

      We prefer undersample if we have a large data set at our disposal.

       
   3. c.

      It is not necessary to have an equal ratio of all the classes. If we have four classes in the target variable, it is not necessary to have 25% of each class. We can target a smaller proportion too.

       
   4. d.

      Use stratified sampling methods while we take the sample (Figure 5-4). The difference between random sampling and stratified sampling is that stratification takes care of distribution of all the variables we want to be used while stratification takes place. In other words, we will get the same distribution of variables in stratified sample as the overall population which might not be guaranteed in random sampling.

       
    

In the population and the sample, stratification maintains the same ratio. For example, as shown in Figure 5-4 the ratio of male to female is 80:20 in the population. In the 10% sample of the population, male to female should be maintained as 80:20, which is achieved by stratification and not necessarily guaranteed by random sampling.

We get the answer as 50%. Hence the dataset has been balanced from less than 1% to 50%.

SMOTE is a fantastic method to treat the class imbalance. It is fast and easy to execute and give good results for numeric values. But SMOTE works erroneously for categorical variables. For example, if we have a categorical variable as “Is_raining”. This variable can only take value in binary (i.e., 0 or 1). SMOTE can lead to a value in decimals for this variable like 0.55, which is not possible. So be cautious while using SMOTE!

With this, we have completed the imbalance dataset treatment. It is one of the most significant challenges we face: it not only impacts the ML model but has an everlasting impact on the accuracy measurement too. We now discuss the outlier issues we face in our datasets.

Outliers in the Dataset

Consider this. In a business the average sales per day is $1000. One fine day, a customer made a purchase of $5000. This information will skew the entire dataset. Or in a weather forecast dataset, the average rainfall for a city is 100 cm. But due to a drought season there was no rain during that season. This will completely change the face of the deductions we will have from this data. Such data points are called outliers.

Outliers impact the ML model adversely. The major issues we face with outliers are as follows:

1. 1.

   Our model equation takes a serious hit in the case of outliers, as visible in Figure 5-5. In the presence of outliers, the regression equation tries to fit them too and hence the actual equation will not be the best one.

    

Hence, we should be cautious while we treat outliers. Blanket removal of outliers is not recommended!

Outliers pose a big challenge to our datasets. They skew the insights we have generated from the data. Our coefficients are skewed and the model gets biased, and hence we should be cognizant of their impact. At the same time, we cannot ignore the relation of the business world with outliers and gauge if the observation is a serious outlier.

Apart from the issues discussed previously, there are quite a few other challenges faced in real-world datasets. We discuss some of these challenges in the next section.

Other Common Problems in the Dataset

We have seen the most common issues we face with our datasets. And we studied how to detect those defects and how to fix them. But real-world datasets are really messy and unclean, and there are many other factors which make a proper analysis fruitful.

This data-cleaning step is not an easy task. It is really a tedious, iterative process. It does test patience and requires business acumen, dataset understanding, coding skills, and common sense. On this step, however, rests the analysis we do and the quality of the ML model we want to build.

Now, we have reached a stage where we have the dataset with us. It might be cleaned to a certain extent if not 100% clean.

We can now proceed to the EDA stage, which is the next topic.

Step 4: EDA

EDA is one of the most important steps in the ML model-building process. We examine all the variables and understand their patterns, interdependencies, relationships, and trends. Only during the EDA phase will we come to know how the data is expected to behave. We uncover insights and recommendations from the data in this stage. A strong visualization complements the complete EDA.

There is a thin line between EDA and the data-cleaning phase. Apparently, the steps overlap between data preparation, feature engineering, EDA, and data-cleaning phases.

EDA has two main heads: univariate analysis and bivariate analysis. Univariate, as the name suggests, is for an individual variable. Bivariate is used for understanding the relationship between two variables.

We will now complete a detailed case using Python. You can follow these steps for EDA. For the sake of brevity, we are not pasting all the results here. The dataset and code are available at the Github link at https://github.com/Apress/supervised-learning-w-python/tree/master/Chapter%205.

Now, we will work on the player of the match and visualize it.

matches_df.player_of_match.value_counts()

matches_df.player_of_match.value_counts()[:10].plot('bar')

matches_df.player_of_match.value_counts()[:10].plot('ba

Deliveries dataset is loaded; check the top five rows.

deliveries_df = pd.read_csv('deliveries.csv')

deliveries_df.head()

Plot the batsmen by their receptive runs scored

batsman_runs = deliveries_df.groupby(['batsman']).batsman_runs.sum().nlargest(10)

batsman_runs.plot(title = 'Top Batsmen', rot = 30)

Same plot as a bar plot can be created using this code.

deliveries_df.groupby(['batsman']).batsman_runs.sum().nlargest(10)

.plot(kind = 'bar',title = 'Top Batsmen', rot = 40, colormap = 'coolwarm')

Let us compare the performance of years 2015 and 2016.

ipl = matches_df[['id', 'season']].merge(deliveries_df, left_on = 'id', right_on = 'match_id').drop('match_id', axis = 1)

runs_comparison = ipl[ipl.season.isin([2015, 2016])].groupby(['season', 'batsman']).batsman_runs.sum().nlargest(20).reset_index().sort_values(by='batsman')

vc = runs_comparison.batsman.value_counts()

batsmen_comparison_df = runs_comparison[runs_comparison.batsman.isin(vc[vc == 2].index.tolist())]

batsmen_comparison_df

batsmen_comparison_df.plot.bar()

batsmen_comparison_df.pivot('batsman', 'season', 'batsman_runs').plot(kind = 'bar', colormap = 'coolwarm')

If we want to check the percentage of matches won and visualize it as a pie chart, we can use the following:

match_winners = matches_df.winner.value_counts()

fig, ax = plt.subplots(figsize=(8,7))

explode = (0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1,0.2,0.3,0.4)

ax.pie(match_winners, labels = None, autopct='%1.1f%%', startangle=90, shadow = True, explode = explode)

ax.legend(bbox_to_anchor=(1,0.5), labels=match_winners.index)

We will now visualize a histogram of dataset where the victory was with wickets and not with runs.

matches_df[matches_df.win_by_wickets != 0].win_by_wickets.hist()

Now we will create box-plots.

team_score = deliveries_df.groupby(['match_id', 'batting_team']).total_runs.sum().reset_index()

top_teams = deliveries_df.groupby('batting_team').total_runs.sum().nlargest(5).reset_index().batting_team.tolist()

top_teams_df = team_score[team_score.batting_team.isin(top_teams)]

top_teams_df.groupby('batting_team').boxplot(column = 'total_runs', layout=(5,1),figsize=(6,20));

top_teams_df.boxplot( column = 'total_runs',by = 'batting_team', rot = 20, figsize = (10,4));

We have developed correlation metrics earlier in the book. Apart from these, there are multiple ways to perform the EDA. Scatter plots, crosstabs, and so on serve as a good tool.

EDA serves as the foundation of a robust ML model. Unfortunately, EDA is often neglected or less time is spent on it. It is a dangerous approach and can lead to challenges in the model at a later stage. EDA allows us to measure the spread of all the variables, understand their trends and patterns, uncover mutual relationships, and give a direction to the analysis. Most of the time, variables which are coming as important during EDA will be found significant by the ML model too. Remember, the time spent on EDA is crucial for the success of the project. It is a useful resource to engage the stakeholders and get their attention, as most of the insights are quite interesting. Visualization makes the insights even more appealing. Moreover, an audience which is not from an ML or statistical background will be able to relate to EDA better than to an ML model. Hence, it is prudent to perform a robust EDA of the data.

Now it is time to discuss the ML model-building stage in the next section.

Step 5: ML Model Building

Now we will be discussing the process we follow during the ML model development phase. Once the EDA has been completed, we now have a broad understanding of our dataset. We somewhat are clear on a lot of trends and correlations present in our dataset. And now we have to proceed to the actual model creation phase.

We are aware of the steps we follow during the actual model building. We have solved a number of case studies in the previous chapters. But there are a few points which we should be cognizant of while we proceed with model building. These cater to the common issues we face or the usual mistakes which are made. We will be discussing them in this section.

We will be starting with the training and testing split of the data.

Train/Test Split of Data

We understand that the model is trained on the training data and the accuracy is tested on the testing/validation dataset. The raw data is split into train/test or train/test/validate. There are multiple ratios which are used. We can use 70:30 or 80:20 for train/test splitting. We can use 60:20:20 or 80:10:10 for train/test/validate split, respectively.

While we understand the usage of training and testing data, there are a few precautions which need to be taken:

1. 1.

   The training data should be representative of the business problem. For example, if we are working on a prepaid customer group for a telecom operator it should be only for prepaid customers and not for postpaid customers, though this would have been checked during the data discovery phase.

    
2. 2.

   There should not be any bias while sampling the training or testing datasets. Sometimes, during the creation of training or testing data, a bias is introduced based on time or product. This has to be avoided since the training done will not be correct. The output model will be generating incorrect predictions on the unseen data.

    
3. 3.

   The validation dataset should not be exposed to the algorithm during the training phase. If we have divided the data into train, test, and validation datasets, then during the training phase we will check the model’s accuracy on the train and test datasets. The validation dataset will be used only once and in the final stage once we want to check the robustness.

    
4. 4.

   All the treatments like outliers, null values, and so on should be applied to the training data only and not to the testing dataset or the validation dataset.

    
5. 5.
   The target variable defines the problem and holds the key to the solution. For example, a retailer wishes to build a model to predict the customers who are going to churn. There are two aspects to be considered while creating the dataset:

   1. a.
      The definition of target variable or rather the label for it will decide the design of the dataset. For example, in Table 5-12 if the target variable is “Churn” then the model will predict the propensity to churn from the system. If it is “Not Churn,” it will mean the propensity to stay and not churn from the system.

      Table 5-12

      Definition of Target Variable Changes the Entire Business Problem and Hence the Model

      -

       
   2. b.

      The duration we are interested in predicting the event. For example, customers who are going to churn in the next 60 days are going to be different from the ones who will churn in 90 days, as shown in Figure 5-6.

       
    

Creation of training and testing data is the most crucial step before we do the actual modeling. If there is any kind of bias or incompleteness in the datasets created, then the output ML model will be biased.

Once we have secured the training and testing dataset, we move to the next step of model building.

Model Building and Iterations

Now is the time to build the actual ML model. Depending on the problem at hand and if we are developing a supervised learning solution, we can either choose a regression approach or a classification algorithm. Then we will choose an algorithm from the list of the algorithms available to use. Figure 5-7 shows the relationship between interpretability and accuracy of all the algorithms.

Generally, we start with a base algorithm like linear regression or logistic regression or decision tree. This sets the basic benchmark for us for the other algorithms to break.

Then we proceed to train the algorithm with other classification algorithms. There is a tradeoff which always exists, in terms of accuracy, speed of training, prediction, robustness, and ease of comprehension. It also depends on the nature of the variables; for example, k-nearest neighbor requires numeric variables as it needs to calculate the distance between various data points. One-hot encoding will be required to convert categorical variables to the respective numeric values.

After training data creation, an ML modeling process can be envisioned as the following steps. We are taking an example of a classification problem:

1. 1.

   Create the base version of the algorithm using logistic regression or decision tree.

    
2. 2.

   Measure the performance on a training dataset.

    
3. 3.

   Iterate to improve the training performance of the model. During iterations, we try a combination of addition or removal of variables, changing the hyperparameters, and so on.

    
4. 4.

   Measure the performance on a testing dataset.

    
5. 5.

   Compare the training and testing performance; the model should not be overfitting (more on this in the next section).

    
6. 6.

   Test with other algorithms like random forest or SVM and perform the same level of iterations for them.

    
7. 7.

   Choose the best model based on the training and testing accuracy.

    
8. 8.

   Measure the performance on the validation dataset.

    
9. 9.

   Measure the performance on out-of-time dataset.

    

There is a fantastic cheat sheet provided by sci-kit learn library at https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html.

At the end of this step, we would have accuracies of various algorithms with us. We are discussing the best practices for accuracy measurement now.

Accuracy Measurement and Validation

We have discussed the various accuracy measurement parameters for our regression and classification problems. The KPI to be used for accuracy measurement will depend on the business problem at hand. Recall the fraud detection case study discussed earlier: in that case, accuracy is not the correct KPI; instead, recall is the KPI which should be targeted and optimized. Hence, depending on the objective at hand the accuracy parameters will change. During the ML model phase, it is a good practice to compare various algorithms, as shown in Table 5-13.

Table 5-13

Comparing the Performance Measurement Parameters for All the Algorithms to Choose the Best One

Then we can take a decision to choose the best algorithm. The decision to choose the algorithm will also depend on parameters like time taken to train, time taken to make a prediction, ease of deployment, ease of refresh, and so on.

Along with the accuracy measurement, we also check time taken to prediction and the stability of the model. If the model is intended for real-time prediction, then time taken to predict becomes a crucial KPI.

At the same time, we have to validate the model too. Validation of the model is one of the most crucial steps. It checks how the model is performing on the new and unseen dataset. And that is the main objective of our model: to predict well on new and unseen datasets!

It is always a good practice to test the model’s performance on an out-of-time dataset. For example, if the model is trained on a Jan 2017 to Dec 2018 dataset, we should check the performance on out-of-time data like Jan 2019–Apr 2019. This ensures that the model is able to perform on the dataset which is unseen and not a part of the training one. We will now study the impact of threshold on the algorithm and how can we optimize in the next section.

Overfitting vs. Underfitting Problem

During development of the model, we often face the issue of underfitting or overfitting. They are also referred to as bias-variance tradeoff. Bias-variance tradeoff refers to a property of the ML model where while testing the model’s performance we find that the model has a lower bias but high variance and vice versa, as shown in Figure 5-8.

Underfitting of the model is when that model is not able to make accurate predictions. The model is naïve and very simple. Since we have created a very simple model, it is not able to do justice to the dataset as shown in Figure 5-9.

Overfitting on the other hand is quite the opposite (Figure 5-10). If we have created a model which has good training accuracy but low testing accuracy, it means that the algorithm has learned the parameters of the training dataset very minutely. The model is giving good accuracy for the training dataset but not for the testing dataset.

Underfitting can be tackled by increasing the complexity of the model. Underfitting means that we have chosen a very simplistic solution for our ML approach while the problem at hand requires a higher degree of complexity. We can easily identify a model which is underfitting. The model’s training accuracy will not be good and hence we can conclude the model is not even able to perform well on the training data. To tackle underfitting, we can train a deeper tree, or instead of fitting a linear equation, we can fit a nonlinear equation. Using an advanced algorithm like SVM might give improved results.

Tip

Sometimes there is no pattern in the data. Even using the most complex ML algorithm will not lead to good results. In such a case, a very strong EDA might help.

For overfitting, we can employ these methods:

1. 1.

   Maybe the easiest approach is to train with more data. But this might not always work if the new dataset is messy or if the new dataset is similar to the old one without adding additional information.

    
2. 2.

   k-fold cross-validation is a powerful method to combat overfitting. It allows us to tweak the respective hyperparameters and the test set will be totally unseen till the final selection has to be made.

   The k-fold validation technique is depicted in Figure 5-11. It is a very simple method to implement and understand. In this technique, we iteratively divide the data into training and testing “folds.”

    

In k-fold validation, we follow these steps:

1. 1.

   We shuffle the dataset randomly and then split into k groups.

    
2. 2.

   For each of the groups, we take a test set. The remaining serve as a training set.

    
3. 3.

   The ML model is fit for each group by training on the training set and testing on the test set (also referred as holdout set).

    
4. 4.

   The final accuracy is the summary of all the accuracies.

   It means that each data point gets a chance to be a part of both the testing and the training dataset. Each observation remains a part of a group and stays in the group during the entire cycle of the experiment. Hence, each observation is used for testing once and for training (k–1) times.

   The Python implementation of the k-fold validation code is checked in at the Github repo.

    
5. 3.

   Ensemble methods like random forest combat overfitting easily as compared to decision trees.

    
6. 4.

   Regularization is one of the techniques used to combat overfitting. This technique makes the model coefficients shrink to zero or nearly zero. The idea is that the model will be penalized if a greater number of variables are added to the equation.

   There are two types of regularization methods: Lasso (L1) regression and Ridge (L2) regression. Equation 5-3 may be recalled from earlier chapters.

   

   (Equation 5-3)

   At the same time, we have a loss function which has to be optimized to receive the best equation. That is the residual sum of squares (RSS).

   In Ridge regression, a shrinkage quantity is added to the RSS, as shown in Equation 5-4.

   So, the modified 

   

   (Equation 5-4)

   Here λ is the tuning parameter which will decide how much we want to penalize the model. If λ is zero, it will have no effect, but when it becomes large, the coefficients will start approaching zero.

   In Lasso regression, a shrinkage quantity is added to the RSS similar to Ridge but with a change. Instead of squares, an absolute value is taken, as shown in Equation 5-5.

   So the modified

   

   (Equation 5-5)

   Between the two models, ridge regression impacts the interpretability of the model. It can shrink the coefficients of very important variables to close to zero. But they will never be zero, hence the model will always contain all the variables. Whereas in lasso some parameters can be exactly zero, making this method also help in variable selection and creating sparse models.

    
7. 5.
   Decision trees generally overfit. We can combat overfitting in decision tree using these two methods:

   1. a.

      We can tune the hyperparameters and set constraints on the growth of the tree: parameters like maximum depth, maximum number of samples required, maximum features for split, maximum number of terminal nodes, and so on.

       
   2. b.

      The second method is pruning. Pruning is opposite to building a tree. The basic idea is that a very large tree will be too complex and will not be able to generalize well, which can lead to overfitting. Recall that in Chapter 4 we developed code using pruning in decision trees.

      Decision tree follows a greedy approach and hence the decision is taken only on the basis of the current state and not dependent on the future expected states. And such an approach eventually leads to overfitting. Pruning helps us in tackling it. In decision tree pruning, we

      1. i.

         Make the tree grow to the maximum possible depth.

          
      2. ii.

         Then start at the bottom and start removing the terminal nodes which do not provide any benefit as compared to the top.

          
       
    

Note that the goal of the ML solution is to make predictions for the unseen datasets. The model is trained on historical training data so that it can understand the patterns in the training data and then use the intelligence to make predictions for the new and unseen data. Hence, it is imperative we gauge the performance of the model on testing and validation datasets. If a model which is overfitting is chosen, it will fail to work on the new data points, and hence the business reason to create the ML model will be lost.

The model selection is one of the most crucial yet confusing steps. Many times, we are prone to choose a more complex or advanced algorithm as compared to simpler ones. However, if a simpler logistic regression algorithm and advanced SVM are giving similar levels of accuracy, we should defer to the simpler one. Simpler models are easier to comprehend, more flexible to ever-changing business requirements, straightforward to deploy, and painless to maintain and refresh in the future. Once we have made a choice of the final algorithm which we want to deploy in the production environment, it becomes really difficult to replace the chosen one!

The next step is to present the model and insights to the key stakeholders to get their suggestions.

Step 6: Deployment of the Model

We have understood all the stages in model development so far. The ML model is trained on historical data. It has generated a series of insights and a compiled version of the model. It has to be used to make predictions on the new and unseen dataset. This model is ready to be put into the production environment, which is referred to as deployment. We will examine all the concepts in detail now.

ML is a different type of software as compared to traditional software engineering. For an ML model, several forces join hands to make it happen. It is imperative we discuss them now because they are the building blocks of a robust and useful solution.

Key elements are shown in Figure 5-12.

1. 3.

   This incoming data is unseen and new to the ML model. The data contains the data in the format expected by the compiled model. If the data is not in the format expected, then there will be a runtime error. Hence, it is imperative that the input data is thoroughly checked. In case we have an image classification model using a neural network, we should check that the dimensions of the input image are the same as expected by the network. Similarly, for structured data it should be checked that the input data is as expected and is checked including the names and type of the variables (integer, string, Boolean, data frame, etc.).

    
2. 4.
   The compiled model is stored as a serialized object like .pkl for Python, .R file for R, .mat for MATLAB, .h5, and so on. This is the compiled object which is used to make the predictions. There are a few formats which can be used:

   1. a.

      Pickle format is used for Python. The object is a streamed object which can be loaded and shared across. It can be used at a later point too.

       
   2. b.

      ONNX (Open Neural Network Exchange Format) allows storing of predictive models. Similarly, PMML (Predictive Model Markup languages) is another format which is used.

       
    
3. 5.

   The objective of the model, the nature of predictions to be made, the volume of the incoming data, real-time prediction or on-off prediction, and so on, allow us to design a good architecture for our problem.

    
4. 6.
   When it comes to the deployment of the model, there are multiple options available to us. We can deploy the model as follows:

   1. a.

      Web service : ML models can be deployed as a web service. It is a good approach since multiple interfaces like web, mobile, and desktop are handled. We can set up an API wrapper and deploy the model as a web service. The API wrapper is responsible for the prediction on the unseen data.

      Using a web service approach, the web service can take an input data and convert into a dataset or data frame as expected by the model. Then, it can make a prediction using the dataset (either continuous variable or a classification variable) and return the respective value.

      Consider this. A retailer sells specialized products online. A lot of the business comes from repeat customers. Now a new product is launched by the retailer. And the business wants to target the existing customer base.

      So, we want to deploy an ML model which predicts if the customer will buy a newly launched product based on the last transactions by the customer. We have depicted the entire process in Figure 5-14.

      

      We should note that we already have historical information on the customer’s past behavior. Since there is a need to make a prediction on a new set of information, the information has to be merged and then shared with the prediction service.

      As a first step, the website or the app initializes and makes a request to the information module. The information module consists of information about customer’s historical transactions, and so on. It returns this information about the customer to the web application. The web app initializes the customer’s profile locally and stores this information locally. Similarly, the new chain of events or the trigger which is happening at the web or in the app is also obtained. These data points (old customer details and new data values) are shared with a function or a wrapper function which updates the information received based on the new data points. This updated information is then shared with the prediction web service to generate the prediction and receive the values back.

      In this approach, we can use AWS Lambda functions, Google Cloud functions, or Microsoft Azure functions. Or we can use containers like Docker and use it to deploy a flask or Django application.

       
    
5. b.
   Integrated with the database: Perhaps this approach is generally used. As shown in Figure 5-15, this approach is easier than a web service–based one but is possible for smaller databases.

   

   In this approach, the moment a new transaction takes place an event is generated. This event creates a trigger. This trigger shares the information to the customer table where the data is updated, and the updated information is shared with the event trigger. Then the prediction model is run, and a prediction score is generated to predict if the customer will make a purchase of the newly launched product. And finally, the customer data is again updated based on the prediction made.

   This is a simpler approach. We can use databases like MSSQL, PostGres, and so on.

   We can also deploy the model into a native app or as a web application. The predictive ML model can run externally as a local service. It is not heavily used. Once the model is deployed, the predictions have to be consumed by the system.

    
6. 7.
   The next step to ponder over is consumptions of the predictions done by the ML model.

   1. a.

      If the model is making a real-time prediction as shown in Figure 5-16, then the model can return a signal like pass/fail or yes/no. The model can also return a probability score, which can then be handled by the incoming request.

       
    

1. b.

   If the model is not a real-time model, then it is possible that the model is generating probability scores which are to be saved into a separate table in the database. It is referred to as batch prediction, as shown in Figure 5-17. Hence, the model-generated scores have to be written back to the database. It is not necessary to write back the predictions. We can have the predictions written in a .csv or .txt file, which can then be uploaded to the database. Generally, the predictions are accompanied by all the details of the raw incoming data for which the respective prediction has been made.

   The predictions made can be continuous in case of a regression problem or can be a probability score/class for a classification problem. Depending on the predictions, we can make configurations accordingly.

    

Congratulations! The model is deployed in the production and is making predictions on real and unseen dataset. Exciting, right?

Step 7: Documentation

Our model is deployed now we have to ensure that all the codes are cleaned, checked in, and properly documented. Github can be used for version control.

Documentation of the model is also dependent on the organization and domain of use. In regulated industries it is quite imperative that all the details are properly documented.

Step 8: Model Refresh and Maintenance

We have understood all the stages in model development and deployment now. But once a model is put into production, it needs constant monitoring. We have to ensure that the model is performing always at a desired level of accuracy measurements. To achieve this, it is advised to have a dashboard or a monitoring system to gauge the performance of the model regularly. In case of non-availability of such a system, a monthly or quarterly checkup of the model can be done.

Model refresh can follow the approach shown in Figure 5-18. Here we have shown the complete step-by-step approach.

Once the model is deployed, we can do a monthly random check of the model. If the performance is not good, the model requires refresh. It is imperative that even though the model might not be deteriorating, it is still a good practice to refresh the model on the new data points which are constantly created and saved.

With this, we have completed all the steps to design an ML system, how to develop it from scratch, and how to deploy and maintain it. It is a long process which is quite tedious and requires teamwork.

With this, we are coming to the end of the last chapter and the book.

Summary

In this last chapter we studied a model’s life—from scratch to maintenance. There can be other methods and processes too, depending on the business domain and the use case at hand.

With this, we are coming to the end of the book. We started this book with an introduction to ML and its broad categories. In the next chapter, we discussed regression problems, and then we solved classification problems. This was followed by much more advanced algorithms like SVM and neural networks. In this final chapter, we culminated the knowledge and created the end-to-end model development process. The codes in Python and the case studies complemented the understanding.

Data is the new oil, new electricity, new power, and new currency. The field is rapidly increasing and making its impact felt across the globe. With such a rapid enhancement, the sector has opened up new job opportunities and created new job titles. Data analysts, data engineers, data scientists, visualization experts, ML engineers—they did not exist a decade back. Now, there is a huge demand of such professionals. But there is dearth of professionals who fulfill the rigorous criteria for these job descriptions. The need of the hour is to have data artists who can marry business objectives with analytical problems and envision solutions to solve the dynamic business problems.

Data is impacting all the business, operations, decisions, and policies. More and more sophisticated systems are being created everyday. Data and its power are immense: we can see examples of self-driving cars, chatbots, fraud detection systems, facial recognition solutions, object detection solutions, and so on. We are able to generate rapid insights, scale the solutions, visualize, and take decisions on the fly. The medical industry can create better drugs, the banking and financial sectors can mitigate risks, telecom operators can offer better and more stable network coverages, and the retail sector can provide better prices and serve the customers better. The use cases are immense and still being explored.

But the onus lies on us on how to harness this power of data. We can put it to use for the benefit of mankind or for destroying it. We can use ML and AI for spreading love or hatred—it is our choice. And like the cliché goes—with great power comes great responsibility!