In the last section, we summarized what is needed to complete our model estimation for our supervised machine learning. Now it is time for us to evaluate these estimated models to see if they fit the client's criterions so that we can either move to the results explanation stage or go back to some previous stages to refine our predictive models.

To perform our model evaluation, in this section, we will need to use Root Mean Square Error (RMSE) to assess our linear regression models of predicting Call Center calls, and use confusion matrix to assess our logistic regression model of predicting customer churn, for which the following numbers are often preferred:

Here, positive means the subscriber departed, and negative means the subscriber stayed.

The preceding four numbers are the building blocks for most classifier-evaluation metrics. A fundamental point in considering classifier evaluation is that pure accuracy (that is, was the prediction correct or incorrect) is not necessarily a best one, as a dataset could be highly unbalanced. For example, if a model is designed to predict fraud from a dataset where 95 percent of the data points are not fraud and 5 percent of the data points are fraud, then a naive classifier that predicts all as not fraud, regardless of the input, will be 95 percent accurate. For our case, the churn ratio is also not very high. For this reason, metrics for precision (positive predictive value) and recall (sensitivity) are typically used because they take into account the type of error. In other words, some balance between precision and recall is needed, which can be captured by combining the two into a single metric called the F-measure that will be calculated here with MLlib.

Just like for model estimation, to calculate RMSEs and to produce a confusion matrix, we need to use MLlib for regression modelling to be implemented with Apache Spark, which, as an exercise, can also be implemented with SPSS on Spark, when we take the MLlib codes to form a SPSS Modeler node. For logistic regression modelling, we will use R notebooks to be implemented in the Databricks environment for Apache Spark. In practice, we tried both MLlib and R for both calculating RMSE and calculating error ratios, because one of the main purposes for this project is to go beyond the limits of our tools of R and MLlib.

val valuesAndPreds = test.map { point =>
  val prediction = new_model.predict(point.features)
  val r = (point.label, prediction)
  r
}
val residuals = valuesAndPreds.map {case (v, p) => math.pow((v - p), 2)}
val MSE = residuals.mean();
val RMSE = math.pow(MSE, 0.5)

accuracy(f, x, test=NULL, d=NULL, D=NULL)

// F-measure
val f1Score = metrics.fMeasureByThreshold
f1Score.foreach { case (t, f) =>
  println(s"Threshold: $t, F-score: $f, Beta = 1")
}

val beta = 0.5
val fScore = metrics.fMeasureByThreshold(beta)
f1Score.foreach { case (t, f) =>
  println(s"Threshold: $t, F-score: $f, Beta = 0.5")
}

model$confusion