Once our model gets estimated as in the preceding section, it is time for us to evaluate these estimated models to see if they fit our client's criteria so that we can either move to the results explanation stage or go back to some previous stage to refine our predictive models.

From the client's perspective, there are two common error types in machine learning for churn prediction.

The first one is False Negative (Type I Error), which is about failing to identify a customer who has a high propensity to depart.

From a business perspective, this is the least desirable error as the customer is very likely to leave, and the company does not know that it lost the chance to act to keep the customers, thus adversely affecting the the company's revenue.

The second one is False Positive (Type II Error), which is about classifying a good, satisfied customer as one who is one likely to churn. 

From a business perspective, this may be acceptable as it does not impact revenue, but will create confusion leading to some negative consequences, and may waste some of the company's expenses, as the company will act or even offer some discounts to save these customers.

To sum up, in order to perform our model evaluation, in this section we will use the aforementioned error numbers that are parts of a confusion matrix, and also RMSE to assess our regression models.

To calculate them, we need to use our test data rather than training data.

As discussed earlier, MLlib has algorithms to return RMSE values, confusion matrices, and even false positive numbers directly.

In MLlib, we can use the following code to calculate RMSE:

val valuesAndPreds = test.map { point =>
    val prediction = new_model.predict(point.features)
    val r = (point.label, prediction)
    r
    }
val residuals = valuesAndPreds.map {case (v, p) => math.pow((v - p), 2)}
val MSE = residuals.mean();
val RMSE = math.pow(MSE, 0.5)

Besides this, MLlib also has some functions in the RegressionMetrics and RankingMetrics classes for us to use for RMSE calculation apart from confusionMatrix and numFalseNegatives()

The following code calculates error ratios:

// Evaluate model on test instances and compute test error
val testErr = testData.map { point =>
  val prediction = model.predict(point.features)
  if (point.label == prediction) 1.0 else 0.0
}.mean()
println("Test Error = " + testErr)
println("Learned Random Forest:n" + model.toDebugString)

The following code may be used to obtain evaluation metrics for the estimated models:

// Get evaluation metrics.
val metrics = new MulticlassMetrics(predictionAndLabels)
val precision = metrics.precision
println("Precision = " + precision)

In our case, we used multiple algorithms on test datasets to obtain RMSEs, a confusion matrix, and error numbers. The following tables shows some example results for two of the models:

Model 1: Decision Tree:

Model 2: Random Forest:

For this project, as we need to evaluate tens of models, the best thing is to calculate error numbers with numFalseNegatives () so that we can easily sort all the models and then compare to select the good ones.

With the preceding two tables as our example, our program will look for Type I Errors first, for which the two models perform the same. In this case, the program will look at Type II Error for which the second model is better than the first one.

In practice, we should use RMSEs to evaluate models, but for this project, the client prefers using error numbers.