In the last section, we completed our model estimation. Now, it is the time for us to evaluate these estimated models to see whether they fit our client's criteria so that we can either move to results explanation or go back to some previous stages to refine our predictive models.
To perform our model evaluation, in this section, we will focus on utilizing confusion matrix and FalsePositive numbers to assess the goodness of fit for our models. To calculate them, we need to use our test data rather than training data.
As discussed before, both MLlib and R have algorithms to return a confusion matrix and even false positive numbers.
MLlib has confusionMatrix and numFalseNegatives() to use.
The following code calculates error ratios:
// Evaluate model on test instances and compute test error
val testErr = testData.map { point =>
val prediction = model.predict(point.features)
if (point.label == prediction) 1.0 else 0.0
}.mean()
println("Test Error = " + testErr)
println("Learned Random Forest:n" + model.toDebugString)To visualize the performance of our classifiers, we can use the R package ROCR. For more information on using ROCR, readers may visit https://rocr.bioinf.mpi-sb.mpg.de/.
In MLlib, we can use the following code to produce a confusion matrix and related false positive ratios:
// compute confusion matrix val metrics = new MulticlassMetrics(predictionsAndLabels) println(metrics.confusionMatrix)
In R, we can produce a confusion matrix and related false positive ratios with the following code:
model$confusion
For the fraud type social engineering, we produced the following confusion matrix, which shows a good result:
|
Type of Fraud |
Predicted as Fraud |
Predicted as NOT |
|---|---|---|
|
Actual Fraud |
81% |
19% |
|
Actual Not |
12% |
88% |
For this project, the preceding table is the most important evaluation as the company wants to increase the ratio in the upper-left cell, which is to catch fraud as much as possible. However, they also need to reduce the ratio in the lower-left cell, which is to reduce the false positive ratio as much as possible.
As discussed earlier, the low ratio in the upper-left cell means that many frauds will not be caught, which may lead to big losses.
The high false positive ratio often leads to wasted labor on the company's side and also incovenience to customers, which could lead to low customer satisfaction and even loss of customers.