Many packages have already included some algorithms for users to assess models quickly. For example, both MLlib and R have algorithms to return confusion matrix for logistic regression models and even get false positive numbers calculated.

Specifically, MLlib has the confusionMatrix and numFalseNegatives() functions for us to use and even some algorithms to calculate MSE quickly, as follows:

MSE = valuesAndPreds.(lambda (v, p): (v - p)**2).mean()
print("Mean Squared Error = " + str(MSE))

Also, R has the confusion.matrix function for us to use. In R, there are even many tools to produce some quick graphical plots that can be used to gain a quick evaluation of models.

For example, we can perform plots of predicted versus actual values and also residuals on predicted values.

Intuitively, the methods of comparing predicted versus actual values are easiest to understand and give us a quick model evaluation. The following is a calculated confusion matrix for one of the company products, which shows a reasonable fit of our model. Take a look at the following table:

In MLlib, we can use the following codes to calculate RMSE:

val valuesAndPreds = test.map { point =>
    val prediction = new_model.predict(point.features)
    val r = (point.label, prediction)
    r
    }
val residuals = valuesAndPreds.map {case (v, p) => math.pow((v - p), 2)}
val MSE = residuals.mean();
val RMSE = math.pow(MSE, 0.5)

Besides the preceding, MLlib also has some functions in the RegressionMetrics and RankingMetrics classes for us to use for RMSE calculation.

In R, we can compute RMSE, as follows:

RMSE <- sqrt(mean((y-y_pred)^2))

Before this, we need to obtain the predicted values with the following commands:

> # build a model > RMSElinreg <- lm(s1 ~ . ,data= data1) > > #score the model > score <- predict(RMSElinreg, data2

After obtaining the RMSE values for all the estimated models, we will compare them to evaluate the linear regression model with the logistic regression model and the decision tree model. For our case, the linear regression model turned out to be the best.

Then we will also compare the RMSE values across products and send back some product models for refinement.

For another example of obtaining RMSE, go to http://www.cakesolutions.net/teamblogs/spark-mllib-linear-regression-example-and-vocabulary.

As an example, we will calculate ROC curves to assess our logistic models.

In MLlib, we can use the MLlib function metrics.areaUnderROC() to calculate ROC once we apply our estimated model to our test data and get labels to test cases.

In R, using the pROC package, we can run the following to calculate and plot the ROC curves:

mylogit <- glm(s2 ~ ., family = "binomial")
summary(mylogit)
prob=predict(mylogit,type=c("response"))
testdata1$prob=prob
library(pROC)
g <- roc(s2 ~ prob, data = testdata1)
plot(g)

As discussed, once the ROC curves get calculated, we can use them to compare our logistic models against decision tree models or compare models across products. For our case, logistic models perform better than decision tree models: