As demonstrated in the previous chapters, turning estimated models into scores is not very challenging, and could be done under non-Spark platforms. However, Apache Spark makes things easy and fast as demonstrated.
With the notebook approach adopted in this chapter, we will fully achieve the advantage to quickly produce new scores when data and customer requirements get changed.
Users will find some similarity to the deployment work in the last chapter—the deployment of scoring for fraud detection.
From coefficients of our predictive models, we can derive a risk score for possible default, which takes some work. But it gives the client the flexibility of changing it whenever needed.
With logistic regression, the process of producing scores is relatively easy—it uses the following formulae for logistic regression:
Specifically, Prob(Yi=1) = exp(BXi)/(1+exp(BXi)) produces the default probability, with Y=1 as the default, and X is a sum of all the features. In R, exp(coef(logit_model)) returns the needed odd ratios.
In R, the quick way is to use the function of predict as follows:
prob=predict(model,x,type="prob")
Specifically, this function will produce a probability value for default, which can be used directly as a score for this project.
However, in order to use the score, we still need to select a cutting out score. For example, we can choose to take action only when the risk score is over 90.
Different score cutting points will produce different false positive ratios, and also the ratios of excluding bad applicants, for which the users need to make a decision about how to balance the results.
By taking advantage of Spark's fast computing, results can be calculated fast, which allows the company to select a cutting point instantly and to make changes when needed.
As similar to other applications, another way to deal with this issue is to use the R package, OptimalCutpoints.