In this chapter, we have turned our focus to a notebook approach to Apache Spark, and specifically developed R notebooks for estimating and assessing models, with which we developed risk scores to help the company XST to improve their risk management.

We first selected a few machine learning methods with our focus on the logistic regression method, along with random forest and decision trees. We then worked on data cleaning and feature development by using a special tool called OpenRefine. Next, we estimated the model coefficients. We then evaluated these estimated models by using a confusion matrix, ROC, and KS. Then we interpreted our machine learning results. And finally, we deployed our machine learning results with a scoring approach.

With a notebook approach, all the preceding machine learning steps are implemented in R, with all the R codes stored in notebooks so that the process is repeatable and can be partially automated. To get everything organized well and integrated with Apache Spark, we used the DataScientistWorkbench here.

After this chapter, readers will have gained a full understanding of the notebook approach to Apache Spark as well as some machine learning techniques for risk scoring and the DataScientistWorkbench. To sum up, readers will gain good knowledge about R, notebook, DataScientistWorkbench, and Spark. For more information on Apache Spark and DataScientistWorkbench, you can go to http://www.db2dean.com/Previous/Spark1.html.