In this chapter, we extended our machine learning on Spark to serve learning analytics, for which we completed a step-by-step process of processing big data obtained from learning management systems and other sources for a rapid development of student attrition prediction models on Apache Spark. With the machine learning results obtained, we developed rules and scores to be used by NIY University for interventions to reduce student attrition.

Specifically, we first selected a supervised machine learning approach with a focus on logistic regression and decision trees as per the special needs of this university and the nature of the project, and after this, we prepared Spark computing and loaded in the preprocessed data. Secondly, we worked on feature development and selection. Thirdly, we estimated model coefficients with the Zeppeline notebook on Spark. Next, we evaluated these estimated models using a confusion matrix and error ratios. Then, we interpreted our machine learning results to the university leaders and technicians. Finally, we deployed our machine learning results with some special effort on scoring students as per attrition probabilities, but we also used insight to develop rules.

This process is similar to the process used in the previous chapters for commercial applications, such as churn modeling. However, in working for educational applications, we made some special considerations for feature development and result explanation.

After reading this chapter, you should have gained a complete understanding of how Apache Spark can be utilized to make our work easier and faster in conducting supervised machine learning to serve educational institutions and, specially, develop student attrition prediction models. At the same time, you gained a good understanding of how fast computing can be turned into analytical capabilities for educational organizations.