In this chapter, using a service request forecasting project, we went through a step-by-step process of utilizing big data to serve city governments as well as related civic organizations, from which we processed open data on Apache Spark and then built several models, including regression and time series ARIMA models to predict service demands. With this, we then developed rules for alerts and scores for zip code zone ranking to help cities prepare resources to measure effectiveness and also rank communities.
Specifically, we first selected a supervised machine learning approach with a focus on time series modeling per use case needs after we prepared Spark computing and loaded in preprocessed data. Secondly, we worked on data and feature preparation by merging a few datasets together and selecting a core set of features from hundreds of features. Thirdly, we estimated model coefficients using the Zeppelin notebook with MLlib and the R notebook on Databricks. Next, we evaluated these estimated models mainly using RMSE. Then, we interpreted our machine learning results with graphs to show trends and tables to show the biggest predictors. Finally, we deployed our machine learning results with a focus on scoring but also used insights to develop rules.
The preceding process is similar to the process as described in the previous chapters. However, in this chapter, we worked on times series modeling, which is a new method. This enabled us to be able to deal with the data with time dimensions and develop insight over time.
After this chapter, readers should have gained a better understanding of how Apache Spark could be utilized not only for commercial use but also for public use to serve cities and universities.
As this is our last chapter, readers may use this real-life project of service request forecasting to review all the modeling methods, such as regression and decision tree, as well as all the Spark computing platforms, such as the Zeppelin notebook with Spark and the R notebook for the Databricks environment. For this purpose, we discussed more methods and platforms in this chapter than in the earlier ones.