As discussed in Chapter 9, City Analytics on Spark, in the United States and worldwide, more and more governments at various levels have made their collected data openly available to the public. As a result of expanding analytics of open data, many governmental and social organizations have used these open datasets to improve service to citizens, with a lot of good results recorded, such as in https://www.socrata.com/video/added-value-open-datas-internal-use-case/. Using data analytics for cities has a huge impact as more than half of us live in urban centers now, and this urban residence percentage is higher and higher every year.

Especially, using big data to measure communities is also favored by researchers and practitioners, as we can see at http://files.meetup.com/11744342/CITY_RANKING_Oct7.pdf. Many cities have policy initiatives to measure communities or even smaller units such as streets with good results and data available for public use, such as that from Los Angeles at http://lahub.maps.arcgis.com/apps/MapJournal/index.html?appid=7279dc87ea9e416d9f90bf844505a54a. Using available open data and computing tools just to create some measurements and rankings may be easy. However, creating an accurate and object ranking of certain properties of some communities is not an easy task. Here, we are asked to use available open data, in combination with other datasets, such as census data and social media data, to improve rankings of communities, with a focus on school districts or schools.

At the same time, we are also asked to explore the available open data and try to model it with available machine learning tools on Spark. In other words, for this project, besides developing a good score to measure and rank communities, we are also asked for special insights to be developed from our dynamic machine learning. Once the ranking is ready, we are even asked to explore the rankings with more machine learning, which makes this project really dynamic, as aided by the ease and speed of Spark computing.

However, everything starts from data, as we found out in Chapter 9, City Analytics on Spark. The datasets are not as good as we expected, and they have the following issues for us to deal with:

Therefore, we will still need to perform a big task of data cleaning and feature preparation. We are lucky that we already have a good process from data to equation, estimation, evaluation, and explanation.

For this work of learning from open data, as we took a dynamic approach, the research team became interested in educational data and gradually turned our focus to the work of ranking school districts with open data.

With regard to this subject, we found some open data about schools at https://www.ed-data.k12.ca.us/Pages/Home.aspx.

The state government of California also has some open data at http://data.ca.gov/category/by-data-format/data-files/.

As discussed earlier, like in the Spark computing section of Chapter 8, Learning Analytics on Spark, you may choose one of the following approaches for our kind of projects:

You have already learned in detail about utilizing them one by one in the previous chapters, mainly from Chapter 3, A Holistic View on Spark to Chapter 7, Recommendations on Spark.

Either one of the preceding four approaches should work very well for our projects of learning from open data here. Specially, you may also take the codes as developed in this chapter, and put them into a separate notebook. You can then implement the notebook with one of the preceding approaches.

As an exercise and also for the best to fit our big amount of open data and project goals of fast ranking, we will need to work with the fourth approach, which is to utilize Apache Spark with MLlib. However, we will also need to use R programming a lot for better visualization and reporting, so that we will utilize our first approach of Spark in Databrick's platform as well. At the same time, to take advantage of some good PCA algorithms in SPSS and to easily develop related workflows, we will also need to use SPSS on Spark to practice a special dynamic approach of utilizing Apache Spark. Finally, to meet the needs of creating many data-cleaning workflows, we will also need to use the DataScientistWorkBench platform, with which we can use OpenRefine.

Let's review the approaches mentioned previously in brief to get us really prepared.

As we discussed in the Spark computing for machine learning section of Chapter 1, Spark for Machine Learning, Apache Spark has a unified platform that consists of the Spark core engine and four libraries, which are Spark SQL, Spark Streaming, MLlib, and GraphX:

As MLlib is Apache Spark's built-in machine learning library, it is relatively easy to set up and scale for our project of learning from open data.

To work within the Databricks environment, we need to perform the following steps to set up clusters:

To utilize IBM Data Scientist Workbench, we need to go to https://datascientistworkbench.com/:

As shown in the preceding screenshot, Data Scientist Workbench has Apache Spark installed and also has a data cleaning system, OpenRefine, integrated so that our data preparation work can be made easier and more organized:

For this project, we will use Data Scientist Workbench for data cleaning and also a little for R notebook creation, as well as Apache Spark implementation. For this setup, some of the Apache Spark techniques described in the previous chapters should apply.

With regards to SPSS on Spark, we will use IBM SPSS Modeler 17.1 and IBM SPSS Analytics Server 2.1, which has a good integration with Apache Spark.

In the previous section, we described our use case of learning from open data, with a focus on using open data to score and rank communities, and also prepared our Spark computing platform with R notebooks, SPSS workflows, and MLlib codes to use. As the next step for our machine learning per our 4E framework, we need to complete a task of mapping our use case to machine learning methods, which is to select our analytical methods or predictive models (equations) for this project of scoring and ranking with open data on Spark.

To turn data into insights, we will need to explore many methods, for which our dynamic approach should work well. To develop scores and rankings, it is not a difficult task with our available analytical tools and fast computing. However, to obtain objective and accurate raking is not an easy job. One approach to achieve this is to ensemble many rankings together, as it will improve results dramatically per past research. Visit http://www.researchmethods.org/Ranking-indicators and http://www.researchmethods.org/InnovativeAnalysisSociety for more information.

Therefore, for this project, we will take a dynamic approach, with which we will explore our open data, with methods including cluster analysis and principal component analysis. We will then use this knowledge to build a few rankings and scores. After that, we will ensemble results to improve rankings and scores. Finally, we will develop models to explain the impact of various features on these rankings and scores. However, as we are taking a dynamic approach, we will jump between these stages to achieve optimal results. As always, once we finalize our decision for analytical methods or models, we will need to prepare the related dependent variable and also prepare for coding.

Both Spark MLlib and R have algorithms available for cluster analysis:

// Cluster the data into two classes using KMeans
val numClusters = 2
val numIterations = 20
val clusters = KMeans.train(parsedData, numClusters, numIterations)

// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)

In R, we can use some R codes. Here is an example:

# K-Means Cluster Analysisfit <- kmeans(schooldata, 5) # 5 cluster solution
# get cluster means aggregate(schooldata,by=list(fit$cluster),FUN=mean)

For more about cluster analysis with Spark MLlib, go to http://spark.apache.org/docs/latest/mllib-clustering.html.

Both Spark MLlib and R have algorithms available for principal component analysis (PCA):

// Compute the top 10 principal components.
val pc: Matrix = mat.computePrincipalComponents(10) // Principal components are stored in a local dense matrix.

// Project the rows to the linear space spanned by the top 10 principal components.
val projected: RowMatrix = mat.multiply(pc)

In R, we can use the prcomp function from the stats package.

For more on PCA with Spark MLlib, go to http://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html.

Besides cluster analysis and PCA, we will also use regression modelling and decision tree modelling to help us understand more about how communities fall into one category or one rank rather than others.

So far, you must know that regression is among the most commonly used models for prediction, and has been utilized for various projects so far.

As we discussed, there are two kinds of regression modeling that are suitable for various kinds of predictions. One is linear regression and another is logistic regression. For this project, linear regression can be used when we take daily service request volume as our target variable, while logistic regression can be used if we want to predict whether or not a certain type of service is requested in a certain location at a certain time period.

For your convenience, in MLlib, for linear regression, we have the following code to be used:

val numIterations = 90
val model = LinearRegressionWithSGD.train(TrainingData, numIterations)

For logistic regresssion, we can use these following code:

val model = new LogisticRegressionWithSGD()
.setNumClasses(2)

In R, as we did earlier, we will use the GLM and LM functions for linear regression modeling and logistic regression modeling.

Once the scores get developed, one of the easy ways of resembling them is to construct a linear combination with special weighting for each. The weights can be developed with subject knowledge or with machine learning.

Besides the preceding things, there are also a few R packages for score ensemble.