In this session, we explored the Bisecting KMeans model, which is new in Spark 2.0. We utilized the glass dataset in this session and tried to assign a glass type using BisectingKMeans(), but changed k to 6 so we have sufficient clusters. As usual, we loaded the data into a dataset with Spark's libsvm loading mechanism. We split the dataset randomly into 80% and 20%, with 80% used to train the model and 20% used for testing the model.

We created the BiSectingKmeans() object and used the fit(x) function to create the model. We then used the transform(x) function for the testing dataset to explore the model prediction and printed out the result in the console output. We also output the cost of computing the clusters (sum of error squared) and then displayed the cluster centers. Finally, we printed the features with their assigned cluster number and stop operation.

Approaches to hierarchical clustering include:

• Divisive: Top down approach (Apache Spark implementation)
• Agglomerative: Bottom up approach