Big data – that was our motivation to explore the world of machine learning with Spark a couple of years ago. We wanted to build machine learning applications that would leverag models trained on large amounts of data, but the beginning was not easy. Spark was still evolving, it did not contain a powerful machine learning library, and we were still trying to figure out what it means to build a machine learning application.

But, step by step, we started to explore different corners of the Spark ecosystem and followed Spark’s evolution. For us, the crucial part was a powerful machine learning library, which would provide features such as R or Python libraries did. This was an easy task for us, since we are actively involved in the development of H2O’s machine learning library and its branch called Sparkling Water, which enables the use of the H2O library from Spark applications. However, model training is just the tip of the machine learning iceberg. We still had to explore how to connect Sparkling Water to Spark RDDs, DataFrames, and DataSets, how to connect Spark to different data sources and read data, or how to export models and reuse them in different applications.

During our journey, Spark evolved as well. Originally, being a pure Scala project, it started to expose Python and, later, R interfaces. It also took its Spark API on a long journey from low-level RDDs to a high-level DataSet, exposing a SQL-like interface. Furthermore, Spark also introduced the concept of machine learning pipelines, adopted from the scikit-learn library known from Python. All these improvements made Spark a great tool for data transformation and data processing.

Based on this experience, we decided to share our knowledge with the rest of the world via this book. Its intention is simple: to demonstrate different aspects of building Spark machine learning applications on examples, and show how to use not only the latest Spark features, but also low-level Spark interfaces. On our journey, we also figure out many tricks and shortcuts not only connected to Spark, but also to the process of developing machine learning applications or source code organization. And all of them are shared in this book to help keep readers from making the mistakes we made.

The book adopted the Scala language as the main implementation language for our examples. It was a hard decision between using Python and Scala, but at the end Scala won. There were two main reasons to use Scala: it provides the most mature Spark interface and most applications deployed in production use Scala, mostly because of its performance benefits due to the JVM. Moreover, all source code shown in this book is also available online.

We hope you enjoy our book and it helps you navigate the Spark world and the development of machine learning applications.