So far, we have only covered components that came along with the standard distribution of Apache Spark (except HDFS, Kafka and Flume, of course). However, Apache Spark can also serve as runtime for third-party components, making it as some sort of operating system for big data applications. In this chapter, we want to introduce Apache SystemML, an amazing piece of technology initially developed by the IBM Almaden Research Lab in California. Apache SystemML went through many transformation stages and has now become an Apache top level project.

In this chapter, we will cover the following topics to get a greater insight into the subject:

• Using SystemML for your own machine learning applications on top of Apache Spark
• Learning the fundamental differences between SystemML and other machine learning libraries for Apache Spark
• Discovering the reason why another machine library exists for Apache Spark