This chapter has introduced Databricks. It shows how the service can be accessed, and also shows how it uses AWS resources. Remember that, in the future, the people who invented Databricks plan to support other cloud-based platforms, such as Microsoft Azure. I thought that it was important to introduce Databricks, because the same people who were involved in the development of Apache Spark are involved in this system. The natural progression seems to be Hadoop, Spark, then Databricks.

I will continue the Databricks investigation in the next chapter, because important features, such as visualization, have not yet been examined. Also, the major Spark functionality modules called GraphX, streaming, MLlib, and SQL have not been introduced in Databricks terms. How easy is it to use these modules within Databricks to process real data? Read on to find out.