Hadoop is an open source software developed by Apache for distributed computing, allowing analysis of big datasets in a secure way. It departs from the principle that every kind of machine has faults that happen often so they need to be cured by the software. Spark is a similar tool, being another kind of big data structure developed by Apache. But while the focus of Hadoop is secure data storage, the focus of Spark is data processing. They are different entities, but they mess things up, mainly because Spark is often used to process data held in Hadoop's filesystem.

In this chapter, we are going to learn about the sparklyr R package: a free and open sourced package developed by RStudio in conjunction with IBM, Cloudera, and H2O. Basically, sparklyr is a dplyr backend to Spark, and it provides an R interface to Spark's distributed machine learning algorithms, so we can handle large scale data using R.