Architecture

Alluxio is a memory-centric distributed storage system and aims to be the de facto storage unification layer for big data. It provides a virtualization layer that unifies access for different storage engines such as Local FS, HDFS, S3, and NFS and computing frameworks such as Spark, MapReduce, Hive, and Presto. Figure 10-1 give you an overview of Alluxio’s architecture.

Alluxio is the middle layer that coordinates data sharing and directs data access, while at the same time providing computing frameworks and big data applications high-performance low-latency memory speed. Alluxio integrates seamlessly with Spark and Hadoop, only requiring minor configuration changes. By taking advantage of Alluxio’s unified namespace feature, applications only need to connect to Alluxio to access data stored in any of the supported storage engines. Alluxio has its own native API as well as a Hadoop-compatible file system interface. The convenience class enables users to execute code originally written for Hadoop without any code changes. A REST API provides access to other languages. We will explore the API’s later in the chapter.

Alluxio’s unified namespace feature does not support relational data stores such as Kudu, relational databases such as Oracle or SQL Server, or document databases such as MongoDB. Of course, writing to and from Alluxio and the storage engines mentioned are supported. Developers can use a computing framework such as Spark to create a Data Frame from a Kudu table and store it in an Alluxio file system in Parquet or CSV format, and vice versa (Figure 10-2).

Why Use Alluxio?

The typical Hadoop distribution includes more than 20 open source components. Adding another component to your technology stack is probably the furthest thing in your mind. Nevertheless, Alluxio delivers substantial benefits that will make you wonder why Alluxio is not part core Apache Spark.

Multiple Frameworks and Applications Can Share Data at Memory Speed

A typical Hadoop cluster has multiple sessions running different computing frameworks such as Spark and MapReduce. In case of Spark, each application gets its own executor processes, with each task within an executor running on its own JVM, isolating Spark applications from each other. This means that Spark (and MapReduce) applications have no way of sharing data, except writing to a storage system such as HDFS or S3. As shown in Figure 10-3, a Spark job and a MapReduce job are using the same data stored in HDFS or S3. In Figure 10-4, Multiple Spark jobs are using the same data with each job storing its own version of the data in its own heap space. ^viii Not only is data duplicated, but sharing data via HDFS or S3 can be slow, particularly if you’re sharing large amounts of data.

By using Alluxio as an off-heap storage (Figure 10-5), multiple frameworks and jobs can share data at memory speed, reducing data duplication, increasing throughput, and decreasing latency.

Provides High Availability and Persistence in Case of Application Termination or Failure

In Spark, the executor processes and the executor memory reside in the same JVM, with all cached data stored in the JVM heap space (Figure 10-6).

When the job completes or for some reason the JVM crashes due to runtime exceptions, all the data cached in heap space will be lost as shown in Figures 10-7 and 10-8.

The solution is to use Alluxio as an off-heap storage (Figure 10-9).

In this case, even if the Spark JVM crashes, the data is still available in Alluxio (Figures 10-10 and 10-11).

Optimize Overall Memory Usage and Minimize Garbage Collection

By using Alluxio, memory usage is considerably more efficient since data is shared across jobs and frameworks and because data is stored off-heap, garbage collection is minimized as well, further improving the performance of jobs and applications (Figure 10-12).

Apache Spark and Alluxio

You access data in Alluxio similar to how you would access data stored in HDFS and S3 from Spark.

You can also access Alluxio from MapReduce, Hive, Flink, and Presto to mention a few. Check Alluxio’s online documentation for more details.

Administering Alluxio

Alluxio provides a web interface to facilitate system administration and monitoring. You get both high level and detailed information on space capacity, usage, uptime, start time, and list of files to name a few. Alluxio provides you with a web interface for the master and workers. Alluxio also provides a command-line interface for typical file system operations.

Master

You can access Alluxio’s master home page by visiting http://<Master IP Address >:19999 (Figure 10-13).

Worker

You can access each Alluxio worker’s web interface by visiting http://<Worker IP Address>:30000 (Figure 10-14).

Apache Ignite

Ignite is another in-memory platform similar to Alluxio. GridGain Systems originally contributed Apache Ignite to the Apache Software Foundation in 2014. It was promoted to a top-level project in 2015. ^xi It is extremely versatile and can be used as an in-memory data grid, in-memory database, in-memory distributed filesystem, streaming analytics engine, and accelerator for Hadoop and Spark to mention a few. ^xii

Apache Geode

Geode is a distributed in-memory database designed for transactional application with low-latency response times and high-concurrency requirements. Pivotal submitted Geode to the Apache Incubator in 2015. It graduated from the Apache Incubator to become a top-level Apache project in November 2016. Gemfire, the commercial version of Geode, was a popular low-latency transactional system used in Wall Street trading platforms. ^xiii

Summary

Spark is a fast in-memory data processing framework. It can be made significantly faster with Alluxio by providing an off-heap storage that can be utilized to make data sharing across jobs and frameworks more efficient, minimizing garbage collection, and optimizing overall memory usage. Not only will jobs run considerably faster, but you also reduce costs due to decreased hardware requirements. Alluxio is not the only in-memory database available; Apache Ignite and Geode are viable options, as well as other commercial alternatives such as Oracle Coherence and Times Ten.

This chapter serves as an introduction to distributed in-memory computing, and Alluxio in particular. Alluxio is the default off-heap storage solution for Spark. You can learn more about Alluxio by visiting its website at Alluxio.org or Alluxio.com.

References