In big data, Hadoop is a core component/technology that cannot be ignored. Elasticsearch has very good support for Hadoop and Elastic Stack has a product for achieving this integration called ES-Hadoop. We will discuss that in some detail in this section. For more details, we urge you to go to the elastic official page here: https://www.elastic.co/products/hadoop.

Hadoop is considered to be good for batch analytics but, for modern applications as well as enterprises dealing with batch and real-time data, performing meaningful analysis is crucial. For dealing with real-time data, this integration of Elasticsearch and Hadoop is apt. Elasticsearch brings the real-time analytics capability on top of Hadoop. ES-Hadoop is a connector shipped as a product by elastic along with the Elastic Stack.

This connector works with Hadoop 1.x and 2.x versions and works like a charm with a variety of other big data technologies by enabling data to flow to and fro between Hadoop and Elasticsearch.

This figure shows the exact working of the ES-Hadoop connector:

Figure 19: ES-Hadoop connector in action (image inferred from www.elastic.co)

As shown in this figure, ES-Hadoop has native support for many of the big data technologies, allowing data to be read/queried from Elasticsearch. ES-Hadoop has a variety of security options, using which security considerations can also be met.

The full details on this library are beyond the scope of this section, so we urge you to read the elastic documentation at https://goo.gl/NSwCVE for more in-depth details on this integration, which is a must for our Data Lake.