Apache Hive & Impala

While we have seen Apache HIVE more as a Data Lake storage and query component, this can as well play a role into the serving layer, if the data exposed via HIVE views is a modelled data and meant for consumption by other applications. Both types of paradigms may exist here, that is, push and pull. Since HIVE supports access via JDBC driver, other application can pull the processed information over JDBC channel. Also, since it is a part of Hadoop storage layer required ETL mechanisms can be put in place for pushing the data out of the HIVE views containing modelled data.

Apache Impala is general purpose SQL query engine (also known as interactive SQL for Hadoop), quite an apt addition to our Data Lake implementation. It has inherent support to a variety of Hadoop file formats like Avro, Parquet and so on It's quite fast in it's operation and that's the main reason for it's inclusion.

It was developed by Cloudera (based on Google’s 2010 published Dremel paper) and then open sourced into ASF and is now incubating with a version of 2.7.0. It has support and integration with a wide range of products in the Hadoop ecosystem. More details on this can be found in http://impala.apache.org/ and https://github.com/apache/incubator-impala. It's highly performant (in-memory query execution and directly accesses data), flexible and horizontally scalable by adding more nodes as needed. It is shipped or packaged with most of the commercial Hadoop distributions, because of these benefits.

You should definitely consider this technology as a good addition to your Data Lake implementation for performing fast analytical queries on large sets of Hadoop data. Impala utilizes the same HIVE metastore and can perform parallel queries on underlying Hadoop storage via Impala tables. Impala tables can also be accessed using JDBC drivers, which make them viable for data access from consuming applications. All this combined with optimized IO usage helps queries to run faster and avoid investments into costly ETL tools. Impala provides comparatively a more real time query execution when compared to HIVE while utilizing the same shared resource as HIVE for shorter running jobs.

Depending on whether the data process resulting into processed data is a long running job or a short running job, choice of HIVE or Impala view can be made however both have very similar data serving mechanisms.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.226.172.200