Apache Flume was developed by Cloudera for handling and moving large amount data produced into Hadoop. Without minimum or no delay (NRT: Near Real Time or Real time) the company wanted the data produced to be moved to Hadoop system, for various analysis to be carried. That was how this beautiful came into existence.

As detailed in previous section, it was initially conceived and developed to take care of a particular use case of collecting and aggregating log data from various source (web servers) into Hadoop for performing various analytics useful for proactive maintenance. Later on it was redesigned and refactored to include different sources and destinations and also design was taken into account for pluggability and extensibility in mind.

Very much similar to Sqoop, Flume also has two major flavours. They are:

• Flume OG (Old Generation: pre 1.0)
• Flume NG (New Generation)

As the name suggests, Flume OG was the initial Flume distribution, which then underwent complete rewrite and refactoring giving rise to Flume NG. Flume NG is the current supported and active Apache project and this is the flavor we will use in our book. As of writing this book, the latest version of Flume NG is 1.7.0, which was released on October 17, 2016. Flume got into Apache Incubator with version 1.0.0 and 1.1.0.