The following table (Table 01) summarizes the main difference which exist between batch and stream data. Our use case does have both these types of data. For handling batch data we selected and used Sqoop as the technology and for stream data handling and transfer to Hadoop we have selected Flume as the choice.
| Batch Data | Stream Data |
| Mostly the volume of data is quite high and deal with a chunk of data in defined period of time. Batch consisting of thousands of records. | The volume of data is not that great. But, deal with high velocity data and it comes continuously but does not have defined period in which they come. Stream data consist of single or few record. Few records are sometimes also called as micro-batch as the batch record size is quite small compared to conventional batch. |
| Analysis or processing is done on a very large period or all the data. | Analysis and processing is done on a small dataset over a moving time window in a continuous manner. |
| Period of operation spans from minutes to hours usually. | Operation period spans from milliseconds to seconds in most of the cases. |
| Since the data set is huge, complex analysis can be performed deriving usual deductions for business decision making. | Simple analysis can be performed and in usual case done with batch analysis to derive usual information. |
| Example, customer who has purchased more electronic items past month when the sale was going on. | Example, detection of credit card fraud as and when a purchase happens OR RBA (Risk Based Analysis) for important transactions in your ecommerce website. According to risk, step up authentication to multi factor (say One Time Password) mechanism. |
Table 01: Details summarizing difference between Batch and Stream data
The preceding table does state quite clearly the varying difference between batch and stream data. This means, these two data need to be handled accordingly when ingesting into the Data Lake.