Stateful/stateless transformations

As seen previously, Spark Streaming uses a concept of DStreams, which are essentially micro-batches of data created as RDDs. We also saw types of transformations that are possible on DStreams. The transformations on DStreams can be grouped into two types: Stateless transformations and Stateful transformations.

In Stateless transformations, the processing of each micro-batch of data does not depend on the previous batches of data. Thus, this is a stateless transformation, with each batch doing its own processing independently of anything that occurred prior to this batch.

In Stateful transformations, the processing of each micro-batch of data depends on the previous batches of data either fully or partially. Thus, this is a stateful transformation, with each batch considering what happened prior to this batch and then using the information while computing the data in this batch.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.216.20.236