Let's do a bit of a deep dive into the speed layer to have a common understanding.

In the batch layer, we run a compute algorithm over the entire dataset. This dataset can be, and usually is, in peta bytes ranges. Clearly, this is a very resource-intensive operation and one that simply throws the concept of latency out of the window. Latency is not a concern for the batch layer. But for the speed layer, latency plays an important role. If the speed layer is not able to produce results in acceptable latency, usually in a few seconds, the speed layer may as well be regarded as a distant cousin of the batch layer. Thus, in order to meet the high demands of low latency, the speed layer takes a fundamentally different approach. It makes use of what is known as incremental computational algorithms. If I am not wrong, the phrase incremental computation was popularized by Natah Marz in his book Big Data.

The batch layer in the Lambda architecture plays a very important role in helping the speed layer do its job. In essence, it eases the responsibilities of the speed layer by relaxing some of the data quality constraints as well as data volume constraints.

The speed layer helps in providing updates to the data views in low latency to keep them recent. And that's the only job of the speed layer. Now this may sound easy, but as we will explore, it can have serious implications if not done correctly.