Criteria for Evaluating Streaming Engines

For a given application, the following criteria should be considered:

What is the latency budget?

If you must process records within milliseconds, this will constrain your options. If your budget is minutes to hours, you have a lot of options; you could even run frequent, short batch jobs!

What is the volume of data to be processed per unit time?

A streaming job may ultimately process petabytes of data, but if this happens one record at a time over a very long time, then almost anything can be made to work. If you’re processing the Twitter firehose, then you’ll need tools that provide lots of scalability through parallelism.

Which kinds of data processing will you do?

Simple record transformations and filtering, like many ETL tasks, can be done with many tools. You’ll need sophisticated tools if you are doing complex GROUP BY and JOIN operations over several streams and historical data. If you need the precise semantics discussed previously in this chapter, that will also constrain your choices.

How strong and dynamic is the community around the project?

Is the project evolving quickly? Are there lots of people contributing enhancements and add-on libraries? Are quality standards high? Does it appear that the project will have a long and healthy life?

How do you build, deploy, and manage services?

If your organization already has a mature CI/CD (continuous integration/continuous delivery) process for microservices, it would be very nice to continue using those tools for stream processing as well. If you are accustomed to running and using a diverse set of tools, then adding new ones will be less of a challenge.

Let’s explore the four tools based on these criteria.

Spark and Flink: Scalable Data Processing Systems

Spark and Flink are examples of systems that automatically manage a group of processes across a cluster and partition and distribute your data to those processes. They handle high-volume scenarios very well and can also operate at relatively low latencies. They offer rich options for stream processing, from simple transformations to subsets of the Beam semantics, either as data flows or SQL queries.

Both support different application deployment options and they are integrated with the popular resource managers: YARN, Kubernetes, and Mesos, which are used to obtain resources on demand. In the most common scenario, you submit a job to a master service, which decides how to partition the data set to be processed and then schedules tasks (JVM processes) to do the work of the job.

All of this work behind the scenes is done for you, as long as your job fits this runtime model. The APIs are reasonably intuitive and promote concise code. Today, they come in two basic forms:

Data flows

A programming model based on sequencing transformations on collections. For Spark in particular, this API was heavily inspired by the Scala language collections API, which also inspired recent changes in Java’s collections API. For problems that are naturally described as a sequence of processing steps, forming a DAG (directed acyclic graph), this approach is very flexible and powerful.

SQL

Recall that SQL is a declarative language, where you specify the relational constraints and the query engine determines how to select and transform the data to satisfy those constraints. We saw earlier that SQL with extensions for streaming has become a popular choice for writing applications, because it is concise and easier to use by nonprogrammers.

Both models produce new streams, even SQL. In batch-mode SQL, the data at rest is processed to return a new, fixed data set. In the Streaming context, the input stream is transformed into an output stream matching the criteria of the query.

Both models require window primitives in order for them to work on infinite streams of data. Beam’s concepts for windowing were influential in the designs of Flink and Spark.

Spark and Flink have different histories. Spark started as a batch-mode system, then added stream processing later. Flink started as a streaming engine first, with batch support added later.

The initial Spark Streaming system is based on mini-batch processing, with windows (called intervals) of approximately 100 milliseconds or more. Also, it only supports processing time windows. This API is based on the original storage model in Spark, RDDs (resilient, distributed data sets), which provides the previously mentioned data flow API inspired by the Scala collections API. This API comes in Scala, Java, Python, and R versions. Spark Streaming is now effectively deprecated, replaced by a new streaming system.

The new system, Structured Streaming, introduced in Spark version 2.0, is based on SparkSQL, Spark’s declarative SQL system, with an idiomatic API in Scala, Java, Python, and R, as well as SQL itself. The declarative nature of SQL has provided the separation between user code and implementation that has allowed the Spark project to focus on aggressive internal optimizations, as well as support for advanced streaming semantics, like event-time processing. This is now the standard streaming API for Spark. A major goal of the Spark project is to promote so-called continuous applications, where the division between batch and stream processing is further reduced. In parallel, a new low-latency streaming engine has been developed to break through the 100-millisecond barrier of the original engine.

Spark also includes a machine learning library, MLlib, and many third-party libraries integrate with Spark.

Because Flink has always excelled at low-latency stream processing, it has maintained a lead over Spark in supporting advanced semantics. Specifically, it supports more of the semantics defined by Beam compared to Spark, as described in the Beam Capability Matrix. It also offers the more mature Beam runner of the two systems.

Flink also provides both a data-flow-style API in Scala and Java, as well as SQL. Integrated machine learning is less mature compared with Spark.

To conclude, Spark and Flink are comparable in the list of criteria we defined. Both provide rich options for processing, good low-latency and very good high-volume performance, and automatic data partitioning and task management. Both have vibrant and growing communities and tool ecosystems. Spark’s community is much larger at this time, while Flink is more mature for stream processing. Both have very good quality.

Akka Streams and Kafka Streams: Data-Centric Microservices

Akka Streams and Kafka Streams are libraries that you integrate into applications, as opposed to standalone services like Spark and Flink. They are designed for the sweet spot of building microservices with integrated data processing, versus using separate services focused exclusively on data analytics. There are several advantages to this approach:

• You would rather use the same microservice-based development and production workflow for all services, versus supporting a more heterogeneous environment.

• You want more control over how applications are deployed and managed.

• You want more flexibility in how different domain logic is integrated together—for example, incorporating machine learning model training or serving into other microservices, rather than deploying them separately.

If you run separate data analytics (for example, Spark jobs), and separate microservices that use the results, you’ll need to exchange the results between the services (for example, via a Kafka topic). This is a great pattern, of course, as I argued in Chapter 3, but also I pointed out that sometimes you don’t want the extra overhead of going through Kafka. Nothing is faster with lower overhead than making a function call in the same process, which you can do if your data transformation is done by a library rather than a separate service! Also, a drawback of microservices is the increase in the number of different services you have to manage. You may want to minimize that number.

Because Akka Streams and Kafka Streams are libraries, the drawback of using them is that you have to handle some details yourself that Spark or Flink would do for you, like automatic task management and data partitioning. Now you really need to write all the logic for creating a runnable process and actually running it.

Despite these drawbacks, because Kafka Streams runs as a library on top of the Kafka cluster, it requires no additional cluster setup and modest additional management overhead to use.

You can use Akka Streams in a similar way, too, although you can leverage Akka Cluster for building highly distributed, resilient applications, when needed. The Akka ecosystem of which Akka Streams is a part—combined with the other Lightbend Reactive Platform projects, Play and Lagom—provides a full spectrum of tools and libraries for microservice development. While Kafka Streams is focused on reading and writing Kafka topics, the connector library Alpakka makes it relatively easy to connect different data sources and sinks to Akka Streams.

Both streaming libraries provide single-event processing with very low latency and high throughput. When exchanging data over a Kafka topic, you do need to carefully watch consumer lag (i.e., queue depth), which is a source of latency.

Kafka Streams doesn’t attempt to provide a full spectrum of microservice tools, so it’s common to embed Kafka Streams in applications built with other tools, like Lightbend Reactive Platform and the Spring Framework.

Akka Streams also implements the Reactive Streams specification. This is a simple, yet powerful, standard for defining composable streams with built-in back pressure, a flow control mechanism. If a single stream has built-in flow control, such that it can control how much data is pushed into it to avoid data loss, then this back pressure composes when these streams are joined together, either in the same process or across process boundaries. This keeps the internal stream “segments” robust against data loss and allows strategic decisions to be made at the entry points for the assembly, where it is more likely that a good strategy can be defined and implemented. Hence, reactive streams are a very robust mechanism for reliable processing. If you don’t use a Kafka topic as a giant buffer between microservices, you really need a back pressure mechanism like this!

Neither library does automatic data partitioning and dynamic task management associated with these partitions, so they are not as well suited for high-volume scenarios as Spark and Flink. Nothing stops you from implementing this yourself, of course, but good luck…

When you are comparing these two libraries, perhaps the most important thing to understand is that each emerged out of a different community, which has influenced the features it offers.

It’s useful to know that Kafka Streams was really designed with data processing in mind, while Akka Streams emerged out of the microservices focus of Akka. Compared to Flink and Spark, neither library is as advanced at the Beam-style streaming semantics, but Kafka Streams is very good at supporting many of the most common data processing scenarios, such as GROUP BY and JOIN operations and aggregations. When you want to see all records in a stream, the usual way of using Kafka topics, Kafka Streams offers a KStream abstraction. When you just want to see the latest value for a key—say, a statistic computed over data in a stream—then the KTable abstraction, analogous to a database table, makes this easy. There is even a SQL query capability that lets you query the state of a running Kafka Stream. In contrast to Spark and Flink, SQL is not offered as an API for writing applications.

Compared to Kafka Streams, Akka Streams is more microservice oriented and less data-analytics oriented. You can still do many operations like filtering and transforming, as well as aggregations like grouping and joining, but the latter are not as feature-complete as comparable SQL operations. Of the five systems we’re discussing here, Akka Streams and Beam are the only ones that don’t offer a SQL abstraction (although Beam’s semantics fit the requirements for streaming SQL very well). Similarly, Akka Streams doesn’t implement many of the Beam semantics that the other engines implement for handling late arrival of data, grouping by event time, and so on.

So why discuss Akka Streams then? Because it is unparalleled for defining complex event processing (CEP) scenarios, such as life cycle state transitions, alarm triggering of certain events, and building a wide range of microservices. Of the five engines we’re discussing, only Akka Streams supports feedback loops in the data flow graph, so you’re not restricted to DAGs. If you think of typical data processing as one end of a spectrum and general microservice capabilities as the other end, then Akka Streams sits closer to the general microservice end, while Kafka Streams sits closer to the data processing end.

Okay, So What Should I Use?

I hesitated including four streaming engines, plus Beam. I was concerned about the paradox of choice, where if you have too many choices, you might panic at the thought of making the wrong choice. The reason I included so many options is because a real environment may need several of them, and each one brings unique strengths, as well as weaknesses. There is certainly enough overlap that you won’t need all of them and you shouldn’t take on the burden of learning and using more than a few. Two will probably suffice—for example, one system like Spark or Flink and one microservice library like Akka Streams or Kafka Streams.

If you already use Spark for interactive SQL queries, machine learning, and batch jobs, then Spark Streaming is a good choice that covers a wide class of problems and lets you share logic between streaming and batch applications. To paraphrase an old saying about IBM, no one gets fired for choosing Spark.

If you need the sophisticated semantics and low latency provided by Beam, then use Flink with the Beam API or Flink’s own API. Choose Flink if you really are focused on streaming and batch processing is not a major requirement.

Since we assume that Kafka is a given, use Kafka Streams for low-overhead processing that doesn’t require the sophistication or flexibility provided by the other tools. The management overhead is minimal, once Kafka is running. ETL tasks like filtering, transformation, and many aggregation tasks are ideal for Kafka Streams. Use it for Kafka-centric microservices that process asynchronous data streams.

Are you coming to streaming data from a microservice background? Do you want one full-featured toolbox with the widest spectrum of options? Use Akka Streams to implement your general-purpose and data-processing microservices.