Complex event processing (CEP) is another analytic engine that is often used for pattern detection. From its roots in discrete event simulation and stock market volatility trading in the 1990s, it is by nature, a method capable of analyzing a live feed of streaming data in near real time. As hundreds and thousands of events enter the system, they are reduced and distilled into higher-level events. These are more abstract than raw sensor data. CEP engines have the advantage of a fast turnaround time in real-time analysis over a stream processor. A stream processor can resolve an event in the millisecond time frame. The downside is a CEP doesn't have the same level of redundancy, or dynamic scaling, as Apache Spark.
CEP systems use SQL-like queries, but rather than using a database backend, it searches an incoming stream for the pattern or rule you suggest. CEP consists of the tuple: discrete data element with a timestamp. A CEP makes use of the different analytic patterns described at the beginning of this chapter, and works well with a sliding window of events. Since it is SQL-like in semantics, and it is designed to be appreciably faster than a regular database query, all the rules and data reside in memory (usually a multi-GB database). Additionally, they need to be fed from a modern stream messaging system such as Kafka.
CEP has operations like sliding windows, joins, and sequence detection. Additionally, CEP engines can be based on forwarding or backward chaining as rules engines are. An industry-standard CEP system is the Apache WSO2 CEP. WSO2 coupled with Apache Storm can process over 1 million events per second, with no storage events needed. WSO2 is a CEP using an SQL language but can be scripted in JavaScript and Scala. The additional benefit is that it can be extended with a package called Siddhi to enable services such as:
- Geolocation
- Natural language processing
- Machine learning
- Time series correlation and regression
- Mathematical operations
- String and RegEx
Streams of data can be queried as in the following Siddhi QL code:
define stream SensorStream (time int, temperature single);
@name('Filter Query')
from SensorStream[temperature > 98.6'
select *
insert into FeverStream;
This all operates as discrete events allowing for sophisticated rules to be applied to millions of events transpiring simultaneously.
Having described a CEP, it is best for the architect to understand where a CEP and rules engine should be used. If the evaluation is a simple state, such as two ranges of temperatures, then the system is stateless, and a simple rules engine should be used. If the system maintains a temporal notion or a series of states, then a CEP should be used.