CHAPTER 5

Markets and Emerging Markets for CEP

This chapter provides a review of markets for complex event processing products and the purposes for which those products are bought.

Event processing is now a basic component in many different IT systems used in business and government. This often includes complex event processing, as explained in Chapter 3, and in particular the section in Chapter 3 entitled “Complex Event Processing and Systems That Use It.” As a result, the number and size of markets for CEP applications is growing at a significant pace.

This chapter is a survey of market areas for commercial CEP products and the uses to which this technology is being put. Our purpose here is to give the reader a feel for the range of CEP applications that are in the field now—as well as future market areas where event processing is being tested experimentally at the moment. But we must add the caveat that this survey is not intended to be complete or exhaustive. It is simply a selection of CEP applications at a moment in 2011. Hopefully it will remain valid for a few years.

There have been a number of research studies of the size of markets in financial terms for CEP products and services. Estimates seemed to agree that the total market for 2009 was somewhere in the region of $190 million in sales of products and service contracts for their maintenance, and that the market rose to roughly $280 million for 2010. This was deemed by the market researchers to indicate “an emerging market in the early stage of development.” Predictions are for an average growth rate of about 30 percent per year to around $580 million in 2013.1

One of the difficulties with these studies is separating event processing products that apply CEP technology2 in some of their operations from those that do not use any CEP at all.

The CEP market studies have made strenuous efforts to specify the markets and have devoted much of their space to classifying the market areas, and mentioning by name the vendors, products, and sometimes consumers that have been included in the study.

In keeping with our guiding principle in this book, we do not mention names of companies making the studies, nor those that are subjects of the market studies. But these studies are easily accessible, possibly at a price, to readers who are interested in more details.

The event processing market in general is a multibillion dollar market—estimated at up to $5 billion in 2010. But a lot of conventional business event processing is for applications that deal with “business events” for recordkeeping and routine operational purposes such as accounting. They simply store events in a database and, later, employ data mining methods for business intelligence.

We exclude this kind of processing of events. We are dealing here with that part of the broad EP market that uses CEP techniques for right now applications. That leaves us with a much smaller market!

The only products that are counted in the estimated $280 million (2010) CEP market are the general-purpose event processing products in which the CEP technology can be configured to deal with different problems and is not hardwired for any particular business problem. These general-purpose products can deal with multiple types of events from diverse sources and can be applied to a wide variety of problems. They contain development tools that make it relatively easy for software developers and expert knowledge workers to create and modify CEP applications. Development facilities include event pattern languages and tools to test and debug systems of event pattern–triggered rules and to replay and analyze the performance of rule-based systems on event streams. The products typically also include adapters to connect to databases, external analytical tools, business dashboards, sensors, actuators, and a variety of event sources, such as messaging systems and market data feeds. Both end users and vendors use these general-purpose platforms to build many kinds of customized CEP applications.

We have excluded the market for hardwired event processing applications—that is, applications intended for a specific use or problem and not generally configurable for use on different problems—which is estimated at $5 billion by some research studies. These hardwired special-purpose event processing applications are to be found among the systems used in supply-chain management, customer contact centers, network management tools, risk management, fraud detection, and manufacturing control systems. The CEP technology in these products is narrowly focused on solving a specific industry or business problem. Their use of events is usually relatively simple; the adapters are oriented toward only a few kinds of input and output; and the development tools are limited or nonexistent, because the buyer is not expected to modify the application by adding new event patterns or changing the configuration of event processing operations. Although such products implement CEP principles, their buyers and users do not typically think of them as CEP tools—they are known by their business purpose. These types of CEP applications are not counted in our market estimates.

Also excluded are CEP applications that compute offline. Generally, these are business intelligence (BI) tools used to generate business reports or answers to queries submitted by an analyst or manager. Virtually all BI tools perform computation on events offline in online analytical processing (OLAP) cubes or other forms of data warehouses in memory or on disk. In the course of their work, they perform many of the same kinds of operations found in the general-purpose event processing platforms. For example, they filter and correlate events, calculate aggregates (sum, average, rank top-k, and so forth) and detect instances of event patterns. These applications do involve CEP, if one uses a broad, academic definition of the term (“any computation involving complex events”). However, they are not considered part of the CEP market, and the term CEP is almost never used to describe their role because they work “after the fact” and not in right now time.

The reader of this chapter will be struck by the underlying similarity of event processing applications in vastly different areas. All of our examples are summaries of actual event processing applications, either fielded or under experimental testing. For example, applications in fraud detection and in, say, health care rely on the same templates for event triggered rules and the same rules engines. Of course, the actual rules are different in each case. The differences lie in the types of events being input and output, the event patterns that trigger the rules, the types of actions and alerts that the rules output when they are triggered, the rates at which events arrive or are output, and the states of the rule processors. But the event processors in each case could be the same rules engine.

We have written the examples so that a reader can skim through and get a good idea of the similarities in the event processing as well as the actual kinds of events and rules in use in each area. To emphasize the similarities, we give as much detail of the event processing in various examples as space will permit.

Market Areas

The overall CEP market can be divided into categories in a hundred different ways. None of them are really satisfactory, and any set of categories overlap. So, with that said, we’ll simply choose one categorization. Our purpose is simply to manage a broad overview by breaking the market into application areas and dealing with some of the main subcategories within each area.

Here are some of the major areas for sales of CEP products and services, not in any significant order:

• Financial systems, operations, and services
• Fraud detection
• Security
• Transportation
• Health care
• Energy
• Consumer relations management (including online sales and marketing)
• Operational intelligence (in business and government)
• Location-based services
• Military applications

We won’t be able to give details of applications in all of these areas, so we’ll take the major ones that have the least overlap.

Financial Systems, Operations, and Services

Financial systems, operations, and services are the largest market for CEP in dollar terms, accounting for about 40 percent of the total revenue for general-purpose event processing platforms. This category covers a variety of application areas with different requirements for CEP processing. Typically, the high-performance event driven applications in terms of event throughputs and processing speeds are in the stock and financial instruments front-office trading sector. They involve processing multiple event feeds in right now time at event speeds on the order of tens of thousands of events per second required to make buy or sell decisions in hundredths of a second. But most of the other applications in the financial sector, such as middle- and back-office applications, involve lower event throughput rates.

Some of the main areas of financial systems, operations, and services are:

• Stock (equity) trading using market data feeds. Systems in this area simultaneously process several event feeds containing reports of stock trades in different markets. They use that information in milliseconds to create further trading events when conditions are judged to be favorable. This category of event processing includes completely autonomous trading systems in algorithmic trading and futures trading, as well as systems that have human traders in the loop.

  Financial trading systems, whether human-in-the-loop or autonomous, mostly process the data carried by events in the feeds. The algorithms applied to the trading data are proprietary, although the trading strategies share many of the same fundamentals.

  Both general-purpose event processing platforms and domain-specific trading platforms are used in exchanges and in virtually all large banks and other financial institutions that participate in capital markets. Usually, the event processing involves applying proprietary algorithms to streams of events in real time as the events arrive. Often the algorithms use of patterns of multiple events, and CEP techniques and pattern detection engines are creeping into these systems.

• Arbitrage, using secret sauces (algorithms). This involves either arbitrage between different exchanges trading in the same instrument or between the price of the same instrument at different times in the future at the same exchange. Again, these are very high-speed, multiple-feed event processors, and may involve some detection of patterns of multiple events—for example, tracking the behavior of predetermined sets of stocks over small sliding time windows on each of several markets. The detection of specified patterns triggers sets of trading orders in the markets being tracked, resulting in sell orders in some markets and buy orders in others.
• Foreign currency trading in multiple markets. Complex and proprietary algorithms are in use to detect favorable trading situations between multiple markets; favorable situations can happen and disappear in seconds, so these applications are often automated with the occasional intervention of human overseers.
• Automated pricing of financial instruments. Many types of securities, derivatives, mortgages, and so on are increasingly subject to real-time pricing that reflects minute-by-minute fluctuations in the markets, as well as demand by consumers. These systems employ event processing methods; some CEP is involved.
• Dynamic credit risk computation and credit rate selection for online credit services offered by banks. This area uses CEP to track customer behavior at ATMs and online banking and to compute customer’s credit ratings in real time. This kind of event processing is used both when customers apply for loans, as well as to make unsolicited loan offers to those who qualify. Event patterns are used to detect behavior that might indicate the need for loans and to determine whether customers qualify for loans. These patterns are becoming more sophisticated as banks increase targeted marketing in this area while attempting to reduce risk.
• Near-real-time (or “right now”) profit and loss accounting. Autonomous trading cannot be allowed to run wild! High-frequency autonomous trading has an increased risk of accumulating either profits or losses very rapidly. The use of automated trading systems has therefore been accompanied by the introduction of automated real-time monitoring of their profit and loss performance. This also enables feedback from the profit and loss system to the trading system to adjust the amount of capital available to trade. Trading systems can now adjust their strategies in real time to attempt to maximize trading profits or reduce risks of losses.

  For example, the more profit that is made with a given strategy, the more resources are made available to the trading system to use that strategy. The performance and risks of different strategies can be tracked in real time, and the balance between the strategies can be adjusted minute-to-minute within the trading system.

• Compliance monitoring for violations of company policies and government regulations in capital markets, health care, and many other industries. The activities of individual traders, workers, companies, and exchanges may be monitored across multiple markets and over time. Generally, very simple searches for violations of regulations and policies have been used in the past, and they have often been done by database searches at the day’s close of business. But real-time CEP is arriving in compliance monitoring. This is in the form of event patterns to detect violations as they happen. Detecting a pattern of activity that adds up to a violation will trigger reporting and, sometimes, automatic enforcement of rules for corrective actions.
• Fraud detection. Amazingly simple-minded event processing has been used to detect fraud in many cases, perhaps because crooks are simple-minded too! However, there are some applications in fraud detection where event patterns, perhaps even patterns whose detection may require time windows of several weeks, are now being used.

Many types of companies are involved in the areas of activities listed above as users of event processing products and CEP. We will refer to them as “users of event processing.” They include large investment banks, smaller hedge funds, savings and loan associations, bank branches, mortgage lenders, credit card issuers and servicers, and security services companies.

Business activity monitoring (BAM) within the financial services area now can encompass all branches of a company, whether it is large or small, in every community across the country. Generally speaking, the EP applications in the areas listed above employ only very simple kinds of event processing and CEP. For example, in the past one found that most of the time, only the data carried in the events were used in the trading algorithms and market opportunity analyses. If patterns of events were used at all, they tended to be looking for a few trades or quote events occurring together within a small time window of a few minutes.

But today there are an increasing number of exceptions to this. More sophisticated event pattern processing is being used involving a wider range of event types, including elementized news feeds, weather reports, and data about user or corporate changes. This is due in part to the improved capabilities of the available CEP products, and also to front-line experience in many areas that points to the need to detect complex patterns involving many events over time.

Example 5.1 illustrates the aggressive use of event processing by a bank in an area that is an aspect of “customer relations management.” In fact, the same system using different event triggering rules would be able to process customer complaints, loan defaults, and suspicious activity. It is thought that the use of right now event processing in marketing banking services, rather than traditional methods based upon data mining customer records, has an advantage in timeliness. The bank has determined from its customer data that this system seems to have improved sales of the services by about 20 percent.

But there are pitfalls with such systems, not the least being to ensure that customers are not put off by overly aggressive tracking of their activities or inundated with offers. So these systems have to have constraints on their marketing activities; for example: “Send an offer only if the customer has not received any offers in the past week.”

Example 5.2 shows another area where CEP can be an asset.

The area of compliance monitoring became a high profile issue with the arrival of government regulations such as Sarbanes-Oxley3 in 2002 and the Basel Accords.4 Large banks and stock market companies have always had their own in-house policy monitoring. Often this goes well beyond the government regulations and includes monitoring of all operations within the company for compliance to the corporation’s policies. However, it is true to say that a lot of this work is simply database monitoring to determine compliance by the day’s close of business, rather than real-time event monitoring. But this is also an area where practices are becoming more right now, because preventing errors from happening costs less than unwinding them after the fact. Consider the error described in Example 5.3.

This is an example in which existing event processing technology, using trading event patterns to monitor stock market behavior in right now time, could have monitored the markets for anomalous behavior and alerted the parties involved when it was detected. Had this technology been used, red alerts would have been going off, with real-time risk analytics highlighting impending problems. The regulators would have been able to see an “early warning” and respond from a central control market watch system.

In summary, the financial systems, operations, and services area involves very sophisticated right now processing of the data in events, often at very high throughput speeds, and, to a somewhat unquantified extent, also the use of event patterns and CEP. It is continuing to evolve as a major market for CEP vendors.

One of the major impediments to standardizing the use of CEP in financial services (e.g., by introducing off-the-shelf libraries of event processing rules and specialized event processing agents; that is, components of EP systems) is the secretive and proprietary nature of much of what goes on in this area. However, the building of CEP applications in financial services does get some help from the wide availability of libraries of commonly used basic functions such as weighted average computation, regression, and other statistical operations. Costs of EP products and services to consumers in this area can be expected to remain high. But then, they can usually afford it!

Fraud Detection

Monitoring real-time event activity for fraud is a second large market for event processing and CEP products. Many instances of fraud detection overlap with both financial operations and services, as well as with the security area, and can legitimately be classified as being in those areas of CEP applications. We describe some fraud applications separately here.

Some market estimates put fraud detection at more than $1 billion per year. But these include all manner of after-the-fact detection methods. We are interested only in right now applications of EP and CEP that are intended to catch fraud in the act, so the market is smaller than that. On the other hand, fraud detection spreads itself into many markets beyond capital markets—as we shall see.

Consumers of fraud detection products include the expected players, such as banks and credit card companies, and also government agencies such as Homeland Security. In addition, they also include all manner of other types of businesses—particularly energy, telcos, health care, insurance, online retailers and auction houses, and online gaming. Internet-based operations are big users of fraud detection.

Many fraud detection systems were homegrown within the user organizations. These systems evolved over time as fraudulent techniques appeared—fraud detection is very much an arms race between the good guys and the bad guys. Some systems have in fact gotten so unwieldy—not much more than very large sets of conditional rules codifying cases of actual fraud in a programming language such as Java—as to be in need of a complete redesign.

More recently, many of these systems have been augmented by CEP products. Fraud-detection systems using CEP are rule-based systems with rules triggered by patterns of events that signify possible fraudulent behavior.

Fraud can originate both in-house and outside a company. CEP products are used to track many different types of events within a company, including user network activity, file accesses, database accesses, authentication, and any other events that might add up to suspicious activity on the company’s networks. Such activity can originate with employees, outside persons with access rights to the company’s networks, or via various types of malware and spyware. Some estimates put the market for fraud-detection products within the credit card industry alone at more than $1 billion, with the caveat that a lot of it is not what we would call right now event processing.

Credit card companies are large consumers of EP and CEP products; Example 5.4 demonstrates why.

Detection of event patterns that indicate the use of a stolen card is most effective if it is done in real time by automated event processing. One problem with automated systems is to reduce false positives.6 The pattern in this example consisted of a timed sequence of card charges, with each charge increasing in value. It was obviously not the holder’s normal pattern of use. And it is also easily codified for automated systems checking—a monotonically increasing set of charges over a small time window.

Surprisingly, the sets of event patterns in common use to detect credit card fraud are often overly simple-minded. Single events can trigger a card’s closure. For example, fraud detectors for credit cards use the location and type of merchant processing the charge as primary triggers. Using a card in a jewelry store or an electronics store away from the card holder’s home location is often enough to trigger a closure of the card.

The simplicity of these event pattern triggers contributes to false positives—so much so that it often leads to very annoyed and sometimes embarrassed customers! Adding in-memory state knowledge of the recent history of card use and location of use to these simple triggering rules would often help in this regard. For example, if the holder has recently used a credit card to settle hotel and restaurant bills in a location away from home, then perhaps use of that card in a jewelry store in the same geographic location should trigger an alert to contact the card holder—but not an immediate closure of the card account.

Credit card fraud often involves theft and use of card information without the actual theft of the card itself. This kind of fraud is enabled by card company policies that allow cards to be used over the telephone or Internet without any physical check that the user is in possession of the card—the three digit security code on the back of the card is usually required in Internet transactions nowadays, and therefore can be stolen just as easily as the card number on the front! As a result, there is a big business in some parts of the world—Eastern Europe is a well known example—in the sale of bundles of stolen credit card information.7 There are websites that auction these card data bundles and even keep customer satisfaction profiles of the sellers. Card issuers consider this type of fraud to be a price of doing business, but they are willing to invest in people and software—typically software with a CEP component—to minimize the damage. Ultimately, however, the card holders pay indirectly in higher fees.

In summary, fraud detection is an expanding area of event processing applications. The currently fielded event processing systems can and should be improved with the introduction of rule-based CEP products. The field is in a continual arms race with the fraudsters! Fraud-detection systems are often antiquated from an event processing viewpoint and are in a continual state of improvement. Secrecy on the part of the event processing users in fraud detection has a negative impact on any efforts to standardize the libraries of high-level CEP rules and reduce the costs of event processing systems for this market area. However, there is a positive trend toward building standard sets of CEP rules, which are being packaged and sold.

Transportation

The airlines, railways, trucking, and shipping industries are all using CEP products and event processing in running their operations.

Airlines

The airlines have been the most obvious and prominent early adopters of event processing products in various parts of their operations. Different airlines have tested and fielded CEP in different aspects of their operations.

Some airline companies use CEP systems to run their aircraft maintenance operations—for example, to schedule the routine maintenance of aircraft, monitor parts inventories, and trigger reordering of parts.

Other airlines are using EP and CEP products in flight operations systems that do real-time tracking of flights, coordination and transfer of passengers making connections, rebooking of passengers who have missed flights or missed connections, and of course baggage handling8 and other aspects of customer services.

Also, there have been cooperative agreements between airlines, typically partnering in flight scheduling and sharing, that involve using CEP in the implementation of the agreements (see Example 5.5).

Railways

Over the past few years, railways have been using CEP to help monitor various aspects of their operations, such as the status of trains, freight cars, shipments, crews, and equipment. For example, CEP is used to help guide crew assignments to reduce the loss of work time due to safety requirements. It is also used in systems that control the scheduling, composition, and operation of the trains.

There is one overarching goal of using event processing systems, which has been summed up succinctly in a quotation by a senior industry official:

If we could increase our average speed by just one mile per hour, it would mean an additional $140 million to our budget.

This quotation was in relation to goods transportation by a national railroad company with operations spanning the United States.

CEP systems have since been deployed within day-to-day operations of some railway companies in the United States and Canada (see Example 5.6 for one company’s experience). A good example of this use is the reduction of fuel costs. Fuel is the second largest cost, after labor, for railways. Using CEP, companies are able to calculate where and when to buy fuel for each train to minimize the prices paid, taxes paid, refueling time, and the cost of carrying extra fuel between places where fuel can be sourced. Airlines make even more intensive use of CEP in their operations to reduce fuel costs.

Future planning is to upgrade the event processing in this train-tracking system so that it is capable of tracking not just the trains and freight cars, but each individual shipment within the freight cars. The advantages are in being able to schedule the movement of freight and the composition of each train in a single system that maintains consistency with operational constraints and cargo schedules over the whole of the railway’s operations. Crew schedules will be created automatically to optimize crew workloads.

Trucking

Trucking is another transportation sector that is using CEP products in the running of operations. There are a few examples in this industry sector of the use of event processing systems, including rule-based CEP, in the management of trucking fleets (see Example 5.7).

Example 5.8 shows the use of a CEP rules engine in trucking operations to monitor and control a fleet of delivery vans within a large and congested metropolitan region.

For example, the rules in the CEP fleet control system monitor:

There are work orders that allow a driver to choose any route. But there are general safety constraints. For example, there are a number of areas of the city that must be avoided, especially at night. The system alerts the fleet supervisor when a truck deviates from its work order, runs behind schedule, or is held up in traffic.

When a driver is approaching a destination, the system will trigger a “package delivery soon” alert to the customer waiting for the delivery. This prepares the customer to sign off on the delivery and shortens the time spent at delivery points.

Typical event pattern rules that are monitored for any work order are:

If a van diverts from its designated route for more than five minutes

then send an alert to the supervisor.

If a van is more than 20 minutes between the waypoints

and it is a weekday

and it is not Friday between 4pm and 6pm

then send an alert to the supervisor.

If a van does not leave the starting point before 1pm or

If a van does not arrive at its planned final destination before 6pm

then send an alert to the supervisor.

These constraints operate as filters on the console input. They focus the operator’s attention on patterns of behavior that violate company rules or workplan requirements.

The advantages for the delivery van company are that the number of supervisors needed has been reduced to one. Before the introduction of the CEP-based rule system, there was a small army of supervisors doing monitoring. Experience shows that there are only a few violations per hour, which show up as alerts to the supervisor on the control screen.

Instead of watching a map with lots of moving vehicles, the supervisor can spend the time trying to resolve violations. This work is more interesting than watching a map. Also, the system improves customer service, because violations are detected early and many times they can be corrected so the customer is not affected.

Shipping

Within this industry, container ship companies have been experimenting with event processing and CEP in the scheduling of ship trip plans. Here the goal, as with trucking, is to reduce costs by increasing the efficiency of overall operations.

A container ship trip plan is somewhat different from trucking. First of all, the size of modern vessels is overwhelming. Some measure 400 meters in length and can carry loads of up to 15,000 truck-size containers, each 20 feet in length. Most containers used today are bigger, and measure 40 feet (12 meters) in length. Loading a container ship is a complex scheduling task in itself. Safety and security constraints must be adhered to. Risk reduction is a prominent concern. Accidents at sea happen. Containers are lost at sea, and container ships have actually capsized.

A container ship’s trip plan might well take several weeks, say across the Pacific, with cargo deliveries and pickup scheduled at ports along the way to the final destination. A trip plan may be changed at almost any time. New orders may come in at company headquarters, triggering a decision to reschedule a ship for a new stop. Or events on board a ship, such as bad weather or accidents, may also trigger changes.

A unique problem is the scheduling of the loading and unloading of containers. This scheduling must optimize the container positions on board the ship so that those to be unloaded earliest are scheduled to be loaded last. Since new orders for cargo transportation are coming in continuously, the shipping companies are experimenting with using real-time event processing as an approach toward monitoring and running all the company’s trip plans. Decisions must be made concerning to which ship the cargo should be assigned. There are real-time delivery constraints to be maintained by the trip plan. As cargo is loaded, the status of a ship and its trip plan may change, as may the overall costs of operating the trip.

From an event processing viewpoint, a container ship trip plan is a continuous event processing engine with input events, operational state, and output events. The event throughput rates for high-level events dealing with cargo and ship operations are slow compared with, say, stock-trading systems. Event inputs to and from a single ship vary from, say, 100 events per hour during certain operations, such as loading and unloading cargo, to 100 events per day while at sea. The state of the trip plan must include all the constraints of the vessel itself, the current state of the load, and many other factors involving actual day-to-day operation. Output events can be updates on the status of operations, alerts about conflicts with cargo schedules, and even refusals to load particular containers.

This is one of many areas in which event processing technology is likely to become so influential as to change the way things are done. The use of event processing technology in the running of container ships will lead to a revolution in container ship operations. Specifically, we may expect changes in the way trips are planned, how trip plans are represented both for humans and automated tracking systems, and how humans interact with trip plans. Event driven technology will be used to construct trip plans and to drive container ship operations according to the trip plans. CEP is just the beginning!

In summary, the use of event processing systems and CEP in the transportation industry is still in its infancy. The potential for EP and CEP in right now operations is clear in every one of the industry sectors. This is another industry in which custom-coded applications were in place in many companies. As a result, CEP vendors have had to work hard to sell to this industry. Many of the current CEP applications are in an experimental proof of concept phase. But there are also many instances where EP and CEP technology is now part of day-to-day operations—in other words, EP and CEP have been adopted. And one may expect the market to open up more to the CEP products and vendors in the future.

Security and Command and Control

The security area is another large market for event processing, closely related to fraud detection. Security is about preventing problems from happening, in contrast to fraud detection which is about detecting problems as they happen or after they happen. Security is the first line of defense of the business—or the country—whereas fraud detection is a working tool, part of doing business. But truth is that similar systems are at work in both areas.

Computer security systems for most businesses used to be glorified firewalls. It was simply a matter of not letting anyone onto the IT system or the company database unless they had the right privileges. But this is changing, and systems that are categorized as security systems have multiple purposes.

Security systems these days do, or should do, a lot more than act as gatekeepers. They must assume that if anyone is allowed in, then those who shouldn’t get in will (as demonstrated in Example 5.9)! So a security system must continuously monitor the event traffic on the company’s networks for unusual or unauthorized traffic. Doing that requires knowledge of the event patterns that signify legitimate traffic, as well as the security constraints against illegitimate traffic. Violation of security constraints must be detected immediately.

Command and Control for Security

The security category is perhaps a bit stretched these days, in that some security systems are now fully fledged command-and-control systems that run the company’s operations as well as protect the company. These kinds of security systems make extensive use of event processing and CEP. They are in use by national and local government agencies and by most large and medium-sized companies, no matter the business area. The Immigration and Naturalization Service (INS) and the national intelligence agencies such as the National Security Agency (NSA) and the Central Intelligence Agency (CIA) use these systems, as do large metropolitan police departments. In the business world, any company with intellectual property to protect, trade secrets, customer lists, or the like will employ an event driven security system as part of its operational IT infrastructure.

The use of CEP in security systems varies; some use a lot, others not so much. For example, some event processing systems in Homeland Security or the INS search for threats that may require detecting patterns of events across sets of very different types of event feeds over long periods of time.

As an illustration of event processing in command-and-control systems, we take an example from a large metropolitan police department’s central operations system. Example 5.10 is a proposed future evolution from the system that is currently in operation. It has all the aspects of event processing for command and control and security.

The overall goal of this system is public safety—to keep the community secure. Crime prevention is a major part. As such, it is categorized as a security application. But a large segment of its application is in dealing with crime and emergencies in progress or that have already happened. As such it could equally well be categorized as command and control.

We are concerned here purely with the right now event processing involved in the system.

In summary, many large businesses with right now time commitments have security systems with features similar to many of those in the example. Large international banks, leading transportation and delivery companies, and energy companies are some examples. As we commented before, security systems are encompassing more and more aspects of command-and-control systems that run the business in a secure manner.

Health Care

Health care is a growth area for event processing applications. Computers and information systems have been used for collecting patient data in health care for fifty years. But progress toward a unified national health care delivery system has been slow. Currently, many different specialized systems are employed, each with part of a patient’s treatment record, and there is no central system with a complete history. As a result, a lot of relevant information is never included in evaluating a patient or a treatment plan. And costs of medical care have not been reduced. This situation obviously has to change.

A vision of the future is one system across all hospitals, all patients, all diseases and conditions and diagnostic tests. One goal of the vision is a system that automates the coordination and delivery of a patient’s treatment over his entire lifetime history. Another is a system that enables remote medical diagnosis and treatment by consultants and specialists who may be far away from a patient’s location. Everything must be done in right now time, and all relevant information must be immediately available to the medical staff who need it. These systems will be event driven.

There are also projects to develop medical event driven systems for the early detection of emerging epidemics across the world. We will discuss these systems later.

In general, progress has been slow. Most event processing applications in health care delivery so far are rather mundane event gathering operations for accounting and records. But there are examples of experimental systems that apply event processing and CEP principles in health care delivery, for example, in the monitoring of the treatment of patients in hospitals. Example 5.11 shows one such system for emergency room processes.

In summary, while information systems have been used in health care systems for many years, their use has been largely old-fashioned. The adoption of event driven architectures and CEP technology where it can obviously help has been slow. Privacy issues are a continuing concern and something of a brake on progress. There are, however, many ongoing experiments both in the United States and Europe introducing event processing methods for right now health care delivery, not only in running healthcare systems efficiently, but also as a support technology in new infrastructures for delivering medical treatment remotely (e.g., when the patient and physician are located in different parts of the country). Health care is a growth area for EP and CEP.

Energy

Control systems for the generation, trading, transmission, distribution, and consumption of energy are another area where the potential for right now business-event processing and CEP is obvious. Perhaps the investment at the moment is not as large as one might hope, but it is a good bet that this will become an important market. It holds many lessons for modern business infrastructures in general, and much can be learned about the kind of event driven technology that is needed from the past failures of these systems.

Control of electricity grids to keep the flow of electricity smooth and stable—so-called smart grids—is a prime example to be studied. This is where the business-event processing should occur. However, the energy area includes other infrastructure resources, such as dams and water supplies, oil and gas transmission lines, power plants and sewage plants. Again, it is difficult to quantify the investment in EP in this area, but there is no doubting the importance of energy systems in the infrastructure of our society. The quickest way for a terrorist to bring the country to a standstill would be to take down the electricity grid!

Not only are energy control systems event driven, but they need to become secure command-and-control systems, very much along the lines described in the security section. There are plenty of examples of what can happen when a control system for an energy resource has inadequate security. Perhaps the most famous of them is the Australian sewage disaster described in Example 5.9.

Electricity Grid Failures

Not so long ago. electricity grids both in the United States and Europe had catastrophic failures.

A notable failure was the great northeastern United States blackout of August 2003. At that time, portions of the grid in different states in that area of the United States were controlled by different ISO15 companies. The grid failure resulted from a sequence of events starting with a simple broken transmission line, then a power generator tripping and going offline, and so on, over a period of four hours. Events cascaded in a domino effect. Final events were very large, such as power transmission reversing direction around Lake Erie. There was speculation among the grid controllers in the various ISO control rooms that the events they were seeing on their control displays were caused by a cyber attack on their SCADA systems. The final result was a blackout across the northeastern United States that left 50 million customers and parts of eight states and Canada without power. The outage cost an estimated $7 billion to $10 billion in financial losses and shut down parts of a two million barrel-per-day oil pipeline and airports in thirteen cities.

This grid failure was a complex event comprised of a cloud of events in the SCADA systems and diagnostic systems that control a number of collaborating grid systems over a four-hour period (12:00 ~PM to 4:15 ~PM). The ongoing Department of Energy (DOE) report (November 2003) lists thirty-five significant events in the failure, but not their causes or relationships.

According to the New York Times,16 nobody had a global view of the developing complex event:

In the end, then, it was not just a circuit breaker tripping or a transmission line sagging into a tree that caused the system to fail. Documents and interviews make clear that the blackout may well have resulted, just as surely, from the fact that the people whose job it was to respond to those failures lacked much of the information about what was happening. They were, that afternoon, like air traffic controllers trying to keep order in the sky without knowing where all the planes were.

Both low-level grid events (e.g., “low voltage online,” “power line open,” “generating element tripped,” “grid pathway closed” events) and higher-level policy events were involved.17 For example, to take the strain off Cinergy’s lines, the Midwest ISO turned to another power company, Allegheny Energy, asking it to help by reducing the electricity it was pumping out. But at 2:24 ~PM, an Allegheny controller told the ISO that the company’s marketing staff wanted to do the opposite of what the ISO was asking so they could make money selling more power!

The overarching problem, of course, is that these electricity grids were one big connected system. Of the thirty-five events over four hours listed in the DOE report’s timeline of events, the last twenty-five or so happened in the last ten minutes leading up the blackout. So a grid event monitoring system has to detect failure event patterns early in their process of happening in order to take remedial action. It has to have a global view of the whole system.

There have been similar grid failure blackouts in Europe as well. For example, most of Italy was blacked out in September 2003 as a result of a grid failure that started in southern France or Switzerland, depending upon which report you believe. However, not everyone in Italy was unhappy with the blackout. One newspaper report of the time quotes a twenty-one-year-old male student caught in a Rome night club when the lights went out: “there was panic, especially among the women.” Similar blackouts in France and Germany have happened more recently.

A common element in all of these failures is lack of a grid event monitoring system that can monitor complex patterns of events to detect and correct possible failures before they happen.

Smart Electricity Grids

Some estimates put the number of current projects to develop smart electricity grids at well over one hundred worldwide. Investments in the United States in 2010 include Department of Energy (DOE) awards of $3.4 billion in stimulus grants toward upgrading the nation’s energy grid, and an additional $4.7 billion in private funds invested through the DOE program.

A smart grid18 is really two grids: a power transmission grid coupled with a two-way event driven monitoring and control grid. It is an essentially event controlled system. The overall goal is efficiency in the generation and use of power combined with reduction of costs.

The two grids are coupled in the sense that the SCADA measuring and control instruments on the power transmission grid are part of the control system, and conversely, the control system can send control events to the power transmission grid.

The control grid is where the complex event processing for the grid takes place.

There are twenty-year plans for developing smart grids that will use superconductive transmission lines to minimize power loss and will include intermittent electricity sources, such as solar and wind. These plans will use business event processing to support “business smarts” to control the grid and encourage the consumers to use power intelligently.

For example, the types of electricity usage and the pricing will vary with levels of consumption. Consumers can allow the grid to schedule power use within the home (e.g., turn on selected home appliances, such as washing machines, at off-peak hours). Factories can allow the grid to schedule selected processes to run at arbitrary hours and turn off those processes at peak times.

Event processing in the control grid will be bidirectional. Smart meters at consumer residences will monitor types of consumption and send report events back to local area control systems. The control system will respond by sending events containing instructions back to the individual meters. This will enable various types of consumption to be turned off or on automatically depending upon factors such as demand, power generation output, weather, and so on. Smart meters are being installed now by municipal utility companies in the United States.

The control grid must be capable of meeting the following requirements:19

• Defend against security threats from either inside or outside the system
• Allow use of alternative power generation sources that are intermittent
• Keep a stable power supply
• Control peak demand surges in order to ensure adequate reserves
• Use EP and CEP technology to control both global (grid control and power generation) and local (e.g., at the household level or even the appliance level in the household) uses of power (see Example 5.12)

Plug-in hybrid vehicles might present a similar problem in the future, since they might tend to be plugged in to electricity sources at roughly the same times within a geographic time zone.

The event processing involved in smart grids and smart metering systems will raise capacity issues that need to be solved. A utility-planning department of a city with a population of about half a million expects that by 2012 the number of events flowing on their smart metering system between the control center and the household meters in their area will be on the order of 6 billion annually20 (which is approaching 750,000 events an hour). Of course, we’re not sure exactly what types of events are included. But the planners know that currently, the event transmission capacity in their networks won’t allow this volume of event traffic.

Types of events flowing in the control grid will include many types of events from:

• Power generators
• Monitoring systems on power grid transmission lines and pathways
• Power transformers
• Local and regional control centers
• Utility company control centers
• Individual consumers (e.g., household and factory smart meters)

and doubtless many others. We can only guess at the types of proactive power consumption control events that may also eventually be carried on the control grid.

Security concerns will also lead to more types of event on the grid. A new generation of computer worms aimed specifically at industrial control software has recently appeared. A recent report on these security concerns is described in Example 5.13.

Summary

We have given a selection of examples of applications of CEP. There are many others and new ones are cropping up all the time. CEP is now being applied in diverse areas such as:

• Monitoring energy trading markets to detect patterns of activities similar to those that led to the Enron disaster22
• Monitoring gambling casinos to detect customer behaviors the house doesn’t like
• Ensuring steady oil flow in oil pipelines

Event processing is a young field, and its potentials are just beginning to be explored.

Notes

1 Based on private correspondence with Roy Schulte, Gartner Corp, and industry research reports.

2 See the section entitled “Complex Event Processing and Systems That Use It” in Chapter 3.

3 PUBLIC LAW 107–204: The Sarbanes-Oxley Act, July 30, 2002. http://www.gpo.gov/fdsys/pkg/PLAW-107publ204/content-detail.html

4 Bryan J. Balin, “Basel I, Basel II, and Emerging Markets: A Nontechnical Analysis,” The Johns Hopkins University School of Advanced International Studies, May 10, 2008. https://jscholarship.library.jhu.edu/bitstream/handle/1774.2/32826/Basel%20I%2c%20Basel%20II%2c%20and%20Emerging%20Markets%20a%20Nontechnical%20Analysis052008.pdf?sequence=1

5 Courtesy of John Bates, CTO, Progress Apama Inc.

6 That is, uses of a card that are flagged as fraudulent but which turn out to be legitimate.

7 See the case of Vladislav A. Horohorin, New York Times, August 24, 2010. www.nytimes.com/2010/08/24/business/global/24cyber.html?_r=1&scp=1&sq=horohorin&st=cse

8 See Chapter 1, “Event Processing in Use.”

9 See Chapter 8 on event abstraction.

10 Based on private correspondence with Marco Seirio, Rulecore.

11 Barton Gellman, “Cyber-Attacks by Al Qaeda Feared,” Washington Post, June 27, 2002. www.washingtonpost.com/wp-dyn/content/article/2006/06/12/AR2006061200711.html (p. 3)

12 National Infrastructure Protection Center, “A publication providing information on infrastructure protection issues, with emphasis on computer and network security matters.” www.iwar.org.uk/infocon/nipc-highlights/2002/highlight02-03.pdf (pg. 6).

13 Based on private correspondence with Gary A. Maio, Public Safety Practice, Data Vision Group, LLC, New Jersey.

14 This example is based upon private correspondence with Leendert W. M. Wienhofen and Andreas D. Landmark of the Norwegian University of Science and Technology.

15 Federally mandated Independent System Operators (ISOs), separate from the power generation companies.

16 Eric Lipton, Richard Pérez-Peña and Matthew L. Wald, “Overseers Missed Big Picture As Failures Led To Blackout,”New York Times, September 13, 2003. www.nytimes.com/2003/09/13/us/overseers-missed-big-picture-as-failures-led-to-blackout.html

17 U.S.-Canada Power System Outage Task Force, “Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations,” April 2004. https://reports.energy.gov/BlackoutFinal-Web.pdf

18 S. Massoud Amin and Bruce F. Wollenberg, “Toward a Smart Grid,” IEEE P&E Magazine September 2005, Vol. 3, No. 5, pp. 34–41.

19 U.S. Department of Energy’s Office of Electricity Delivery & Energy Reliability, The Smart Grid: An Introduction, 2008, p. 37. www.oe.energy.gov/DocumentsandMedia/DOE_SG_Book_Single_Pages(1).pdf

20 Peter Alpern, “Smart Grid Inches Its Way Toward Reality,” Industry Week, August 18, 2010. www.industryweek.com/articles/smart_grid_inches_its_way_toward_reality_22550.aspx?Page=2

21 Joseph Menn and Mary Watkins, “Stuxnet Worm Causes Worldwide Alarm,” Financial Times, September 23, 2010. www.ft.com/cms/s/0/cbf707d2-c737-11df-aeb1-00144feab49a.html#axzz1TjqZYRGK

22 “The smartest guys in the room,” http://www.pbs.org/independentlens/enron/.