12
The Future of Event Processing
Where is event processing going? What new technologies will drive event processing forward? What consumer and industry demands will lead to more event-processing products and businesses? What are the barriers to adoption of the technology? What are the dangers of event-processing technologies? Let’s answer these questions based on the material in the previous chapters.
All animal species—from bacteria to humans—are event-driven. They sense and respond to threats and opportunities as well as routine events such as heartbeats. Storks that do not detect frogs in the reeds die of starvation. Frogs who are forever jumping away from imagined storks die of exhaustion. Let’s learn from nature.
Packs of animals—herds of zebras, prides of lions, schools of fish, and tribes of humans—detect events collectively. To use the vernacular, they “crowd-source” events. When a white-tailed deer runs from danger, its white tail signals others in the herd about danger in the vicinity. The presence of buzzards in the sky informs others about food in the area. The “wisdom of crowds” starts with the “sensing of crowds” and leads to collective responses of crowds.
Information technologies help societies advance by amplifying their collective intelligence and by facilitating time-, request-, and event-driven interactions among their members. Let’s look at how we can help solve (some of) the world’s problems using event processing in conjunction with other information processing technologies. Nature suggests how we should use event-processing technologies: we should amplify our collective ability to sense, analyze, and respond to situations appropriately. The key word is “appropriately.” There is no point in responding faster than is appropriate, in using more energy than is appropriate, or in responding to situations that need no response. Peace of mind and a balanced view are no less important than speed and responsiveness.
Ancient man, hunting on the savannah, obtained situation awareness by using his sight, hearing, and smell and signals from members in his hunting party. In the next decades, we will achieve global situation awareness by using event-processing technologies and our collective ability to sense, analyze, and respond. Buildings will respond in seconds to slippage along earthquake faults; electric grids will respond to shifts in wind and clouds; governments will respond to E. coli contamination in food supplied from anywhere on the globe; telescope systems will respond to events in the far reaches of the universe; and parents on trips will remain aware of the situation at home.
Solving the World’s Problems One Application at a Time
What are the scarcest resources on the planet and how can event processing help husband them?
Surely, one of the most precious resources is your time: time in which you can think, do activities, relax, and enjoy things without interruption. We let ourselves be driven by events that should be handled later or ignored altogether. Devices spawned by information technology are driving us to distraction. A frog that continually jumps away from imagined threats, or jumps toward imagined opportunities, suffers from exhaustion; so do we. The most significant scarce resources of the 21st century are your time and attention. We are beginning to realize that we are frittering away this precious nonrenewable resource. Event-processing technologies will save the day: they can help give you undistracted time during which you can pay attention to things that matter.
Your Attention
An attention amplifier that helps you pay attention to what matters to you now must know what does matter to you now: the application must know your current context. Do you want to pay attention now to your sales forecast, to your presentation on enterprise software, to reading a book, or to finding a proximate restaurant? Applications will estimate your current context by analyzing your history—the items you’ve searched for in web searches, the books you’ve read, the movies you’ve downloaded, your pattern of movements—and by monitoring your current activity—what you are doing now, what documents you are working on now, what web searches you are conducting now, where you are now and where you came from. Applications will proactively acquire information and tools you need, and either push them to you or have them ready the instant you ask for them.
Personal information managers will become attention amplifiers. They will distinguish events that you care about from those that you don’t care about; they will determine whether you should be interrupted immediately or not; and they will organize information about events that don’t merit your immediate attention into documents for your later perusal. Our sensory capacity will be magnified manyfold by thousands of sources of information across the globe, and attention-amplifier applications will help us process the information without suffering from sensory overload.
Expert Attention
Attention from experts is a scarce resource. Event-processing applications will deliver expert attention to where it is most needed.
The old and infirm need attention, but costs of caring for the aged threaten treasuries of many nations. Getting expert medical attention to the aged immediately when they need it—such as after a fall—is difficult. The costs of home-healthcare nurses have been increasing exponentially while the costs of event-processing devices have been dropping exponentially. Supporting experts (social workers, nurses, and doctors) with event-processing technologies will improve quality while reducing costs. Sensors in smart wheelchairs and smart houses will help direct expert attention to where it is needed urgently. Blood sugar levels of diabetics in Washington D.C. will be monitored by health centers in Maryland or even in the Philippines. Long-term-care insurance will promote use of smart walking sticks, smart pill dispensers, and other sensor-rich devices.
Commercial off-the-shelf (COTS) devices and applications will monitor internal events; provide plug-and-play interfaces that enable other applications to listen to events; and have dashboards that display key performance indicators (KPIs). Air conditioners, refrigerators, washers, dryers, cooking ranges, and cars will monitor and report internal events. Appliances are getting more complex, and most people don’t want to analyze data generated by hundreds of sensors monitoring the internal workings of their appliances; however, maintenance experts can exploit such detailed information. The costs of experts to maintain increasingly complex appliances are increasing too. So, appliances and cars in expensive areas such as Manhattan will be monitored from less expensive locales such as Detroit, Biloxi, and Sarajevo.
You will gain situation awareness about your entire home and family by means of applications that integrate and display information about events from multiple appliances, people, and pets. This integrated display will help you manage power, gas, and water consumption and keep track of children, dogs, and things that are important to you. (A downside, however, is that the technology will enable couch potatoes to remain on their couches even longer! Imagine having a remote control that not only allows you to surf hundreds of TV channels and thousands of YouTube videos but also allows you to check on your children, pets, and spouse and whether the washer has finished its load.)
Nonrenewable Resources
Some resources cannot be renewed. World reserves of petroleum, particularly light sweet crude, are shrinking. Several hundred animal species and plant species are endangered. Recovering carbon dioxide from the atmosphere is expensive, and so the atmosphere cannot be renewed and returned to its state of 100 years ago. Event processing will help humanity manage its precious nonrenewable resources.
Sensors in critical habitats of endangered species will detect poachers and other threats. EDA applications will detect schools of fish in rivers and respond by turning off water intakes of factories to ensure that fish are not sucked in. Sensors and satellite tags on animals will enable scientists to see and hear what sharks, turtles, and other animals see and hear. Data gathered about animals over their lifetimes coupled with CEP will help scientists manage recovery of threatened species. Event-processing applications will help reduce consumption of fossil fuels by enabling effective use of renewable—but occasionally erratic—energy resources, such as wind and solar power. Humankind’s increasing awareness of the fragility of planet earth will drive use of event-processing technologies.
Renewable Resources: Food, Water, and Energy
The essentials of life include clean water, food, energy, health, shelter, security, and peace of mind. In the next decades event-processing technologies will help conserve and replenish these essentials.
Departments of agriculture will rely on event-processing technologies—sensors, simulation models, pattern detection, and machine intelligence—to ensure the quality of food delivered to tables in London from farms in Chile, Australia, and Israel. Event-processing applications that detect and analyze events over the lifetimes of farm animals in herds in different states and countries will help prevent substandard meat from reaching supermarkets.
Clean water is essential for life but is in short supply in many parts of the world. Regions of the world suffer from drought. Irrigation is being threatened by demands from rapidly-growing cities. In the next decades you will see massive efforts to improve water supplies and to help humanity consume water more wisely—and event-processing technologies will play a major role in those efforts. Uniform irrigation over large regions will be replaced by location-specific irrigation tailored to the needs of small areas. Data from sensors that measure soil conditions, flow rates in rivers, snow packs in mountains, and weather forecasts will be fed to computational models to determine the ideal amounts of water to use and the ideal times to irrigate.
Event-processing technologies enable the use of renewable, intermittent, energy sources by detecting and responding to changes in demand and supply. The smart grid cannot function without these applications. The next quarter century will see massive use of event-processing applications for managing power generation, power markets, power transmission and distribution, across the world.
EDA applications will reduce energy consumption in homes and offices by monitoring activity and turning appliances on and off appropriately. These applications will detect from GPS readings when you are on your way home and turn on heaters, air conditioners, computers, and other appliances. Humankind has been profligate in its use of energy—from cutting down trees for firewood to burning coal in power plants—and event-driven systems can help humanity consume this resource wisely.
Healthcare
Countries around the globe will make hard choices about healthcare in the next quarter century. Healthcare costs have been rising faster than the rate of inflation for decades while the costs of sensors, responders, and computers have been dropping. U.S. healthcare costs are expected to grow to 20 percent of gross domestic product (GDP) in a decade. Companies are going bankrupt because they can no longer afford the costs of providing healthcare to their employees.
Governments and companies want individuals to take more responsibility for their own health. Better healthcare begins with self-awareness—knowing what you eat, how much you exercise, and what your blood pressure, pulse rate, and blood sugar levels are. Your self-awareness will be raised by using applications that detect, record, and analyze the events you generate. Devices ranging from software that logs the foods you consume to exercise calorie counters are mechanisms for monitoring and recording events. Noninvasive, or minimally invasive, sensor devices will become available in the next decades that monitor a variety of physiological vital signs. The costs of these devices will keep decreasing and their power will keep increasing, enabling employers to reduce the healthcare dollars spent on employees.
Feedback control theory tells us that the way to improve the quality of actions is to measure KPIs and then make corrections to actions based on the deviations of measured values from ideal values determined by a model. Future healthcare applications will embody control-theoretic principles. Sensors will measure your physiological vital signs and send data to computational engines that analyze deviations of measurements from desired values; based on this analysis, the computational engines will send recommendations to you and, when appropriate, send information to your doctors. Unfortunately, event-processing applications are limited: they do not solve the overwhelming problem of making us follow recommendations!
In the next 20 years, you will benefit from plug-and-play connectivity standards for medical devices. Event objects generated by different medical devices will be encoded using standard data structures. This will help provide holistic situation awareness of a patient by fusing data from all the devices monitoring the patient. In the next decade, hospitals will deploy complete, shrink-wrapped applications for monitoring and managing patients, hospital resources, and medication. These applications will use RFID tags and will have event-, time-, and request-driven components.
Jet travel has increased concern that healthcare systems have insufficient time to respond effectively to new contagious diseases. Swine flu in Mexico or avian flu in Southeast Asia can spread to Europe in weeks. Event-processing systems will be used (sometimes inappropriately) to reduce spread of contagious diseases. Airports will have sensors that screen arriving passengers for above-normal temperatures, and passengers with symptoms detected will be required to undergo further tests on the spot. Concerns about pandemics as well as bioterrorism will result in the development of sensors that can make measurements some distance away from the subject so that people can be analyzed as they walk through doors. Healthcare will be a fruitful area for event-processing technologies in the next two decades.
Security
Unfortunately, antisocial elements use technology, too. Without a doubt, antisocial elements will exploit event-processing technologies just as they exploit the Internet and mobile phones. Progress promotes a technological arms race between those who would protect society and those who would destroy it—and this is no less true of progress in event processing.
Event processing will be widely used in the next decades to protect countries against adversaries with dangerous technologies. EDA applications will be used by customs and immigration agencies to combat terrorists with access to nuclear material, drug lords with “narco-subs” (crude submarines built surreptitiously) and planes, and people smugglers.
The defense industry and the military have been pioneers in event-driven systems and will continue to lead in the future. Departments of defense have been drivers for global situation awareness and for integrating event-driven applications across multiple services and “stovepipe” applications within services. Companies, nongovernmental organizations, and other agencies of government, such as Homeland Security, will adopt technologies and procedures pioneered by the Department of Defense. Little further needs to be said about the future of EDA and CEP applications in the Defense industry; however, a point worth mentioning briefly is the use of EDA and CEP in robotic systems and in the exploitation of space.
Several books have been written recently on robots in warfare. The books predict that the number and variety of robots used in defense will increase in the coming years. Robots extend military sense and response capabilities: robots detect and defuse remote improvised explosive devices (IEDs), and unmanned aerial vehicles (UAVs) provide reconnaissance from battlefields across the world. Networks of military reconnaissance satellites are sophisticated remote-sensing devices that provide detailed images of the earth using different types of imaging sensors. UAV and satellite reconnaissance applications are prototypical examples of streaming intelligence—they derive intelligence continuously from data streams generated by video cameras and other sensors. Increased resolution of ground images from cameras in satellites and UAVs provides invaluable information, but the total volume of data generated is overwhelming without efficient CEP analysis.
The Workforce and Science in the 21st Century
The workforce of this century will have to deal with developing and maintaining increasingly complex products, from credit default swaps to pluggable hybrid electric vehicles. New employees will have to be trained quickly; lifetimes of service in steel mills, coal plants, and automobile factories are giving way to mastering complex new situations rapidly. Just-in-time information will enable the workforce to manage complex systems. Some just-in-time learning will be request-driven and some will be eventdriven. When an electric utility worker inspects a transformer, all the information about that transformer will be pushed to the worker; the system will detect where the utility crew is and what components are being inspected and respond by pushing information to help the crew solve the particular problem that they are facing at that point.
Science progresses, in part, by observing and then understanding the unexpected. Telescopes will become even more event-driven as they are retargeted to follow unexpected transient phenomena. Biological instruments will become event-driven as biologists focus their measurements on interesting but infrequent intervals of activity interspersed among long periods of less interest. Instrumentation in high-energy physics will also have more embedded event-driven capability that detects significant events and retargets instruments to track significant events when they do occur.
Infrastructure
Road travel is fast and pleasurable when roads aren’t congested, but can be a prolonged nightmare otherwise. The transition from acceptable to excruciatingly slow occurs as traffic increases by the small amount that takes the system from uncongested to congested states. The same situation holds in the electricity grid: the system works perfectly when demand is less than capacity, but a small surge in demand or a small cut in supply can push it over the edge.
Infrastructure is expensive. Moreover, construction of infrastructure such as prisons, freeways, and transmission lines in many regions is limited by the not-in-mybackyard (NIMBY) syndrome. Infrastructure usage is getting perilously close to its limits in many areas, and this will increase demand for dealing creatively with the tensions between “nimbyism” on the one hand and poor service on the other. Event-processing applications will help deal with this tension.
Users of certain types of infrastructure, such as roads, have an incentive to use the infrastructure when it is not congested. In other cases, users are given incentives—for example, you can get paid for allowing the electric utility to turn off your air conditioner during peak loads. Event-processing applications will detect and respond to congestion. A response might be to raise rates for using the infrastructure; inform potential users of the congestion so that they can delay their use of the infrastructure or choose alternates; or manage demand by disallowing customers from using the infrastructure for short periods.
A resource that is increasingly congested is the sky. There were about 13 million commercial flights in the United States in 2005 and about 9 million in 1985. The demand for more flights is likely to increase. The next generation of air traffic control system, called Next Generation Air Transportation System (NextGen), uses satellite navigation systems to keep pilots in planes in touch with each other and with ground controllers. Distributed event processing is an important component of NextGen. You will see increasing use of event-processing technologies as NextGen is deployed over the next quarter century.
An example of the NIMBY syndrome is the aversion that communities have to prisons and convicted sex offenders in their neighborhoods. For example, sex offenders in Dade County, Florida, are reported by National Public Radio and the Miami Herald as living in tent cities under a causeway because living anywhere else is difficult given county ordinances (see Appendix A for references). Electronic monitoring of people on probation can help ensure that they don’t enter designated areas such as schools while giving the probationers more flexibility. Given growing costs of managing prisons and people on probation, governments will use sense-and-respond applications to help deal with the problem.
Businesses
Online retailers already use event processing extensively and will increase use of the technology in the future. Retailers monitor customer behavior at their websites and make recommendations that help steer customers to valuable interactions. EDA, CEP, and real-time response will become absolutely critical for successful retailing.
Web portal and search companies attempt to deliver advertisements that are meaningful to you, and what is meaningful to you depends on your context. Web portal companies will use increasingly sophisticated, near-real-time CEP to estimate your long-term interests and your current context and determine the advertisements that are most likely to result in sales.
The more a business knows about you, the better it can estimate your context and your needs. So, businesses have an incentive to provide you with services such as ecalendars, e-mail, and document storage that enable them to learn more about you. These services are useful to the business to the extent that they can use business intelligence (BI) to offer you the services and products you want just when you want them. Web-based companies will use BI to build models of your long-term behavior and use CEP to estimate your current context; knowing your long-term range of interests and your current focus will enable them to give you information that both interests you and is relevant to your immediate needs.
Finance and logistics applications will increasingly use event-driven components. Mutual fund companies, banks, package-shipping companies, and airlines will allow you to specify events that interest you and the actions to be taken in response to those events. Today, responses to events are mostly notifications by e-mail or text messaging. In the future, companies will offer responses that are more varied, such as triggering adjustments to your investment portfolio when specified events occur or changing your flight schedule when a plane is late.
Impact on Society
People have always depended on others—the hunting party, tribe, or town—to look out for dangers and alert the community. In the future we will depend on social networks to inform us about threats and opportunities. An alert may be as commonplace as being told that an author will autograph books at the local bookstore or as important as being warned about an armed deranged person on the loose. Indeed, people will join social networks partly because they can collectively sense and respond to situations better than they can individually.
EDA applications will promote even more globalization. Events in one part of the world will create responses in another part. A patient in San Francisco may depend on doctors in the vicinity and also on a team of nurses in the Philippines dedicated to monitoring that patient for 24 hours each day. Round-the-clock care in San Francisco is expensive, and getting work visas to enter the United States is hard; so, EDA applications will be used to help deal with the problem. Likewise, IT systems in Frankfurt will be managed by engineers in Bangalore.
A great deal has been written about the digital divide: the poor have less access to information technology; as a consequence, they don’t acquire adequate IT skills and thus can’t compete as well at school and work. The “event divide” will become an aspect of the digital divide. Those who master event-driven technology will be able to better focus their attention on the things that matter to them, sense situations that are important to them, and respond more effectively.
Barriers and Dangers
Concerns about security and privacy are major barriers to the adoption of event-processing technology. Security is a concern because successful attacks on systems such as air traffic control, medical devices, finance, and the electric grid will be devastating for society. A related concern is that insiders, or others who learn how the systems work, will exploit their knowledge to misuse the system for their own ends. Organizations will postpone implementations of event-processing applications until they are sure that the threat to security can be reduced to manageable levels, and they will exercise constant vigilance after the application is deployed.
Event-processing applications push relevant information to you by knowing what information is relevant to you. The more that the application knows about you the better it will serve you. Many people are willing to cede private information to an organization if the organization promises not to release the information to anybody else or misuse the information; however, once private information becomes public it may remain nonprivate forever. Somebody who obtains a copy of your private information may hold on to it despite copyright rules and privacy laws. You have to trust the organization as an entity, the individual people in the organization who have access to your information, and governments who can demand that information.
Most people in well-governed countries trust that companies will obey laws and keep private information private. This trust is essential for growth of event-processing web applications.
A danger of widespread use of event processing is system fragility. Water, food, health, security, energy, shelter, and finance may be controlled by software applications that have components in common. Systems that appear to be independent will, in reality, be dependent on common substrates. A failure of—or a successful attack on— some components may affect all the applications that use those components. These applications may include those on which life depends. Therefore, government agencies for verifying and testing critical components will enforce regulations that are shared by many applications.
Drivers for Adoption
The adoption of event-processing technologies is driven by several forces, including the PC-cubed (price, performance, pervasiveness, celerity, complexity, and connectedness) trends discussed in Chapter 2. The prices of sensors, responders, communication bandwidth, and computing devices in event-processing systems will continue to drop for several reasons. Consumer applications, such as computer games and mobile phones, increase demand for sensors, responders, computers, and displays. The use of accelerometers in phones and car airbag systems has increased demand and reduced price for them. GPS devices are now commodity items. Instant messaging systems are almost universally available. Consumer electronics and consumer Internet applications will keep driving down the costs of components used in event-processing systems.
The integrated circuit industry may be reaching the limits of Moore’s Law—the number of transistors on an integrated circuit may not double every two of years— but that limit will not hinder the growth of event-processing applications. Unlike many applications that are inherently sequential, event-processing applications can exploit increasing numbers of processors by executing different steps concurrently on different processors. Even if single processors do not double in speed every two years, the cost of processors will drop, and thus hardware platforms for event-processing applications will continue to get more powerful and less expensive.
Models play central roles in event-processing applications. Solutions for problems in business, social, or natural systems are based on models of these systems. Some models are complex and require a great deal of computing power. For instance, models of stock markets, roads, water systems, hurricanes, food transportation, and the spread of disease require high-performance computing. Great strides in computer modeling and simulation over the past 25 years will pay off for event-processing applications.
Massive efforts have been made in neural networks, machine learning, BI, statistics, natural-language processing, and rule engines. Powerful tools have been encapsulated as services, and some of these services can be used free of charge. The availability of tools (especially open source) and services is increasing interest in these areas. The technology-push trends for event processing have been growing stronger.
The PC-cubed trends of celerity, connectedness, and complexity show no signs of abating. The celerity trend is not that systems and people will respond faster year after year, reducing responses from seconds to milliseconds to microseconds; rather, the celerity trend is that increasing numbers of systems will respond in a timely fashion whether timeliness requirements are in microseconds or hours.
The interconnectedness of people and systems across the globe drives demand for event-processing applications that enable systems in one part of the world to respond to events in another part. Event-processing technologies, in turn, create more interconnectedness. Event-processing technologies will also drive demand for a different form of interconnectedness: location-based, opportunistic connections. When two mobile phones are near each other, one of them can be used to relay messages to the other. Networks of mobile phones will be used as publish-and-subscribe systems. Cars on a road will communicate information to each other about accidents and congestion.
Event processing will be used to monitor and respond to complex systems. Detecting fraud and noncompliance will get more complex as rules get more intricate and rule evaders exploit technologies more skillfully. Smart buildings, smart electricity grids, and smart management of water resources are complex, event-driven systems.
Consumer applications are driving development and innovation in many components of event-processing applications. Features of instant messaging systems are used in alert engines. Applications such as Twitter that support social networks can be thought of as social activity monitors analogous to business activity monitors. Massive multiplayer online games process high volumes of events each second. Many consumers use event-processing technologies such as instant messaging routinely. Technology support for event-driven processing was rare 50 years ago; today, the young in many countries cannot function without it.
Most people have heard the adjective “smart” applied to nouns such as the electric grid, building, city, phone, and healthcare. A characteristic of smartness is the ability to sense and respond to events. Event-processing businesses—customers, vendors, and products—will gradually become more mature in the next decade. One indication of market maturity is widespread acceptance of standards, nomenclature, and reference architectures. The U.S. National Institute of Standards and Technology (NIST) is developing standards for the smart grid, and organizations such as the World Wide Web Consortium (W3C) are developing standards for events (see Chapter 9).
Technology push, enterprise pull, widespread consumer demand for event-processing features such as agility and celerity, and a generation growing up with eventprocessing consumer applications all suggest that the use of event-processing technologies will continue to grow rapidly.
Summary
Event-processing technologies will play critical roles in managing the essential stuff of life—water, food, health, security, energy, and shelter—as well as activities that are not necessary for basic survival but are extremely important such as finance, logistics, sales, and marketing. Growth will be driven by advances in technology, reductions in price, demand from enterprises, and increasing familiarity with event-processing applications.
The introduction of powerful technologies can have dangerous side effects. The dangers of event processing include possible loss of security and privacy, and a possible increase in fragility of the systems that serve society. Moreover, antisocial elements will acquire this powerful, but low-cost, technology. Deployment of event-processing applications must be done carefully, with particular attention paid to security, privacy, and reliability.
People who master event-processing technologies will have an advantage over those who don’t. Just as mastery of request-driven applications such as search engines, online encyclopedias, and online forums provides an advantage to people today, facility with using event-processing services will provide an added advantage in the future.
Event processing will have a marked impact on society over the next quarter century. Companies in IT as well as in other businesses will incorporate event-processing technologies into their services and products. All government agencies and many nongovernmental organizations will provide essential services by using event-driven components. The IT industry and educators have the opportunity and the responsibility to ensure that this powerful technology helps deal with society’s most pressing problems.