Hershey Foods Corporation is the leading North American manufacturer of quality chocolate and nonchocolate confectionery and chocolate-related grocery products. In 1999, Hershey spent $112 million on an enterprise resource planning (ERP) integration project involving software from SAP, Manugistics, and Siebel. Unfortunately, it blew up in Hershey's face.

Admittedly Hershey was taking a huge gamble because it was trying to deploy a four-year project in only 30 months. It started the project in 1997 and went live with SAP financial, purchasing, materials management, and warehousing modules in January 1999. In July it added SAP modules for order processing and billing as well as planning and scheduling modules from Manugistics and a pricing promotions package from Siebel.

That's when things started to go wrong. Just as the back-to-school and Halloween season set in, Hershey started experiencing severe order fulfillment problems. Orders weren't being processed properly. Shipments were delayed or delivered incomplete. And Hershey had no idea what was going on. It had to call its customers to find out how much candy they had received.

Chocolate started to pile up in the warehouse. By the end of September, product inventories were up more than 25 percent over those of the preceding year. Hershey blamed the subsequent 19 percent drop in third quarter profits on these system problems.

Hershey didn't say whether the problems were caused by software quality, implementation issues, or a combination of the two. It did indicate, though, that one of its highest priorities for fixing the problem was to improve the way information flowed between the different applications.

Three years later, Hershey is much happier with its systems. Staying with SAP, it started another ERP upgrade in July 2001 and went live in May 2002. This upgrade went smoothly, and it came in 20 percent under budget. Hershey has consolidated more than 95 percent of its revenue and business transaction processing within a single system, and this arrangement gives the organization greater consistency, visibility, and real-time access to critical business information.

The new integrated system helps Hershey reduce costs, improve quality, increase efficiency, and make better decisions. The upgrade includes a number of process enhancements such as materials management invoice verification, military distributor credit processing, and pick list processing. These enhancements help reduce costs and processing times, and Hershey now has a near zero-defect production environment. Using business analytics, Hershey can perform real-time impact analysis of sales and marketing activities, resulting in reduced packaging stock outs.

As Hershey can attest, when application systems don't communicate effectively, your business process suffers. When application systems do communicate effectively, your business works better. From a tactical point of view, application integration can reduce errors and friction in the production process, thereby improving quality, increasing productivity, and reducing costs. From a strategic point of view, application integration helps people get access to the information they need to be more effective at their jobs. Employees can work more efficiently. Analysts can get a better sense of the big picture. Decision makers can make more informed decisions. Better knowledge makes your business more competitive, resulting in higher profits.

Application integration is at the heart of nearly every IT project. Very few application systems work in a vacuum. In many cases, a “new” application aggregates information from some number of legacy applications or perhaps provides a new, integrated interface to the business process functionality implemented by an assortment of legacy applications. In both cases the integration makes employees more efficient and effective at their jobs.

Application integration forms the foundation for business process automation and workflow applications. The goal of these applications is to streamline operations by automating the process by which applications interact with humans and with other applications. A set of business rules guides the process, determining the next task and specifying which application or human should perform it.

A portal is another example of an application integration project. A portal provides a consolidated doorway into countless application systems and information resources. Thus, a portal must have integrated connections to every system and resource.

Knowledge management (KM) and business intelligence (BI) systems aggregate information from multiple business applications and make it available to analysts and decision makers via reports, multidimensional databases, and data mining tools. You must give these tools access to all applications if you want to achieve a unified view of your business.

Most major IT initiatives, such as ERP, product lifecycle management (PLM), customer relationship management (CRM), sales force automation (SFA), field force automation (FFA), supply chain management (SCM), and demand chain management (DCM), are, in fact, application integration initiatives. These initiatives have far-reaching goals to improve employee efficiency, reduce errors, streamline operations, and give decision makers more accurate and timely information.

Application integration can also extend beyond your corporate boundaries, enabling your application systems to converse directly with those of your suppliers, distributors, partners, and customers. Business-to-business (B2B) integration is inherently more challenging than internal integration because you have control over only half of the equation.

In some cases it's quite easy to measure the return on an application integration investment. If you deploy a new customer care application that makes your telephone support staff more efficient, the application can reduce the amount of time workers spend on each call, allowing fewer people to handle more calls. In this case you have a clear metric (time per call) that you can measure before and after an integration project.

Perhaps, over time, you might also notice a drop in your customer attrition rate. It's hard to say definitively what generates this benefit. Was it the new customer care application? Perhaps the quality of the product improved. Perhaps the competition wasn't working as aggressively to woo your customers away. Perhaps your new customer-retention marketing program should get the credit.

It's often difficult to measure the ROI for strategic investments. When managers have the right information, they make better decisions. They can determine which products or services are most or least profitable. They can recognize conditions that might affect demand for a particular product in a particular geographic region and adjust inventory accordingly. Without the right information, they might make poor decisions that could increase costs or jeopardize an important contract. They also might not recognize a golden opportunity. How can you measure the cost of an unknown lost opportunity?

As you've no doubt experienced, nearly every application, at some point in its lifecycle, needs to share information with at least one other application. Unfortunately it isn't always easy to make your application systems talk to one another. Web services make it easier to connect your applications.

Most likely your enterprise has a hodgepodge of application systems, deployed over the years as necessary. Some of these applications were developed in-house, others purchased from a variety of software vendors, and still others acquired in mergers and acquisitions.

If yours is like most companies, you probably have to deal with a heterogeneous environment. No doubt you've built your corporate application systems using a variety of hardware platforms and software technologies. Having heterogeneous systems makes integration hard. Your application systems simply don't speak the same language. Web services provide a common, platform-independent language that simplifies heterogeneous application integration.

One of the biggest issues, though, is that most application systems aren't designed to talk to other application systems. When you build in-house systems, expedience tends to drive development. You have a set of well-defined requirements for the application system, and your goal is to deliver that functionality as quickly as possible. It takes a lot more time and effort to design an extensible application that can support future, unknown requirements.

Packaged application systems are notorious for being closed and proprietary. Vendors have strong incentives to limit the integration capabilities of their applications. They want you to buy all your software from them. During the 1980s and 1990s software packages started growing in size and complexity as the vendors strove to provide a theoretically “complete” solution. Unfortunately, no single vendor supplies a complete solution, so you're always faced with the task of external integration. Only recently have these vendors started to respond to customers' demands for more versatile integration capabilities.

Application integration strategies fall into two basic categories: data-level integration and application-level integration. Figure 1-1 shows the difference.

Figure 1-1. Allocating inventory for an order. In data-level integration, the order processing application reaches directly into the inventory database and deletes the inventory, bypassing the business rules in the inventory management application. In application-level integration, the order processing application calls the inventory management application and asks it to allocate inventory.

With data-level integration, applications can share information simply by sharing their databases. This approach is less costly and less invasive than application-level integration. Generally you don't need to modify your existing application systems to implement data-level integration. It's a useful way to implement retrieval-only information aggregation. An application can extract information from multiple databases, aggregate it, and produce useful reports. You frequently use this type of integration to build a data warehouse.

But you need to be very careful using this approach for update operations. I view data-level integration as a quick and dirty way to integrate systems. As a rule, you should not attempt to implement transactional or update applications using this technique. Data-level integration doesn't let your applications share business logic—the code that implements your business rules. For example, in your inventory application, your business logic implements the business rules used to allocate inventory. If you build an integration application that accesses the inventory database directly without using the inventory application, that application will bypass the business rules that you have defined to establish priorities and resolve conflicts. You must either duplicate the business logic in the new application or risk violating the business rules. A number of vendors of packaged applications view direct manipulation of the database as a violation of your support agreement.

In many cases it's a much better idea to integrate at the application level. Compared with data-level integration, application-level integration takes more time and effort, but it offers much more versatility and maintains business consistency. In this approach, an application makes its information and its business logic available to other applications through an application programming interface (API).

An API is a programming mechanism that allows an application or system function to expose its capabilities (make them available) to other applications. Hence an API supports application-to-application communication. An API is a bit like a two-sided coin. One side faces outward, providing an interface to external applications. The other side reaches inward, hooking into the private capabilities of the application.

Most application systems provide a few built-in interfaces. Generally there's some type of human interface available, but it's hard to get another application to think and act like a human. Sometimes, particularly in the case of purchased applications, an application includes a set of APIs designed to support internal integration among its related modules. These APIs often support proprietary formats, and that makes it difficult to use them for external integration. Sometimes you're lucky, and the application provides an interface designed for external integration. But rarely will you find a comprehensive set of APIs that lets you do everything you want to do. At some point along the way, you'll probably have to build your own integration hooks and then expose them through an API. You can use Web services to build open, nonproprietary APIs.

You can build an integration hook either by leveraging an existing interface or by modifying the application to create a completely new interface. These integration hooks are often called application adapters. Conceptually, an application adapter is similar to a power adapter. If you want to power an electrical appliance, you need an electrical outlet that matches the power plug on your appliance. If the power plug and the electrical outlet don't match, you can use a power adapter to make them work together. In some cases the power adapter acts as a pass-through, simply realigning the physical outlet configuration to allow the positive, negative, earth, and ground wires to connect. In other cases the power adapter actually modifies the voltage, switching between, for example, AC and DC power or 110 and 220 volts. An application adapter can work the same way. In some cases it simply realigns the programming interface. In other cases it adds additional processing. Figure 1-2 shows a conceptual overview of an application adapter.

Figure 1-2. An application adapter acts as a go-between, resolving incompatibilities between application interfaces.

If your application has internal APIs and if you have access to the proprietary formats used by these APIs, you can build an application adapter that makes these APIs accessible to outside applications. In this case the application adapter acts as a simple translator. Outside applications call the adapter using an open API. The adapter then translates the request into the proprietary API that's native to the application.

In some cases you may want to add business or navigation logic to the application adapter that either simplifies or adds value to the native application interface. For example, you can encapsulate a human interface using a technique known as screen scraping. Essentially you build an application that simulates human interaction. Outside applications call this screen scraper application, which initiates a dialog with the application through its native graphical user interface (GUI) or browser interface. This technique is the least invasive method for creating an interface to an existing application, but it is also the least efficient method, and it tends to generate the most errors and be the hardest to maintain. Imagine the maintenance headaches you'll get when you upgrade to the latest revision of a packaged application and you realize that the GUI has completely changed.

If an existing interface can't supply you with the integration hooks that you need, you can always build a custom API into your application. A custom API is new application code that you develop to support a particular business requirement. Obviously, a custom API requires more time and effort, and it is by far the most invasive integration technique, but it does exactly what you need it to do.

Quite a few enterprise application integration^[2] (EAI) vendors provide connectivity frameworks that can help you build application adapters for your applications. Adapter framework vendors include Attunity, GE Global eXchange Services, IONA, iWay Software, Software AG, TIBCO Software, webMethods, and many more. Traditionally these vendor frameworks have generated adapters that rely on the vendors' proprietary protocols. Most of these vendors are now adopting Web services so that the adapters can be more widely used.

So far we've talked about building integration hooks that reach into an application to access its information and business logic. An application adapter also needs to expose those integration hooks to the outside world through an open, network-enabled API. You normally create a network API using some type of communication middleware. Web services represent a new type of communication middleware.

Middleware is a software package that sits between your application code and its underlying platform, providing easy access to core system facilities such as the network, storage, and processors. Middleware hides the complexities of the underlying facilities from the programmer and provides a consistent interface across various hardware and software environments. Middleware allows programmers to concentrate on solving business problems and not waste time trying to make the business application run properly on a particular machine. Middleware systems include file systems, database systems, application servers, transaction processing monitors, message brokers, and distributed computing systems. Communication middleware is middleware that lets applications talk to one another across the network.

There are many kinds of communication middleware, each offering slightly different features and capabilities, but the basic goal of all communication middleware is to allow applications to communicate while hiding the complexity associated with using the network. Figure 1-3 provides an overview of communication middleware functionality. The middleware performs all the work associated with establishing network connections, creating network messages, and transferring the messages on behalf of the applications. Because different machines and different programming languages use different formats to represent data, middleware often performs data translation.

Figure 1-3. Communication middleware performs all the work necessary to create and exchange messages over a network.

Each type of communication middleware communicates in its own way. Each system supports a specific set of programming languages, operating systems, and network protocols. Most traditional middleware systems use a binary data format to represent the information as it gets sent across the network. A binary data format is a compact way to represent information. Unfortunately, each middleware system uses a different binary data format, and this means that different types of middleware can't talk to each other. For the most part, two applications must both use the same type of middleware to communicate.

The tricky part is that it's hard to find one middleware solution that supports all your applications. Each middleware product supports a limited set of operating systems and programming languages. All these differences make it hard to support heterogeneous environments. Web services solve this problem by supporting any language on any platform.

Historically, most people use a mixture of MOM and RPC-style middleware to integrate their application systems. But there are a number of challenges associated with integrating your applications based on these traditional middleware technologies. I refer to these challenges as the Traditional Middleware Blues. I can summarize these challenges in two basic themes. The first theme relates to pervasiveness and heterogeneity. The second theme relates to the total cost of ownership associated with integration.

So far I've confined my discussion to the issues associated with internal application integration. But your business processes don't stop at your firewall. Integration benefits your B2B processes as much as it does your internal business processes. Supply chain management and demand chain management initiatives focus on streamlining interbusiness interactions. These initiatives require extensive application-to-application integration efforts.

Unfortunately, traditional middleware solutions aren't designed to support intercorporate communication across the Internet. Middleware messages are frequently blocked by firewalls. Firewalls inspect incoming messages and determine whether they may pass. Unfortunately, most firewalls can't interpret the binary formats used by traditional middleware, so they block the traffic. In addition, these middleware solutions rely on domain-based security, reliability, and transaction systems to ensure the integrity of the environment, and these domain-based systems don't effectively extend across the open Internet. It has become obvious that to effectively conduct business-to-business application integration, we need a new type of middleware that inherently supports the Internet.

Web services address the challenges associated with traditional middleware. Probably the most powerful feature of Web services is that they provide a common way for any application, written in any language, running on any platform, to talk to any other application. In this way, Web services solve the heterogeneous communication problem.

As an added benefit, Web services technology is pervasive. You don't have to install a special software package on every one of your systems. Web services rely on standard Internet protocols, which are already resident on every Internet-enabled system. All major software vendors have bought into the idea of Web services. Operating systems, application servers, database systems, application systems, language platforms, and development tools now come prepackaged with Web services support.

One thing that's remarkable about Web services software is that it is much less expensive than most other middleware. Most vendors treat Web services as a new feature of their existing products, such as application servers and development tools. The primary impetus behind Web services is heterogeneous communication, so vendors can't develop lock-in strategies that led to high price tags. Also, the open source community has been deeply involved in the development of Web services technologies right from the beginning. These factors have kept software prices very low.

Best of all, Web services are easier to use than most traditional middleware, so development costs are lower. Developers with advanced skills in CORBA, RMI, DCOM, or MOM are expensive—if you can find them at all. Professional training classes often cost $500 or more per day. In contrast, Web services technology is much easier to use. Many developers are learning the technology on their own, knowing that these new skills have broad application and will make them more marketable. You can expect to find a large supply of talent at a reasonable price.

The cost savings also apply to long-term maintenance and total cost of ownership. Web services combine the ease of use of RPC-style middleware with the loose coupling of MOM. Application adapters built with Web services are much more flexible and adaptable than those built with traditional RPC-style middleware. These adapters are cheaper and easier to maintain.

In case it isn't obvious by now, I'm recommending that you adopt Web services to help you battle the integration crisis. The arguments I've cited thus far give you a host of tactical reasons to consider this technology. Web services technology is cheap and easy, and it works with any application language on any platform. It works for both internal and external integration projects. It's a safe choice because all vendors support it, and they are committed to ensuring interoperability. You won't find yourself locked in to a proprietary solution. These tactical advantages, by themselves, should be enough to convince you to at least consider Web services for your next integration project.

Web services also have profound strategic value. This value stems from the concept of reusability. One of the most costly aspects of integration is that application adapters are rarely reusable. Each new point-to-point connection between applications causes an exponential increase in the total cost of ownership of your applications. Reusability has been an elusive goal for many generations of application development technology. Web services represent the latest generation of technology that tries to deliver reusability. And this generation is likely to be much more successful than previous generations, such as objects, components, and frameworks. A Web service is a larger piece of application functionality than objects or components. This larger granularity gives Web services the potential to finally deliver on the promise of reusability.

Web services aren't automatically reusable. You must put some thought into the design to ensure that more than one application can effectively use a service. So to gain the strategic value of Web services, you may need to make some changes in your application design process.^[5] Even so, it is easier to design for reusability when you're working at the service granularity level than at the object or component level. It's relatively easy to build a general-purpose Web service application adapter for one of your legacy applications. After you have built this general-purpose adapter, the legacy application becomes “integratable.” Any application can use the adapter to talk to the legacy application. At this point you have one adapter to maintain for this application rather than one for every application that wants to use it.

The real strategic value appears after you have created general-purpose adapters for all your application systems. When all your applications are integratable, you then have the ability to dynamically configure your systems to support your business. You can dynamically modify or manipulate your business process. You can dynamically aggregate data from multiple systems. You can dynamically use your data mining tools and analytics to discover new trends. You can turn your business into a dynamic enterprise, one that can respond to business drivers in real time.