Chapter 15. System Access Patterns
It’s a pretty sure bet that whatever you implement in TIBCO ActiveMatrix Service Bus is going to have to interact with an external system. This chapter examines the different approaches that can be taken to external system access. It then explores the implementation options for accessing systems, including:
• Direct interaction via an ActiveMatrix-supported protocol (e.g., SOAP service calls)
• Indirect interaction via an ActiveMatrix adapter
• Direct interaction by a component using a non-ActiveMatrix-supported protocol
Approaches to Accessing External Systems
There are four commonly available approaches available for accessing external systems:
• Application Programming Interface (API) interaction
When considering which approach to employ, you need to consider how the approach works in both the inbound (to the external system) and outbound (from the external system) directions (Figure 15-1). A closely related issue is that of event recognition: recognizing the event that should trigger the interaction. The following sections explore the four interaction options with respect to both the direction of interaction and event recognition.
Figure 15-1. External System Interactions
Application Programming Interface (API) Interaction
An application programming interface (API) is a programming-language interface provided by an application to support interactions with the application. By definition, an API is provided in some programming languages.
APIs provide a reasonable way to implement interactions in the inbound direction. In the design of your ActiveMatrix component, you (a) recognize that an interaction is required, and (b) call (directly or indirectly) the API provided by the external system.
APIs do not work as well in the outbound direction. Getting an external system to call your code is not straightforward. It typically requires the external system to have a means of registering callbacks, pieces of your code that get called when specific events occur in the external system. Few systems have such a capability.
Database Interaction
Many applications have underlying databases in which they store information. Provided that the schema of the database is understood, it is simple to query the database and extract information. Furthermore, with the addition of a database trigger, you can actually recognize that a change has occurred in the database (i.e., recognize an event) and use the trigger to extract and publish information about the change. Virtually any significant event in the application results in a database change, so this makes it a fairly straightforward process to recognize application events and publish outbound notifications that these events have occurred.
In the inbound direction, however, database interactions have serious limitations. In general, applications have business logic either in the application itself or embedded in stored procedures in the database. It is generally unacceptable to make direct changes to these databases. Thus, database interactions do not provide a good mechanism for inbound communication.
File-Based Interaction
Many older applications (and some newer ones as well) have mechanisms for generating and consuming files. In the inbound direction, applications typically will have a well-defined file format that you can create with an ActiveMatrix component (directly or indirectly). In addition, most have a corresponding command-line interface that you can invoke that will cause the application to consume the file.
In the outbound direction, many applications have the ability to generate files that contain extracts of their information. Many also have a command-line interface that can be invoked to generate this file. Event recognition, however, remains a problem in the outbound direction. Outbound files are typically not generated in response to an event, but rather in response to some external action (e.g., calling the command-line utility) or a scheduled activity. In practice, files are not generated very often and do not serve well as an event-recognition mechanism.
Protocol-Based Interaction
A protocol-based interaction is, logically, the combination of an API with a transport. The SOAP interfaces provided by many modern systems are typical examples.
Protocol-based interactions tend to have the same limitations as API-based interactions: strong in the inbound direction and weak in the outbound direction. Most systems do not have the capability for you to register your protocol-based interface and have it called by the system when a particular event occurs.
The Event Recognition Challenge
Event recognition is a key challenge when interacting with external systems. Business-significant events typically result in changes to these systems, and recognizing these changes is an important means of triggering activity elsewhere when those events occur. In an event-driven enterprise you want to be able to trigger activity upon arrival of a customer, creation of a new account, placement of an order, or change of inventory level. An inability to recognize these events makes it very difficult to trigger work elsewhere.
One historical solution to event recognition has been extract-transform-load (ETL) interactions between systems. Periodically, one system will do a mass extract of information that is subsequently transformed and loaded into a second system. Although some source systems are capable of identifying what has changed since the last extract, others simply extract it all and leave it up to the receiving system to determine what, if anything, has changed.
ETL interactions generally move large amounts of information (particularly when changes are not identified in the source system). Processing large amounts of information is costly, and consequently ETL interactions tend to be performed infrequently. The result can be a delay in event recognition (and action) and data inconsistencies between the systems.
An alternative to ETL interactions is to use database triggers to initiate the information exchange between systems. The database trigger recognizes the change and initiates an outbound communication that can trigger work elsewhere. With this approach only the changes are published, and the published information can be limited to only that which is required elsewhere. The TIBCO ActiveMatrix Adapter for Database provides a simple means of implementing such interactions.
There are two arguments typically leveled against the use of database triggers: efficiency and the need to modify the database to insert the trigger. Both can be readily addressed.
From the perspective of efficiency, it is almost certainly the case that more work is required to process an individual record when the record is being transmitted via a database trigger than when it is being processed via an ETL exchange. However, the peak resource demand for the trigger approach is liable to be substantially less than for the ETL approach. In other words, you’ll need less machine horsepower for the trigger approach. Why? Because the work for the trigger approach will be spread out over time (usually—you’ll need to verify this for your case), whereas the ETL exchange happens all at once.
As for modifying the database, many database administrators are loath to add triggers for fear of performance implications. But if you are trying to recognize events, what is the alternative? The alternative is a query that will examine large numbers of records to identify changes. Furthermore, this query will have to be repeated periodically, and the longer the period, the greater the delay in recognizing events. In most situations, the use of the database trigger places less of a burden upon the database.
Many database vendors include the infrastructure for this type of event recognition, using the term Change Data Capture to label these capabilities. An overview of these capabilities can be found on Wikipedia.1
Combining API and Database Interactions
Combining API and Database interactions provides an effective means of implementing bi-directional interactions with many systems. The two approaches complement one another: Each is strong in the direction in which the other is weak. The idea is that you use the API approach in the inbound direction and use the database approach in the outbound direction.
There is some synergy in this combination as well. The database approach requires understanding the database schema, which is often proprietary and subject to change. APIs, on the other hand, tend to be publicly supported interfaces. You can reduce the risk of schema dependency by having the trigger only publish the primary key of the record that has changed and then use the publicly supported API to retrieve the remaining information and publish it.
There is, however, a minor drawback to this two-step publication process: There is a time delay between initial publishing of the key and subsequent invocation of the API to retrieve the remaining information. If there is a change to the database during this interval, the API call will return the latest information, not the information that was present at the time the trigger was invoked. Nevertheless, the approach is so powerful that it is actually the design basis for a number of the TIBCO ActiveMatrix adapters to commercial systems.
Direct Interaction via ActiveMatrix-Supported Protocols
Interaction with an external system that uses one of the protocols supported by ActiveMatrix is straightforward. There are two possible patterns, depending upon which party is initiating the interaction. Figure 15-2 shows the architecture pattern when the external system initiates the interaction. This pattern generally arises with front-end systems interacting with users or other systems belonging to customers or business partners. Note that this pattern requires the external system to recognize the triggering event and initiate the interaction.
Figure 15-2. System-Initiated Direct Interaction via ActiveMatrix-Supported Protocol
Figure 15-3 shows the architecture pattern when it is the Active-Matrix component that recognized the triggering event and initiates the interaction. This pattern generally arises when the external system is either a back-office internal system or an external customer or partner system.
Figure 15-3. ActiveMatrix-Initiated Direct Interaction via ActiveMatrix-Supported Protocol
The following protocols can be used with these patterns:
• SOAP over HTTP
• SOAP over JMS
• XML over JMS
Note that SOAP over ActiveMatrix Virtualization cannot be used since that protocol can only be used to communicate with ActiveMatrix-hosted components.
Because these patterns use ActiveMatrix-supported protocols, both the services and references are usable as policy enforcement points for access control policies. Furthermore, the ActiveMatrix framework gathers detailed data about each interaction. This enables both the statistical monitoring of the interactions and the detection of breakdowns.
Indirect Interaction via ActiveMatrix Adapters
Although it is convenient to interact with external systems via ActiveMatrix-supported protocols, many external systems do not support such interfaces. However, TIBCO provides a wide range of adapters for interacting with external systems. An adapter is a component that acts as an intermediary between ActiveMatrix and the external system. It interacts with ActiveMatrix with one of the ActiveMatrix-supported protocols and with the external system using one of its native interfaces. As a rule, using adapters takes considerably less effort than writing code to interact with systems.
All TIBCO adapters can be run as stand-alone processes. Some additionally offer the option of running within an ActiveMatrix node. This gives rise to four different architecture patterns. Figure 15-4 shows the available patterns when the external system is the initiator of the interaction. Beyond simple request-response interactions, these patterns are very useful for systems of record to announce information changes.
Figure 15-4. System-Initiated Indirect Interaction via Adapters
Figure 15-5 shows the architecture patterns available when the ActiveMatrix component initiates the interaction.
Figure 15-5. ActiveMatrix-Initiated Indirect Interaction via Adapters
The following adapters can be hosted either stand-alone or within an older TIBCO ActiveMatrix 2.x environment. Most of these adapters will, over time, become available in the 3.x environment.
• TIBCO ActiveMatrix® Adapter for Database
• TIBCO ActiveMatrix® Adapter for Files (Unix/Win)
• TIBCO ActiveMatrix® Adapter for IBM i
• TIBCO ActiveMatrix® Adapter for Kenan BP
• TIBCO ActiveMatrix® Adapter for LDAP
• TIBCO ActiveMatrix® Adapter for Lotus Notes
• TIBCO ActiveMatrix® Adapter for PeopleSoft
• TIBCO ActiveMatrix® Adapter for SAP
• TIBCO ActiveMatrix® Adapter for Siebel
• TIBCO ActiveMatrix® Adapter for Tuxedo
• TIBCO ActiveMatrix® Adapter for WebSphere MQ
The following adapters can be only run in the stand-alone mode.
• TIBCO® Adapter for CICS
• TIBCO® Adapter for Clarify
• TIBCO® Adapter for COM
• TIBCO® Adapter for CORBA
• TIBCO® Adapter for EJB
• TIBCO® Adapter for Files i5/OS
• TIBCO® Adapter for Files z/OS
• TIBCO® Adapter for Infranet
• TIBCO® Adapter for JDE OneWorld XE
• TIBCO® Adapter for Remedy
• TIBCO® Adapter for SWIFT
• TIBCO® Adapter for Teradata
• TIBCO® Adapter SDK: Build your own!
It is worth noting that the last adapter, the Adapter SDK, is really not an adapter at all—it is a software development kit for writing your own adapters. What it contains are the class libraries that TIBCO uses to write its own adapters. Using these libraries simplifies the process of implementing the Adapter-to-ActiveMatrix side of the interface and building an adapter that is deployed, monitored, and managed in the same manner as the other TIBCO stand-alone adapters.
Finally, it should be mentioned that other components can play the role of adapter as well. This is appropriate when the mapping between the ActiveMatrix communications and the interactions with the external systems is not one-to-one. TIBCO ActiveMatrix BusinessWorks is often employed in this role.
Direct Interaction via Non-ActiveMatrix-Supported Protocols
The third category of interaction patterns involves direct interaction, but using protocols that are not represented by SCA services and references. Figure 15-6 shows the architecture pattern when the external system initiates the interaction. This is a common pattern when the ActiveMatrix WebApp implementation type is being used and the external system (in this case, a browser) is interacting with the HTTP interface on the WebApp.
Figure 15-6. System-Initiated Direct Interaction via Non-ActiveMatrix Protocol
Figure 15-7 shows the architecture pattern when the interaction is initiated by the ActiveMatrix component. This is a common pattern when the external system is a database and the ActiveMatrix component is accessing the database via JDBC.
Figure 15-7. ActiveMatrix-Initiated Direct Interaction via Non-ActiveMatrix Protocol
While simple, direct interaction via a non-ActiveMatrix protocol has its limitations. With some notable exceptions (e.g., the HTTP interface on the WebApp implementation type), most of these interfaces cannot be policy enforcement points and no data is collected by ActiveMatrix about the usage of these interfaces.
General Considerations
Each of the interaction patterns has its strengths and weaknesses, and each has its appropriate applications. Here are a few things to be considered:
• As a rule of thumb, it is good practice to minimize the number of moving parts in a solution. This consideration favors the direct interaction patterns.
• If a single external system requires both types of interaction (system-initiated and ActiveMatrix-initiated), the design and administration will be simpler if the same type of pattern is used in both directions. In many cases, this consideration favors the use of adapters.
• The use of an ActiveMatrix-supported protocol provides both a policy enforcement point and the infrastructure to gather data regarding the use of the interface.
Database Interactions
Interacting with databases is a common requirement. There are two readily available options for this interaction: direct interaction via JDBC and indirect interaction via the Database adapter. Each of these patterns has appropriate usages.
Direct access via JDBC is generally appropriate when the ActiveMatrix component is, in some sense, the “owner” of the database, or at least some of the information in it. When this is the case, interaction only occurs in one direction, with the ActiveMatrix component always initiating the interaction. This strategy is particularly effective when combined with the BusinessWorks implementation type with its readily configured JDBC activities. This option, however, is not appropriate when the ActiveMatrix component needs to be informed of changes to the database arising from other sources.
Indirect access via the Database adapter is generally used when ActiveMatrix components need to be informed of changes to a database. Configuring the adapter places triggers in the database that initiate interactions when database changes occur. However, this strategy does involve the use of an extra component, the adapter, and its interface.
File Interactions
There are two common options for interacting with files: direct interaction with the file system using the BusinessWorks implementation type and the File adapter. Once again, each has its appropriate usages.
Direct interaction with BusinessWorks provides interaction capabilities in both directions. BusinessWorks can detect changes to files (creation and modification) and initiate a wide variety of activities in response. It can also coordinate the creation, updating, renaming, and moving of files with other activities.
The File Adapter provides similar capabilities. In the outbound direction (from the external system), it can detect the creation and modification of files, and optionally rename or move them after they have been processed. In the inbound direction, it can create and append to files, and optionally call a command-line executable to trigger the system’s consumption of the file.
There are some differences in capability between BusinessWorks and the File Adapter. Although BusinessWorks generally provides the simpler solution, its parsing capabilities are not as rich as those of the file adapter. It also has limitations when it comes to the size of the file that it can handle, whereas the file adapter can handle arbitrarily large files.
Summary
Interactions with external systems depend upon the interfaces provided by those systems. Four common mechanisms are used for such interactions: application programming interfaces (APIs), databases, files, and protocols.
APIs are useful for interactions that are inbound to the system, but few systems provide callbacks to allow the system to call external code. Many applications have underlying databases, and these provide another mechanism for interaction. Databases tend to be good in the outbound direction, as extracting data from a database is straightforward (provided you know the schema). However, in the inbound direction, it is generally unacceptable to bypass application logic and update the database directly.
Many systems are able to both produce and consume files. For these systems, file-based interactions can be a good mechanism.
A protocol is logically an API tied to a transport, and it has the same limitations. Protocols tend to provide a good mechanism in the inbound direction, and little or no capability in the outbound direction.
One of the biggest challenges in systems’ interaction is recognizing events within the system—events that you would like to have trigger an outbound communication. The Adapter for Database provides a convenient mechanism for event recognition, placing triggers in the database that initiate the publication of database changes.
The API and database mechanisms can be effectively combined to provide good system interactions in both directions: Each is strong where the other is weak. This combination provides the architectural basis for a number of TIBCO adapters.
When it comes to interactions between ActiveMatrix components and external systems, there are three primary design patterns:
• Direct interaction via ActiveMatrix-supported protocols, such as SOAP over HTTP or JMS, and XML over JMS
• Indirect interaction via TIBCO adapters
• Direct interaction via non-ActiveMatrix-supported protocols, such as HTTP and JDBC
When it comes to database and file interactions, there are two viable strategies for each. For databases, you can interact with the database directly using JDBC or you can use the database adapter. For files, you can interact with the file system directly or you can use the file adapter. In both cases, the TIBCO ActiveMatrix BusinessWorks implementation type makes direct interactions easy to implement.