When introducing service-oriented architectures into legacy environments or when delivering services via bottom-up approaches, it is not uncommon to encounter interoperability challenges that can only be overcome by programs with brokerage features.
These three common patterns have been part of integration architectures for many years and continue to be valuable and even essential in modern-day SOA implementations. Each addresses an aspect of data exchange:
• Communications Protocol – Software programs can be built to communicate with different technologies or different versions of the same technology. Either way, when disparity between wire-level protocols exist, some form of intermediary program is required to “bridge” these differences.
• Data Format – Even if the underlying communications protocol is compatible, two programs that need to exchange data may offer this data via different character sets, file formats, or data formats (such as XML and CSV). For this, an additional type of intermediary processing logic is required to convert one format to the other.
• Data Model – Even when the communications protocol and data formats are compatible, for software programs to exchange any form of structured data (such as business documents), the definition of the model that the data structure is based on (the data model) can be different at transmitting and receiving ends. This leads to a requirement for the runtime transformation of information from one data model into another.
Protocol Bridging (687), Data Format Transformation (681), and Data Model Transformation (671) respectively address these three interoperability obstacles. Although their methods are well-established and proven, the required use of these patterns can indicate an inability to achieve the standardization required to realize service-orientation to its full extent within a given environment.
Each pattern introduces a layer of intermediary processing that has similar impacts on performance, design complexity, and development effort. The overuse of these patterns is therefore considered an anti-pattern.
For any service-enabled part of an enterprise where Canonical Schema (158) has not been successfully applied, services delivered as part of different projects run a high risk of representing similar data using different schemas (Figure 21.1).
This will typically not affect immediate service implementations, as the schemas may very well have been standardized across the initial solution or composition. However, when these same services need to be recomposed, schema incompatibilities can impose significant interoperability challenges.
Data model transformation logic can be introduced to carry out the runtime conversion of data, so that data complying to one data model can be restructured to comply to a different data model (Figure 21.2). This extends a non-standardized messaging framework, enabling it to dynamically overcome disparity between the schemas used by a service contract and messages transmitted to that contract.
Figure 21.2 An XSLT style sheet containing data model mapping logic (2) is added as a form of intermediary processing that is executed at runtime. With each transmission, the data model of the claims document is converted from the schema used by the Process Claims service (1) to the data model compliant with the schema used by the Claims service (3). This runtime transformation logic can reside with either service architecture or as part of a separate middleware platform.
Formalized data transformation is an established concept that dates back to the EAI era where data model incompatibilities were often resolved via broker services that existed as part of middleware platforms.
When services are implemented as Web services, XSLT is generally used to define the mapping logic that is subsequently executed to perform the transformation at runtime. In fact, the use of XSLT style sheets represents the most common application of this pattern.
Although Data Model Transformation is a fundamental and essential part of most service-oriented architectures, it represents a pattern that is used only out of necessity. Many of the service contract patterns described in this book, in fact, result in a reduction or even an elimination of data transformation requirements. However, the realities of legacy encapsulation, the application of Domain Inventory (123), and common governance challenges usually introduce schema disparity that still must be dealt with.
This pattern solves an important problem, but it is equally important to be constantly aware of the fact that it is only applied when other patterns cannot be realized to their full potential. Due to the constant emphasis on “transformation avoidance” throughout service-orientation, it is recommended to consider this pattern as a last resort to solving interoperability problems.
The use of this pattern has several consequences:
• The development of the mapping logic adds time and effort to the creation of service compositions.
• The incorporation of data model transformation logic introduces design complexity into a service composition and the service inventory as a whole.
• Data model transformation logic introduces a runtime layer wherein the execution of mapping logic adds performance overhead every time services with disparate schemas need to exchange data.
Canonical Schema (158) has a well-known “tug-of-war” relationship with Data Model Transformation, in that this pattern generally needs to be applied to whatever extent Canonical Schema (158) is not realized.
When in use, it can help overcome disparity between services that may be introduced by patterns like Domain Inventory (123) and Inventory Endpoint (260). It also addresses conversion requirements that may result from Legacy Wrapper (441) and Rules Centralization (216).
Although it provides important functionality, its overuse is an anti-pattern that can undermine the goals of Capability Recomposition (526).
As shown in Figures 21.4 and 21.5, Data Model Transformation is a core part of Service Broker (707) and Enterprise Service Bus (704) and also the only broker-related pattern associated as an optional extension of Orchestration (701).
Figure 21.5 Data Model Transformation is also one of the optional patterns associated with Orchestration (701).
Services can be fundamentally incompatible with resources or programs that only support disparate data formats. For example, a service may have been standardized to send and receive XML-formatted data but is required to also retrieve data from a legacy environment that only supports the CSV format (Figure 21.6).
A layer of data format transformation logic is introduced. This logic is specifically designed to convert one or more data formats into one or more different data formats (Figure 21.7).
Figure 21.7 A Format Conversion utility service is added to the architecture. This service abstracts the legacy application API and provides XML-to-CSV and CSV-to-XML functions. Note that in the depicted architecture, the Format Conversion service exists as a component being reused by multiple components that are part of Web services, as per Dual Protocols (227).
Data format transformation logic can exist within a service’s internal logic, within a service agent, or (as shown earlier in Figure 21.7) as a separate service altogether. Middleware with brokerage features will often provide out-of-the-box services and agents that perform a variety of data format conversions.
Furthermore, because the conversion of one data format to another will naturally transform the source data model, the application of this design pattern can be seen as also applying Data Model Transformation (671). However, as previously explained, Data Model Transformation (671) is most commonly used separately for the conversion of one formal data model to another when both source and destination formats are the same.
As with any of the three design patterns associated with the parent compound pattern Service Broker (707), the introduction of a transformation layer for data format conversion will have the following impacts:
• An increase in solution development effort when the required transformation logic needs to be custom-programmed. However, custom development effort may not be necessary when using a middleware platform already equipped with the required conversion functionality.
• An increase in design complexity, as this pattern will introduce a new layer in an already distributed environment.
• An increase in performance requirements because the format conversion will need to be executed every time interaction between disparate format sources is required. This is especially a concern with the transformation of data formats because of the tendency to carry out bulk or batch format conversions.
These impacts are often considered natural requirements that come with introducing a service layer into an enterprise with legacy applications and resources.
Data Format Transformation is typically utilized to help integrate custom service logic with legacy systems, which makes it commonly required when applying Legacy Wrapper (441). The actual transformation logic may be the responsibility of a façade component that is tasked with converting proprietary to standardized formats, as per Service Façade (333).
When protocols are standardized using Canonical Protocol (150), data formats are usually standardized as well. This is because most communication protocols are associated with common data formats. For example the CSV format is often used with FTP and SCP protocols, and XML is primarily associated with HTTP. Therefore, when Data Format Transformation is required, it is often in conjunction with Protocol Bridging (687).
Figure 21.8 Data Format Transformation is commonly employed to deal with traditional integration issues resulting from legacy encapsulation, which is why this pattern relates to Legacy Wrapper (441) and broker-related patterns.
Also worth noting is that this pattern represents one of the three core patterns of Service Broker (707). Because Service Broker (707) is a also a core part of Enterprise Service Bus (704), ESB platforms are fully expected to support the conversion of different data formats.
Case Study Example
Besides commercial forestry activity, another area the FRC is involved in is forestry-related disaster prevention. A large part of this program regulates critical response plans for forest wildfires.
One business process in particular governs the issuance of flight plans from fixed-wing and helicopter aircrafts capable of acting as “water bombers” for aerial firefighting. To coordinate such an effort, especially when having to call in aircrafts from different regions, a complex flight planning process is encapsulated and carried out via a central Flight Plan Validation service.
Because this service is responsible for processing flight plans sent from different sources, it has been designed to handle different data input formats. For example, the default flight plan format is XML, but the service is capable of receiving proprietary formats, such as AFTN (Aeronautical Fixed Telecommunication Network), AMHS (Aeronautical Message Handling System), as well as SMTP.
As shown in Figure 21.10, the Flight Plan Validation service converts AFTN, AMHS and SMTP message attachments to the expected XML message format, which is then forwarded to additional services.
Figure 21.10 Different organizations providing critical assistance in aid of a forestry fire supply aircrafts that become part of an aerial firefighting effort. Flight plans are submitted to the FRC, which is responsible for the coordination effort. The FRC Flight Plan Validation service is designed to support the receipt of multiple flight plan format types and to then convert these formats into XML.
Note
If you haven’t already, be sure to read the section What Do We Mean by “Protocol?” in the description for Canonical Protocol (150) before proceeding.
Services delivered by different project teams or at different times can be built using different communication technologies (Figure 21.11). For example, services can use completely disparate frameworks (such as JMS and DCOM) or different versions of the same communications technologies (such as HTTP plus SOAP versions 1.1 and 1.2).
This is a common scenario when design standards are not prevalent in an enterprise and services are primarily designed in support of tactical requirements. As a result, service interoperability outside of immediate composition boundaries is severely diminished, leading to missed opportunities to reuse and recompose services for new purposes. Over time, this leads to “islands” of disparate service compositions that are reminiscent of traditional, silo-based application environments.
Protocol bridging technology is used to overcome the disparity between different communications frameworks by enabling the runtime conversion of protocols (Figure 21.12).
Figure 21.12 The consumer programs interact with a middle-tier broker that provides protocol bridging features. Separate protocol adapters are used to translate the two incompatible protocols to the required SOAP version 1.2 over HTTP. The broker then transmits the messages to the service on behalf of the consumers.
This pattern is commonly employed when legacy systems need to act as service consumers or when legacy logic needs to be encapsulated by services. As mentioned earlier, it is also used to overcome communication gaps when disparate sets of services are delivered.
A protocol bridging layer is composed of a set of adapters that act as on-/off-ramps for a given transport protocol. These adapters may be off-the-shelf products provided by the broker or a third-party vendor, or they may be generated to mirror specific service contracts using a proprietary protocol dialect.
When a protocol bridge receives a message, it transforms it and performs whatever other work is required to make the message comply with the target protocol (or protocol version).
Protocol bridging can be used to expose or access services via multiple protocols, depending on whether adapters are implemented on the consumer or provider end. A multi-protocol conversion program can offer support for numerous protocols (HTTP, JMS, UDP, TCP, etc.) that can be natively exposed as independent protocol transformation services.
While necessary to overcome baseline communications disparity, Protocol Bridging is a pattern only used out of necessity. From an architectural perspective, it is considered an undesirable option because of the design complexity and runtime performance overhead it imposes. Even though it can support interoperability between services, it does so at a less than optimal level. Service compositions relying on this pattern often end up having decreased levels of availability and reliability.
Protocol Bridging originated with early EAI platforms and is therefore still considered an integration-related design approach. The manner in which it involves protocol adapters and translation logic can be overly complex because of the binary encoding used by many transport protocols. In some cases, protocol conversions may even result in inaccurate semantic meanings being applied to message contents.
Though an expected part of brokerage functionality, some platforms may support limited sets of protocols with an emphasis on conversion to and from SOAP and HTTP only.
Protocol Bridging is a pattern that has long been used to solve traditional integration problems and therefore finds itself closely related to Legacy Wrapper (441) and Dual Protocols (227). The nature of its bridging technology can also be influenced by Canonical Resources (237) when choices are available.
The extent to which this pattern is applied to intra-service bridging requirements only is based on how successful the application of Canonical Protocol (150) has been in achieving a sole inter-service communications technology.
As a part of Service Broker (707), Protocol Bridging is commonly associated with the compound pattern Enterprise Service Bus (704), as shown in Figure 21.14.
