Service Orchestration is one of the terms that is most commonly associated with the Enterprise Business Flows framework and SOA implementation in general. At the same time, Orchestration is one of the two most common compound patterns (the other is ESB) employed for addressing reoccurring problems related to application integration. In this chapter, we will discuss these common problems and the ways of mitigating them using different patterns that are related to Orchestration, and find ways to turn integration into service collaboration. Not all debated patterns will be related to the Orchestration realm directly.
However, as long as they are quite universal and applicable to every framework, we will put them here; this is because the Orchestration layer can be a good demonstration ground to begin with this exercise. Some basic SOA Suite skills (for 11g composites: BPEL, Mediator, and RE) and some Java skills (MDB and JMS) would be useful, as we simply do not have enough room here to provide a complete technical tutorial. The main focus in this chapter will be on patterns, ensuring reusability and composability. Therefore, architecting the agnostic Composition Controllers and subcontrollers will be the most interesting part.
Another reason to start with Orchestration first is that due to the popularity of Oracle BPEL Process Manager (initially Collaxa and later 10g and 11g), in the last 9 years, quite a few enterprises have maintained a considerable amount of orchestrated services with different degrees of service orientation. It is not too common for an enterprise to start something from scratch these days (such as the St. Matthews Hospital example in Oracle SOA Suite 11g Handbook, Lucas Jellema, McGraw-Hill), practicing a pure top-down approach. Thus, the patterns related to Service Inventory Analysis and Modeling will be essential for understanding the presented example.
Orchestration is always related to task-orchestrated services and non-agnostic workflows that are devised for covering a single (but complex) business operation, which we discussed in Chapter 1, SOA Ecosystem – Interconnected Principles, Patterns, and Frameworks. This master service is a chain of service invocations and simple request-responses and/or complex transactions that involve several services at a time (ACID- or BASE-style). As mentioned earlier, this master service fulfills the runtime role called Composition Controller, which controls the service's composition members or other subcontrollers. This is a pure runtime role, and the next time you decide to include this master-controller into a more complex composition, the new role will be different (becomes a subcontroller). We also know that Composition Controller does not always start the service interaction (or service activities) within the composition. More often than not, a composition is commenced by a composition initiator that is not part of the composition itself.
In relatively static business models (big savings/deposit banks and some governments institutes), compositions could be pretty static as well with fixed data models and business rules. Some conservative businesses can rely on traditional BPEL processes, especially if geographic areas are well set and related service endpoints are permanent. As we all know, the static business model is rarely used these days, even in government structures. This is because there are many aspects such as technological, commercial, and organizational that shake our organizations. Interestingly, even small companies with a single line of business also like to stay more and more agile and adaptable in modern competitive surroundings. Task services that are placed at the top of the whole IT stack, most closely related to concrete operations and business people, must not pose bottlenecks for adapting the enterprise principle. How can we achieve that? Yes, you are right, all eight principles must be applied, but with some preconditions as follows:
- Composition members must logically encapsulate a concise business context and introduce a standard service contract (among the other seven principles), that is, stay composable
- Controllers that compose these members into a business process fabric must be agnostic
The second outcome is possible when all the business rules are abstracted from Composition Controller (remember that theoretically, we are compromising the controller's autonomy), and it can be accessed and consumed by the controller dynamically using the message context. Following this logic, we can assume that Composition Controller can recursively act as a subcontroller for just another instance. That's it! Just by changing the rules, we can change the sequence of the invocations and consequently implement new processes on the fly. So simple, isn't it? Who would think about it? Here, we can conclude the chapter and probably the entire book.
Talking about it seriously though, dynamic Composition Controllers are not new at all, and we have all coded and implemented them many times. However, here we are going to choose examples that are specific to Orchestration, where we have two distinct properties: processes should be long running and asynchronous and the controller must be truly agnostic for dynamically brokering other service members. Therefore, the problems we are facing are obvious too:
- There is no problem in abstracting the rules, centralizing them, and accessing them dynamically. What we must remember is that these rules could evolve while the process is running for days and weeks. Thus, rule versioning must be handled seamlessly by Composition Controller together with Rule Engine. A single process cannot start using one ruleset and finish with another.
- The process can fail for many reasons. With a static non-agnostic controller, you can set precise compensations and error handler(s), anticipating concrete faults. An agnostic controller alone has no idea about what to do in error situations, which means that you will again need to employ Rule Engine in order to compensate for the error.
- A compensation itself is rather specific as we cannot always apply generic compensation flow(s) in a generic controller. We should also be careful in propagating the error to the upper level (initiator or master-controller) for many reasons.
- Actually, abstraction and centralization of the rule is the problem as we technically present a single point of failure, and the infrastructure must provide a redundant solution for this.
- While abstracting the controller itself, we must not forget that these types of controllers are more suitable for synchronous and fast-running compositions. Long-running compositions will require persisting the process state, and agnostic controllers are not the best candidates for this. Actually, BPEL was invented to solve this problem, presenting a task-orchestration service as a non-agnostic Composition Controller and one of the means for the implementation of the Process Centralization SOA pattern.
So, why would we step aside from the Process Centralization pattern based on a very comfortable and easy-to-use BPEL and then try to decentralize the processes using an agnostic controller? The following practical example based on a counterfeit, yet quite realistic, telecommunication primer will demonstrate the reasons for this approach. Jumping ahead a little, we would like to substantiate that we are not actually breaking the concept of the Process Centralization pattern with the implementation of an agnostic controller, as we are enforcing the process of centralizing composition's rules. Secondly, we are not going to abandon the idea of using BPEL as the Orchestration language/engine or reinvent BPEL's syntax. Instead, we will use SOA Suite tools (11g PS6) in this example; since SOA patterns have very generic solution approaches, you can roll it up on any other platform.
It is difficult to find a business sector with more rapidly changing commercial and operational environments than telecom these days. Reasons such as active mergers of mobile operators with cable and content providers, new technologies and geographic diversities, and high demands from customers for new products set the highest requirements for the agility and adaptability of business processes. Intense competition will easily put any company of any size out of business if it cannot keep up with these types of shifts. Let's see how the agnostic controller can address these challenges, and at the same time, mitigate the problems described in the bulleted list in the previous section.
Some of the basic facts about our fictitious telecommunication enterprise are mentioned in the following table:
|
Telecommunication enterprise |
Business domain |
Governing and type of ownership |
|---|---|---|
|
CableTelUnlimited Inc. (CTU), HQ: Brasilia |
Telecommunication |
Public |
CTU was started 20 years ago as a cable company, providing TV/video for customers in one geographic area. Quite soon, through a series of acquisitions, they became a large telecom enterprise, delivering various combinations of three main traditional-for-cable-company products:
- Voice/phone (landline VoIP)
- Video entertainment
- Internet provisioning
Following this business development strategy, geographic operations were expanded all over South America during the last ten years, effectively covering ten countries and turning the company into one of the biggest telecom market players.
Currently, business operations are still cable-oriented and mobile products are not in the company's portfolio yet, but there are some plans for business expansion in this area.
Traditionally, this telecom enterprise is not a "software house", so all the development work is done by external vendors. The two main factors, namely the technically diversified products' portfolio (video/voice/Internet) and past acquisition, have shaped the company into three main departments (offices) with a relatively low level of collaboration.
Also, local affiliates in all the countries where the enterprise operates have a considerable level of freedom with regards to standards implementation; additionally, the burden of legacy applications is also quite substantial. Some legacy applications have been in operation for almost ten years due to budget constraints or lack of life cycle governance, so the level of adaptation to the current business environment is below expectations.
CTU has a massive inventory of products from various vendors. In operational HQ, these products are unevenly distributed between three main operational departments:
- Network (responsible for Internet and voice), handled by the CNO
- Technology (content delivery and Video on Demand), handled by the CTO
- IT and internal IT systems (responsible for order management and provisioning, billing, and customer management), handled by the CIO
The last department was also involved in providing internal system integration to some extent to the other two departments, especially the CRM domain. Only a few solutions were developed in-house, mostly for integration and service abstraction. The number of applications in the application's portfolio is unknown but can be roughly estimated at 50 in CIO, 200 in CTO, and 100 in CNO.
The HQ's application farm is deployed, maintained, and administered on two business data centers with more than 600 virtual machines, each in clustered environments, handling multitenant and individual accesses for regional offices. The level of multitenancy is low at the present moment; about 90 percent of all VMs are country-specific, although their business logic is similar with very small variations.
Regional offices have their own local application infrastructure that supports business applications in regional offices and maintains integration with core HQ apps related to the affiliates.
It must also be mentioned that in some countries, a corporate entity has more than one affiliate, depending on the line of business. The level of an affiliate's technical efficiency varies, reflecting business proficiency. Each of the three HQ administration offices has an IT division with its own departmental structure and organizational hierarchy. There are regular meetings between the IT managers from all divisions, but outside of that, there is infrequent communication or coordination.
The vendor selection process and new product procurement routines are formalized by policies, which are specific for each department. The standard RFI/RFP process could take up to 10 months according to these policies; at the same time, the requirements for new product/projects' implementation are about 6 months. IT resources are rarely shared between departments.
The recent annual financial report demonstrated the best corporate earnings over the last 10 years. At the same time, a detailed analysis revealed that operational costs have increased considerably compared to the last year by almost 10 percent. The reasons can be identified as follows:
- Two new strategic products were released last year with the first stage covering one third of the countries in operations
- Some applications have been migrated from local premises to a private cloud that is built on a corporate data center
These reasons are naturally positive, but some considerable drawbacks should also be mentioned:
- New products were under development for three years and the implementation was out of budget (40 percent) and time (the deadline was moved three times).
- Due to the delay, some first-on-market advantages were lost, and during the same period, competitors managed to deliver two to four products more in different geographic areas.
- The architectural delivery models for these products were presented in the application-silo style at the last stage of the products' delivery with very limited service collaboration efforts. Most of the applications were integrated in a point-to-point (P2P) manner.
- Although migration of some applications to a private cloud was considered successful, it didn't reduce the operational cost, as none of the cloud delivery models (Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS)) were presented clearly. VM provisioning still takes up the same amount of time that it used to with on-premise deployment.
Considering all of the stated reasons, the Board of Directors decided to transform the IT process, aiming to initiate the following:
- Streamline the RFI/RFP process for optimizing PoC time and narrowing down the vendors list. They decided that having two approved vendors would be the common rule for all departments.
- Bridge the gaps between architectural groups in the main operational departments, and establish a new enterprise architecture office. A new Chief Architecture Office (CAO) head was appointed.
- This new department was made responsible for establishing standards, policies and design rules, and controlling E2E delivery from the enterprise architecture standpoint. Even common terminologies were supposed to be consolidated, agreed upon, and conveyed down to all the related parties, from the project management and delivery office to the coders from a number of vendors.
- The physical implementation of this approach was required to be materialized in the PoC prototype for every key delivery.
Suppose that the new CAO designates you as the lead architect for the first SOA initiative. This will be a highly visible project that your colleagues and superiors will be observing with high interest. The goal of this project will be to assess the services currently in development and to upgrade the CTU runtime platform using proven SOA design solutions in order to demonstrate that SOA can solve a series of problem areas. In support of this goal, you will be provided with funding to implement a modern enterprise service bus platform.
As the starting point of this challenging IT transformation program, the Operations System support domain is selected. Traditionally, setting up the Order Management (OM) functionality is the primary objective for the implementation of service-oriented computing within this domain.