Chapter 10
Combining the Services
10.1 Introduction
In Part Two of the book we started out with the architecture of the Service Availability (SA) Forum interface specifications and the information model used in systems compliant to the specifications. These chapters together painted the overall picture of such systems and provided the basics for the discussions to follow.
In subsequent chapters we looked at the different service specifications in details starting from the platform level up toward the applications and the management functionalities.
Our suspicion is that these chapters even in this format presented quite a bit of challenge for readers not particularly familiar with the domain. Nevertheless we can assure these readers that reading the specifications themselves would have been a more daunting task.
In any case, we have done the decomposition and the analysis of our architecture and this is the point where we need to pull everything we learnt about our system together and do the synthesis.
We can look at our system from the perspective of the designers and the developers of an application, who need to build an application that will be able to provide its services in a highly available manner; and for that purpose who want to take advantage of the SA Forum specifications and their implementations by using the SA Forum services instead of developing the same functionality from scratch.
The other perspective is of the system architects and designers who need to come up with the platform on which such applications can be deployed. Their primary goal is to come up with a flexible platform that can accommodate the anticipated range of applications effectively and also support the operational and maintenance activities that are inevitable throughout the system life-cycle.
Finally we also take a quick look from the perspective of system administrators who have the task of managing and maintaining the system. They would like to see a rather homogeneous picture in spite of all the diversity that may exist in the actual system.
10.2 Application Design and Development
After reading Part Two of the book, probably the question one ponders about is: Do I really need to deal with all these services and that kind of complexity?
We are happy to say that no, application designers and developers do not need to deal with all the different services. They can focus only on those that provide functionality needed in their application implementation.
Application designers of course need to understand the SA Forum services enough to be able to decide which one of them offers the functionality they need in their applications and if there are more than one, which one of them is the most suitable. Here we provide some hints on how to go about these choices.
Unless an application manages or monitors some hardware in the system it does not need to use the Hardware Platform Interface (HPI) [35].
If it does, for example, the application service is actually provided using some hardware elements then the use of the HPI interface becomes handy as it allows the application to manage this piece of hardware in an implementation independent way and therefore the application can control these hardware elements on any system providing the HPI interface.
If the application does not manage the hardware, but still needs to know its status or it needs to influence certain administrative actions applicable to platform elements then using the Platform Management service (PLM) [36] may be enough. Through the PLM API (application programming interface) applications are able to track the status changes and vote—so to speak—whether the removal or the locking of a platform element impacts their services. The prerequisite of this tracking is that the piece of platform (hardware or software) the application is interested in is reflected in the PLM configuration.
Even though we emphasized at each service that the Cluster Membership service (CLM) [37] is the authority on the membership, this information is more important for the Application Interface Specification (AIS) services themselves than for applications. Applications that are managed by the Availability Management Framework (AMF) [48] will always run on member nodes, so using the CLM API is not needed to determine membership. But it can provide some additional information by passing through the node related PLM tracking information. If this is needed the application has the choice to listen to CLM or go directly to PLM.
Using these APIs and being managed by AMF may actually create confusion as both the application and AMF may receive the same validation requests resulting in contradictive reactions. The application needs to coordinate its actions with the AMF high-availability (HA) state assignments and provide feedback toward AMF using the HA readiness state settings.
Considering that the subject of this book is SA we expect that application designers and developers interested in the subject will want to use the AMF and therefore its APIs.
As we have pointed out already in the Chapter 6, for a developer there is no need to understand all the details of AMF. After all AMF can manage an application even if it does not implement any of its APIs (see non-proxied-non-SA-aware components and proxied components in Section 6.3.2.5. Obviously the SA aspect may not result in the best possible user experience, but for some applications these might be a perfectly viable options as we will see in Part III of the book where we present different integration options.
Applications that need to communicate within or without the cluster in a location independent way that also decouples the senders and the receivers can benefit from the described in Chapter 7 Event service (EVT) [43] and Message service (MSG) [44].
If the communication is intra-cluster and related to some state synchronization among different application instances, the Checkpoint service [42] becomes an option to alleviate some of the burden the developer needs to face.
We have to emphasize that none of these or any other AIS Utility Service is mandatory by any means. So, for example, if high performance is an requirement for the application communication, the use of other methods may be more appropriate than either EVT or MSG. However such choice also means that the developer will have to resolve at least some of the challenges we presented for these particular services.
Besides the EVT, Messaging, and Checkpoint services the SA Forum also standardized—and an application designer may want to consider—the Lock [40], the Naming [46], and the Timer services [47]. Chapter 3 presented a brief overview of each of these AIS Utility Services.
The AIS Management Services are there to unify the system management. As a result most applications that need to be integrated with such systems will need to use at least the Notification service (NTF) [39] for their notifications and alarms to provide the system administration with some online and offline status information about their operation. Such data becomes essential in the root cause analysis of failures and outages. Since notifications and alarms are logged by the middleware the application needs to use the Log service (LOG) [40] only if additional data needs to be recorded.
If the application requires some configuration and it exposes some administrative control, it may be beneficial to integrate it with the Information Model Management service (IMM) [38] as object implementer for the configuration object classes defined as an extension to the SA Forum information model. The objects of these configuration classes can offer a configuration interface and also expose the application status. The AMF and PLM are good examples that application designers may study.
Alternatively an application may chose to expose its status through runtime object classes in a similar way as we have seen in case of the AIS Utility Services.
Finally applications that need to coordinate their actions with their own and system upgrades may also use the Software Management Framework (SMF) [49] API. The anticipation though is that it will be mostly management applications, system components and such that have the task of coordinating different activities in the system that would benefit from the possibility offered by the SMF API. For example, if an application needs to backup its application level data or deploy additional safety measures for the duration of an upgrade the application designer may consider the callback mechanism offered by the SMF.
To facilitate the application developers' task in Part Three of the book we review the basics of the programming model for both the C and Java APIs. As mentioned already we will present an example of integrating a relatively simple application with some AIS services. We will also look at legacy applications how they can utilize the SA Forum services in the hope that these examples convince everyone that the SA Forum specification did indeed make life easier for many application developers.
10.3 Application Platform Design
While the application developers can pick and choose from the SA Forum Services as connoisseurs from a smorgasbord, system architects who design the platform to support these applications need to follow the dependencies existing among the SA Forum services at the specification level, as well as at the implementation level and also the needs of the targeted applications.
In Chapter 3 we have pointed out that modularity is the basis of the architectural design of the Service Availability Interface (SAI) specification. This means that in a particular system not all the services need to be present all the time. That is, application platform designers also have some choices, but instead of the smorgasbord allowing any choice it is more like an ‘à la carte’ menu where complete entrées are defined.
The SAI specification defines some dependencies among its services. These need to interact and rely on each other for proper operation and therefore they need to be included in the application platform together.
An important assumption about these service interactions is that a dependent service should be able to obtain everything it needs from the required sponsoring service using the standard APIs as defined in the specification. That is, generally there is no need to extend the service interface to enable service interactions, which provides the necessary condition for interchangeable service implementations. Of course, it depends on the particular implementation whether this remains to be the case or not.
At the discussion of each of the services we dedicated a separate section to present their interaction with other SA Forum services. Therefore here we only focus on the global picture and we will not go into details.
Among the service interactions the most important one is the dependency of the AIS services on the CLM with the exception of the PLM and the Timer services. The result is that for all practical reasons we can assume that CLM is part of the middleware providing AIS services.
But the CLM specification does not define its dependency on the PLM as mandatory, nor does PLM's dependency on the HPI. Service implementers are free to make their own choices.
Since our focus is on SA our target applications require availability management and therefore the AMF becomes a must. Note that this does not require that all managed applications are SA-aware. AMF can manage applications without them implementing the AMF APIs.
AMF receives its configuration and exposes its administrative API via the IMM which needs to be included in the application platform.
Considering the HA requirement—approximately five minutes down time in a year—it is easy to see that there is no opportunity for shutting down the system for maintenance or upgrade. This also makes the SMF an essential part of the system. During upgrades SMF introduces the AMF configuration changes and the administrative control again using IMM.
The same applies to all AIS services which define an information model or expose administrative control. Therefore IMM becomes just as important in the application platform as CLM.
Throughout Part Two the other recurring service in the service interaction sections was the NTF. The different SA Forum services use NTF as the carrier of their notifications and alarms. The expectation is the same toward applications developed to be deployed in SA Forum clusters.
Finally all SA Forum services are expected to use the LOG to record significant events that may facilitate system analysis and in particular root cause analysis. This interaction we did not mention in the service interaction sections as the specifications—except for a few cases—do not elaborate on the information that needs to be collected. The middleware implementation should LOG at least the notifications and alarms generated in the system by the system and also by applications. Therefore LOG is also considered as a fundamental application platform service.
From the perspective of applications besides availability management they need communication and synchronization facilities as we presented in Chapter 7. However from the perspective of the SAI architecture all these services are optional. The question is whether any and which one of them needs to be included in the application platform.
The good news is that since the specifications do not define dependency between these services, they could be added at any time when a new application requires them provided that the service implementations follow the specifications.
This brings us to the perspective of the middleware implementation. Different middleware implementations may define their own requirements on how the service implementations are coupled. For example, in the OpenSAF [91] implementation of the SA Forum specifications, some AIS service implementations rely on the AMF implementation for managing their HA. This means that even if the target applications do not require AMF, the middleware itself does and needs to be included in the application platform.
It actually makes complete sense as the middleware services are expected to be highly available which means that one way or another, their availability management needs to be implemented. So it is natural that they rely on one of their own, which was designed for exactly this purpose, it is ‘the professional’ for the job. This also means that the functionality is not duplicated in each service creating potentially significant overhead. This comes with the added benefit that applications can also use the services of AMF, reducing the overhead even further.
In Part Three we will also look at more details and considerations of the system level design again in the context of migrating complex applications requiring complete solutions.
To provide some insight to the middleware itself we will have a peek in Chapter 13 at OpenHPI [71] and OpenSAF [91] and their architectural solutions to implement the SA Forum services. This chapter is only a teaser for these implementations matured to be carrier grade.
10.4 Operation and Maintenance
If the SA Forum did its job well and the application designers aligned with the concepts correctly we should have a homogeneous administrative interface to our system. The administrative control is exposed through the IMM object management API, while status information is available also via the NTF. The LOG can collect additional data delivered in terms of log files.
The main question though is who are the expected consumers of all this information?
Looking at the content of the SA Forum information model it is hard to imagine a human administrator tweaking the system by adjusting dozens of attribute values in hundreds of configuration objects. In fact many have the view that most of these attributes should not be exposed to humans as human errors are a, if not the major, source of outages. Complex and overwhelming management data backfires easily.
That is to say that most of the management information exposed by IMM is there for management applications rather than for the human operator. A very good example of a management application is the SMF presented in Chapter 9, which expects an upgrade campaign as an input and uses it to deploy all the configuration and other changes in the system in a systematic manner. By keeping track of all the operations performed and the possible recovery actions which it can deploy, SMF is able to perform systematically an upgrade or a reconfiguration. As we mentioned already to protect the system from human errors SMF is considered the only way to modify system configurations except for complete system initialization from a new configuration file.
Now looking at the information model as the source of the system configuration and the upgrade campaigns as scripts to manipulate this configuration one still may not be at ease. Someone somehow needs to come up with these artifacts. The only reassurance the SA Forum specifications can provide is that they can be designed and developed offline and tested just like one would do with software to be deployed in the system.
The idea of using scripts and automating these administrative tasks is not completely new. However the systematic end-to-end approach taken by the SA Forum is quite revolutionary in our opinion. Some of the details of the solution have been presented already in Chapter 9 and we will take another look in Chapter 16 of Part Three from the perspective of how software engineering techniques known better in the field of software development can be deployed here too for these tasks to further increase SA.
With that let us embark on our adventure and get a taste of the use of the SA Forum specifications. Let see them in action.