Preface
How This Book Came About
Maria's Story
I joined the Service Availability (SA) Forum in 2005 with the mandate of representing Ericsson in the efforts of the SA Forum Technical Working Group (TWG) to define the Software Management Framework. This is where I met Francis and the representatives of other companies working on the different specifications. The standardization has been going on already for several years and I had a lot to learn and catch up with. Unfortunately there was very little documentation available besides the specifications themselves, which of course were not the easiest introduction to the subject.
Throughout the discussions it became even more obvious that there was an enormous ‘tribal knowledge’—as someone termed it—at the base of the specifications. This knowledge was not written anywhere, not documented in any form. One could pick it up gradually once he or she started to decipher the acronym ridden discussions flying high in the room and on the email reflectors. There were usually only a handful who could keep up with these conversations at the intensity that was typical at these discussions. For newcomers they were intimidating to say the least. This was an issue for the SA Forum from the beginning and for the years to come even though there was an Educational Working Group with the mandate to prepare training materials. Many TWG members felt that it would be good to write a book on the subject, but with everyone focusing on the specifications themselves there was little bandwidth to spare for such undertake.
Gradually I picked up most of the tribal knowledge and was able to participate in those discussions, but preparing educational materials or writing a book still did not come to my mind until Ericsson started a research collaboration with Concordia University. Suddenly I had to enlighten my students about the mysteries of the SA Forum specifications. These specifications are based on the years of experience of telecom and information technology companies in high-availability cluster computing. These systems evolved behind closed doors in those companies as highly guarded secrets and accordingly very little if any information was available about them in the public domain. This also meant that the materials were not taught at universities nor were books readily available to which I could refer my students. Soon the project meetings turned into an ad-hoc course where we went through the different details, the intricacies of the specifications and the reasoning behind the solutions proposed. These solutions were steeped in practice and brewed for production. They reflected what has worked for the industry as opposed to theoretical models and proofs more familiar to the academia. This does not mean that they lack theoretical basis. It just means that their development was driven by practice.
Understanding all these details was necessary before being able to embark on any kind of research with the students and their professors. These discussions of course helped the students but at the same time they helped me as well to distill the knowledge and find the best way to present it. Again it would have been nice to have a book, but there was none, only the specifications and the knowledge I gathered in the TWG discussions.
A few years later OpenSAF, the open source implementation of the SA Forum specifications reached the stage when people started looking at it from the perspective of deployment. They started to look for documentation, for resources that they could use to understand the system. OpenSAF uses mostly the SA Forum specifications themselves as documentation for the services compliant to these specifications.
These people faced the same issue I had experienced coming to the world of the SA Forum. I was getting requests to give an introduction, a tutorial presentation so that colleagues can get an idea what they are dealing with, how to approach the system, where to start. After such presentations I would regularly get the comment that ‘you should really write a book on this subject.’ At this time I saw the suggestion of writing a book more realistic and also with the increasing demand for these presentations it made a lot of sense.
In a discussion with my manager I mentioned the requests I was getting to introduce the SA Forum specifications and the suggestions about the book. He immediately encouraged me to make a proposal. This turn of events transformed the idea I have toyed with for some time into a plan and the journey has begun. I have approached Francis and others I knew from the SA Forum to enroll them in the book project. This book is the realization of this plan, the end of this journey. It is a technical book with a rather complex subject that we, the authors and editors tried to present in a digestible way.
Francis' Story
My contribution related to the SA Forum specifications in this book was based on the project titled ‘High Availability Services: Standardization and Technology Investigation’ that I worked on during 2001–2006 in Nokia Research Center. The project was funded by Strategy and Technology, the then Nokia Networks (now part of Nokia Siemens Networks), with the objective to support the company's standardization effort in the SA Forum and contribute to a consistent carrier-grade base platform architecture for the then Nokia Networks' business. I became one of the Nokia representatives to the SA Forum and took part in the development of the first release of the Availability Management Framework specification with other member companies' representatives. Subsequently, I took up the role of co-chairing with Maria the Software Management specification development group. Regrettably I had to stop my participation in the SA Forum at the end of 2006 before the Software Management Framework was published.
Parallel to my full-time employments over the years, I have been giving a 12-hour seminar course on highly available systems to the fifth (final) year Master of Engineering students in Computer Science at INSA Lyon (Institut National des Sciences Appliquées de Lyon) in France almost every year since 1993. It has been widely recognized in the academic community that there is a lack of suitable books for teaching the principles and a more pragmatic approach to designing dependable computer systems. Very often such materials have to be gathered from various sources such as conference proceedings, technical reports, journal articles, and the like, and put together specifically for the courses in question. On a number of occasions, the thought of writing such a book came to my mind but it left rather quickly, probably due to my senses were warning me that such an undertaking would have been too much.
I remember it was a few years ago when Maria asked me if I could recommend a book in this area for her teaching. After explaining to her about the general situation with regard to books in this subject area, I half-jokingly suggested to her that we could write one together. She left it like that but only returned in January 2010 and asked if I would be interested in a book project. As they say, the rest is history.
The Goal of the Book
Our story of how the book came about has outlined the need that has built up and which it was time to address with a book. It was clear that the approach to the subject should not be too theoretical, but rather an explanation of the abstractions used in the SA Forum specifications that would help practitioners in mapping those abstractions to reality; it also needed to make the knowledge tangible, to show how to build real systems with real applications using the implementations of the SA Forum specifications. The time was right as these implementations were reaching maturity fast.
At the same time we did not want to write a programmers' guide. First of all a significant portion of the specifications themselves is devoted to the description of the different application programming interface (API) functions. But there is so much reasoning in these systems and the beauty of their logic cannot be delivered just by discussing the APIs, which are like the scrambled puzzle pieces do not reflect the complete picture, the interconnection and interdependencies until they are put together piece by piece. They give little information on the reasoning which animates the picture and fills in even missing puzzle pieces.
The specifications may not be perfect at this time yet but they bring to the light this technology that has been used and proved itself in practice to provide the magic five-nine figures of in service performance, but has been hidden from the public eye. At this time they already come with open source implementations meaning that they are available for anyone to experiment with or to use for deployment, and also to evolve and improve.
The concepts used in these specifications teach a lot about how to think about systems that need to provide their services continuously 24/7 in the presence of failures. Moreover they are designed to evolve respecting these same conditions, that is, these systems and their services develop without being taken out for planned maintenance, they evolve causing minimal service outage. They are ideal for anyone who needs to meet stringent service level agreements or SLAs.
The concepts presented in this book remain valid whether they are used in the framework of the SA Forum specifications or transpired to cloud computing or any other paradigm that may come. The SA Forum specifications provide an excellent basis to elaborate and present the concepts and the reasoning. They also set the terminology allowing for a common language of discussion, which was missing for the area.
We set out to explain these concepts and their manifestation in the specifications and demonstrate their application through use cases.
So who would benefit from this book? The obvious answer is that applications and systems designers who intend to use the SA Forum middleware. However since we look at the specifications more as one possible manifestation of the concepts, ultimately the book benefits anyone who needs to design systems and applications for guaranteed service availability, or who would like to learn about such systems and applications. We see this book as a basis for an advanced course on high service availability systems in graduate studies or in continuous education.
The Structure of the Book
The book is divided into three main parts:
Part One introduces the area of service availability, its basic concepts, definitions, and principles that set the stage for the subsequent discussions. It also delivers the basic premise that makes the subject timely. Namely that in our society the demand for continuous services is increasing in terms of the number and variety of services as well as the number of customers. To meet this demand it is essential to make the enabling technologies widely available by standardizing the service APIs so that commercial off the shelf components can be developed. Enabling such an ecosystem was the mission of the SA Forum, whose coming about is presented also in this part.
Part Two of the book focuses on the specifications produced by the SA Forum to achieve its mission. The intention was to provide an alternative view of the specifications, a view that incorporates that ‘tribal knowledge’ not documented anywhere else and which provides some insight to the specifications, to the choices that were made at their design.
We start out with the architectural overview of the SA Forum middleware and its information model.
The subsequent chapters elaborate on the different services defined by the SA Forum Architecture. Among them the Availability Management Framework and the Software Management Framework each has their own dedicated chapter while the other services are presented as functional groups: the Platform services, the Utility services, and the Management Infrastructure services.
Rather than discussing all the SA Forum services at a high level we selected a subset on which we go into deeper discussions so that the principles become clear. We do not cover the Security service in our discussions as it is a subject crosscutting all the services and easily filling a book on its own.
The presentation of the different services and frameworks follow more or less the same pattern:
First the goals and the challenges addressed by the particular service are discussed, which are followed by an overview of the service including the service model and architecture supporting the proposed solution.
Rather than presenting the gory details of each of the API functions like it would be in a programmer's guide we decided to explain the usage through the functionality that can be achieved by using the APIs. This approach reveals better the complete picture behind the puzzle pieces of the API functions. We mention the actual API functions only occasionally when it makes it easier to clarify the overall functionality.
Whenever it is applicable we also present the administrative perspective of the different services. The goal of these sections is to outline what a system administrator may expect to observe in a running system and what control he or she can obtain through configuration and administrative operations according to the specification. Sometimes these details could be overwhelming, so the anticipation is that different implementations of the standard services may restrict this access while other vendors may build management applications that enhance the experience by assisting the administrator in different ways.
Subsequently the service interactions are presented inserting the service discussed thus far in isolation into the environment it is expected to operate. Since the specifications themselves are written in a somewhat isolated way, these sections collect information that are not readily available, which require the understanding of the overall picture.
Finally the open issues and recommendations conclude each of the service overviews.
Particularly the open issues deserve some explanation here: even though the SA Forum specifications are based on the best practice developed in the industry over the years, the specifications themselves are not the reflection of a single working implementation. Rather they are based on the combined knowledge derived by the participants from different working implementations. So at the time of the writing of the different specifications the SA Forum system existed only in the heads of the members of the SA Forum TWG. It was this common vision that was scrutinized in the process of the standardization that obviously reshaped and adjusted the vision.
As the work progressed and people started to implement the different specifications the results were fed back to the standardization process. In case of the simpler services most of the issues found through these implementations have been resolved by the time of the writing of this book. But for the more complex services there are still remaining open issues.
There are also a few cases where the TWG deliberately left the issues open so that the implementations have the freedom to resolve them in a way most suitable for the particular implementation; for example, the system bootstrapping was left implementation specific. These are usually cases that do not impact applications using the services, but for which service implementers would like to have an answer (but typically not the one the specification would offer).
Part Three of the book looks at the SA Forum middleware in action, that is, at the different aspects of the practical use of the specifications presented in Part Two.
It starts with the overview of the programming model used throughout the definition of the different service APIs. There is a system in the API definitions of the different specifications and Chapters 11 and 12 serve as Ariadne's thread in what seem to be a labyrinth. This is followed by a bird's-eye view at the two most important open source implementations of the SA Forum specifications: OpenSAF and OpenHPI.
To help integrators and application developers to use these middleware implementations in Chapter 14 we discuss different levels of integration of the VideoLAN Client (VLC) application originally not developed for high availability. This exercise demonstrates in practice how an application can take advantage of the SA Forum Availability Management Framework even without using any of its APIs. Of course better integration and better user experience can be achieved using the APIs and additional services, which is also demonstrated.
After this ‘hands on’ exercise the problem of migrating large scale legacy applications is discussed. This chapter gives an excellent insight not only for those considering such migration, but also to designers and developers of new applications. It demonstrates the flexibility of the SA Forum specifications which people usually realize only after developing an intimate relationship with them. The mapping of the abstractions defined by the specifications is not written in stone and it is moldable to meet the needs of the situation. This is demonstrated on the example of two different database integrations with the SA Forum middleware depending on the functionality inherent in the database.
The final chapter of Part Three takes yet again a different perspective. It discusses the issues complementary to the specifications but necessary for the operation of the SA Forum middleware. It introduces the use of formal models and techniques to generate system configurations and upgrade campaigns necessary for the Availability and the Software Management Frameworks to perform their tasks. This approach was part of the vision of the SA Forum specifications as they defined the concepts enabling such technology opening the playground for tool vendors.
We could have continued exploring the subject with many exciting applications, but we had to put an end as we reached our page limit as well as the deadline for delivering the manuscript. So we leave the rest of the journey to the reader who we hope will be well equipped after reading our book to start out with their own experimentations.