Chapter 11

SA Forum Programming Model and API Conventions

Francis Tam

Nokia Research Center, Helsinki, Finland

11.1 Introduction

In Part Two of the book, we have discussed the founding principles, motivation behind the design, usage scenarios, and limitations of the Service Availability (SA) Forum services and frameworks. In this chapter, we turn our attention to how we can use these services and frameworks in practice.

For developers, one of the first places to look for implementation related information is the service's Application Programming Interface (API) definition sections of the specification. In the SA Forum specifications, all the service and administrative APIs are defined using the C programming language [92]. The question typically asked by people has been: why C? It was a decision based on the member companies at the time when the specifications were developed. This represented a common ground for the participating companies, one of the key characteristics of open standards development. This decision was chiefly backed by the evaluation of the language's suitability from a technical perspective. C is a well-known and popular candidate system implementation language for operating systems and embedded system applications, primarily due to its efficiency. It has been recognized for code portability across many different platforms. Its flexibility such as ease of access to hardware and relatively low demand on system resources at runtime add to the list of advantages for implementing the SA Forum middleware. The fact that the C++ programming language is source and link compatible with C offers another edge that the services can also be used by an object oriented language.

As we have seen, the whole SA Forum specification has been divided into a number of services. This was due to the complexity of the entire system to achieve modularity, as well as a way to develop the ecosystem such that different participating member companies could take part in just a subset of the specifications and their implementation, instead of the all-or-nothing or monolithic approach. This called for a need to have a consistent naming conventions and coherent behavioral patterns in similar functions across all the services. As a consequence, the usage model established a generic interaction between a service user process and a library for a service area. It is important to note that this division is a logical one. The implementation of service areas could be physically combined into a single library.

In the SA Forum specifications, the definition of a process is based on that of Portable Operating System Interface for Unix (POSIX) [73]. However, this does not mandate the middleware be implemented in a POSIX system, although at the time when the specifications were developed, POSIX was the choice of implementation platform amongst most member companies. This explains why many interfaces have this influence behind their designs.

We begin this chapter by describing the programming model, which is applicable across all the Application Interface Specification (AIS) services. This includes the interface relationships between service users and implementers, naming conventions and type definitions, real-time support, the usage model, and the tracking capability. As the API specifications are typically the first place where developers would look for programming related information, we outline what one can expect from the service and administrative API sections of the specifications. Finally we discuss a number of topics and issues that are frequently considered by developers in practice. These include the interaction with POSIX, memory management, the handling of pointers, finding out the implementation limits, the availability of area servers, and the concern of backward compatibility.

11.2 Programming Model

In this section, we explain the general C programming model of the SA Forum specifications [63], which is applicable to all the AIS services. This includes the AIS area service interfaces, naming conventions, predefined types and constants, real-time support, library life-cycle, and the concept of tracking.

11.2.1 AIS Area Service Interfaces

The SA Forum specifications have been divided into a number of services according to their functions. As we have seen in Part Two, each service provides the corresponding functions through standardized interfaces. Each of these services can therefore be regarded as an area of its own. In the rest of this chapter, we use the term area service to refer generically a SA Forum AIS service, without specifically naming the service.

Besides the consistent naming conventions and type definitions, interactions between a service user process and the area service must also be consistent. Figure 11.1 shows the interface relationships among a user process, the area service interface implementation library, and an area server. It must be noted that all area service packages have the same logical structure.

Figure 11.1 Interface relationships.

The SA Forum Application Interface for an area has been defined to be between a process and an area library that implements the area service interface. On behalf of the user process, an area library invokes the appropriate functions at an area server that carries out the actual service functionality as described in the corresponding AIS service specification. It is important to point out that this interaction is a logical one. This is due to the freedom an implementer has to construct a separate physical module for each area server or combine a few into a single physical one; and the choices of centralized or distributed implementation of an area server. It must also be noted that the method of communication between an area library and area server has not been specified, therefore, it can either be local or remote, depending on how an area server has been implemented.

11.2.2 Real-Time Support

The SA Forum interface has been designed to accommodate both unthreaded and threaded models. In a single-threaded program, if the main execution thread blocks on a long-running task, the entire application can appear to be frozen, thus giving the impression that the service is not being provided and the application does not exhibit the characteristic of guaranteeing a response in real-time. A multi-threaded programming model allows for multiple threads of execution within a process. The threads share the same address space, state information, and other resources within a process. Context switching between threads within a process is typically faster than that of processes. By moving long-running tasks to a thread that runs concurrently with the main execution thread, it is possible for the application to remain responsive to user input while executing tasks in the background.

Multi-threading is not the only way to keep program responsive, nonblocking interactions can also be used to achieve the same result. This is one of the reasons why the SA Forum interface supports both synchronous and asynchronous programming models. A synchronous programming model is easy to use by developers. An application invokes an interface in the implementation library and blocks until the call returns. In an asynchronous programming model, an application invokes an interface in the implementation library, but control returns to the caller immediately. The caller has to specify a callback function that will be invoked when the operation is completed. On the middleware side, there is also a need to use the callback mechanism to initiate interactions to an application. There are also operations that have an asynchronous nature, for example, a change in the cluster membership.

Since the C programming language only offers a single-threaded control flow, support is therefore needed by the underlying operating system to provide the asynchronous programming model. The notion of a selection object (see Section 11.2.4) has been introduced in the programming model such that the operating system dependency is hidden in the area service implementation library. This leaves the user application to be as portable as possible across other AIS implementation platforms.

11.2.3 Naming Conventions and Type Definitions

Since the SA Forum specifications have been broken down into a number of areas corresponding to the AIS services, an interface tag is therefore needed to identify an individual service. The assignment used to tag each area interface to its corresponding services is shown in Table 11.1.

Table 11.1 Interface area tags

For the AIS services, a function declaration is expressed as:

type sa<Area><Object><Action><Tag>(<arguments>);

where

sa – prefix for ‘service availability’

<Area> – interface area

<Object> – name or abbreviation of object, or service

<Action> – name or abbreviation of action

<Tag> – tag for the function such as Async or Callback

An example is:

SaAisErrorT saClmClusterNodeGetAsync(
      SaClmHandleT clmHandle,
      SaInvocationT invocation,
      SaClmNodeIdT nodeId
);

where

Other declarations are expressed as:

• Type

typedef <...> Sa<Area><TypeName>T;

• Macro

#define SA_<Area>_<MACRO NAME> <macro definition>

• Enumeration type

typedef enum {
     SA_<Area>_<ENUMERATION_NAME1> [= <value>],
     SA_<Area>_<ENUMERATION_NAME2> [= <value>],
     ....
     SA_<Area>_<ENUMERATION_NAMEn> [= <value>]
} <enumeration type name>;

Table 11.2 lists all the standard and predefined types and constants for the AIS services.

Table 11.2 Standard and predefined types and constants

11.2.4 Usage Model and Library Life Cycle

The logical interface relationships introduced in Section 11.2.1 has the premise that a user process must associate itself with an area service library for the duration when the area service functions can be used. This duration is expected to be between an area service library is initialized and finalized by the service user process. Figure 11.2 shows an example interaction sequence among a process, application interface implementation, and an area server. It is important to stress that this is just an example scenario and it should not be interpreted as the only way to implement an area service library and its interactions with the area server.

Figure 11.2 Example interaction sequence.

Actions 1 and 2 show the sequence of an area service library initialization. As part of the library association step, the callback functions that the user process has implemented and can be invoked by the library are passed as parameters. The returned handle is used to represent the association between the user process and the library in subsequent interactions. It is worth pointing out that a user process is allowed to call the library initialization multiple times. Each call would return a different handle for a different instance of the same library.

Actions 3 and 4 illustrate the sequence of obtaining a selection object for making the callback mechanism usable in C. A selection object is an operating system dependent entity that allows for a process to wait until there are callbacks pending. In the POSIX environment, the selection object is a file descriptor that is used with either a select() or poll() system call to detect incoming callbacks. It is important to point out that a process may be able to obtain different selection objects for the same library handle. This is used to support the multi-threaded dispatching of callbacks.

The motivation behind the introduction of a selection object concept is to hide this operating system dependency in the library implementation. For an AIS implementation on a platform other than POSIX, the equivalent function of a selection object to support the callback mechanism in C must be implemented in the library.

Assuming that the wait is over for the user process (Actions 5, 6, and 7), Action 8 signals to the library that the process is ready to process the pending callback. Here a user process has the options to choose how to handle the pending callbacks according to the supplied dispatch flags in the dispatch call. The flags have the following behaviors and provide for flexibility to the user process when combined with using the multi-threaded model:

• SA_DISPATCH_ONE — Invoke a single pending callback and then return.
• SA_DISPATCH_ALL — Invoke all the pending callbacks before returning.
• SA_DISPATCH_BLOCKING — One or more threads execute callbacks as they become pending. The threads remains in the dispatch and stay until a finalize call is made by one of the threads in the process.

According to the user-selected dispatch flags, a callback function is invoked by the library (Action 9) in order for the user process to execute the command (Action 10). The appropriate response generated by the user process as a result is then returned to the library (Action 11), which relays it back to the server of the area service (Action 12).

Finally, the user process enters the finalization stage when it does not require the use of the area service library by calling sa<Area>Finalize() in Action 13. This results in disassociating the user process from the library. It must be noted that multiple handles obtained by calling the sa<Area>Initialize() multiple times must be finalized separately.

11.2.5 Tracking

Tracking is a useful capability that enables a service user process to follow the changes that are of interest. An application requests a tracking operation to be performed on an entity, or a group of entities. A track callback is then invoked when the requested changes on the entity or entities have occurred, or about to occur. There are many scenarios in which AIS services require the support of tracking, for example, when the cluster membership changes in Cluster Membership service (CLM) (see Section 5.3.4), protection group changes in Availability Management Framework (AMF) (Section 6.3.2.4), and readiness status changes in the Platform Management service (PLM) (see Section 5.4.4).

The format of a tracking function is:

sa<Area><Xxx>Track[<Func>]()

where

• start tracking an entity or an entity group;
• stop a previously initiated tracking on an entity or entity group;
• callback notification;
• respond to a callback notification.

A service user process can also specify the different tracking behaviors according to the different tracking flags as shown in Table 11.3.

Table 11.3 Tracking flags

When the area service informs a service user process that the changes requested to be tracked have occurred, the information about the changes as specified by the tracking flags are returned in a callback function. The required memory for this information is allocated by the area service library. It is the responsibility of the called back process to invoke the corresponding sa<Area><Xxx>NotificationFree() function to free the memory allocated by the area service library.

In some use cases, a service user process needs be able to accept or reject a request to change the status of a tracked entity, or to be able to perform actions prior to the change takes effect. This allows for graceful reaction to certain administrative operations and is supported by the enhanced tracking.

The enhanced tracking can occur in three different ways. As shown in Figure 11.3, the first one begins at the top with the following steps:

Figure 11.3 Enhanced tracking steps.

The second way begins at the start step. This happens when a subscribed process does not subscribe for the validate step or if the pending change cannot be rejected. Therefore, such processes are directly notified at the start step without the validate step. They then continue to the completed step as described in the first way above.

The third one occurs at the completed step, which means that this is the same as the non-enhanced tracking as described in the beginning of this section, that is, processes are notified after the change has taken effect.

It is useful to note that the use of the three different paths of enhanced tracking can be generally thought of as:

Both PLM and CLM offer the enhanced tracking mechanism to track their corresponding entities. Interested readers are referred to Sections 5.3.4 and 5.4.4 for more details.

11.3 Making Sense of the API Specifications

What is in the API sections for each defined AIS service? In this section, we explain the API conventions for AIS services and administrative operations.

11.3.1 Structure of Service API Specification

In the specification of each of the AIS services the API section consists of the following subsections:

• Service model — the subsection describes the model of interaction between an application and the specific service. It also discusses the communication and interaction of the specified service with other AIS services. The conditions in which the area service API is deemed unavailable are described here.
• Include file and library — the subsection lists the names of the service-specific include file and library to be used with the service.
• Type definitions — the subsection gives all the definitions of types and constants specific to the service.
• Library life cycle — the subsection describes the library life-cycle APIs.
• Defined operations — the subsection contains one API description for each defined operation for the service. Each API description follows a consistent format across all the defined operations in the specific service, as well as across the other AIS services, as shown in Figure 11.4.

Figure 11.4 Sample service API.

At the top of each service-defined operation is the name of the API.

• Prototype

• Parameters

• Description

• Return values

• See also

11.3.2 Administration API

For some AIS services, there are administrative operations and possible states defined and exposed by the Information Model Management service (IMM) to the system management applications. The administration API for each such AIS service has the following subsections in the specification:

• Administration API model covering the administrative states and operations for the specific AIS service.
• Include file and library name — the subsection lists the names of the service-specific include file and library to be used with the administration API. Since for all AIS services IMM exposes the administration API, they are the include and library files for IMM.
• Type definitions — the subsection gives all the definitions of types required by the administration API.
• Defined operations — each of which consists of an administrative API definition following the format, as shown in Figure 11.5.

Figure 11.5 Sample administrative API.

At the top of each service-defined administrative operation is the enum value for the corresponding administrative operation as defined by the particular AIS service. It is passed as a parameter to the IMM OM-API (information model management service object management application programming interface) function defined to invoke an administrative operation. Since it is always the same IMM API — a variant of the saImmOmAdminOperationInvoke() — which is used to invoke an administrative operation the function prototype is not given. It has been discussed in Section 8.4.3.5.

11.4 Practical Topics

In this section we look at a number of topics that are related to the use of the SA Forum middleware in practice. These include short notes on interacting with the POSIX environment, memory management, pointers handling, finding out the limits of an implementation, conditions in which an area service is deemed unavailable, and the issue of backward compatibility.

11.4.1 Interacting with POSIX

For those applications that are developed in a POSIX environment, greater application portability from one AIS implementation to another can be achieved by following the SA Forum guidelines for developing applications related to the use of signals and threads.

In connection to the asynchronous programming model, signals in a POSIX environment are widely used by applications as a mechanism to be notified when an event is occurring in the system. Examples of such events include detection of hardware exceptions, timer expirations, terminal activity, and user-defined signals. Upon receiving a signal, an application responds by taking an appropriate action, which can be a default one as specified by the system, ignore the signal or invoke a user-specified function known as a signal handler to deal with the event.

Since the SA Forum specifications do not require that the AIS functions can be safely invoked from a signal handler, the following should be observed:

• Avoid using any SA Forum API from a signal handler.
• Do not assume that SA Forum APIs are interruptible by signals.

As stated earlier, SA Forum APIs have been designed to support both the single-threaded and multi-threaded models. Therefore, the implementation of area service libraries must be thread-safe for those applications that may have different threads of a process concurrently calling the area service APIs. Thread-safety is a property that attempts to eliminate the risk of interference among threads and unwanted modifications to another thread's data elements.

One of the many mechanisms of ensuring thread-safety when terminating a thread is to carry out a thread cancellation operation in a controlled manner. This involves the thread to be cancelled is allowed to hold cancellation requests pending until it reaches a thread cancellation point. At which point, the cancellation request is acted upon and the thread to be terminated is allowed to perform application-specific cleanup processing. In other words, thread cancellation may only occur at these cancellation points. Many POSIX functions have been specified as thread cancellation points. However, it must be pointed out that the SA Forum APIs cannot be assumed to be thread cancellation points.

There is a general problem of making fork() work in a multi-threaded process, which boils down to deciding what to do with all the threads in the newly created process. One approach is to copy all the threads to the new process. This however must prevent those threads that are suspended on system calls or just about to execute systems calls from executing in the new process. An alternative is to just copy the thread that calls fork() but this might create a situation in which a thread that holds a resource will not be present in the new process, resulting in the resource is never released. The guidance from POSIX [73] to this problem is to use an appropriate thread creation call when a new thread of control is needed, and only use fork() to create a new process to run a different program. Since we cannot assume that the AIS functions are fork-safe, if a process using AIS functions forks a child process in which AIS functions will also be called, the child process should therefore exec() a new program immediately.

11.4.2 Allocating and Freeing Memory

When the amount of memory required for a program cannot be determined beforehand, dynamic memory allocation at run time is needed. The allocated memory must be freed and returned to the memory pool for re-use, otherwise a memory leak may exhaust the available memory in a system. In the context of the SA Forum APIs, where there are interactions between a user process and an area service library, either party may be responsible for the dynamic memory allocation.

The SA Forum API specifications have a ground rule which states that memory dynamically allocated by one entity, either the user process or area service library, must be freed by the same entity. However, it is sometimes simpler or even better to have the area service library allocating the memory and the service user freeing the memory afterwards. For example, it is difficult to provide a buffer of the appropriate size by the invoking process when it is problematic to determine how much memory is required for an operation in advance. There are also situations where excessive copying can be avoided if this approach is followed, as a result, the performance is improved. In either case, each area service that provides a function to dynamically allocate memory for a user process also provides a function to be called by the user to free the memory. A user process must therefore call these memory free functions to avoid any memory leaks.

This type of usage is clearly documented in the SA Forum APIs. Table 11.4 lists the memory allocation calls and their corresponding memory free calls for those relevant area services.

Table 11.4 Per service memory allocation and free calls

11.4.3 Handling Pointers

When parameters of the type of pointers are passed between a service user process and an area service library, the lifetime of these pointers must be taken into account in the program for both synchronous and asynchronous APIs.

In synchronous APIs where pointers are passed as parameters by a service user process to the area service, the area service must not keep any pointer beyond the function has returned.

In asynchronous APIs where pointers are passed as parameters by a service user process to the area service, the area service must not keep any pointer after the area service has invoked the corresponding asynchronous callback function. On the user process side, it must not use that pointer after the callback function has returned. If the user process needs to keep the information passed by the pointer beyond its lifetime, a copy of the information must be made locally before the callback function returns.

11.4.4 Finding Out Implementation Limits

An area service implementation usually has limits, for example, the maximum number of supported entities of a certain type, the maximum size of certain objects, and so on. It is important for developers to find out what these limits are, so that they can be taken into account of the overall application design. Some AIS services offer a function to a user process to obtain these implementation limits, each of which is defined by an enumeration type Sa<Area>LimitIdT:

typedef enum {
      SA_<Area>_<NAME1>_ID = 1,
      ...
      SA_<Area>_<NAMEn>_ID = n
} Sa<Area>LimitIdT;

Table 11.5 lists the AIS services that have defined implementation limits.

Table 11.5 Per service implementation limits

A process can obtain at runtime the current value of a particular implementation limit by invoking the sa<Area>LimitGet() function:

SaAisErrorT sa<Area>LimitGet(
      Sa<Area>HandleT <Area>Handle,
      Sa<Area>LimitIdT limitId,
      SaLimitValueT *limitValue
);

The handle <Area>Handle is the handle which identifies this particular initialization of the <Area> Service, which was obtained by a previous invocation of the sa<Area>Initialize() function.

limitId identifies the limit and the <Area> service returns the implementation-specific value in the memory area to which limitValue points.

These limits typically refer to the maximum number or size of entities of a certain type that the implementation can support, however, other limits such as thresholds can also be used.

11.4.5 When an Area Service is Unavailable

Since we are dealing SA applications, it is therefore important for developers to understand when the underlying availability middleware may not be available. There are primarily two conditions under which an area service is not available to a service user process. The first one, which is signaled by the returned error code SA_AIS_ERR_UNAVAILABLE, indicates that the service user process is not allowed to use the operations offered by the area service. The second one, which is indicated by the returned error code SA_AIS_ERR_TRY_AGAIN, gives the information that the provider is currently experiencing some difficulties in delivering the service, although the service user process is allowed to use the operations in this area service. The reasons may be due to transport, communications, or other issues but they tend to be of a temporary nature.

In order to understand why a service user process is not allowed to use the operations in an area service, one has to go back to the premise of the SA Forum solution relating to a cluster. As defined in the CLM (see Section 5.4), a cluster is formed by a set of cluster nodes. Each cluster node is the logical representation of an execution environment that has been administratively configured into the cluster configuration. A member node is a cluster node that CLM has recognized to be healthy and well-connected to be used for deploying high availability applications and services. CLM is the authority that decides when a cluster node can be transitioned into a member node.

Another important distinction related to a cluster is whether a logical entity is cluster-wide or node-local. The former indicates that an entity spans one or more cluster nodes and its name is unique in the entire cluster, while the latter defines an entity on a cluster node and accessible by processes on this node only. A checkpoint in the Checkpoint service is an example of a cluster-wide entity whereas a timer of the Timer service is a node-local entity example.

Operations of an area service that are meant for cluster-wide entities are in general not allowed for processes running on cluster nodes that are not in the cluster membership. Accordingly the membership status reported by CLM determines whether the SA_AIS_ERR_UNAVAILABLE or the SA_AIS_ERR_TRY_AGAIN code is returned at the time the area service is unavailable. Exceptions are those operations that enable or detect the formation of the cluster membership.

For those operations that are intended for node-local entities, the cluster node does not need to be a member node at the time when the operation is performed.

Guidelines for service implementers include the use of CLM to determine the membership status of a node before returning the appropriate error code, and specify the behavior of the service API functions when they are placed under various conditions that cause the service to be unavailable within the scope of a node.

11.4.6 Backward Compatibility

Backward compatibility can generally be defined as the ability of a newer version of a system to interoperate with an older version of the same system. For the SA Forum specifications, this is an appropriate concern as the specifications evolve over time. In the context of the SA Forum specifications development, backward compatibility is achieved by following a set of rules when inevitable changes must be made in the newer releases.

A ground rule is that a function or type definition never changes for a specific SA Forum release. If changes are unavoidable in a function or type definition, a new function or type name is defined in the new version. This new name must be built from the original name in the previous version with a suffix that indicates the version where the change is made. This new name then replaces the old one in the new version, that is, there is only one and only entity providing the function or type definition. As an example, saAmfComponentErrorReport_4() was introduced in the AMF's B.04.01 release. The function replaces an earlier version of the component error report function saAmfComponentErrorReport() in previous releases. As a result, there is only one such function, the latest one in the new version of the specification. This newest version of the function interoperates with all the other existing functions that did not change in the library.

There is one exception though. The type name does not need to be changed if the size of the enum, flag, or union type does not change when new enum values, flag values, or union fields are added to an existing enum, flag, or union types.

It is important to note that the library can provide more than one version of a service implementation. In this case, different versions can be used by different service user processes according to the corresponding version request in the library initialization call. AIS implementers must also ensure that they respect the version numbers provided by the application when the library is initialized:

• New enum values must not be exposed to applications using older versions.
• New or modified error codes that only apply to newer functions are not exposed to applications using older versions.
• Only functions for the specific version are accessible by applications.

11.5 Concluding Remarks

In this chapter we have focused on the aspects related to using the SA Forum specifications from a developer's perspective. We have described the generic programming model for AIS services, in particular the interface relationships between service users and service implementers, naming conventions and type definitions, real-time support, its usage model and library life cycle, and the capability of tracking. We have outlined the type of information one can expect from the API sections for each service in the SA Forum specifications. A number of issues associated with using the SA Forum APIs in practice have been discussed, which included POSIX interactions, memory management, pointers handling, obtaining implementation limits, availability of area service libraries, and the concern of backward compatibility.

The C APIs are the definitive specifications of the programming interfaces in the SA Forum's services and frameworks. However, there is a need for other programming languages as well in practice. Java is one good example and the SA Forum has responded by defining a Java language mapping standard. In the next chapter we will be looking at the Java language mapping specification, its design rationale and usage example.