Chapter 29. Case Study: Highly Available Platform

“You just call out my name,

And you know wherever I am

I’ll come running, oh yeah baby

To see you again.”

Carole King, “You’ve Got a Friend”

Many corporate software systems depend on a highly available platform. This chapter shows increasingly detailed acceptance tests for such a platform.

Context for Switching Servers

A highly available platform has at least two independent computers. If one goes down, the other available computers take over the load. If the servers are running close to capacity, not all the applications may be able to run. A predetermined priority mechanism determines which applications get to run. In addition to switching applications when a server goes down, in the case study, the system administrator is notified either via email or a text message.

The capacity of a server to run applications depends on the demands of the applications, such as memory, processor usage, and input-output operations. Instead of being overwhelmed by all the details at once, this study shows how to introduce the details gradually. This is another manifestation of the separation of concerns guideline.

Test for Switching Servers

The first test uses a simplified capacity that considers just the number of applications that can be run on each server. This test demonstrates that the servers switch applications properly when one of them goes down. There will be a lot of technical work to perform to make that happen. Starting with this simple acceptance test keeps the developer unit working while the customer unit examines more detailed capacity issues.

A second test ensures that the next part of the flow is proper. Now that an event has occurred, the administrator should be notified in the appropriate way.

You can create a similar test for three servers if one of them is going down. That test would show how to distribute the applications among the remaining two servers.

Test for Technical Rule

Now that application switching works, more complex rules can be applied to the capacity of each server. The selection of what applications to run is a separable concern. The results can be tested independently of testing the switching functionality.

For example, if a single server is running, the applications that can be run depend on selecting the highest priority applications that can run within the capacity. It’s possible that an application that has a higher priority cannot be run because there is insufficient capacity. Then a lower priority application that does not require as much capacity might be run.

In the following table, CPU usage is measured in millions of instructions per second (MIPS). Memory usage is calculated in megabytes (MB). Input-output usage is measured in total of reads and writes per second (RWS).

In this test, “Lost Episode Watching” requires 5MIPS. The two higher priority applications—“CEO’s Pet” and “MP3 Download”—use 11MIPS of the 15MIPS available. So “Lost Episode Watching” cannot be run, but a lower priority application “External Web” requires only 3MIPS, so it can be run.

You can create similar tests for two or more servers. The creation of these tests brings up issues of how to balance applications between two servers. Given this test, it’s clear that if there were a second server, at least “Lost Episode Watching” should be running, as long as the server’s capacity was greater than that application’s needs. Or perhaps “CEO’s Pet” will run on the second server, allowing “Lost Episode Watching” to run on the first one.

The tests for what applications should be run on what servers can become fairly complicated. There are usually more issues, such as applications that need specific devices that are only available on some of the servers, so they cannot be run on the other servers.

At some point, an additional test that combines the switching and the selection is created. From the acceptance point of view, this test need only demonstrate that the switching takes into account selection based on the more complex selection rule. Lots of other combinations of applications and servers may be tested to ensure that the code and design do not have more esoteric defects. There would also be acceptance tests for the switching time performance.

Summary

• Separate tests so that each checks a different part of the flow.

• Separation of concerns makes for simpler testing.