The basic tenets of prevention are deceptively simple:

Both tenets provide a solution to only one aspect of the technology part of the equation—the server itself. What about the network connecting to the server? What about an application’s ability to handle the switch to a secondary server during an outage? These are just two examples of possible problem areas; there will be more for your environment that you need to prepare for. The bottom line, though, is that planning, implementation, and administration of systems are equally important aspects of preventing an availability problem.

The more redundancy you have, the better off you are. However, there is a conundrum. No system is completely infallible in and of itself, but adding too many levels of redundancy might make an availability solution so complex that it becomes unmanageable.

The designers of the Titanic thought she was unsinkable, yet the ship went to the bottom of the ocean on her maiden voyage. For protection, the Titanic had 16 watertight compartments that could be sealed with 12 watertight doors. In theory, this would have kept the ship afloat despite flooding in three or four compartments. Unfortunately, the watertight compartments themselves only went to a certain level above the water line. When the iceberg struck the Titanic, the damage it caused was, in reality, only six noncontiguous small gashes totaling about 12 square feet. But they made it possible for water to overflow from one compartment into another (think of the way an ice cube tray fills with water) and eventually caused the ship to sink.

What turned out to be a fatal flaw was addressed to the best of the designers’ ability at the time, but the unknown always lurks in the shadows. You must do your best when you are seeking to prevent a catastrophe.

Technology is only one part of the equation. Environmental and human aspects also contributed to the Titanic’s fate. She was caught in a set of circumstances—an ice field that affected most ships in the North Atlantic at that exact time, the lookout saw the iceberg too late, no other ships were close enough to assist after the Titanic struck the iceberg, and so on—for which no amount of technology could compensate.

Similarly, you might have two exact copies of your production database, but what happens if they are located in two data centers in the same state and are affected by an earthquake simultaneously? Do not overthink your availability solutions; there is only so much you can plan for.

When the iceberg struck, the Titanic was unprepared. There had been no safety drills to test an emergency plan (if one existed), not enough lifeboats for all passengers, and so on. Survivor accounts of the chaos during the last few hours reinforce the importance of proper planning in ensuring that more good than harm occurs during a crisis. Consider that each database, or even the entire system, could need to be rescued but not have to be salvaged from the bottom of the ocean. Rash decisions for the solution could have dire consequences. The only way to execute a rescue or salvage operation is to have the right people in place with a complete, well-tested plan to direct them.

When the crisis has passed, a postmortem should be done to determine the lessons learned to prevent such an outage from occurring in the future. Chapter 12, includes an in-depth discussion on disaster recovery and the plans needed.

Keep in mind that when it comes to disaster recovery, there can be another extreme. You can be proactive to the point of being negative. The person who is constantly worried about everything might be considered obsessed. There is a happy balance somewhere. If the crew of the Titanic made each passenger walk around with life preservers, a ration kit, and some sunblock, would that have been practical? No. Being prepared does not equate to paranoia. Assume that there is always something that could happen to your systems, anticipate and plan to the best of your ability, but go about your day knowing that you cannot control every factor.

If something happens—and something will happen at some point during your career—ensure that you will not go down with the ship. Whether you are the captain or just one of the passengers or crew, a disaster situation might make or break your career with your employer. Not being able to answer a question such as the one posed in the beginning of the chapter—"When will it be up?"—because, for example, the plan was never tested or the backups themselves were never tested and timed, is generally considered unacceptable. Keep in mind that fallout might still occur when the dust settles, but as long as you were prepared and handled the situation properly, all should be fine.

Assemble a project team that will own the overall responsibility for the availability of the solution. The leader should be the business sponsor—a person from the management team who will ultimately answer for the success or failure of the operation and have the greatest influence on its budget. Representatives from all parts of the organization—from management to end users—should be part of the team, as each of them will be affected in one way or another by the availability of the systems or solution that will be put in place.

Once the project team is assembled, it should meet to decide the principles that will govern how the solution will be designed and supported. The following are some sample questions to ask; there might be more for your environment.

Some of the questions you ask might not have answers until other pieces of the puzzle are put in place, such as the specifications for each application, because they will drive how SQL Server and other parts of the solution (for example, Microsoft Windows 2000 Server) are used and deployed. Others might be answered right away. Ensure that both the business drivers and the more detailed "what if" scenarios are well documented, as they are crucial in every other aspect of planning for high availability. It might even be a good idea when documenting to divide the questions into separate lists: one that pertains to the requirements of the business independent of technology, and one that is technology-dependent.

Having listed questions, it is a safe assumption that each person in the room will have a different answer for each question. The goal is to have everyone on the same page from the start of the project; otherwise the proverbial iceberg might start ripping holes in the hull of your solution. Compromise will always be involved. Compromise can only be achieved if there is a business sponsor who is driving and ultimately owning the solution at all levels. It is the responsibility of this person to gather consensus and make decisions that result in the compromise that everyone can live with. As long as all parties agree on the compromise, the planning, implementation, and support of the solution will be much smoother than if the voices of those two steps down the road are not heard or are ignored.

High availability is not synonymous with other vital aspects of any production system, such as performance, security, feature sets, and graphical user interfaces (GUIs). Achieving high availability is ultimately some form of a trade-off of availability versus performance versus usability. All aspects also need to be considered when doing overall system, infrastructure, and application design. Designing a highly available system that is not usable will not satisfy anyone. This is where the trade-offs come into play.

Is buying a single 32-processor server to support a larger amount of concurrent users for a database that is used 24 hours a day the most important business factor, or is it more important to ensure that the server is going to be up 24 hours a day to support the continuous business? Chances are people will say both are equally as important, but in reality, a budget dictates that some sort of trade-off must occur. Having slightly lower performance to ensure availability might be a reasonable trade-off if the cost of downtime is greater than the ability to have 10 additional users on the system. The same could be said for security—if the system or solution is so secure that no one can use it, is it really available? Conversely, if a developer coded the database server’s security administrator account and password into the application to make things more convenient, this may compromise security, as well as the application’s ability to work with certain high-availability technologies.

Think of it another way: If money was no object and you had to purchase a car, would you buy a fast, sleek sports car or a sensible four-door sedan with airbags for all passengers? Many would choose style over substance—that is human nature. High availability is like buying the sedan; it might not be the best looking car on the block, but it is a solid, reliable investment. The airbags and sturdy roll bar, among other safety features, will make you as prepared as you can be for a possible accident.

For a clear example of a trade-off, briefly consider the Titanic again and that the White Star Line valued luxury over lifeboats. That was their trade-off, and at the time it made sense to the people funding the ship as well as the designers. That decision ultimately proved to be fatal. You need to determine what the acceptable trade-offs are for each situation so that the solution will meet the needs of everyone, especially those responsible for administering it and, most important, the end user or customer.

Once the basic principles governing the solution have been put into place, it is time to mitigate the known and unknown risks by asking the "what if" questions to the best of the group’s ability. You might know what the risk and its associated questions are, but not the solution to mitigate them. Even more risks will become apparent as the solution moves from conception to planning, on through to implementation, and as more and more technology and application decisions are undertaken. Whenever a risk is identified, even if there is no answer at the time, make sure it is documented. Continually check the documented list of risks to see that there has been a corresponding answer to the question recorded. By the time the solution hits production, all identified risks should have a response, even if it is that nothing can be done to mitigate the risk.

Although there are many more possibilities, here are some common questions to jump-start the risk management process:

Your guiding principles are now documented, along with some risks that might or might not have answers. At this point, the principles should be reviewed and debated to ensure that they are correct. If something is not right, now is the time to correct it, as these principles will live through the entire project life cycle. They can be modified and reassessed if necessary, but the initial principles should be the measuring stick against which the success or failure of a solution is measured. There should be a formal signing off by the entire team, because availability is the responsibility of everyone involved.

Calculating availability is simple:

For example, assume:

Plug that into this formula:

which results in .999658 availability of the individual system.

Keep in mind that the individual system rarely exists alone: it is part of a whole solution. You have now calculated the individual system’s availability, but what is the entire solution’s overall availability? That number is only as good as the weakest critical link, so if one of the essential servers in the solution only has .90497 uptime, that is the uptime of the entire solution. The number for the overall solution also has to factor in such things as network uptime, latency, and throughput. These critical distinctions bring the availability of a solution into focus. Having said that, qualify numbers when they are calculated and explained. This definition might be too simplistic for some environments—if the server that has .90497 uptime is not considered mission-critical and does not affect the end user’s experience or the overall working of the solution, it might not need to be counted as the number for the overall solution. The bottom line is that multiple components in a solution can fail at the same time or at different times, and everything counts toward the end availability number.

There are also the related concepts of mean time between failures (MTBF) and mean time to recovery (MTTR). MTBF is the average expected time between failures of a specific component. A good example is a disk drive—every disk has an MTBF that is published in its corresponding documentation. If a drive advertises 10,000 hours as its MTBF, you then need to think about not only how long that is, but also how the usage patterns of your application will shorten or lengthen the life (that is, a highly used drive might have a shorter life than one that is used sparingly). An easy way to think of MBTF is predicted availability, whereas the calculation detailed earlier provides actual availability.

MTTR is the average time it takes to recover from a failure. Some define the MTTR for only nonfatal failures, but it should apply to all situations. MTTR fits into the disaster recovery phase of high availability.

The calculation example yielded a result of 99.9 percent uptime. This equates to three nines of availability, but what exactly is a nine? A nine is the total number of consecutive nines in the percentage calculation for uptime, starting from the leftmost digit, and is usually measured up to five nines.

The following table shows the calculations from one to five nines.

Most environments desire the five nines level of availability. However, the gap between wanting and achieving that number is pretty large. How many companies can tolerate only 5.26 or less minutes of downtime—planned and unplanned—in a given calendar year? That number is fairly small. Good-quality, properly configured, reliable hardware should yield approximately two to three nines. Beyond that, achieving a higher level of nines comes from operational and design excellence. The cost of achieving more than three nines is high, not only in terms of money, but also effort. The cost might be exponential to go from three, to four, and ultimately, to five nines. Realistically, striving to achieve an overall goal of three or four nines for an individual system and its dependencies is very reasonable and something that, if sustained, can be considered an achievement.

Consider the following example: Microsoft SQL Server 2000 is installed in your environment. Assume that Microsoft releases a minimum of two service packs per year. Each service pack installation puts the SQL Server into single-user mode, which means it is unavailable for end user requests. Also assume that because of the speed of your hardware, the service pack takes 15 minutes to install. At the end of the install process, it might or might not require a reboot. For the sake of this example, assume it does. A reboot requires 7 minutes, for a total of 44 minutes of planned server downtime per year. This translates into 99.9916 percent uptime for the system, which is still four nines, but you can see how something as simple as installing a service pack can eliminate five nines of availability in short order. One possible way to mitigate this type of situation is to have one of the standby systems brought online to take requests, synchronize the primary server when the service pack is finished, and then switch back to the primary server. Each switch will result in a small outage, but it might wind up being less than the 22 minutes it takes for one service pack. That process also assumes that it has been tested and accurately timed on equivalent test hardware.

If you ask the CEO, CFO, CIO, or CTO, each one will most likely tell you that they require five nines and 24/7 uptime. Down in the trenches, where the day-to-day battles of system administration are fought, the reality of achieving either of those goals is challenging, to say the least.

Keep in mind that the level of nines and the requirement might change during the day. If the company is, say, an 8-to-5 or a 9-to-6, Monday-to-Friday type of shop, those are the crucial hours of availability. Although the other hours might require the system to be up, there might not be a 100 percent uptime guarantee for the outlying hours. When the solution is designed, the goal should be to achieve the highest level of nines required, not shooting for the moon (five nines)—that might be overkill. When the question is asked during the business drivers meeting, such situations need to be taken into account. Five nines of availability from Monday to Friday might only mean 9 hours a day, which translates into 2340 hours per year. This number is well below the 8760 hours required for 24/7 support, 365 days a year. All of a sudden, what once seemed like a large problem might now seem much easier to tackle.

Also be mindful that each person—from members of management right down to each end user—will have different goals, work patterns, and agendas. In good times, the demand for high availability might be someone’s priority, but if other factors and goals, such as high profitability, are currently driving the business, availability might not be number one on the list of priorities. This also relates to the nature of some industries or different organizations—financial and medical institutions by default require availability, whereas service or retail organizations might see availability only as an aspect of the business that might or might not be important. This could mean that in the negotiations, a variable rate of availability might be discussed, and might confuse people.

For example, if you need your systems to have five nines of availability from 8 A.M. to 8 P.M., that should be your target for availability. Stating different rates of availability for different hours can potentially lead to the design of an inferior system that will not be able to support the highest required availability number. In the end, it is important to understand all of the factors when putting together the business drivers and guiding principles that ultimately will govern the solution. Ignoring any one of the groups touching the systems is a mistake.

There are two approaches for assessing your environment for availability:

As noted earlier, Chapter 2 covers more specifics about the basics of high availability, but it is very important to keep in mind the differences in the two approaches and how they really are not the same. Keep this in mind as you go through each chapter in this book, as well as when you start to plan, design, and implement highly available solutions in your environment.

In assessing availability, it is important to keep in the forefront of everyone’s mind that applications directly impact the availability of systems. One could even argue that in spite of massive amounts of other software and hardware redundancy, in the end the application might prove to be the weak point in the availability chain if it is not designed to handle such things as a server name change if it should be required during disaster recovery. The application drives the requirements of how each production server, its related hardware, and third-party software—such as database platforms and operating systems—will be selected and assembled; the infrastructure should not dictate the application. Designing the perfect infrastructure does no good if the one component accessing it cannot properly utilize the new multimillion-dollar infrastructure. This is also the reason that retrofitting high availability into an already existing environment is much more challenging than starting from scratch. Sometimes, it can be akin to putting a square peg into a round hole. A bad application means low availability.

No ones likes the cost aspect of assessing availability and, to a large degree, it is the toughest aspect to handle. The question is not so much the cost of availability, but the cost of not having availability and, subsequently, the actual cost of downtime. Money is always a factor, as achieving availability is not cheap. Speaking realistically, achieving five nines of availability on a limited budget is just about impossible.

Consider the following example: A high volume e-commerce Web site generates, on average, $10,000 in sales per hour. For a day, that equates to $240,000, and weekly, $1,680,000. On the busiest day, the Web site encountered an unexpected availability problem around noon. When all was said and done, the Web site was down for six hours, which is the equivalent of $60,000 in sales. That was the surface cost of downtime.

Analyzing the downtime further, when the outage happened, no plan was in place, so an hour of that time was spent gathering the team together. This also meant that whomever was called in to work on the problem had to stop working on whatever they were doing. Because no plan was in place, there was no clear chain of command, resulting in many people attempting the same thing, further tying up system resources. This impacted other schedules and mission-critical timelines, a cost that must also be measured.

When the company went to call support, they realized that they never renewed their Microsoft Premier Support agreement because they did not really use support services in the past, so they felt it was not needed. Instead of getting the quick response engineers, they were left waiting in the queue with other pay-per-incident customers. Also hindering quick time to resolution, certain information was either not known about the systems or could not be gathered because of stringent processes. It took a committee a half hour to decide to allow the information to be gathered. The technical issue was eventually solved, but due to the delays caused by the lack of a support contract and the information problem, it took longer than it should have. These were tangible costs and direct problems.

Perhaps even more important than the tangibles are the intangibles: how many existing customers went to the Web site to buy something, could not access it, and did not come back because they knew they could get the item from another e-tailer that was up and running? Those users might be enticed by discount coupons to return—but perhaps not. An even bigger intangible was the loss of prospective customers who visited the site during the downtime, saw it was down, and never visited again. That might factor into the $10,000 hourly rate, but can you put a price on future business? Customer loyalty is directly associated with a brand name. A bad experience generates negative word of mouth. If word spreads that the site is unreliable, the more people that spread that message, the more potential there is of losing current and future business.

For this example, there are a few conclusions that should be highlighted as part of a postmortem to prevent the problem from happening again:

Chapter 12 walks you through putting together a complete disaster recovery plan. Because the downtime cost at least $60,000 in this case (again, this is the only concrete number), would a $10,000 support contract have paid for itself? Most likely it would have. If the technical problem was found to be an issue of configuration, was it completely thought out during the planning process, or was the system rushed into production? Correcting these kinds of issues can save money and time to resolution.

The preceding example shows that the lack of a proper disaster recovery plan, no support contract, and the possible missed configuration point are potential barriers to availability, or roadblocks to achieving the level of availability required. These barriers include, but are not limited to, the following:

During the planning phase, you must take into account barriers to availability and mitigate any risks associated with them.