Assessing Risk in the Enterprise

Enterprise Risk Management (ERM) is not a template that can be given to every company to meet their needs and fit their business structure. Proper risk assessment identifies the risks throughout the organization and specifies the external and internal sources that the organization may face. The organization engages members from each organizational unit (Executives, HR, Finance, etc.), and asks questions such as “What do you perceive to be the largest risks to the company in terms of significance and likelihood?” and “What do you perceive to be the biggest risks within your control?” After a common understanding is met and all are aware of the risks, such risk assessments should be linked to strategic objectives as shown in Figure 5.1.

Once the company has an understanding of the top risks that can impact the organization, the executive team determines the company’s risk appetite and risk tolerance. Risk appetite is the amount of risk, on a comprehensive level, that an entity is willing to accept in pursuit of value. Risk tolerance, on the other hand, is the range of acceptable variation around the company’s objectives. The key is to determine the degree of maturity that meets the needs of your organization.

Steps in the Risk Process

There are five steps to consider for a company to come out ahead when a disaster hits to avoid risk and protect your data. The following checklist (see checklist: An Agenda for Action for Risk Assessment) is a list of these steps, from assessment to planning, architecting, specifying and implementing a full-bodied disaster recovery (DR) solution.

Downtime presents serious consequences for businesses, no matter what their function may be. It is difficult, if not impossible, to recoup lost revenue and rebuild a corporate reputation that is damaged by an outage. While professionals cannot expect to avoid every downtime event, the majority of system downtime is caused by preventable failures. Distinguishing between planned and unplanned system downtime allocates different procedures as both present vastly diverse paths when bringing systems back up.

While both planned and unplanned downtime can be stressful, planned downtime must be finished on time. Unplanned is the worst. Unplanned can be good for teaching troubleshooting techniques to junior IT staff, but can be very frustrating to the workforce. The authors see fewer and fewer techs with troubleshooting experience, and more and more senior techs launching their own consultancies. The biggest cost is how it affects the customers and the impressions of the company that a severe outage can make on the organization’s customers and clients. Planning, having people on staff with good troubleshooting skills, and documenting how the issue was found and fixed will help resolve the issue faster next time.

Matching the Response to the Threat

Separate each of your significant processes into one of three categories: Mission Critical, Business Critical or Organizationally Critical. By classifying processes into categories, you are able to define which parts of the organization would be recovered first in the event of an outage or disaster. How long can your organization or group live without access to a particular system? Should this system fail, how much data can the business realistically handle losing? Define, succinctly, what the organization considers a disastrously disruptive event and set the maximum amount of time you can go without access to your system. Then set the acceptable amount of data loss from the most recent backup, or replication. Repeat this step for each system and you will soon realize which systems have the highest priorities and highest impact. Look back in history and identify types of outages the firm has experienced and how those outages were dealt with. If one is more relevant than another, plan for that incident first.

Identifying Business-Critical Activities

An organization must have a thorough understanding of the critical business processes and the tolerance of a business outage to define objectives to succeed in the event of an outage. A successful solution employs a vertical and horizontal, or top/down approach to understand, identify, and map critical business processes, functions, IT systems, resource dependencies, and delivery channels. The organization must analyze the cost of disruptions and place them into resilience tiers to assist in defining operational availability and disaster recovery requirements from a business perspective.

Additionally, Recovery Point Objectives (RPOs) and Recovery Time Objectives (RTOs) are perhaps the most important key metrics when architecting a disaster recovery solution. An RTO is the amount of time it takes to recover from a disaster event, and an RPO is the amount of data, measured in time, that your organization lost from that same event. The two business-driven metrics will set the stage for:

Keep in mind that there are several intricacies to consider when assessing RTOs and RPOs. First, the objective in both stands for “objective” and should be defined as the target. If an RPO is five hours, then the architecture must ensure data loss of five hours or less. Therefore, when testing or recovering from a disaster, document and track actual thresholds achieved, including recovery point and recovery time. In many test cases, the time to recover does not meet the objective due to overhead time. Examples of overhead time are as follows:

When tracking and documenting actual versus objective, especially during testing, you will understand what is being accomplished in a given period of time. Figure 5.2 illustrates a flowchart of conflict resolution in the BIA and shows how time can be calculated when following the flow of the dependencies. Ultimately, this will allow a firm to defend future investment by honing your recovery methodologies and processes to better meet or exceed those objectives. Once the recovered data is made available and back to the application, the end users and owners of the applications only understand the RPO and RTO specific to usability of the application with an understood and acceptable amount of data loss in a specified amount of time.

Specifying Required IT Support from Technical Staff

There are challenges associated with managing a high-density infrastructure coupled with the technology each department integrates into that infrastructure. To meet the continually increasing demand for faster, better, and more powerful technology, IT directors are deploying high-density equipment with great processing powers encompassing a small footprint in the data center.

Incongruent IT teams include storage, server, network, and application teams; all understand their specific roles in recovering from a disaster, which is why tests are so crucial. However, once the infrastructure is recovered with the associated application data, many more tasks are required to make the application usable and available to the end users. Consider, for example, what the DBAs need to do to the databases and what the application and software teams need to do in order to validate functionality. Thus, having these objects defined and metrics in place will help with the testing and checkpoints in the recovery process to ensure that the RTOs and RPOs are met successfully.

Designing Recovery Solutions

Many kinds of “disasters” can occur in business. Typically, one thinks mostly of natural disasters when one thinks of disasters. To do this when planning a disaster recovery solution is a fatal mistake. For example, consider a small company (or possibly even a large company!) where one or a small number of people control access to all data. Perhaps only one person has access to critical systems that, one day, may require reconfiguration or repair. What if that day comes sooner than expected, and what if that person was involved in a fatal car accident on the way to work? This sort of scenario and myriad others plague the information technology industry. The amount of risk tied up in IT director fiefdoms could, if exercised by a wide-scale disaster one day, bring about a nationwide business calamity. Think about the assorted systems that could be affected by the following “disasters:”

The final point, while not directly related to information technology and the preservation of business continuity through tragic, business-altering events is included because, at the heart of all the systems exist the human beings who operate and understand them. Recovery solutions should consider the human element. More importantly, a disaster recovery should be able to do just what the name implies—it should allow complete recovery, if not business continuity, through any disaster.

Consider, for example, a software company called Knowledge Inc. that provides critical software services to many of the Fortune 500 companies. On a Friday afternoon, a terrorist attack destroys their one, and only, location. The 400+people who ran the company, except for those who were out sick or were on the road, are no more. Granted, the owner of the company has nothing to worry about anymore; he was in his office, but those 300 Fortune 500 companies, what about them? From their perspective, they are likely asking themselves why they never asked to see the DR plan for Knowledge Inc. They should have asked about the business recovery plan. Ostensibly, then, a DR plan contains two parts:

No business should ever close down due to a disaster. Yes, closing a business’s books and winding down after a disaster could be the outcome of a disaster, but that should be a business decision, not a forced event. That is, the disaster itself, followed by a poorly executed or nonexistent DR plan, should not be a business-ending event. Closing the business or selling its assets may be the decision made by survivors and beneficiaries, but by preserving the business through proper DR planning, this should be just one of many options, not the only option after a disaster. As intimated earlier, disasters come in many forms, which can be loosely grouped into three categories:

Force Majeure, or catastrophe, is obvious. Events such as hurricanes, earthquakes, fire, flood, war, volcanic eruptions, and terrorist acts all fall into this category.

Conditional is less obvious and revolves around the circumstances of an unexpected change in infrastructure conditions. For example, on a Monday afternoon, the Internet “goes down.” Service doesn’t’ resume until Wednesday evening because it took that long for the service provider to find the problem. In another example, a construction crew saws through all three trunks that service the city of Chicago. As a result, 99% of the city loses its Internet. The other 1%? They had a recovery plan that included routing phones and Internet through a satellite dish on their roof.

The human category means loss of life and the business impact. For example, what if a strange new virus decimated more than half the organization’s staff, and for some strange reason, it wiped out all but the most itinerant staff members, who have little knowledge of operations. A solid DR plan will consider that a disaster can occur, and all infrastructure may remain intact.

Establishing a Disaster Recovery Site

Once a disaster recovery plan has been created that includes both fail and no-fail infrastructure circumstances, a plan must be made that allows for varying degrees of infrastructure failure. For example, consider the conditions that must exist before someone says “Break out the black book!” and the organization shifts its mind-set into one of disaster recovery. For example, consider the following course of events:

In the event of a disaster, DR sites should be as logically separated from a catastrophe point of view as possible. To establish what makes a good DR site, let’s explore disaster a little bit.

Site Choices: Configuration and Acquisition

Since this is a book about computer and information security, and not about business alternative planning, these next sections will focus primarily on force majeure and conditional disasters. The assumption it makes is that the business critical functions have been deemed complex and necessary enough that a geographically disparate location is desirable and necessary. For many, disaster recovery is about backup planning. A distinction between the two must be made. Backup and recovery addresses one thing, and only one thing: For example, a hacker accesses the system and deletes a small but critical table from a database. This is a backup and recovery option. You simply restore from backup, and you are up and running again. Disaster recovery is more severe, and it implies that the recovery of data after an incident that destroys equipment or data in tandem with an event of some sort has rendered equipment and communications at a particular site unusable. In this case the definition of unusable is one of a business nature:

Over time, some piece of equipment may become unserviceable, but in terms of disaster recovery, something is unusable only if its unserviceability inflicts damage on the business. Disaster recovery assumes one thing is true: that a major, business critical function has ceased operation due to one of the aforementioned reasons, and that it can no longer continue.

The following list details a number of disasters and chooses alternative locations. In the end, one may consider that the disaster is too unlikely to occur, and that if the disaster were to occur (such as an asteroid strike), there would likely be no point in continuing business anyway—simply surviving will be everyone’s concern, and whether or not customers can purchase concert tickets will be moot if nobody is going to be going to concerts any time soon. Table 5.1 provides a sample listing of DR failover locations, the disaster that occurs, and the reason why it is a bad choice. This table seeks to provide a thought experiment framework whereby a planner may base a similar table for her location.

Choosing Suppliers: In-House Versus Third Party

As evidenced by Table 5.1, the complexity of choosing a DR location that is a perfect ying to your primary locations yang is not an easy task. If one assumes that the third-party provider has been in business for a while, has experience in the field, and is not just an investor who purchased an underground quarry in Kansas City and simply didn’t know what to do with it, then the primary reason to contract a DR provider will be one of reliability.

Furthermore, the DR provider will host other companies, which means that they will likely be performing the sort of monthly and manual testing of their failover electrical and uplink capabilities that is required. This results in a savings due to the economy of scale. A company that has to test its systems understands that testing takes time, is expensive, and may result in unanticipated damage to equipment should a faulty failover mechanism result in a massive surge through the power grid.

Typically, businesses that aren’t savvy or interested in the facts of disasters— that they do happen, and they can happen to them—may make a rather cursory attempt at DR. They may blend some non-DR site functions with the DR site. The mistake they make in doing this relates directly to redundancy—inherent to the DR strategy is that, essentially, both sites require each other to act as backup. If the DR site is being used, and it fails, it would need to failover to the primary site. Organizations that are greatly concerned about business continuity will segregate business functions completely. When contracting a vendor to handle DR, the business will drive the requirements. Things like email, database applications, HR, and finance systems may not have the same impact of loss as, for example, a Web site that takes customer orders.

When building a DR plan, the planners must consider the scope of the disaster as well, how the vendor charges, and whether or not multiple vendors are required to fully comply with business needs. For example, for some companies, it is perfectly acceptable that employees could work for home in case the DR triggering event also closed the physical office. Other companies may have differing compliance issues and may require a secure facility. They may even require that employees travel to a distant location and work there. Now, arrangements have to be made for both PC access to network systems and food and lodging. This author recommends a nice resort that has conference center capabilities that could be set up to provide workstations. Do bear in mind that some employees will require family housing. Choosing to provision DR through in-house versus choosing an external vendor requires comparing and determining a number of factors:

Figure 5.3 diagrams a failover site. Data flows are unidirectional to the DR site. It is a common fallacy that the DR site can also provide some functionality to the enterprise and serve as more than just a graveyard for servers, waiting for the day they must spring to life and perform a critical duty.

Specifying Equipment

The challenge of a DR site is primarily cost, followed closely by configuration. Setting it up, planning redundancy, creating strategies for rerouting voice, messaging, email, delivery of goods, and so on are things that can be written and kept within arm’s reach of all employees. ALL employees should be aware of the DR plan. No single employee should be left wondering what his role is during a disaster.

Configuration and cost are closely related. In just one example (of many similar), consider a complex federated database where many applications and many databases commingle information and exist in harmony on a home-grown system across 30 or 40 SQL servers. Picking up something so complex, and moving it, is not a simple task. It may take many weeks of reconfiguration to get such as system to be functional at the DR site. The complexity of an application and the ease of setting it up in a DR site should always be considered during the purchasing and application review phase. Waiting until afterwards to think about the DR site can be a very costly mistake. A DR-ready application will be able to collapse down to just a couple of servers and will be able to provide core functionality, with additional functionality brought online with the addition of more servers as needed.

A DR site is a site that is to be used for a temporary period of time. Typically, the hope is that the DR failover will not be permanent. However, this should be a planned contingency—that it will be permanent and that you may need to rapidly scale the environment. This is not to say that a DR site should be a fully functional, duplicate site of the primary site that failed. Rather, it should be able to support essential business functions. Suppose, for example, an enterprise that hosts a document review platform. Many people would like to access the system, but in the contract with its clients, the business is only committed to support minimal, required activities, and they are listed. Contractually, activity should be restricted. Only known, business-critical activity should be conducted. Users should only be performing activities that are business critical for them or their customers.

From an equipment purchasing standpoint, then, only minimal hardware need be purchased. Some companies may even refer to their DR site as “the graveyard” and the DR plan as a “Dawn of the Dead” plan. However, from a strategic standpoint, rack space MUST be available for rapid growth. Should the disaster become lengthy, or should the outage be permanent, the enterprise must be able to scale rapidly. Many business computing sales companies will configure a “standing order.” This is something that, for a price, can be held in a sort of “escrow” until needed. When an outage becomes extensive, or permanent, or is recognized as being permanent the moment it happened, then pulling the trigger on such an order could actually be automatic and scripted in the failover plan. After all, in the event of a Dawn of the Dead, the graveyard will need to be fed more brains.

True/False

Multiple Choice

Problem

What are the differences among a Continuity of Operations Plan (COOP), a Business Continuity Plan (BCP), a Critical Infrastructure Protection (CIP) Plan, a Disaster Recovery Plan (DRP), an Information System Contingency Plan (ISCP), a Cyber Incident Response Plan, and an Occupant Emergency Plan (OEP)?

Hands-On Projects

Case Projects

Optional Team Case Project