Developing a Disaster Recovery Plan

Although the BCP folks develop a plan to keep business operations rolling, the DRP people develop a plan to restore the damaged facility (or facilities) so that the critical business functions can operate in their original location(s).

Preparing for emergency response

Emergency response teams must be prepared for every possible scenario. Members of these teams need a variety of specialized training to deal with such things as water and smoke damage, structural damage, flooding, and hazardous materials.

You must document all the types of responses so that the response teams know what to do. The emergency response documentation consists of two major parts: how to respond to each type of incident, and the most up-to-date facts about the facilities and equipment that the organization uses.

In other words, you want your teams to know how to deal with water damage, smoke damage, structural damage, hazardous materials, and many other things. Your teams also need to know everything about every company facility: Where to find utility entrances, electrical equipment, HVAC equipment, fire control, elevators, communications, data closets, and so on; which vendors maintain and service them; and so on. And you need experts who know about the materials and construction of the buildings themselves. Those experts might be your own employees, outside consultants, or a little of both.

It is the DRP team’s responsibility to identify the experts needed for all phases of emergency response.

Responding to an emergency branches into two activities: salvage and recovery. Tangential to this is preparing financially for the costs associated with salvage and recovery.

Salvage

The salvage team is concerned with restoring full functionality to the damaged facility. This restoration includes several activities:

Damage assessment: Arrange a thorough examination of the facility to identify the full extent and nature of the damage. Frequently, outside experts, such as structural engineers, perform this inspection.

Salvage assets: Remove assets, such as computer equipment, records, furniture, inventory, and so on, from the facility.

Cleaning: Thoroughly clean the facility to eliminate smoke damage, water damage, debris, and more. Outside companies that specialize in these services frequently perform this job.

Restoring the facility to operational readiness: Complete repairs, and restock and reequip the facility to return it to pre-disaster readiness. At this point, the facility is ready for business functions to resume there.

The salvage team is primarily concerned with the restoration of a facility and its return to operational readiness.

Recovery

Recovery comprises equipping the BCP team (yes, the BCP team — recovery involves both BCP and DRP) with any logistics, supplies, or coordination in order to get alternate functional sites up and running. This activity should be heavily scripted, with lots of procedures and checklists in order to ensure that every detail is handled.

Financial readiness

The salvage and recovery operations can cost a lot of money. The organization must prepare for potentially large expenses (at least several times the normal monthly operating cost) to restore operations to the original facility.

Financial readiness can take several forms, including

Insurance: An organization may purchase an insurance policy that pays for the replacement of damaged assets and perhaps even some of the other costs associated with conducting emergency operations.

Cash reserves: An organization may set aside cash to purchase assets for emergency use, as well as to use for emergency operations costs.

Line of credit: An organization may establish a line of credit, prior to a disaster, to be used to purchase assets or pay for emergency operations should a disaster occur.

Pre-purchased assets: An organization may choose to purchase assets to be used for disaster recovery purposes in advance, and store those assets at or near a location where they will be utilized in the event of a disaster.

Letters of agreement: An organization may wish to establish legal agreements that would be enacted in a disaster. These may range from use of emergency work locations (such as nearby hotels), use of fleet vehicles, appropriation of computers used by lower-priority systems, and so on.

Standby assets: An organization can use existing assets as items to be re-purposed in the event of a disaster. For example, a computer system that is used for software testing could be quickly re-used for production operations if a disaster strikes.

Notifying personnel

The Disaster Recovery Plan team needs to have communication plans prepared in advance of any disaster. Employees need to be notified about closed facilities and any special work instructions (such as an alternate location to report for work). The planning team needs to realize that one or more of the usual means of communications may have also been adversely affected by the same event that damaged business facilities. For example, if a building has been damaged, the voice-mail system that people would try to call into so that they could check messages and get workplace status might not be working.

Organizations need to anticipate the effects of a disaster when considering emergency communications. For instance, you need to establish in advance two or more ways to locate each important staff member. These ways may include landlines, cell phones, spouses’ cell phones, and alternate contact numbers (such as neighbors or relatives).

Mobile text messaging (also known as SMS or Short Messaging Service) is often a reliable means of communication even when cellphone communications systems are congested.

Many organizations’ emergency operations plans include the use of audio conference bridges so that personnel can discuss operational issues hour by hour throughout the event. Instead of relying on a single provider (which you might not be able to reach because of communications problems or because it’s affected by the same disaster), organizations should have a second (and maybe even a third) audio conference provider established. Emergency communications documentation needs to include dial-in information for both (or all three) conference systems.

Facilitating external communications

The corporate departments that communicate with customers, investors, government, and the media are equipped with pretty much the same information as for Business Continuity Planning. There are really no differences in logistical planning for external communications between DRP and BCP. See the section “External communications,” earlier in this chapter for a summary of communications with external entities.

Maintaining physical and logical security

Looting and vandalism sometimes occur after significant disastrous events. The organization must be prepared to deploy additional guards, as well as erect temporary fencing and other physical barriers in order to protect its physical assets until damaged facilities are secured and law and order are restored. And we’re not just concerned with physical assets: personnel (if any are present) require protection too.

When developing DR plans, keep in mind the need to protect information from unauthorized access as well as accidental or deliberate damage. The security controls used in main production systems need to be implemented on recovery systems as well. These controls will probably include

Access controls

Authorization

Audit logging

Intrusion detection

Firewalls

Encryption (including data in motion, as well as data at rest)

Backup

Physical access controls

Environmental controls

Personnel controls (background checks, security training, and so on)

We discuss these controls throughout this entire book.

Information that resides on disaster recovery systems is the same data that resides on normal production systems, so you must protect it by using the same or similar controls.

Personnel safety

The safety of personnel needs to be addressed, as there are often personnel working in areas with damage and safety issues, usually right after a disaster, during salvage and damage assessment.

An organization’s number-one priority is the safety of its personnel.