One of the most important concepts in all of information security is that of the principle of least privilege. The principle of least privilege dictates that persons have no more than the access that is strictly required for the performance of their duties. The principle of least privilege may also be referred to as the principle of minimum necessary access. Regardless of name, adherence to this principle is a fundamental tenet of security, and should serve as a starting point for administrative security controls.

While the principle of least privilege is necessary for sound operational security, in many cases it alone is not a sufficient administrative control. As an example, imagine that an employee has been away from the office for training, and has submitted an expense report indicating $1,000,000 was needed for reimbursement. This individual happens to be a person who, as part of her daily duties, had access to print reimbursement checks, and would therefore meet the principle of least privilege for printing her own reimbursement check. Should she be able to print herself a nice big $1,000,000 reimbursement check? While this access may be necessary for her job function, and thus meet the requirements for the principle of least privilege, additional controls are required.

Rotation of Duties, also known as job rotation or rotation of responsibilities, provides an organization with a means to help mitigate the risk associated with any one individual having too many privileges. Rotation of duties simply requires that one person does not perform critical functions or responsibilities without interruption. There are multiple issues that rotation of duties can help begin to address. One issue addressed by job rotation is the “hit by a bus” scenario: imagine, morbid as it is, that one individual in the organization is hit by a bus on their way to work. If the operational impact of the loss of an individual would be too great, then perhaps one way to assuage this impact would be to ensure that there is additional depth of coverage for this individual’s responsibilities.

An additional operational control that is closely related to rotation of duties is that of mandatory leave, also known as forced vacation. Though there are various justifications for requiring employees to be away from work, the primary security considerations are similar to that addressed by rotation of duties; reducing or detecting personnel single points of failure, and detection and deterrence of fraud. Discovering a lack of depth in personnel with critical skills can help organizations understand risks associated with employees unavailable for work due to unforeseen circumstances. Forcing all employees to take leave can identify areas where depth of coverage is lacking. Further, requiring employees to be away from work while it is still operating can also help discover fraudulent or suspicious behavior. As stated before, the sheer knowledge that mandatory leave is a possibility might deter some individuals from engaging in the fraudulent behavior in the first place, because of the increased likelihood of getting caught.

A non-disclosure agreement (NDA) is a work-related contractual agreement that ensures that, prior to being given access to sensitive information or data, an individual or organization appreciates their legal responsibility to maintain the confidentiality of that sensitive information. Job candidates, consultants or contractors often sign non-disclosure agreements before they are hired. Non-disclosure agreements are largely a directive control.

Background checks (also known as background investigations or pre-employment screening) are an additional administrative control commonly employed by many organizations. The majority of background investigations are performed as part of a pre-employment screening process. Some organizations perform cursory background investigations that include a criminal record check. Others perform more in-depth checks, such as verifying employment history, obtaining credit reports, and in some cases requiring the submission of a drug screening.

The preparation phase includes steps taken before an incident occurs. These include training, writing incident response policies and procedures, providing tools such as laptops with sniffing software, crossover cables, original OS media, removable drives, etc. Preparation should include anything that may be required to handle an incident, or which will make incident response faster and more effective. One preparation step is preparing an incident handling checklist. Figure 8.4 is an incident handling checklist from NIST Special Publication 800-61r2.

One of the most important steps in the incident response process is the detection phase. Detection (also called identification) is the phase in which events are analyzed in order to determine whether these events might comprise a security incident. Without strong detective capabilities built into the information systems, the organization has little hope of being able to effectively respond to information security incidents in a timely fashion. Organizations should have a regimented and, preferably, automated fashion for pulling events from systems and bringing those events into the wider organizational context. Often when events on a particular system are analyzed independently and out of context, then an actual incident might easily be overlooked. However, with the benefit of seeing those same system logs in the context of the larger organization, patterns indicative of an incident might be noticed. An important aspect of this phase of incident response is that during the detection phase it is determined as to whether an incident is actually occurring or has occurred. It is a rather common occurrence for potential incidents to be deemed strange, but innocuous after further review.

The response phase (aka containment) of incident response is the point at which the incident response team begins interacting with affected systems and attempts to keep further damage from occurring as a result of the incident. Responses might include taking a system off the network, isolating traffic, powering off the system, or other items to control both the scope and severity of the incident. This phase is also typically where a binary (bit by bit) forensic backup is made of systems involved in the incident. An important trend to understand is that most organizations will now capture volatile data before pulling the power plug on a system.

The mitigation phase (aka eradication) involves the process of understanding the cause of the incident so that the system can be reliably cleaned and ultimately restored to operational status later in the recovery phase. In order for an organization to be able to reliably recover from an incident, the cause of the incident must be determined. The cause must be known so that the systems in question can be returned to a known good state without significant risk of compromise persisting or reoccurring. A common occurrence is for organizations to remove the most obvious piece of malware affecting a system and think that is sufficient. In reality, the obvious malware may only be a symptom, with the cause still undiscovered.

The reporting phase of incident handling occurs throughout the process, beginning with detection. Reporting must begin immediately upon detection of malicious activity. Reporting contains two primary areas of focus: technical and non-technical reporting. The incident handling teams must report the technical details of the incident as they begin the incident handling process, while maintaining sufficient bandwidth to also notify management of serious incidents. A common mistake is forgoing the latter while focusing on the technical details of the incident itself: this is a mistake. Non-technical stake holders including business and mission owners must be notified immediately of any serious incident, and kept up to date as the incident handing process progresses.

The recovery phase involves cautiously restoring the system or systems to operational status. Typically, the business unit responsible for the system will dictate when the system will go back online. Remember to be cognizant of the possibility that the infection, attacker, or other threat agent might have persisted through the eradication phase. For this reason, close monitoring of the system after it is returned to production is necessary. Further, to make the security monitoring of this system easier, strong preference is given to the restoration of operations occurring during off or nonpeak production hours.

Remediation steps occur during the mitigation phase, where vulnerabilities within the impacted system or systems are mitigated. Remediation continues after that phase, and becomes broader. For example: if the root-cause analysis determines that a password was stolen and reused: local mitigation steps could include changing the compromised password and placing the system back online. Broader remediation steps could include requiring dual-factor authentication for all systems accessing sensitive data. We will discuss root-cause analysis shortly.

Unfortunately, the lessons learned phase (also known as post-incident activity, reporting, or post mortem) is the one most likely to be neglected in immature incident response programs. This fact is unfortunate because the lessons learned phase, if done right, is the phase that has the greatest potential to effect a positive change in security posture. The goal of the lessons learned phase is to provide a final report on the incident, which will be delivered to management.

There are four types of IDS events: true positive, true negative, false positive, and false negative. We will use two streams of traffic, the Conficker worm (a prevalent network worm in 2009) and a user surfing the Web, to illustrate these events.

A Network-based Intrusion Detection System (NIDS) detects malicious traffic on a network. NIDS usually require promiscuous network access in order to analyze all traffic, including all unicast traffic. NIDS are passive devices that do not interfere with the traffic they monitor; Figure 8.5 shows a typical NIDS architecture. The NIDS sniffs the internal interface of the firewall in read-only mode and sends alerts to a NIDS Management server via a different (read/write) network interface.

Host-based Intrusion Detection Systems (HIDS) and Host-based Intrusion Prevention Systems (HIPS) are host-based cousins to NIDS and NIPS. They process information within the host. They may process network traffic as it enters the host, but the exam’s focus is usually on files and processes.

A Pattern Matching IDS works by comparing events to static signatures. According to Cisco, “The worm may also contact the http://www.maxmind.com domain and download the geoip.dat.gz and geoip.dat files.”[6] Based on that information, the following pattern can be used to detect Conficker: If the strings “geoip.dat.gz” or “geoip.dat” appear in Web traffic: alert.

A Protocol Behavior IDS models the way protocols should work, often by analyzing RFCs (Request for Comments). RFC 793 (TCP, see: http://www.ietf.org/rfc/rfc0793.txt) describes the TCP flags. A SYN means synchronize, and FIN means finish. One flag is used to create a connection, the other to end one.

An Anomaly Detection IDS works by establishing a baseline of normal traffic. The Anomaly Detection IDS then ignores that traffic, reporting on traffic that fails to meet the baseline.

The most commonly deployed endpoint security product is antivirus software. Many of the full endpoint security suites evolved over time from an initial offering of antivirus. Antivirus products are often derided for their continued inability to stop the spread of malware. However, most arguments against antivirus seem to bemoan the fact that these products alone are not sufficient to stop malware. Unfortunately, there is no silver bullet or magic elixir to stop malware, and until there is, antivirus or antimalware products will continue to be necessary, though not sufficient. Antivirus is one layer (of many) of endpoint security defense in depth.

Application Whitelisting is a more recent addition to endpoint security suites. The primary focus of application whitelisting is to determine in advance which binaries are considered safe to execute on a given system. Once this baseline has been established, any binary attempting to run that is not on the list of known-good binaries is prevented from executing. A weakness of this approach is when a “known good” binary is exploited by an attacker, and used maliciously.

Another recent endpoint security product to find its way into large suites assists with removable media control. The need for better controlling removable media has been felt on two fronts in particular. First, malware infected removable media inserted into an organization’s computers has been a method for compromising otherwise reasonably secure organizations. Second, the volume of storage that can be contained in something the size of a fingernail is astoundingly large, and has been used to surreptitiously exfiltrate sensitive data.

Another endpoint security product found with increasing regularity is disk encryption software. Organizations have often been mandating the use of whole disk encryption products that help to prevent the compromise of any sensitive data on hard disks that fall into unauthorized hands, especially on mobile devices, which have a greater risk of being stolen.

Standardizing on a security configuration is certainly important, but there is an additional consideration with respect to security baselines. Security baselining is the process of capturing a point in time understanding of the current system security configuration. Establishing an easy means for capturing the current system security configuration can be extremely helpful in responding to a potential security incident. Assuming that the system or device in question was built from a standardized security baseline, and also that strong change control measures are adhered to, then there would be little need to capture the current security configuration. However, in the real world, unauthorized changes can and will occur in even the most strictly controlled environment, which necessitates the monitoring of a system’s security configuration over time. Further, even authorized system modifications that adhere to the change management procedures need to be understood and easily captured. Another reason to emphasize continual baselining is because there may be systems that were not originally built to an initial security baseline. A common mistake that organizations make regarding system security is focusing on establishing a strong system security configuration, but failing to quickly and easily appreciate the changes to a system’s security configuration over time.

One of the most basic, yet still rather difficult, tasks associated with maintaining strong system security configuration is patch management, the process of managing software updates. All software has flaws or shortcomings that are not fully addressed in advance of being released. The common approach to fixing software is by applying patches to address known issues. Not all patches are concerned with security; many are associated with simple non security-related bug fixes. However, security patches do represent a significant piece of the overall patch pie. Software vendors announce patches both publicly and directly to their customers. Once notified of a patch, organizations need to evaluate the patch from a risk management perspective to determine how aggressively the patch will need to be deployed. Testing is typically required to determine whether any adverse outcomes are likely to result from the patch installation. From a timeline standpoint, testing often occurs concomitantly with the risk evaluation. Installation is the final phase of the patch management process, assuming adverse effects do not require remediation.

Security patches are typically intended to eliminate a known vulnerability. Organizations are constantly patching desktops, servers, network devices, telephony devices and other information systems. The likelihood of an organization having fully patched every system is low. While un-patched systems may be known, it is also common to have systems with failed patches. The most common cause of failed patches is failing to reboot after deploying a patch that requires one.

The most basic and obvious measure to increase system or data fault tolerance is to provide for recoverability in the event of a failure. Given a long enough timeframe, accidents, such as that in Figure 8.7, will happen. In order for data to be able to be recovered in case of a fault some form of backup or redundancy must be provided. Though magnetic tape media is quite an old technology, it is still the most common repository of backup data. The three basic types of backups are: full backup, incremental backup and differential backup.

Even if only one full backup tape is needed for recovery of a system due to a hard disk failure, the time to recover a large amount of data can easily exceed the recovery time dictated by the organization. The goal of a Redundant Array of Inexpensive Disks (RAID) is to help mitigate the risk associated with hard disk failures. There are various RAID levels that consist of different approaches to disk array configurations. These differences in configuration have varying cost, in terms of both the number of disks required to achieve the configuration’s goals, and capabilities in terms of reliability and performance advantages. Table 8.1 provides a brief description of the various RAID levels that are most commonly used.

Though redundancy and resiliency of data, provided by RAID and backup solutions, is important, further consideration needs to be given to the systems themselves that provide access to this redundant data.

Errors and omissions are typically considered the single most common source of disruptive events. Humans, often employed by the organization, unintentionally cause this type of threat. Data entry mistakes are an example of errors and omissions. These mistakes can be costly to an organization, and might require manual review prior to being put into production, which would be an example of separation of duties.

Natural disasters include earthquakes, hurricanes, floods, tsunamis, etc. In order to craft an appropriate response and recovery strategy in the BCP and DRP, an understanding of the likelihood of occurrence of a natural disaster is needed. The likelihood of natural threats occurring is largely based upon the geographical location of the organization’s information systems or datacenters. Natural disasters generally have a rather low likelihood of occurring. However, when they do happen, the impact can be severe. See Chapter 4, Domain 3: Security Engineering for additional information on these risks as well as specific strategies for mitigating them.

While natural disasters are often associated with the most catastrophic events that an organization might ever have to deal with, power problems represent much more commonly occurring threats that can cause significant disruptions within an organization. When power problems do occur, they typically affect the availability of a system or organization. Integrity issues can also crop up on disk drives as a result of sudden power loss; however, modern transaction-based or journaling file systems have greatly reduced these integrity issues.

Temperature and humidity are critical controls that must be managed during a disaster. While it is obvious that information systems must have a regular clean power supply in order to maintain their availability, the modern datacenter must also provide sufficient heating, cooling, ventilation, and air conditioning. Proper cooling and humidity levels are critical.

The height of human-intentional threats is found in the examples of warfare, terrorism, and sabotage. The threat of traditional warfare, terrorism, and sabotage to our organizations can vary dramatically based on geographic location, industry, brand value, as well as the interrelatedness with other high-value target organizations. While traditional physical attacks are still quite possible, an even more likely scenario is cyber-warfare, terrorism, or sabotage. The threat landscape for information systems has rapidly evolved over the years.

Another recent trend that impacts threat analyses is the greater presence of financially motivated attackers. The attackers have come up with numerous ways to monetize attacks against various types of organizations. This monetization of cybercrime has increased the popularity of such attacks. Whether the goal is money via exfiltration of cardholder data, identity theft, pump-and-dump stock schemes, bogus anti-malware tools, or corporate espionage, the trend is clear that attackers understand methods that allow them to yield significant profits via attacks on information systems. One of the more disturbing prospects is the realization that organized crime syndicates now play a substantial role as the source of these financially motivated attacks. The justification for organized crime’s adoption of cybercrime is obvious. With cybercrime, there is significant potential for monetary gain with a greatly reduced risk of being caught, or successfully prosecuted if caught. With respect to BCP and DRP, an appreciation of the significant changes in the threat sources’ capabilities and motivations will help guide the risk assessment portions of the planning process.

Another threat source that can result in disaster is found in issues related to personnel shortages. Though most of the discussions of threats until this point have been related to threats to the operational viability of information systems, another significant source of disruption can come by means of having staff unavailable. While some systems can persist with limited administrative oversight, most organizations will have some critical processes that are people-dependent.

Dependence upon communications without sufficient backup plans represents a common vulnerability that has grown with the increasing dependence on call centers, IP telephony, general Internet access, and providing services via the Internet. With this heightened dependence, any failure in communication equipment or connectivity can quickly become disastrous for an organization. There are many threats to an organization’s communications infrastructure, but one of the most common disaster-causing events that occur with regularity is telecommunication lines being inadvertently cut by someone digging where they are not supposed to. Physical line breaks can cause significant outages.

In order to begin the disaster recovery process, there must be an initial response that begins the process of assessing the damage. Speed is essential during this initial assessment. There will be time later, should the event warrant significant recovery initiatives, to more thoroughly assess the full scope of the disaster.

If during the initial response to a disruptive event a disaster is declared, then the team that will be responsible for recovery needs to be activated. Depending on the scope of the disaster, this communication could prove extremely difficult. The use of calling trees, which will be discussed in the “Call Trees” section later in this chapter, can help to facilitate this process to ensure that members can be activated as smoothly as possible.

After the successful activation of the disaster recovery team, it is likely that many individuals will be working in parallel on different aspects of the overall recovery process. One of the most difficult aspects of disaster recovery is ensuring that consistent timely status updates are communicated back to the central team managing the response and recovery process. This communication often must occur out-of-band, meaning that the typical communication method of leveraging an office phone will quite often not be a viable option. In addition to communication of internal status regarding the recovery activities, the organization must be prepared to provide external communications, which involves disseminating details regarding the organization’s recovery status with the public.

Though an initial assessment was carried out during the initial response portion of the disaster recovery process, the (now activated) disaster recovery team will perform a more detailed and thorough assessment. The team will proceed to assess the extent of the damage to determine the proper steps necessary to ensure the organization’s ability to meet its mission and Maximum Tolerable Downtime (MTD). Depending upon whether and what type of alternate computing facilities are available, the team could recommend that the ultimate restoration or reconstitution occurs at the alternate site. An additional aspect of the assessment not to be overlooked is the need to continually be mindful of ensuring the ongoing safety of organizational personnel.

The primary goal of the reconstitution phase is to successfully recover critical business operations either at primary or secondary site. If an alternate site is leveraged, adequate safety and security controls must be in place in order to maintain the expected degree of security the organization typically employs. The use of an alternate computing facility for recovery should not expose the organization to further security incidents. In addition to the recovery team’s efforts at reconstitution of critical business functions at an alternate location, a salvage team will be employed to begin the recovery process at the primary facility that experienced the disaster. Ultimately, the expectation is (unless wholly unwarranted given the circumstances), that the primary site will be recovered, and that the alternate facility’s operations will “fail back” or be transferred again to the primary center of operations.

It goes without saying that any BCP/DRP is worthless without the consent of the upper level management team. The “C”-level managers must agree to any plan set forth and also must agree to support the action items listed in the plan if an emergency event occurs. C-level management refers to people within an organization like the chief executive officer (CEO), the chief operating officer (COO), the chief information officer (CIO), and the chief financial officer (CFO). C-level managers are important, especially during a disruptive event, because they have enough power and authority to speak for the entire organization when dealing with outside media and are high enough within the organization to commit resources necessary to move from the disaster into recovery if outside resources are required. This also includes getting agreement for spending the necessary resources to reconstitute the organization’s necessary functionality.

The BCP/DRP project manager is the key Point of Contact (POC) for ensuring that a BCP/DRP is not only completed, but also routinely tested. This person needs to have business skills, be extremely competent and knowledgeable with regard to the organization and its mission, and must be a good manager and leader in case there is an event that causes the BCP or DRP to be implemented. In most cases, the project manager is the Point of Contact for every person within the organization during a crisis.

Building the BCP/DRP team is essential for the organization. The BCP/DRP team comprises those personnel that will have responsibilities if/when an emergency occurs. Before identification of the BCP/DRP personnel can take place, the Continuity Planning Project Team (CPPT) must be assembled. The CPPT is comprised of stakeholders within an organization and focuses on identifying who would need to play a role if a specific emergency event were to occur. This includes people from the human resources section, public relations (PR), IT staff, physical security, line managers, essential personnel for full business effectiveness, and anyone else responsible for essential functions. Also, depending on the type of emergency, different people may have to play a different role. For example, in an IT emergency event that only affected the internal workings of the organization, PR may not have a vital role. However, any emergency that affects customers or the general public would require PR’s direct involvement.

Remember, the BIA and Critical State Asset List is conducted for every IT system within the organization, no matter how trivial or unimportant. This is to ensure that each system has been accounted for. Once the list is assembled and users and user representatives have received input, the critical asset list can be created. The critical asset list is a list of those IT assets that are deemed business-essential by the organization. These systems’ DRP/BCP must have the best available recovery capabilities assigned to them.

The BCP/DRP-focused risk assessment determines what risks are inherent to which IT assets. A vulnerability analysis is also conducted for each IT system and major application. This is done because most traditional BCP/DRP evaluations focus on physical security threats, both natural and human. However, because of the nature of Internet-connected IT systems, the risk of a disruption occurring is much greater and therefore, must be mitigated.

The primary goal of the BIA is to determine the Maximum Tolerable Downtime (MTD), which describes the total time a system can be inoperable before an organization is severely impacted. It is the maximum time it takes to execute the reconstitution phase. Reconstitution is the process of moving an organization from the disaster recovery to business operations.

A number of metrics are used to quantify how frequently systems fail, how long a system may exist in a failed state, and the maximum time to recover from failure. These metrics include the Recovery Point Objective (RPO), Recovery Time Objective (RTO), Work Recovery Time (WRT), Mean Time Between Failures (MTBF), Mean Time to Repair (MTTR), and Minimum Operating Requirements (MOR).

Acquisition of computer equipment and business systems can be fairly straightforward during normal business operations. This can change drastically during a disaster. For example, an organization plans to equip a cold site in the event of disaster and purchase 200 computer servers in the event of a disaster.

Telecommunication management ensures the availability of electronic communications during a disaster. Communications is often one of the first processes to fail during a disaster. In the event of a widespread disaster, electricity, landlines, and cell phone towers may be inoperable, as they were in Louisiana in the aftermath of Hurricane Katrina. In that case, satellite phones were the only means of electronic communication immediately after the hurricane.

Utility management addresses the availability of utilities such as power, water, gas, etc. during a disaster. Specific utility mitigating controls such as power availability, generators, and uninterruptible power supplies are discussed in Chapter 4, Domain 3: Security Engineering.

Once an organization has determined its maximum tolerable downtime, the choice of recovery options can be determined. For example, a 10-day MTD indicates that a cold site may be a reasonable option. An MTD of a few hours indicates that a redundant site or hot site is a potential option.

The Continuity of Operations Plan (COOP) describes the procedures required to maintain operations during a disaster. This includes transfer of personnel to an alternate disaster recovery site, and operations of that site.

The Business Recovery Plan (also known as the Business Resumption Plan) details the steps required to restore normal business operations after recovering from a disruptive event. This may include switching operations from an alternate site back to a (repaired) primary site.

The Continuity of Support Plan focuses narrowly on support of specific IT systems and applications. It is also called the IT Contingency Plan, emphasizing IT over general business support.

The Cyber Incident Response Plan is designed to respond to disruptive cyber events, including network-based attacks, worms, computer viruses, Trojan horses, etc. For example, self-propagating malicious code such as worms has the potential to disrupt networks. Loss of network connectivity alone may constitute a disaster for many organizations.

The Occupant Emergency Plan (OEP) provides the “response procedures for occupants of a facility in the event of a situation posing a potential threat to the health and safety of personnel, the environment, or property. Such events would include a fire, hurricane, criminal attack, or a medical emergency.” [25] This plan is facilities-focused, as opposed to business or IT-focused.

The Crisis Management Plan (CMP) is designed to provide effective coordination among the managers of the organization in the event of an emergency or disruptive event. The CMP details the actions management must take to ensure that life and safety of personnel and property are immediately protected in case of a disaster.

Organizations must ensure that there is always an executive available to make decisions during a disaster. Executive Succession Planning determines an organization’s line of succession. Executives may become unavailable due to a variety of disasters, ranging from injury and loss of life, to strikes, travel restrictions, and medical quarantines.

A full system backup means that every piece of data is copied and stored on the backup repository. Conducting a full backup is time consuming, bandwidth intensive, and resource intensive. However, full backups will ensure that any necessary data is assured.

Incremental backups archive data that have changed since the last full or incremental backup. For example, a site performs a full backup every Sunday, and daily incremental backups from Monday through Saturday. If data are lost after the Wednesday incremental backup, four tapes are required for restoration: the Sunday full backup, as well as the Monday, Tuesday, and Wednesday incremental backups.

Differential backups operate in a similar manner as the incremental backups except for one key difference. Differential backups archive data that have changed since the last full backup.

A common tape rotation method is called FIFO (First In First Out). Assume you are performing full daily backups, and have 14 rewritable tapes total. FIFO (also called round robin) means you will use each tape in order, and cycle back to the first tape after the 14^th is used. This ensures 14 days of data is archived. The downside of this plan is you only maintain 14 days of data: this schedule is not helpful if you seek to restore a file that was accidentally deleted 3 weeks ago.

Electronic vaulting is the batch process of electronically transmitting data that is to be backed up on a routine, regularly scheduled time interval. It is used to transfer bulk information to an offsite facility. There are a number of commercially available tools and services that can perform electronic vaulting for an organization. Electronic Vaulting is a good tool for data that need to be backed up on a daily or possibly even hourly rate. It solves two problems at the same time. It stores sensitive data offsite and it can perform the backup at very short intervals to ensure that the most recent data is backed up.

A database journal contains a log of all database transactions. Journals may be used to recover from a database failure. Assume a database checkpoint (snapshot) is saved every hour. If the database loses integrity 20 minutes after a checkpoint, it may be recovered by reverting to the checkpoint, and then applying all subsequent transactions described by the database journal.

Database shadowing uses two or more identical databases that are updated simultaneously. The shadow database(s) can exist locally, but it is best practice to host one shadow database offsite. The goal of database shadowing is to greatly reduce the recovery time for a database implementation. Database shadowing allows faster recovery when compared with remote journaling.

Increasingly, systems are being required to have effectively zero downtime, an MTD of zero. Recovery of data on tape is certainly ill equipped to meet these availability demands. The immediate availability of alternate systems is required should a failure or disaster occur. A common way to achieve this level of uptime requirement is to employ a high availability cluster.

The DRP Review is the most basic form of initial DRP testing, and is focused on simply reading the DRP in its entirety to ensure completeness of coverage. This review is typically to be performed by the team that developed the plan, and will involve team members reading the plan in its entirety to quickly review the overall plan for any obvious flaws. The DRP Review is primarily just a sanity check to ensure that there are no glaring omissions in coverage or fundamental shortcomings in the approach.

Read-Through (also known as checklist or consistency) testing lists all necessary components required for successful recovery, and ensures that they are, or will be, readily available should a disaster occur. For example, if the disaster recovery plan calls for the reconstitution of systems from tape backups at an alternate computing facility, does the site in question have an adequate number of tape drives on-hand to carry out the recovery in the indicated window of time? The read-through test is often performed concurrently with the structured walkthrough or tabletop testing as a solid first testing threshold. The read-through test is focused on ensuring that the organization has, or can acquire in a timely fashion, sufficient level resources on which their successful recovery is dependent.

Another test that is commonly completed at the same time as the checklist test is that of the walkthrough, which is also often referred to as a structured walkthrough or tabletop exercise. During this type of DRP test, usually performed prior to more in-depth testing, the goal is to allow individuals who are knowledgeable about the systems and services targeted for recovery to thoroughly review the overall approach. The term structured walkthrough is illustrative, as the group will talk through the proposed recovery procedures in a structured manner to determine whether there are any noticeable omissions, gaps, erroneous assumptions, or simply technical missteps that would hinder the recovery process from successfully occurring. Some structured walkthrough and tabletop exercises will introduce various disaster scenarios to ensure that the plan accommodates the different scenarios. Obviously, any shortcomings discovered through this testing process will be noted for inclusion in an updated recovery plan.

A simulation test, also called a walkthrough drill (not to be confused with the discussion-based structured walkthrough), goes beyond talking about the process and actually has teams to carry out the recovery process. A pretend disaster is simulated to which the team must respond as they are directed to by the DRP. The scope of simulations will vary significantly, and tend to grow to be more complicated, and involve more systems, as smaller disaster simulations are successfully managed. Though some will see the goal as being able to successfully recover the systems impacted by the simulation, ultimately the goal of any testing of a DRP is to help ensure that the organization is well prepared in the event of an actual disaster.

Another type of DRP test is that of parallel processing. This type of test is common in environments where transactional data is a key component of the critical business processing. Typically, this test will involve recovery of critical processing components at an alternate computing facility, and then restore data from a previous backup. Note that regular production systems are not interrupted.

Arguably, the highest fidelity of all DRP tests involves business interruption testing. However, this type of test can actually be the cause of a disaster, so extreme caution should be exercised before attempting an actual interruption test. As the name implies, the business interruption style of testing will have the organization actually stop processing normal business at the primary location, and will instead leverage the alternate computing facility. These types of tests are more common in organizations where fully redundant, often load-balanced, operations already exist.

Though it might seem simple, converting a datacenter to emergency power, such as backup generators that will begin taking the load as the UPS fail, is not to be taken lightly. Specific training and testing of changing over to emergency power should be regularly performed.

Another example of combination training and testing is in regard to calling trees, which was discussed previously in the “Call Trees” section. The hierarchical relationships of calling trees can make outages in the tree problematic. Individuals with calling responsibilities are typically expected to be able to answer within a very short time period, or otherwise make arrangements.