The first step in responding to a security incident is to decide what response, if any, needs to be made immediately. Ask these questions:

If the incident looks like an aggressive attack on your system, you probably want to take strong steps quickly. These steps might include shutting down the system or your Internet connection until you figure out how to deal with the situation.

On the other hand, if the incident is a less aggressive one — perhaps someone has just opened a Telnet connection to your machine and is trying various login/password pairs — then you may want to move more slowly. If you’re reasonably confident that the attack won’t succeed (e.g., you can see that the attacker is trying passwords that consist of all lowercase letters, and you know for certain that no account on the system has such a password), you might want to leave things alone and just watch for a while to see what the attacker does. This may give you an opportunity to trace the attack. (However, see the Section 27.3 section, later in this chapter, for a discussion of the issues involved in tracing an attack.)

Whatever you do, remember Rule 1: Don’t panic!

As soon as you determine that you actually have a problem that you need to respond to, start documenting what’s going on. You don’t need to get fancy at this point (you don’t have time to, until you’ve taken the next step), but you should at least start a log by making a note of what time it is.

Once you’ve evaluated the situation, your next priority is to give yourself the time to respond without risking your systems further. The least disruptive alternative is usually to disconnect the affected machine from all networks; this will shut down any active connections. Shutting down active connections may make it harder to trace the intruder, but it will allow the rest of the people at your site to continue to do their work, and it will leave the intruder’s programs running. This may help you to identify who the intruder might be.

If you’re afraid that other machines have been compromised or are vulnerable to the same attack, you’ll probably want to disconnect as many machines as you can as a unit. This may mean taking down your connection to the Internet, if possible. If your Internet connection is managed elsewhere in your organization, you may need to detach just your portion of the network, but you’ll also need to talk to other parts of your organization as soon as possible to let them know what’s happening.

In some situations, you may want to shut down the compromised system. However, this action should be a last resort for a number of reasons:

Even if you’re responding to an incident that has already ended, you still might want to disconnect or shut down the system, or at least close it to users, while you analyze what happened and make any changes necessary to keep it from happening again. This will keep you from being confused by things users are doing, and it will prevent the intruder from returning before you’re done.

Your next priority is to start to fix what’s gone wrong. The first step in actually correcting the problem is to relax, think for a while, and make sure you really understand what’s happening and what you’re dealing with. The last thing you want to do is make the situation worse by doing something rash and ill considered. Whatever corrective actions you’re contemplating, think them through carefully. Will they really solve the problem? Will they, in turn, cause other problems?

When you’re working in an unusual, high-stress situation like this, the chances increase of making a major error. Because you’re probably going to be working with system privileges (for example, working as root on a Unix system), the consequences of an error could be serious.

There are several ways you can reduce the chances of making an error. One good way is to work with a partner; each of you can check the other’s commands after they’re typed but before they’re executed. Even if you’re working alone, many people find that reading commands aloud and checking the arguments in reverse order before executing them helps avoid mistakes. Resist the temptation to try to work fast. You will go home sooner if you work slowly and carefully.

Try not to let your users get in the way of your response. You may want to give someone the specific job of dealing with user inquiries so the rest of your response team can concentrate on responding to the incident.

Also, try to keep your responders from tripping over each other. Make it clear which system managers and investigators are working on which task, so they won’t step on each other’s toes (or wind up unintentionally chasing each other as part of the investigation!).

You’re not the only person who needs to know what’s going on. A number of other people — in a number of different places — have to be kept informed.

Another early step to take is to make a “snapshot” of each compromised system. You might do so by doing a full backup to tape or by copying the whole system to another disk. In the latter case, if your site maintains its own spare parts inventory, you might consider using one of the spares for this purpose, instead of a disk that is already in use and might itself turn out to have been compromised.

The snapshot is important for several reasons:

Because the snapshot may become important for legal proceedings, you need to secure the evidence trail. Here are some guidelines:^[1]

It’s a good idea to set aside an adequate supply of fresh media just for snapshots because you never know when you’re going to need to produce one. It’s very frustrating to respond to an incident, and be ready to do the snapshot, only to discover that the last blank tape got used for backups the day before and the new order hasn’t come in yet.

Finally, you’re at the point of actually dealing with the incident. What do you do? It depends on the circumstances. Here are some possibilities:

At worst, you may need to rebuild your system from scratch. Sometimes you end up doing this because the intruder damaged things, purposefully or accidentally. More often, you’ll rebuild your system because it’s the only way to ensure you have a clean system that hasn’t been booby-trapped. Most intruders start by making sure they’ll be able to get back into your system, even if you close their initial entry point. As a result, your systems may be compromised even if the intruder was present for only a short time.

If you need to rebuild your system, first ensure that your hardware is working properly. You want to make sure it passes all relevant self-tests and diagnostics; you don’t want to restore onto a flaky system. A reinstall may reveal previously unnoticed hardware problems. For instance, a disk may have bad spots that are in unused files. When you reinstall the operating system, you will attempt to write over the bad parts, and the problem will suddenly become apparent.

Next, make sure you are using trusted media and programs, not necessarily your last backup, to restore the system. Unless you are absolutely sure that you can accurately date the first time the intruder accessed your system, you don’t know whether or not programs had already been modified at the time the backups happened. It’s often best to rebuild your system from vendor distribution media (that is, the tapes or CD-ROM your operating system release came on) and then reload only user data (not programs that multiple users share) from your backup tapes.

If you need programs you didn’t get from your vendor (for instance, packages from the Internet), then do one of the following:

Do not recompile software until you’ve reinstalled the operating system, including the compiler; you don’t know whether the compiler itself, and the libraries it depends on, have been compromised.

This implies that if you’re heavily customizing your system or installing a lot of extra software beyond what your vendor gives you, you need to work out a way of archiving those customizations and packages that you’re sure can’t be tampered with by an attacker. This way, you can easily restore those customizations and packages if you need to. One good way is to make a special backup tape of new software immediately after it’s installed and configured, before an attacker has a chance to modify it.

You may have programs that were locally written, and in these cases, you may not be able to find even source code that’s guaranteed to be uncontaminated. In this situation, someone — preferably the original author — will need to look through the source code. People rarely bother to modify source code, and when they do, they aren’t particularly subtle most of the time. That’s because they don’t need to be; almost nobody actually bothers to look at the source before recompiling it.

In one case, a programmer installed a back door into code he expected would run on only one machine, as a personal convenience. The program turned out to be fairly popular and was adopted in a number of different sites within his university. Years after he wrote it, and long after the original machine was running a version without the back door, he discovered that the back door was still present on all the other sites, despite the fact that it was clearly marked and commented and within the first page of code. You can’t make a comprehensive search of a large program, but you can at least avoid humiliation by looking for obvious changes.

Life gets very confusing when you’re discovering, investigating, and recovering from a security incident. A good chain of communication is important in keeping people informed and preventing them from tripping over each other. Keeping a written (either hardcopy or electronic) record of your activities during the incident is also important. Such a record serves several purposes:

From a legal standpoint, the best records are hardcopy records generated and identified at the time of occurrence. Just about anything else (particularly anything kept online) could be tampered with or falsified fairly easily — or at least a judge and jury could be convinced of that. You need to produce records on pieces of paper, label, date, and sign them. Furthermore, unless the pages are actually bound together, so that pages can’t be inserted or removed without indication, you’ll need to date and sign every page. (And you thought continuous tractor-feed paper was useless these days!)

You need to have legal documentation even if you aren’t completely certain you’re going to need it. An incident that initially looks fairly simple may turn out to be serious. Don’t assume it isn’t going to be worth bringing in the police.

For both legal and practical reasons, it’s useful to put in exact times when things occurred. Legally, this helps to show that entries were being made in order. Practically, it’s extremely helpful when you need to correlate multiple sources of information (for instance, when you need to compare your logs against event logs on computers or against somebody else’s actions).

Here are several useful documentation methods you might want to consider:

You will probably want to use multiple methods, one to record what’s happening online and one to record what’s happening outside of the computer. For example, you might have a typescript of the commands you were typing, but a handwritten log for phone calls.

It’s easy to decide what to record online; you simply record everything you do. Remember to use the terminal or session that’s being recorded. (With some methods, like script, you can record every session you’ve got going; just make sure you record each session in a separate file.) It’s harder to decide what to record of the events that don’t just get automatically captured. You certainly want to record at least this much:

In addition to the journal, a log of time spent for everyone working on an incident can be invaluable. You may need to justify some level of “loss” in order for some law enforcement agencies to be able to open an investigation, and if the intruder didn’t do any damage to the machines, the time that was spent cleaning up is the main loss.

Time logs may also be useful if you are having difficulty in convincing management that the organization needs to allocate additional resources to be prepared to deal with incidents. It’s a way of showing how much these incidents cost. It’s particularly helpful if you can show which areas could have been anticipated and mitigated by planning.

An incident starts when somebody detects an intruder or attacker. That person might be a system administrator, but more often it’s someone with no official responsibility. If you’ve properly educated the people who use your computers, they know they’re supposed to report weird events. Somebody then needs to sort run-of-the-mill peculiarities from a security incident in progress. Who are the users going to report to? Who are those people going to report to if they’re still not sure? What are they authorized to do if they are sure?

The two cases you really want to plan for are these:

In the first case, you need a procedure that is going to reliably start a full incident response immediately. Don’t waste any time. It’s going to be embarrassing and expensive if you don’t actually get around to doing anything until your senior security person arrives in the next morning, takes in enough caffeine to become able to think, and gets around to looking at some report. (And that’s if there is a report in the first place; without a response plan, it may be weeks before anyone actually tells someone who can begin to do something about the situation.)

In the second case, it’s going to be embarrassing and expensive if you disconnect the network and get five people out of bed, all to prevent somebody from doing the work they’re paid to do.

Either way, it’s not a decision you probably want made by a night operator, or by a user acting alone because he or she can’t figure out how to call somebody who knows how to tell a real incident from a false alarm.

At a small site, you might want to simply post a number that users can call to get help outside of office hours (for instance, a pager number). Users might be encouraged to shut down personal machines if they suspect an attack and know how to shut the machine down gracefully. You want to be very cautious about this, however, because an ungraceful shutdown, particularly of a multi-user machine, may be more damaging than an intruder.

At a larger site, one that has on-site support after hours, you should instruct the on-site support people to call a senior person if they see a possible security incident. They should be told explicitly not to do anything more than that unless circumstances are extreme, but to keep trying to contact senior personnel until they get somebody who can take a look at what’s going on.

Who’s going to decide that you don’t just have a suspicious situation — you actually have a security problem? You need to designate one specific person who will have responsibility for making the important decisions. It’s tempting to pick one specific person in advance and put his or her name in your plan. But, what if that person isn’t available in the event of an actual incident? Who, then, will have the responsibility?

Teamwork is great, but emergencies call for leadership. You don’t want to have everybody doing their own thing and nobody in charge, and you certainly can’t afford to stand around arguing about it. If your senior technical person is absent, do you want someone less senior but more technical to do the evaluation, or do you want someone more senior but less technical? How much time are you going to spend searching for the senior technical person when you have an emergency to deal with, before proceeding to your next candidate for the hot seat?

At a small site, you may not have a lot of options; if only one person has the skills necessary to do something about an attack, your policy will simply list that person as the one in charge in case of a security incident. If that person is unavailable, authority should go to somebody levelheaded and calm who can take stopgap actions and arrange for assistance (for example, from a relevant response team). In this situation, technical skills would be nice, but resourcefulness and calm are more important.

At a larger site, probably more than one person could be in charge. Your plan may want to say that the most senior will be in charge by default or that whoever is specified as being on call will be in charge. Either way, the plan should state that if the default person in charge is unavailable, the first of the other possible people to respond is in charge. Specifying what order they’re going to be contacted in is probably overkill; let whoever is trying to reach these people use his or her knowledge of the situation. If none of those people are available, you’ll usually want to work up the organizational hierarchy rather than down. (A manager, particularly a technical one, is probably better equipped to cope than an operator.)

In a small organization, you will pick your fallback candidates by name. In a large one, you will usually specify fallbacks by job title. If job title is your criterion, it’s important to base your decision on the characteristics of the job, not of the person currently in it. Don’t write into your plan that the janitor should decide, on the theory that the current janitor also is the most sensible and technical of those who aren’t system administrators. The next janitor might be an airhead with a mop.

Your response plan needs to specify what kind of situation warrants disconnecting or shutting down, and who can make the decision to do it. Most importantly, as we’ve discussed in “Pursuing and Capturing the Intruder”, are you ever willing to allow a known intruder to remain connected to your systems? If you’re not, are you going to take down the system, or are you going to disconnect from the network altogether?

If you are at a site with multiple computer facilities, do you want to take the entire site off the Internet if one facility has been compromised, or is it better (or even possible) to take just that facility off the Internet?

At most sites, the reasonable plan is to disconnect the site as a whole from the network as soon as you know for sure that you have an intruder connected to your systems. You may have a myriad of internal connections, with a triply redundant, diversely cabled, UPS-protected routing mesh, which can make “disconnecting” a daunting prospect (the system keeps “fixing” itself). On the other hand, you probably have only one (or a small handful) of connections to the outside world, which can be more easily severed.

Your plan needs to say how to disconnect the network, and how the machines should be shut down. Be very careful about this. You do not want to tell people to respond to a mildly suspicious act by hitting the circuit breakers and powering off every machine in the machine room. On the other hand, if an intruder is currently removing all the files on the machine, you don’t want them to give that intruder a 15-minute warning for a graceful shutdown.

This is one case in which you need clear, security-specific instructions in your plan. Here’s what we recommend you do:

Whoever is going to disconnect the network needs to know how to do that. The safest and easiest way often is to unplug cables and clean up the side effects afterwards. With networks, this tends to result in voluminous error messages but to cause no actual damage. You do have to unplug the relevant cables, however, and the voluminous error messages may make it difficult to determine whether or not the cables that were unplugged were actually the correct ones. Your plan needs to tell people what to unplug and how to make things functional afterwards.

Your incident response plan needs to specify who you’re going to notify, who’s going to do the notification, when they’re going to do it, and what method they’re going to use. As we described earlier in this chapter, you may need to notify:

Your own organization

To start with, you need to notify the people who are going to be involved in the response. You’ll have an urgent need to get hold of them, so you need telephone and pager numbers. Be sure you have all the relevant phone numbers; in addition to home and work numbers, check to see if people have mobile phones at which they might be reached. This list includes anybody who manages computers within your site and anybody who manages those people, plus anybody else who might be needed to provide resources (to sign off on emergency purchases or to unlock doors, for example). Ideally, the list — or at least the key portions of it — should be reduced down so it’s small enough to carry easily (for example, it might be laser-printed onto business card-sized stock). Obviously, the list isn’t much use unless it’s kept up to date.

If many people must be notified, you may wish to use a phone tree or an alert tree. In such a tree, shown in Figure 27.2, each person notifies two or three other people; it is a geometric progression, so a large number of people can be rapidly notified with relatively little work to any one person. Everybody should have a copy of the entire tree, so that if people are unavailable, their calls can be taken over by someone else (usually the person above them on the tree). It’s best to set it up so that as many calls as possible are toll-free, and so that people are notifying other people they know relatively well (which increases their chances of knowing how to get through). There’s no need for an alert tree to reflect an organizational chart or a chain of command.

Figure 27.2. An alert tree

Next, you’re going to notify other people within your organization who need to know, starting with the users of your computer facility. For that, you’ll use whatever your organization normally uses for relatively urgent notifications to everybody, whether that’s memos or electronic mail. Your plan should specify how to do it (system administrators rarely send memos to all personnel and may not know how).

Your plan should also show a sample notification message for the users of your systems, which can sometimes be tricky. Your message needs to contain enough information so that legitimate users understand what’s happening. They need to know:

• What has been taken out of service

• Why you’re making their lives miserable

• Exactly which things that they normally do aren’t going to work

• When service will be restored

• What they’re supposed to do (including leave you alone so that you can concentrate on the response)

• That you realize you’re making life unpleasant for them

• That you’re doing everything possible to improve matters

• That you’re going to tell them the details later

Things that are obvious to you may not be obvious to your users (e.g., they might not even understand why it’s so bad to have an intruder). Writing an appropriate message (see Figure 27.3) is not easy, particularly if you’re busy and tired.

Figure 27.3. A notification message

For the remaining people within your organization — people from other computer facilities, legal, audit, public relations, or security — the plan needs to specify who gets notified. Do you need to call the legal department? If so, who should you talk to? Who are the administrators for other sites within your organization? During the Morris worm incident in 1988, at least one large government lab was reduced to having the guards hand out flyers at the gate to people as they came to work, asking “Are you a system administrator?” because they had no idea who all the system administrators were, much less how to contact them.

Think about how you are going to send your message. If you send it via electronic mail, remember that the intruder may see it. Even if you know that your own systems are clean, don’t assume that other people’s are. Don’t say anything in your message that you don’t want the attacker to know. Even better yet, use a telephone.

Some sites use a simple code phrase to announce a system attack that they can include in electronic mail. This can rapidly degenerate into bad spy fiction, but if you have an agreed-upon phrase that isn’t going to alert an intruder (and isn’t going to cause people who don’t know it or don’t remember it to give the game away by asking what on earth you’re talking about), it can be effective. Something like “We’re having a pizza party; call 3-4357 to RSVP” should serve the purpose.

Should you contact your organization’s security department? At some organizations, the security department is responsible only for physical security. You’ll want to have a contact number for them in case you need doors unlocked, for example, but they are unlikely to be trained in helping with an emergency of this kind, so you probably won’t need to notify them routinely of every computer security incident. However, if a group within your organization is responsible for computer security, you are probably required to notify that group. Find out ahead of time when the members of the group want to be notified and how, and put that information in the plan. Even if that group cannot help you respond to your particular type of incident (perhaps because they may be personal computer specialists or government security specialists), it’s advisable to at least brief them on the incident after you have finished responding to it.

Your incident response plan should specify how you’re going to do snapshots of the compromised system. Make sure that your plan contains the answers to these questions:

Different incidents are going to require different amounts of recovery. Your response plan should provide some general guidelines.

Reinstalling an operating system from scratch is time consuming, unpleasant, and often exposes underlying problems. For example, you may discover that you no longer know where some of your programs came from. For this reason, people are extremely reluctant to do it. Unless your incident response plan says explicitly that they need to reinstall the operating system, they probably won’t. The problem is, this leads to situations where you have to get rid of the same intruder over and over again because the system hasn’t been properly cleaned up. Your response plan should specify what’s acceptable proof that the operating system hasn’t been tampered with (for instance, a comparison against cryptographic checksums of an operating system known to be uncompromised). If you don’t have those tools, which are discussed in Chapter 10, or if you can’t pass the inspection, then you must install a clean operating system, and the plan should say so.

The plan should also provide the information needed to reinstall the operating system; for example:

Your plan should include the basic instructions on what documentation methods you intend to use and where to find the supplies. If you might pursue legal action, your plan should also include the instructions on dating, labeling, signing, and protecting the documentation. Remember that you aren’t likely to know when you start out whether or not there will be legal action, so you will always need to document if you ever want to be able to take legal steps; this is not something you can go back and “fix” later on.

However solid your security incident response plans may seem to be, make sure to review them periodically. Changes — in requirements, priorities, personnel, systems, data, and other resources — are inevitable, and you need to be sure that your response plans keep up with these changes. The right question to ask about each item isn’t “Has it changed?,” but "How has it changed?”

A good time to review your incident response plan is after a live drill, which may have exposed weaknesses or problems in the plan. (See Section 27.5.7 at the end of this chapter.) For example, a live drill may uncover any of the following:

Your filesystem backups are probably the single most important part of your recovery plan. Before you do anything else (including writing your response plan), make sure that your site’s backup plan is a solid one and that it works. Don’t assume that it’s OK just because you haven’t had a problem yet. It is entirely possible to go for months without noticing that you have no backups at all, and it may take you years to notice that they’re only partially broken. Unfortunately, when you do notice, it’s often when you need the backups most, and the outcome is likely to be disastrous.

Backups are vital for two reasons:

Every organization needs a backup plan and not just for security reasons. If you don’t have one, that’s probably a sign that your current backup system is not OK. When you are doing incident response planning, however, pay special attention to your backup plan.

For your security-critical systems (e.g., bastion hosts and servers), you might want to consider keeping your monthly or weekly backups indefinitely, rather than recycling them as you would your regular systems. If an incident does occur, you can use this archive of backup tapes to recover a “snapshot” of the system as of any of the dates of the backups. Snapshots of this kind can be helpful in investigating security incidents. For example, if you find that a program has been modified, going back through the snapshots will tell you approximately when the modification took place. That may tell you when the break-in occurred; if the modification happened before the break-in, it may tell you that it was an accident and not part of the incident at all.

If you’re not sure whether or not you should be worried, try testing your backup system. Play around and see what you can restore. Ask these questions:

Even the best backup system won’t work if the backup images aren’t safeguarded. Don’t rely on online backups and keep your media in a secure place separate from the data they’re backing up.

As organizations grow, they acquire hardware; they configure networking in different ways; and they add or change equipment of various kinds. Usually only one or two people really know what a site’s systems look like in any detail.

Information about system configuration may be crucial to investigating and controlling a security incident. While you may know exactly how everything works and fits together at your site, you may not be the person who has to respond to the incident. What if you’re on vacation? Think about what your managers or coworkers would need to know about each system in order to respond effectively to an incident involving that system.

Labels and diagrams are crucial in an emergency. System labels should indicate what a system is, what it does, what its physical configuration is (how much disk space, how much memory, etc.), and who is responsible for it. They should be attached firmly to the correct systems and easily legible. Use large type sizes and put at least minimal labels on the back as well as the front (the front of a machine may have more flat space, but you’re probably going to be looking at it from behind when you’re trying to work on it). Network diagrams should show how the various systems are connected, both physically and logically, as well as things like what kind of packet filtering is done where.

Be sure that labels are kept up to date as you move systems around; wrong labels are worse than no labels at all. It’s particularly important to label racked equipment and equipment with widely scattered pieces. There’s nothing more frustrating than turning off all the equipment in a rack, only to discover that some of it was actually part of the computer in the next rack over, which you meant to leave running.

Information that’s easily available when machines are working normally may be impossible to find if machines are not working. For example, you’ll need disk partition tables written down in order to reformat and reinstall disks, and you may need a printed copy of the host table in order to configure machines as they’re brought back up.

Once you’ve had a break-in, you need to know what’s been changed on your systems. The standard tools that come with your operating system won’t tell you; intruders can fake modification dates and match the trivial checksums most operating systems provide. You will need to install a cryptographic checksumming program (these are discussed in Chapter 10), make checksums of important files, and store them where an intruder can’t modify them (which generally means somewhere offline). You may not need to checksum every system separately if they’re all running the same release of the same operating system, although you should make sure that the checksum program is available on all your systems.

An activity log is a record of any changes that have been made to a system, both before an incident and during the response to an incident. Normally, you’ll use an activity log to list programs you’ve installed, configuration files you’ve modified, or peripherals you’ve added. During an incident, you’ll be doing a lot more logging.

What is the purpose of an activity log? A log allows you to redo the changes if you have to rebuild the system. It also lets you determine whether any of the changes affect the incident or the response. Without a log, you may find mystery programs; you don’t know where they came from and what they were supposed to do, so you can’t tell whether or not the intruder installed them, if they still work the way they’re supposed to, or how to rebuild them. Figure 27.4 shows a sampling of routine log entries and incident log entries.

Figure 27.4. Activity logs

There are a variety of easy ways to keep activity logs, both electronic and manual; email, notebooks, and tape recorders can also be used. Some are better for routine logs (those that record your activities before an incident occurs). Others may be more appropriate for incident logs (those that keep track of your activities during an incident).

Email to an appropriate staff alias that also keeps a record of all messages is probably the simplest approach to keeping an activity log. Not only will email keep a permanent record of system changes, but it has the side benefit of letting everybody else know what’s going on as the changes are made. The email approach is good for routine logs, whereas manual methods are likely to work more reliably during an incident. During an actual security incident, your email system may be down, so any messages generated during the response may be lost. You may also be unable to reach existing online logs during an incident, so keep a printed copy of these email messages up to date in a binder somewhere.

Notebooks make a good incident log, but people must be disciplined enough to use them. For routine logs, notebooks may not be convenient because they may not be physically accessible when people actually make changes to the system. Some sites use a combination of electronic and paper logs for routine logs, with a paper logbook kept in the machine room for notes. This works as long as it’s clear which things should be logged where; having two sets of logs to keep track of can be confusing.

Pocket tape recorders make good incident logs, although they require that somebody transcribe them later on. They’re not reasonable for routine logging.

Well before a security incident, collect the tools and supplies that you are likely to need during that incident. You don’t want to be running around, begging and borrowing, when the clock is ticking.

Here are some of the things you’ll need in order to respond appropriately to an incident. (Actually, you ought to have these things around at all times; they come in handy in all sorts of disasters.)

Set aside basic supplies (e.g., a full backup’s worth of media, networking cables, the most critical tools, notebooks or tape recorders for incident logs) in a cache to be used only in case of disaster. This cache should be separate from your normal stock of spare parts and tools.

If a serious security incident occurs, you may need to restore your system from backups. In this case, you will need to load a minimal operating system before you can load the backups. Are you equipped to do this?

Make sure that you:

Testing your ability to reload the operating system is a good idea, and too few organizations ever do it. You can learn a lot by doing this. While you’re trying to reload a dead system is not a good time to discover that you’ve got a bad copy of the distribution media. It’s also not a good time to discover that the people who have to do the reload can’t figure out how to do it. The best way to test is to designate the least experienced people who might have to do the work, and let them try out the reload well ahead of time.

Most organizations find that the first time they try to reinstall the operating system and restore on a completely blank disk, the operation fails. This can happen for a number of reasons, although the usual reason is a failure in the design of the backup system. One site found that people were doing their backups with a program that wasn’t distributed with the operating system, so they couldn’t restore from a fresh operating system installation. (After that, they made a tape of the restore program using the standard operating system tools; they could then load the standard operating system, recover their custom restore program, and reload their data from backups.)

Don’t assume that responding to a security incident will come naturally. Like everything else, such a response benefits from practice. Test your own organization’s ability to respond to an incident by running occasional drills.

There are two basic types of drills:

You might also test only parts of your response. For example, before configuring a new machine, use it to test your recovery procedures by recovering an existing machine onto it. If you have down time scheduled for your facility, you may be able to use it to test what happens when you disconnect from the network. Run your checksum comparison program before and after you install changes to the operating system to see what changes it catches when you think everything’s the same, and what it does about the things you know have changed. Coordinate with another site to see what messages are logged when various types of attacks occur (pick someone you know and trust and who’ll reliably tell you exactly what they did, or do it yourself). Try taking down all of your central machines at the same time and see whether they’ll all come back up in this situation. (Do this when you have a few hours to spare; if it doesn’t work, it often takes a while to figure out how to coax the machines past their interdependencies.)

This is all a lot of trouble, but a certain amount of perverse amusement can be had by playing around with fictitious disasters, and it’s much less stressful than having to improvise in a real disaster.