Appendix C. Incident Response
Your router has been hacked. Now what? This chapter covers the basics of emergency response when dealing with a router compromise. Ideally, you should have an incident response plan that is tailored to your organization. If you are reading this chapter because you have just been hacked and don’t know what to do, first promise that as soon as this incident is over, you will develop a complete incident response plan. Then keep reading for help on responding to incidents involving router compromises.
The goals of incident response are to:
Determine if the incident is an attack or an accident
Discover what happened and the scope of the incident
Preserve all the evidence
Recover from the incident
Take the steps necessary to prevent this incident from happening again
Warning!
If you do not have a detailed incident response plan in place and you have been hacked, it is best to do nothing yourself and to call law enforcement. They are trained to preserve the evidence and investigate the incident and can track down attackers through means you don’t have access to. Therefore, the first recommendation is to do nothing and call law enforcement.
However, many attacks may look like accidental outages (and vice versa). The following information is provided for those who are still trying to determine if an incident is due to a hacker or an accident or for those who must get the compromised router operational as soon as possible. So please read this entire chapter—especially the section on preserving evidence—to collect enough evidence to provide law enforcement with leads if necessary. When you reconfigure or reboot the router, you destroy the original evidence, so how you make copies of this evidence is extremely important to having any chance of holding up in a court of law.
Keys to Investigating
Your mission while investigating an incident is to:
Change nothing.
Record everything.
Even if you suspect the incident was accidental, it is best to follow these two rules until you are sure. Once you start modifying your router, you destroy your ability to use any information on it in the future.
Change Nothing
Many administrators’ first step when a router goes down is to reboot the system. It is amazing how many times this seems to fix a problem, but if the router malfunctioned because of an attack by a hacker, rebooting the system can cause the loss of valuable evidence, sometimes all evidence of the attack. Additionally, while investigating the incident, until you have determined that it was indeed an accident, do not make any changes to the router. These changes can cause significant problems if the evidence is ever needed in court.
Record Everything
The most unobtrusive way to log into a router is through the console port. Thus, for investigation purposes, use terminal emulation software—like HyperTerminal—to connect to the router’s console port. Before you even log in, configure your terminal emulation software to capture your current session. This will record everything you do and can be helpful in proving that you did not make any changes to the router during your investigation. HyperTerminal can be configured to capture your session though the menu option Transfer → Capture Text. This option will bring up a dialog box that lets you choose the name and location of the capture file. Once you have chosen it, click the Start button to begin recording. You can now log into the router and use read-only commands—show commands—to investigate the incident.
Tip
Make sure you record the date and time inside your terminal session somehow. To do this, right after you connect to the router, run the show clock command. Run this command about every five minutes or so to establish a time record, and then run it one more time just before you log out of the router.
Attack Versus Accident
When many administrators start getting involved in information security, they tend to get very worried and excited about the state of their networks. First, they get worried because they realize how vulnerable their systems are; second, they get excited by the challenge of protecting those systems. The Holy Grail for many system and network administrators who move into InfoSec is catching the bad guy. This provides the ability to impress friends with tales of how your cunning outsmarted the wily hacker.
This excitement can make these administrators jump to conclusions and see accidental incidents as attacks. In their excitement, they inform management that the systems have been hacked, and they are quickly tracking down the attacker. This can become embarrassing when it turns out that the janitor accidentally tripped over a power cord.
So, before you run to management claiming that you have been hacked, take the time to rule out accidental causes. They are more often responsible for router problems than are intentional compromises, and caution can save you much embarrassment.
Discover What Happened and the Scope of the Incident
People request a nice checklist when they reach the step of determining what happened and how big the problem is. Networks are so complex and types of attacks are changing so fast that such a checklist will never exist. This type of work is what separates those who truly understand routers and networking from those who don’t. To determine what happened, you need to go through your router logs, configurations, access points, and so on. Once you decide that your router was actually compromised by an attacker, you need to determine details such as:
What parts of your organization are impacted, and how much damage is the impact causing?
How did the attacker do it?
Who is the attacker?
Is the incident ongoing, or has it stopped?
What other systems or routers have been accessed from the compromised router?
What version of IOS are you running, and are there any known vulnerabilities to this version?
What IP addresses have recently accessed the router?
Have the running-config or startup-config been changed?
This list is far from complete, but will hopefully get you thinking in the right direction. More often than not, answering every question on this list necessarily involves law enforcement. If you are not sure how to start looking for answers to the preceding questions, you are probably over your head and it is time to call in a professional.
Evidence Preservation
If you must get your router functional as quickly as possible, it is vitally important that you record any volatile information that may be lost upon reconfiguration or reboot of the router. Before you make any changes to, shut down, or reboot the router, follow these steps to gather as much of this volatile evidence as possible:
Connect to the router’s console port. This is the least-intrusive way to access the router. It doesn’t require network access and will not tip off your attackers if they are sniffing your network.
Configure your terminal emulation software to record your session.
Log in to the router.
Enter enable mode (enable).
Show the current date and time (show clock detail).
Write down the time from a trusted time source—atomic clock, NTP server, etc.
Show the IOS, uptime, and hardware information (show version).
Show the current running configuration (show running-config).
Show the current startup configuration (show startup-config).
Show scheduled reload time (system may auto reboot, if set) (show reload).
Show the routing tables (show ip route).
Show the ARP tables (show ip arp).
Show who is logged in (show users).
Show current logs (show logging).
Show current interface ip configuration (show ip interface).
Show current interface configuration (show interfaces).
Show TCP connections (show tcp brief all).
Show open sockets (show ip sockets).
Show NAT translations (show ip nat translations verbose).
Show NetFlows (show ip cache flow).
Show CEF forwarding table (show ip cef ).
Show SNMP v3 users (show snmp user).
Show SNMP v3 groups (show snmp group).
Show date and time again (show clock detail).
Write down the time from a trusted time source again.
Disconnect from the router.
End your terminal recording session.
Print out your recording session.
Write the two times you recorded from the trusted time source on the printout.
Sign and date the printout.
Get a witness to sign and date the printout.
Keep both the electronic copy and the hardcopy in a secure location until you can turn them over to law enforcement.
Next, you need to gather information from the router externally:
Port scan the router from an external system.
Record the time of the port scan from a trusted time source.
Print out the port scan and write the time on the printout.
If the router is running SNMP, get a copy of the current SNMP tree. This can be done with a command such as snmpwalk (from NetSNMP http://net-snmp.sourceforge.net).
Record the time of the SNMP walk from a trusted time source.
Print out the SNMP tree info and write the time on the printout.
Sign and date both printouts.
Get a witness to sign and date both printouts.
Keep all copies in a secure location until you can turn them over to law enforcement.
The worst-case scenario is when the router’s enable password has been changed by either an accident or an attacker. In these situations your ability to collect forensic information is severely limited. Password recovery procedures require rebooting the router, which destroys much of the evidence you are interested in. If this happens, attempt to log in with a lower privileged account and run as many of the preceding commands as possible. When you cannot log into the router at all, the information gathered externally becomes much more important because it is all you have. Therefore, be sure to try to use SNMP and port scans to gather as much information about the router as possible.
Recovering from the Incident
Once law enforcement officials have completed their initial analysis of the router, they may return it to you or keep it for more detailed forensic investigation. Whether you are using the original router or a replacement router, the next step is to recover from the incident. This is why it is so important to have current documentation on your network and backup copies of all your router configuration files. With backup copies, recovery may be as simple as reloading the backup configuration onto the router. However, this configuration has already been compromised once; it is imperative that you move on to the next step—preventing future incidents.
Preventing Future Incidents
Having finally recovered, your job isn’t over. In the course of your response and investigation, you should have determined how the attacker compromised your router. Chances are it will have been compromised due to a known vulnerability that hadn’t been patched, an attacker sniffing the wire for passwords, or poorly chosen router passwords that the attacker simply guessed.
Whatever the cause, now is the time to do a postmortem and come up with a plan to close the security hole and prevent such holes from appearing in the future. Finally, if you responded to this incident by the seat of your pants, consider this your wake-up call and develop a documented and tested incident response policy.
Incident Response Checklist
Here is a quick overview of responding to an incident:
Follow your established incident response plan, if you have one.
Determine if the problem was due to an accident or malicious attack.
While determining the cause of the problem:
Change nothing.
Record everything.
If you don’t have an incident response policy and you determine you have been hacked, touch nothing and call law enforcement.
If you cannot call or wait for law enforcement, understand the risks you take by modifying or rebooting the router.
If you must modify or reboot the router, first record all volatile evidence from the router in a well-documented manner.
Recover from the incident by getting the router functional again.
Perform a postmortem and implement changes to prevent future compromises.
If you don’t have a documented and tested incident response plan, develop one now.