Spam and Abuse

We are not going to say a lot about spam in this chapter other than to point out a few things. If you get abuse e-mail that is from the outside world telling you that you are sending spam, you should carefully check it out. It might be evidence of botnet activity. There are a number of considerations here:

SNMP and Netflow: Network-Monitoring Tools

In this section we briefly discuss tools typically used for network monitoring and management. Here the primary focus is usually learning just how full the network “pipes” are, in case you need to buy a bigger WAN connection or bigger routers or Ethernet switches. You might also be interested in knowing who is talking to whom on your network, or traffic characterization, or whether a heavily used server needs a faster interface. From the anomaly detection point of view, it is often the case that these tools can be useful in terms of detecting network scanning, botnet spam outbursts, and, of course, the ever-popular DoS or DDoS attack. All these may be botnet manifestations. For the most part we will confine ourselves to mentioning open-source tools. However, it is reasonable to point out that Cisco is the market leader for network infrastructure gear when it comes to netflow-based tools.

SNMP

In Figures 5.2 and 5.3 we show two examples of DoS attacks as captured with an open-source SNMP tool called Cricket (see http://cricket.sourceforge.net). Cricket uses RRDTOOL to make graphs (see http://oss.oetiker.ch/rrdtool/rrdworld/ for other possible tools that use RRDTOOL). Figure 5.2 graphs an SNMP MIB variable that shows router CPU utilization. This is an integer variable that varies from 0 to 100 percent, the latter of which means that the CPU utilization is very high. This router is “having a bad day” due to a DoS attack that has forced its CPU utilization to be astronomical for a long period of time. This can impact the router’s performance in many ways, including damaging your ability to log into it as an administrator, reducing its ability to route, and possibly damaging its ability to respond to SNMP probes from SNMP managers trying to learn about the attack. Note that the attack went on for at least 12 hours and was finally caught and eliminated. You can see that the load finally dropped drastically around noon.

Figure 5.2 DoS Attack: Cricket/SNMP Router CPU Utilization

Figure 5.3 DoS Attack: Cricket/SNMP Graph of Single Host Traffic

Figure 5.3 shows a switch port graph. Here the SNMP system is graphing bytes in and bytes out from a given switch port hooked up to a single host. Graphing input and output (of bytes or packets) is probably the most traditional SNMP measurement of all. Here a host has been hacked and has launched a DoS attack outward bound. We know it is outward bound because this graph is taken from the switch’s point of view. For the switch, “in” means “out from the host” because traffic is coming into the switch port. Probably this host only has a 100 megabit Ethernet card; otherwise, the DoS attack would have been worse. (But it is still pretty bad.) A router CPU utilization graph, of course, does not tell which host launched the attack. But the correct switch port graph is a pretty useful giveaway. If nothing else, you can physical or remotely access the switch and disable the switch port.

SNMP setup pretty much follows our discussion about probes and analysis boxes in the previous section. Cricket runs on a collection (analysis) box and probes switches and routers with SNMP requests every 5 minutes. Results are made available on the Web as graphs. Information is baselined over a year. As a tool, Cricket has a nice setup that is object-oriented in terms of configuration commands. This allows bits of configuration that are more global to be easily applied to subsets of switch or router hosts.

In practice, it is a very good idea to put every router or switch port in an enterprise (and every router or switch that has an SNMP CPU utilization variable) into your SNMP configuration. As a result, by looking at graphs like those produced by Cricket, you might be able to actually find an internal attacking host. Sometimes the problem with an attack is that if you do not have other sources of information, you may not know the IP address of the attacker. (Netflow or ourmon in the next chapter might help here, but large DoS attacks can put some tools out of commission.) Worse, you might also not know where the attacking host is physically located. In extreme cases, network engineers have had to chase hosts down through a hierarchy of switches in wiring closets using a sniffer. Sometimes SNMP-based tools might be able to extract configuration labels from network interfaces in switches and routers and display them with the relevant graph. Thus labeling interfaces in switches and routers with location information, IP addresses, or DNSNAMES can be extremely useful in a crisis situation. This is especially important when you have a DoS attack, as in Figure 5.3. If this attack is headed out to the Internet, it can easily plug up a more external WAN circuit because WAN circuits typically have less bandwidth than internal Ethernet NICs. A host with a gigabit NIC launching an attack outward bound is both very possible and very traumatic for both you and any upstream ISP.

Firewalls and Logging

During the Blaster and Welchia worm outbreaks, the first signs of the outbreak were not picked up by our AV tools; rather, they were noticed in the firewall logs. The outbound traffic from these worms trying to recruit others was blocked and recorded by the firewall. In our daily examination of the previous night’s traffic, we noted a dramatic increase in the number of blocked messages, all on the same port. Because the information security profession had recently warned about the potential vulnerabilities, we knew exactly what it was as soon as we saw it. It was several days before our AV product began to detect the worm. The point is that firewall logs can be very useful in spotting infected hosts, especially when you are denying bad things from getting in or out. I am not a lawyer, but since there are firewalls to fit every size organization and budget, not having one is probably grounds for claims of negligence. This is the modern-day equivalent of a tug boat operator whose tug sank because he didn’t purchase a weather radio even after all of his colleagues had bought one. The argument of “having a high-speed pipe and therefore a firewall wouldn’t keep up” reminds me of a recent bumper sticker stating that “you should never drive faster than your guardian angel can fly.” It doesn’t matter how fancy your firewall is—whether it a host firewall, a commercial version, or just router-based access control lists (ACLs). If you just monitor them, you will see “interesting” traffic.

One thing, though, is that if you have been paying attention, you probably have noticed that the Internet is attacking you 24/7. Given that situation, it makes sense to watch your firewall or router ACL logs to see if you are attacking the Internet. For example, look at the following Cisco router log:

Be grateful. Only a few entries for this particular incident are shown; we deleted thousands more and have laundered the IP addresses. 192.168.1.1 is an infected internal “spambot” host trying to send spam outside the network, presumably to a list of external hosts elsewhere. It can’t connect, so all we see are TCP SYN packets aimed at port 25 on external hosts. Essentially the Cisco router spotted and stopped it from getting to the Internet. This is because port 25 for ordinary DHCP-using hosts inside the network was blocked. It is considered a best practice to require all outbound SMTP traffic to go through official e-mail gateways to get to the Internet. Blocking all other port 25 traffic will also give you a warning whenever a spambot takes up residence.

To reinforce this point, consider the following absolute barebones firewall policy in terms of botnet activity. Of course, it represents the past, but the past has a tendency to repeat itself. It also is not necessarily entirely botnet related, but it exemplifies malware still lurking on the Internet. For example, SQL-slammer at UDP, port 1434, is still out there waiting to get in:

By blocking these ports and logging the results, you can gain a warning when some of your internal hosts become infected. You can also configure the firewall to alert you when these occur, to improve your response time to these infestations.

Remember, this list is a minimum, but it is effective, given botnet attacks against Microsoft File Share (CIFS) and spammers as well as certain historic attacks. A local site with a small set of rules that falls into the bad security practice called “access all, deny a few” should also factor in local experience based on local incidents. On the other hand, if you are blocking nearly everything with the classic corporate firewall and you log the blocked traffic, you will see interesting things. If a bug gets loose on the inside, it might get loose again, due to either fan-out or the fact that once hackers discover a local hole, they could try to see if you repeated that hole elsewhere on your site. This is because infection may arrive over VPNs, mobile hosts (or USB earrings), email attachments, Web surfing, and even P2P applications. Firewall logging is an essential part of defense in depth.

Layer 2 Switches and Isolation Techniques

Layer 2, meaning Ethernet switches, might be a topic that most people do not consider very much or very long in terms of security. But some attacks can take advantage of weaknesses at Layer 2. For example, consider the popular Ettercap tool (http://ettercap.sourceforge.net), which fundamentally relies on attacks such as ARP spoofing or filling a switch forwarding table full of fake MAC addresses to enable password sniffing. (See www.securitypronews.com/securitypronews-24-20030623EtterCapARPSpoofingandBeyond.html for more discussion of Ettercap-based attacks.)

We need to define a few terms before we go on:

The next worst thing to having a malefactor standing physically next to a protected computer is to have the attacker within the same ARP broadcast range of a protected host. Until recently there has been little useful protection against some forms of attack in the same broadcast domain. One could also point out that ARP and DHCP as fundamental networking protocols lack authentication. Moreover, other protocols might assume that nearby hosts are “safe” and hence use plain-text passwords to contact those systems, or simply send in the clear data that’s possibly useful for identity theft.

Some have called having only a border firewall and no other defenses “M&M security,” meaning that the border firewall represents a hard, crunchy shell that, once pierced, leads to a soft, chewy middle. In a recent blog entry (http://blogs.msdn.com/larryosterman/archive/2006/02/02/523259.aspx), Larry Osterman took a rather humorous slant on this in comparing a DMZ firewall to the French Maginot Line in World War II. The French built a great defense wall to keep the Germans out. Unfortunately, the Germans simply drove north around it. The lesson is that it is reasonable to consider defense in depth for hosts within a firewall enclave. These techniques can include host firewalls and cryptographic protocols. They can also include Layer 2 techniques as one more form of defense in depth. The good news about Layer 2 techniques is that they are not per host but can be centrally administered by a network engineer.

Malware spread via botnets or other means could choose to launch attacks, including:

This is not an exhaustive list of Layer 2 attacks, but we will confine ourselves to this list for the time being, since the first two scenarios are more common in our experience.

So, do the good guys have any tricks up their sleeves? Yes, a few. The tricks can be divided into two categories: switch configuration, which must rely on vendor features, and infrastructure tricks, which hopefully can be done by any network engineer with most hardware.

Cisco switches have long supported a port security feature in a number of variations. For example, a switch can be configured to statically lock down a MAC address, or it can be configured to dynamically learn the first MAC address it sees. This makes flooding the switch table unlikely. A number of the switch configuration features are relatively new in the world and can be found in recent Cisco Catalyst switches. See Charlie Schluting’s excellent article, Configure Your Catalyst For a More Secure Layer 2, for more information: www.enterprisenetworkingplanet.com/netsecur/article.php/3462211. Schluting tells us that:

These new features, along with traditional port security, can help make the Layer 2 switched environment much safer.

From the infrastructure point of view, here are several techniques that could help security:

One final point is that switches can have logging as well. Logging based on various Layer 2 isolation violations can thus alert you to a hacked system.

Virus Detection on Hosts

How do you manage the botnet problem—or indeed, any security problem? Here’s a simplification of a common model describing controls for an operational environment:

You can see from this list that detection is only part of the management process. In fact, when we talk about detection as in “virus detection,” we’re often using the term as shorthand for an approach that covers more than one of these controls. Here we consider antivirus as a special case of a HIDS, but it doesn’t have to be (and, in enterprise terms, it shouldn’t be) restricted to a single layer of the “onion.” The antivirus industry might not have invented defense in depth or multilayering, but it was one of the first kids on the block (Fred Cohen: A Short Course on Computer Viruses, Wiley). In a well-protected enterprise, antivirus sits on the desktop, on laptops, on LAN servers, on application servers, on mail servers, and so on. It’s likely to embrace real-time (on-access) scanning at several of those levels, as well as or instead of on-demand (scheduled or user-initiated) scanning. It might include some measure of generic filtering (especially in e-mail and/or Web traffic) and should certainly include some measure of heuristic analysis as well as pure virus-specific detection (see the following discussion).

Nowadays full-strength commercial antivirus software for the enterprise normally includes console facilities for central management, reporting, and logging as well as staged distribution of virus definitions (“signatures”). Properly configured, these facilities increase your chances of getting an early warning of malicious activity, such as a botnet beginning to take hold on your systems. Look out for anomalies such as malicious files quarantined because they could not be deleted or files quarantined because of suspicious characteristics. Many products include a facility for sending code samples back to the vendor for further analysis. And, of course, antivirus products can be integrated with other security products and services, which can give you a better overview of a developing security problem.

Antivirus is often seen as the Cinderella of the security industry, addressing a declining proportion of malware with decreasing effectiveness and tied to a subscription model that preserves the vendor’s revenue stream without offering protection against anything but known viruses. What role can it possibly have in the mitigation of bot activity? Quite a big role, in fact, not least because of its ability to detect the worms and blended threats that are still often associated with the initial distribution of bots.

You should be aware that modern antivirus software doesn’t only detect viruses. In fact, full-strength commercial antivirus software has always detected a range of threats (and some nonthreats such as garbage files, test files, and so on). A modern multilayered enterprise antivirus (AV) solution detects a ridiculously wide range of threats, including viruses, jokes, worms, bots, backdoor Trojans, spyware, adware, vulnerabilities, phishing mails, and banking Trojans. Not to mention a whole class of nuisance programs, sometimes referred to as possibly unwanted programs or potentially unwanted applications. So why don’t we just call it antimalware software? Perhaps one reason is that although detection of even unknown viruses has become extraordinarily sophisticated (to the point where it’s often possible to disinfect an unknown virus or variant safely as well as detect it), it’s probably not technically possible to detect and remove all malware with the same degree of accuracy. A vendor can reasonably claim to detect 100 percent of known viruses and a proportion of unknown viruses and variants but not to detect anything like 100 percent of malware. Another reason is that, as we’ve already pointed out, not everything a scanner detects is malicious, so maybe antimalware wouldn’t be any better.

To this end, modern malware is distributed inconspicuously, spammed out in short runs or via backdoor channels, the core code obscured by repeated rerelease, wrapped and rewrapped using runtime packers, to make detection by signature more difficult. These technical difficulties are increased by the botherder’s ability to update or replace the initial intrusive program.

Snort as an Example IDS

Snort, written in 1998 by Martin Roesch, is often still described as a lightweight NIDS, though its current capabilities compare very favorably to heavyweight intrusion detection systems such as ISS RealSecure, Cisco’s Secure IDS, eTrust IDS, and so on. Snort is available for most common platforms, including Windows, Linux, BSD UNIX, Solaris, and Mac OS X. You can get the software at a very attractive price—well, free (it’s open source, to be precise). However, Sourcefire does market a commercial version (the Sourcefire Intrusion Sensor), which is based on the Snort detection engine but adds other components such as a friendlier interface, reporting, policy management, and a full support package (www.sourcefire.com).

Snort is claimed at the time of writing to have well over 150,000 active users and to have been downloaded over 3 million times (www.snort.org). Although the superiority of open-source software, especially in the security arena, is sometimes overstated, Snort is a fine example of how continuing review and testing by a community of experienced programmers and administrators can benefit a product.

Tripwire

Tripwire is an integrity management tool that was originally created by Professor Eugene Spafford and Gene Kim in 1992 at Purdue University, though the project is no longer supported there. In 1997, Gene Kim cofounded Tripwire Inc. (www.tripwire.com) to develop the product commercially, and the company continues to be a leading player in commercial change-auditing software for the enterprise, monitoring changes and feeding reports through enterprise management systems. However, the Open Source Tripware project at Sourceforge (http://sourceforge.net/projects/tripwire/) is based on code contributed by Tripwire Inc. in 2000 and is released under Gnu General Public License (GPL), so there is a clear line of succession from the original academic source release (ASR). See www.cerias.purdue.edu/about/history/coast/projects/ for more on the origins of Tripwire at Computer Operations Audit and Security Technology (COAST).

The original product has been described as an integrity-monitoring tool, using message digest algorithms to detect changes in files. This is under the assumption that such changes are likely to be due to illegal access by an intruder or malicious software. Although it was originally intended for UNIX systems and is widely used on Linux systems, Mac OS X, and so forth, it has been ported commercially to other platforms, notably Windows. Open Source Tripwire, however, is available only for POSIX-compliant platforms and has a more restricted range of signing options, for example. The commercial product range is nearer an integrated integrity management system.

Tripwire is also sometimes claimed to be an intrusion detection system. In a general sense, it is, though the tripwire detection concept is strictly reactive. It can tell you that there’s been a change that might be due to malicious action, but only once the change has been made.

The idea is to create a secure database (ideally kept on read-only media) of file “signatures.” In the midst of discussion about attack signatures, this use of the term signature might be confusing. It doesn’t refer here to attack signatures, the usual use of the term in intrusion detection. Instead, it refers to a set of encoded file and directory attribute information called a digital signature. The information is captured as a “snapshot” when the system is in a presumed clean state, the “signature” is in the form of a CRC, or cryptographic checksum.

“Secure” in the context of Tripwire signatures is a comparative term, however. In recent years a number of flaws in MD5 have been discussed that bring into question its continuing fitness for some applications. Although snefru is theoretically vulnerable to differential cryptanalysis, the attack is currently still considered practically infeasible.

If a subsequent snapshot comparison with the stored signature indicates that the file has been altered or replaced, this might give you your first warning of an attack. However, you can also use this facility, in tandem with other measures such as firewall logs and other system logs, to investigate and analyze a known breach or infection.

Clearly, Tripwire detects intrusion. It doesn’t, by itself, prevent it. Its purpose is to alert you to a breach that has already taken place and assist in analyzing the extent of that breach. Irrespective of the version of Tripwire you use, when you initialize the database by taking your first directory snapshot, you need the file system to be intact and clean. If it’s already been compromised, Tripwire is of very little use to you. Ideally, the system should just have been installed (what we used to call a “day-zero” installation, before the term zero-day became popular as a description of something more sinister).

Tripwire is an example of a defensive technique that has been referred to as object reconciliation, integrity detection, change detection, integrity checking, or even integrity management, though these terms are not strictly interchangeable. It was at one time seen as the future of virus detection, when the main alternative was exact identification of viruses, resulting in an inevitable window of vulnerability between the release of each virus or variant and the availability of detection updates. For a while, most mainstream antivirus packages included some form of change detection software, and many sites used it as a supplement to known virus detection. However, Microsoft operating environments became bigger, more sophisticated, and more complex, and the processing overhead from ongoing change detection and changes in the threat landscape meant that the range of places that a virus could hide grew fewer. It’s probable that the disappearance of change detectors from antivirus toolkits is as much to do with a lack of customer enthusiasm. Nonetheless, the continued popularity of Tripwire suggests that there is still a ready place for some form of change detection in security, especially in integrity management.

Exactly what is protected (or rather monitored; for full protection, you need to call on backups and/or reinstallation media) depends on which files and directories you configure it to monitor. In principle, it can be set to monitor every—or any—file or directory on a monitored system, not just system files and directory trees. In general, though, this can be counterproductive. Even on a server on which system files stay fairly static and contain no user data, you’ll need to make exceptions for files that are changed dynamically, such as log files. On a system that contains dynamic data, you need to set up a far more discriminating system.

Tripwire configuration and policy files are signed using the site key, whereas the database file and probably the report files are signed with the local key. Once the database is initialized and signed, Tripwire can be run from cron according to the settings in the configuration file, which specifies which files and directories are to be monitored and in what detail. Ignore flags specify the changes that are considered legitimate and that should generate an alert. In check mode, the file system objects to be monitored are compared to the signatures in the database: Apparent violations are displayed and logged and can also be mailed to an administrator. Apparent violations can, if found to be valid, be accepted by selectively updating the database.

Process

In the real world of computer forensics, each job begins with an ops or operations order that provides the details for managing the case as well as describing what you are expected to do. When gathering intelligence about botnet clients, you should do the same. Develop a naming convention for all case-related files and folders so that the mountain of data you gather can be useful two to three months later.

Each case is different, so in this section we will describe actions taken in a real botnet infestation. The basic ideas will be the same as presented here, but the problem-solving aspect will vary significantly.

In this infestation we got our first indication of its existence when a server began scanning for other recruits. Using the investigative techniques described here, we found, over a period of four months, 200+ botnet clients that were not detected by our network sensors. This infestation was either Rbot or Phatbot or both. Both of these botnet types use password-guessing attacks featuring the same list of default userids. They are both capable of exploiting other vulnerabilities, but it was the password-guessing attack that we detected.

Management made a decision very early in the incident response that we would not engage law enforcement unless the case met some pre-established criteria, such as:

However, for the majority of the cases, we performed a quick forensic, intended to extract information about the attack vectors, other infected systems, the botnet architecture (bot server, payload, functions, C&C method), and code samples that can be sent for further analysis. The steps we take in these cases are as follows:

To prepare for gathering this information, we prepared 1G USB memory sticks. We chose a set of very useful tools, mostly from the sysinternals tools located at www.microsoft.com/technet/sysinternals/default.mspx. In our tool chest, we included Process Explorer (now called Process Monitor),TCPView, Autoruns, Rootkit Revealer, and a small application called AntiHookExec (www.security.org.sg/code/antihookexec.html), which the author claims will let you execute an application in a way that is free from stealth application hooks. In other words, it lets them see hidden applications. Unfortunately, it works only with XP or newer operating systems. We also included a batch file (find.bat, described in Chapter 2), conveniently provided by the botherder and edited by us, that searched through the computer to locate where he had put his files. It seems that when you have thousands of computers to manage, you forget where you put things.

Next we chose a naming scheme for the folders that would be collected. This was an important step because the data was going to be collected by many people—some security staff but mostly help desk support and business liaison IT staff. Our folder-naming convention consisted of the computer name (the NetBios name of the computer), the date (in yymmdd format), and the help desk ticket number. Log files and picture images we created were named in the format Computer Name Date Description. So the security event log for a computer called Gotham that was gathered on December 27, 2006, would be called GOTHAM 061227 Security Event.evt. Within the main folder you want to make a distinction between files that actually existed on the computer and analysis files gathered about the computer (such as the files saved by Process Explorer).

Since we are not gathering the information as evidence, we can attempt to use the tools present on the computer with the caveat that the bot may interfere with the reliability of what we see. If we have external confirmation that a computer is part of the botnet, yet we find nothing during this examination, we perform an external virus scan of the hard drive using another system. In our case, we do a PXE boot of the system on an isolated network using a clean computer that is used only for virus scanning. We only do this if we find nothing on the computer, since the virus scanner will actually delete some of the intelligence data we are looking for. In our sample case, the intelligence data we were looking for was found on the computer, so we did not run a virus scan until after we completed the forensics.

First we open a help desk ticket. We use the RT ticketing system to track all virus infections. This permits us to know whether a system has been reinfected after it has been cleaned. The ticket first goes to the network team to place that computer’s network connection in a network quarantine area, to prevent further spread of the bot while permitting the user to do some useful work. Then we track down the computer and begin to gather the event logs and the virus scanner logs. The order of the data isn’t important. We chose this order to ensure that we had gathered the static data before we started chasing the interesting stuff.

Event Logs

The event logs are located in Windows or WINNT directory under %WinDir%system32config. These files end in .evt, but we have seen them with different capitalization schemes (.evt, .EVT, .Evt).

The security event log is controlled by the Local Policy | Audit Policy settings. For this type of analysis, the following policies should be set to success, failure:

In practice, we usually gather all the logs and then examine them one at a time in real time, then later analyze them in nonreal time. Here we describe the examination process as we tell how to locate each log. Use the Administrative tool and Event Viewer to examine the security event log. In the security event log you are looking first for failed logins (see Figure 5.4). You can sort the file by clicking the Type column. This will divide the log into successes and failures. In our case the entries of interest are the failed logins with a login type 3, the network login. You can find more information about the login types listed in the event log at http://technet2.microsoft.com/WindowsServer/en/library/e104c96f-e243-41c5-aaea-d046555a079d1033.msp, or search Microsoft for audit logon events.

Figure 5.4 Failed Login Record

In addition, we looked for instances of logon type 3 in which the originating workstation name differed from the victim’s computer and where the domain name is the name of the attacking computer. In most environments, this should be a rare occurrence. The victim’s computer would have to be actively sharing files and adding local accounts from the other computer as users on the victim’s computer.

To clinch the deal, password-guessing attacks occur much more rapidly than any human can type. This won’t be the case every time. The password-guessing tools we have captured can throttle down the attack frequency (x attacks over y hours), so it might not be so obvious (see Figure 5.5).

Figure 5.5 A Password-Guessing Attack

Both Phatbot and Rbot provide other clues that a password-guessing attack is real. Earlier in the book we listed the default userids they both can use. You might not see this in every attack, but if the bot hasn’t gathered any userids locally yet, or if the gathered userids haven’t gotten in, the bot might try userids from the default list. They almost always try Administrator, so if you have renamed this account, its appearance in a failed login attempt raises the probability that this is an attack. If you see attempts using userids of Administrador, then administrateur as the login ID, you can be sure that this is password-guessing attack and that a bot (likely Phatbot, Rbot, or another related bot family) is attacking the victim’s computer. If the attempts happen to take place during times that no one is supposed to be working in that department, you can be even more certain.

So, what’s the point of analyzing this data? You are examining this computer because someone already said it was virus infected or because one of your intelligence sources spotted it talking to a known C&C server. Here’s the value of this analysis: The computers listed in the workstation field of the failed login records type 3 login, where the workstation field differs from the victim’s computer name, are all infected computers. Using this technique during the analysis phase, we have found over 200 infected computers that were part of one botnet. This is despite the fact that we actively scan for bot C&C activity. This is defense in depth at its finest. However, that is during the analysis step, which we will cover later in this chapter. In this step we are trying to determine the attack vector, the time of the successful attempt, and the userid that successfully logged in (which should now be considered compromised).

Finding these failed login attempts tells us that password guessing was one of the attack vectors. Finding a successful login among the attempts using one of the attempted userids or immediately following the last attempt is valuable because it marks the time of the actual break-in. Take note of this time because you will use it later to look for files associated with the break-in (see Figure 5.6).

Figure 5.6 A Successful Break-in

During the analysis phase you can use a log processor such as Log Parser from Microsoft to process multiple log files at once. At the time of this printing, Log Parser can be downloaded from www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en. Log Parser reads the event files and permits the analyst to craft SQL queries to extract information.

We created a batch file containing a single line:C:“Program FilesLog Parser 2.2”LogParser.exe -o:CSV file:LogonFailuresDistinct2.sql?machine=*”

This line says, “Run log parser, read the file LogonFailures.sql, execute the SQL commands you find there, report what you find for all machines, and place the results in a comma-separated value file.”

The SQL query LogonFailures says:

FROM .logs2*.evtWHERE EventType = 16 AND EventCategory = 2 AND Attacking_Workstation <> ComputerName

This query will cause Log Parser to:

Log Parser is to do this from all the event logs in .logs for all logon events (Event Category 2) that failed (Event Type 2) and where the attacking workstation name doesn’t match the ComputerName field.

Table 5.1 shows a sample of output from this SQL query. You can see that attacks came from two computers, ATTACKER1 and ATTACKER2. ATTACKER2 shows the pattern consistent with an automated password-guessing attack, with attempts coming one a second for an hour. It is also a bit of a clue that there were 2200 attempts during that hour. You can also see that the attacker in our greatly modified example used a dictionary containing five passwords to try for each userid. When you consolidate all the logs like this for analysis, you can see the attack pattern. Find an attacker and then look for the attacker in the Victim column. You can note which computer infected that one and trace it backward in the Victim column, thus reconstructing the timeline of the spread of the botnet. This will often show the pattern called “fan out,” where the botnet infects a single computer in a new subnet, then that computer fans out to infect others in the same subnet. Using this technique we are able to turn the bot client attack vector into an intelligence source.

Table 5.1 Sample Output from Log Parser SQL Query

You can find basic explanations in the accompanying help file and by searching the Microsoft site for Logparser. There is also a much more in-depth treatment of uses of Log Parser in the Syngress book, Microsoft Log Parser Toolkit, written by Gabriele Giuseppini and Mark Burnett. Guiseppini is one of the Microsoft developers of the tool.

The computers listed in the Attacking Workstation column are the infected systems, unless you can discover a legitimate reason for the failed attempt to connect two workstations. For example, you might discover that a small group of workstations in a lab have set up shares between them, and users periodically connect workstations. For this reason, we include as much of the following information as we can in the help desk ticket for this incident:

We discovered that it was necessary to know what was solid information (found in the logs) and what was derived (e.g., IP address from NSLookup of computer name). The time last observed is important, especially in environments using DHCP, since you are only interested in the computer that held a particular IP address during the time of the event observed in the logs. In our case, the lookup table we used for building, room number, and jack number was horribly out of date and consequently inaccurate. If the computer was online, the networking team could confirm the room number and data jack by reading the switch that detected the computer. The most difficult part of this process proved to be matching the infected machine with a user and location.

Several critical pieces of our infrastructure are missing. There is no asset management system, so the asset database is not linked to the help desk system. The database that links the building room and data jack information to a switch port has not been kept up to date. The building maps to room and data jacks haven’t been kept up to date, so we keep sending techs out to rooms that no longer exist. There is no simple way to correlate the computer’s NetBios name to its IP address and MAC address. Although there is a standard naming convention for computers, it is loosely followed by other departments. It is next to impossible to find a computer of the name LAPTOP in a population of 27,000 users. In XP, the security event log record only contains the computer NetBIOS name, not the IP address; the way our DNS is setup, few of these NetBIOS names are found using nslookup.

Under these circumstances, we have had to find creative ways to locate these infected computers. If the userid has portions of a name, we try student and faculty records to see if there is a match or a short list of candidates. Sometimes the computer name is somewhat unique, and a search of the university’s Web pages can win the prize. One tough case was a computer called ELEFANT. Searching through the university’s Web pages revealed a Web page for the chemistry department’s lab network that touted ELEFANT as the most important computer in their lab. The Web page also identified the lab manager’s name, phone number, and e-mail address.

Once we are confident in the IP address associated with an attacker, the help desk ticket is assigned to our networking group. The networking group places the switch port associated with the attacker into a network jail, although our kindler, gentler customer service interface calls it a “network quarantine” when speaking to our customers. The networking group then confirms the building and room information directly from the switch, to confirm the data base entries we posted earlier.

Once the computer’s location has been deter mined, the help desk ticket is assigned to our desktop support techs, who arrange for it to be retrieved for our quick forensic exam and reimaging. We had determined early in the process that with this bot, reimaging was preferable to attempting to remove the virus and chancing that we would miss something. Reimaging also gave us the opportunity to remove the offending local administrator accounts.

As we processed systems, we realized that we needed to collect and correlate information about all the systems we had identified. For that we established a spreadsheet that brings together all the relevant information. That way, if we see a system in an event log two months from now, we can confirm whether the system was reimaged since the time of the new sighting or if this is a reinfection.

We are now experimenting with using a tool called NTSyslog, available for download at http://sourceforge.net/projects/ntsyslog, to automatically forward the Security Event logs to a central syslog server. The central syslog server formats the data for an SQL database and then will run the above query in near real time. This has the effect of turning this approach into an early warning tool instead of a recovery tool.