Chapter 5

Intrusion-Detection Systems

IDSs have become much more widely used in the last few years. An IDS inspects all inbound and outbound port activity on a machine/firewall/system and looks for patterns that might indicate an attempted break-in. For example, if the IDS finds that a series of packets were sent to each port in sequence from the same source IP address, this probably indicates that a system is being scanned by network-scanning software such as Cerberus (scanners are discussed at length in Chapter 12, “Assessing System Security”). Since this is often a prelude to an attempt to breach a system’s security, it can be very important to know that someone is performing preparatory steps to infiltrate a system. The IDS may also detect an abnormally large flow of packets from the same IP address, all in a brief period of time. This may indicate a DoS attack. In either case, these are situations the network administrator should be aware of and should take steps to prevent.

Let’s begin our discussion of intrusion-detection systems with a look at the concepts. How do IDSs work? Well, to explore this, we must first review how networks function. Historically, when IDSs were first developed, hubs were used very frequently. Today, switches are used rather than hubs. With a hub, after a packet has travelled from its source network to the destination network (being routed by its destination IP address), it finally arrives at the network segment on which the target is located. After it gets to that final segment, the MAC address is used to find the target. All the computers on that segment can see the packet, but because the destination MAC address does not match the MAC address of their network card, they ignore the packet.

At some point, enterprising individuals realized that if they simply chose not to ignore packets not destined for their network card, they could see all the traffic on the network segment. In other words, one could look at all the packets on that network segment. Thus the packet sniffer was born. After that it was just a matter of time before the idea came about of analyzing those packets for indications of an attack, thereby giving rise to intrusion-detection systems.

This sort of intrusion detection and avoidance can be quite complicated, and there is the potential of blocking a legitimate user by mistake. The complexity arises from distinguishing legitimate traffic from that indicative of an impending attack. This can lead to the problem of false positives, in which the system mistakenly identifies legitimate traffic as some form of attack. Usually, a software system will simply alert the administrator that suspicious activity has taken place. A human administrator will then make the decision whether or not to block the traffic. If the software automatically blocks any addresses it deems suspicious, you run the risk of blocking out legitimate users. It should also be noted that nothing prevents the offending user from moving to a different machine to continue his or her attack. This sort of approach should only be one part of an overall intrusion-detection strategy and not the entire strategy.

Threshold monitoring

Resource profiling

User/group work profiling

Executable profiling

Executable profiling enables the IDS to identify activity that might indicate an attack. Once a potential danger is identified, the method of notifying the administrator, such as by network message or e-mail, is specific to the individual IDS.

An activity is an element of a data source that is of interest to the operator.

The administrator is the person responsible for organizational security.

A sensor is the IDS component that collects data and passes it to the analyzer for analysis.

The analyzer is the component or process that analyzes the data collected by the sensor.

An alert is a message from the analyzer indicating that an event of interest has occurred.

The manager is the part of the IDS used to manage, for example a console.

Notification is the process or method by which the IDS manager makes the operator aware of an alert.

The operator is the person primarily responsible for the IDS. This is often the administrator.

An event is an occurrence that indicates a suspicious activity may have occurred.

The data source is the raw information that the IDS uses to detect suspicious activity.

Beyond these basic components, IDSs can be classified either based on how they respond to detected anomalies or based on how they are deployed. An active IDS, now called an IPS (intrusion prevention system), will stop any traffic deemed to be malicious. A passive IDS simply logs the activity and perhaps alerts an administrator. The problem with IPS/active IDS is the possibility of false positives. It is possible to have activity that appears to be an attack, but really is not.

You can also define IDS/IPS based on whether a single machine is monitored or an entire network segment is monitored. If it is a single machine, then it is called a HIDS (host-based intrusion-detection system) or HIPS (host-based intrusion prevention system). If it is a network segment then it is called a NIDS (network-based intrusion-detection system) or NIPS (network-based intrusion prevention system).

Snort works in one of three modes: sniffer, packet logger, and network intrusion-detection.

Snort works primarily from the command line (Shell in Unix/Linux, command prompt in Windows). Configuring Snort is mostly a matter of knowing the correct commands to enter and understanding their output. Anyone with even moderate experience with either Linux shell commands or DOS commands can quickly master the Snort configuration commands. Perhaps Snort’s greatest advantage is its price: It is a free download. For any organization to not be using some IDS is inexcusable when a free product is available. Snort is a good tool when used in conjunction with host-based firewalls or as an IDS on each server to provide additional security.

There are a number of products in this group, notably the Firepower 4100 series, the Firepower 8000 series, and the Firepower 9000 series. All the products include malware protection as well as sandboxing. These Cisco products also integrate cyber threat intelligence features. The Firepower 4100 is shown in Figure 5-1.

The 4100 series is meant for smaller networks. The 9000 series is designed for large-scale networks. You can see the Firepower 9000 series in Figure 5-2.

One of the chief benefits of using Cisco security products is their widespread use across the industry and the availability of good training. The fact that so many organizations use Cisco indicates a high level of successful field testing, which generally indicates a reliable product. Cisco also sponsors a range of certifications on its products, making it easier to determine whether someone is qualified on a particular Cisco product.

The underlying premise of the honeypot is that any traffic to the honeypot machine is considered suspicious. Because the honeypot is not a real machine, no legitimate users should have a reason to connect to it. Therefore, anyone attempting to connect to that machine can be considered a possible intruder. The honeypot system can entice him to stay connected long enough to trace where he is connecting from. Figure 5-3 illustrates the honeypot concept.

Specter works by appearing to run a number of services common to network servers. In fact, in addition to simulating multiple operating systems, it can also simulate the following services:

SMTP

FTP

TELNET

FINGER

POP3

IMAP4

HTTP

SSH

DNS

SUN-RPC

NETBUS

SUB-7

BO2K

GENERIC TRAP

Even though Specter appears to be running these servers, it is actually just monitoring all incoming traffic. Because it is not a real server for your network, no legitimate user should be connecting to it. Specter logs all traffic to the server for analysis. Users can set it up in one of five modes:

Open: In this mode the system behaves like a badly configured server in terms of security. The downside of this mode is that you are most likely to attract and catch the least skillful hackers.

Secure: This mode has the system behaving like a secure server.

Failing: This mode is interesting in that it causes the system to behave like a server with various hardware and software problems. This might attract some hackers because such a system is likely to be vulnerable.

Strange: In this mode the system behaves in unpredictable ways. This sort of behavior is likely to attract the attention of a more talented hacker and perhaps cause her to stay online longer trying to figure out what is going on. The longer the hacker stays connected, the better the chance of tracing her.

Aggressive: This mode causes the system to actively try and trace back the intruder and derive his identity. This mode is most useful for catching the intruder.

In all modes, Specter logs the activity, including all information it can derive from the incoming packets. It also attempts to leave traces on the attacker’s machine, which can provide clear evidence should civil or criminal action later be required.

Users can also configure a fake password file in all modes. These are particularly useful because most hackers attempt to access a password file to crack the passwords. If they are successful they can then log on as a legitimate user. The holy grail of hacking is getting the administrator’s password. There are multiple ways to configure this fake password file:

Easy: In this mode the passwords are easy to crack, leading a would-be intruder to believe that she has actually found legitimate passwords and usernames. Often a hacker with a legitimate logon will be less careful covering her tracks. If you know that logon is fake and the system is set up to monitor it, you can track it back to the hacker.

Normal: This mode has slightly more difficult passwords than the easy mode.

Hard: This mode has even harder passwords to crack. There is even a tougher version of this mode called mean, in which the passwords are very difficult to break so that the hacker can be traced while he is taking time to crack the passwords.

Fun: This mode uses famous names as usernames. In my opinion this one, and the related one named Cheswick, have dubious security value.

Warning: In this mode the hacker gets a warning telling him he has been detected if he is able to crack the password file. The theory behind this mode is that most hackers are simply trying to see if they can crack a system and do not have a specific objective. Letting this sort of hacker know he has been detected is often enough to scare him off.

The cost of this software system is about $900, and it requires a PC to install it on. The purpose of honeypots like Specter is not preventing intrusion. Instead, they minimize the damage once someone is in. They serve to direct the hacker’s attention away from critical systems. They also can be very helpful in tracking down hackers.

As the Decoy Server works as a honeypot, it also works as an IDS monitoring the network for signs of intrusion. If an attack is detected, all traffic related to that attack is recorded for use later in whatever investigative, criminal, or civil procedures that may arise.

Decoy Server is designed to be part of a suite of enterprise security solutions that work together, including enterprise versions of Symantec’s antivirus software, firewall software, and antispyware. The product is usually purchased as part of a volume licensing agreement for a complete security package.

This is often done by using what is commonly referred to as a honeypot. Essentially, you set up a fake system, possibly a server that appears to be an entire subnet. The administrator makes that system look attractive to hackers, perhaps making it appear to have sensitive data, such as personnel files, or valuable data, such as account numbers or research. The actual data stored in this system is fake. The real purpose of the system is to carefully monitor the activities of any person who accesses the system. Because no legitimate user ever accesses this system, it is a given that anyone accessing it is an intruder.

This sort of system can be difficult to set up and maintain. It also presupposes that someone is able to successfully compromise security. Intrusion deflection systems are typically only employed at sites requiring very high security. They should only be a part of the overall IDS strategy—not the entire strategy.

Because this approach costs almost nothing to implement and is relatively easy to set up, it is a good option for any system when used in conjunction with other strategies.

To implement this strategy, warn the user at every step in the process of connecting that her activities are being closely monitored, whether they are or are not. In addition, avoid advertising that the system or machine contains sensitive data by giving it an innocuous name. For example, a database server that contains research material might be named “print_server 1” rather than “research_server” to make it less attractive. When using this type of naming approach, maintaining a master list and developing a naming scheme is important. For example, all real print servers might end with X and all false print server names end in Y so that staff know that “print_server1x” is a real print server and “print_server1y” is actually a sensitive server being hidden from intruders. Some way must exist for keeping track of the real purpose of the servers.

The purpose of the multiple warnings is to scare off less skilled hackers. Although such people might not have a great deal of technical prowess, their attempts to invade a system are a nuisance and can cause problems. Many of these attackers are new to hacking and appropriate warnings can scare off a significant percentage of them.

A complete IDS solution should have a perimeter IDS working in conjunction with a perimeter firewall. The most complete IDS solution includes multiple sensors for each subnet. Ideally, an administrator places some IDS on each major server and implements a honeypot solution.

Clearly, such a level of expenditure and complexity is not possible in all circumstances. This level certainly provides the greatest security, but many organizations do not require, nor can they afford, this level of security. At a minimum, an organization should have an IDS running with the perimeter firewall. Because free IDS solutions are available, there is no reason not to have one.