Essentials First: Intrusion Detection

Networks of all sizes are designed to enable the sharing of information, and only rarely is security a part of that design. Many businesses are leveraging IP-based networks, such as the Internet, to bring remote offices, mobile workers, and business partners into their trusted internal network environments. The Internet is continuously growing and connecting more and more places; as it becomes increasingly reliable, companies can redefine how corporate applications function. The clearest example is how almost everything is becoming based on HTML. Although this enables businesses to have broader interaction with customers, streamline operations, reduce costs, and increase revenues, it also comes at a price and with risks.

The reach and openness that make the Internet such a powerful business tool also makes it a tremendous liability. Simply put, the Internet was designed to connect and share, not to secure and protect. This bears repeating: The Internet was not designed to secure and protect—period—the Internet is a web connecting the world together; it is not a super highway with law enforcement.

The websites and portals that welcome remote sites, mobile users, customers, and business partners into the trusted internal network might also be welcoming attackers who would misappropriate network resources for personal gain. As discussed in Chapter 10, “Wireless Security,” the growth of wireless networks is compounding this problem.

The question becomes this: How are these mission-critical communications protected from an inherently insecure medium such as the Internet? This book covers various means of increasing the security of these resources by adding layers of protection. The most common layers of security in a network are an Internet router prescreening packet and a stateful firewall. However, your organization has both a web and email server that must be accessible from the Internet to function. You cannot block this traffic because your business depends on it. You also know that, as the Internet has grown, so have the sophistication of the attacks; however, the knowledge level required to conduct these attacks has decreased, as shown in Figure 11-1.

Figure 11-1 Attack Sophistication and Attacker Skills

Neither the router nor the firewall can tell you whether that WWW packet actually contains an attack or a customer request; unfortunately, many people have placed their trust in these devices, which can fall short in the detection arena. Perhaps your organization has a talented system administrator who is trusted to secure and lock down business-critical servers or implement thorough security policies and procedures. None of the security solutions discussed so far address the need to detect attacks or intrusion attempts!

In Internet terms, IDSs are rather young; research began in the 1980s with the efforts and writing of Anderson and Denning. In the 1980s, the government first began using basic IDS functionality on what was then still the ARPANET. Late in the 1980s, members of the Haystack Project formed Haystack Labs as a commercial venture into developing host-based intrusion detection. Network-based intrusion detection followed in the 1990s with Todd Heberlein leading the charge. By then, several organizations were developing IDS tools, Haystack Labs, SAIC; in 1993 the United States Air Force implemented Automated Security Incident Measurement Systems (ASIM), and the team that developed this solution formed the Wheel Group in 1994, as shown in Figure 11-2.

Figure 11-2 IDS Development Timeline

This is relevant to the discussion because in 1994, Cisco purchased the Wheel Group; this acquisition formed the core of the IDS and security services.

IDS Functional Overview

Whether an attacker’s motive is intellectual challenge, espionage, political, financial, or even just to make trouble; your network will face an attack. Not only is it common sense to monitor these attacks, but in many cases, it is also a business imperative. Starting in the early 1990s, new products began to appear to deal with this aspect of network security: intrusion detection systems (IDS). An IDS is like an alarm system for your network. The network is protected, but without the IDS (alarm), you would never know whether an attacker was trying to get entry. The goal of intrusion detection is to monitor network assets to detect unusual behavior, inappropriate activity, and attacks, or stop the attack intrusion and even provide information to prosecute the attacker.

IDSs that are available on the market today promise a plethora of feature sets and capabilities. In evaluating an IDS for your organization, the following capabilities should generally be the focus, beyond traditional event logging:

• Event correlation: When an IDS is deployed in a busy network with multiple IDSs, the ability to correlate events (attacks) is crucial to ensure that your network is secure. Consider that an attack could span multiple segments as one host is compromised and then used to attack another, and so on. Without proper event correlation, this attack could cause great confusion and lead to many hours of wasted resources attempting to isolate the cause of the outage. Event correlation enables the IDS administrator to quickly track down and relate events that occur across multiple sensors deployed in different subnets, or perhaps in different geographical locations and over extended periods of time.

• Centralized sensor management: Having an IDS correlate events is important, and having all the IDS managed via centralized management is just as critical. In the real world, every device (server, router, and firewall) creates logs; however, they are rarely checked, let alone reviewed. Therefore, having a centralized management platform that enables event correlation and response control over multiple sensors and the ability to run detailed reports on your network’s security is crucial for success.

• Customizable signatures and thresholds: Company or business-specific applications, software upgrades, new operating systems, viruses, and intelligent hackers are always looking for and discovering new vulnerabilities. There is always a delay from the time a new vulnerability is discovered and when IDS developers release a new signature that detects the attack used to exploit the vulnerability. Therefore, an IDS must provide administrators with the ability to create attack signatures to deal with any eventuality.

• Elimination of false positives: Just like every operating system (such as Windows) that comes with all the features enabled, so do IDS devices. In other words, they are overly sensitive out of the box and provide a lot of false positives, thereby resulting in fear, uncertainty, and doubt (FUD) about your network security. You can understand, then, that every good IDS must have the capability to eliminate false positives. Having too many alarms sounding and gathering too much information can be a hazard in and of itself. An actual attack can be overlooked by getting drowned in a log-file. Take caution; however, to eliminate only a rule or feature set if you are sure of its impact. If you are sure, wait 24 hours after implementation and review. Double-check that you are capturing the data you want to capture.

• Standards-based implementation: An important aspect of deploying any technology is choosing a standards-based implementation. Many vendors create products that perform wonderful security services, but few are interoperable or provide the framework for future implementations. An IDS is no exception to this rule, and few standards currently exist. Because the most important aspect of integrating an IDS and managing it are its reporting capabilities; a standard has emerged based on the Common Vulnerabilities and Exposures (CVE) database. The CVE database both classifies and groups vulnerabilities into an easily referenced system. CVE compatibility is important for IDS because it provides reporting capabilities that far surpass the typical cryptic reporting historically found in IDS. By integrating CVE-compatible IDSs, organizations can use other CVE-compatible tools, such as vulnerability assessment (VA) tools, to further enhance the accuracy and criticality of event reporting. CVE has become widely adopted and will continue to be a standard method of reporting and classifying network security events (http://cve.mitre.org/cve/).

• Intrusion prevention functionality: Intrusion prevention is essentially the ability to actively respond to and prevent intrusions and unwanted traffic. The term intrusion prevention has recently been the subject of much confusion and is often marketed as a competing technology to intrusion detection; however, the reverse is true. In today’s market, an IDS must support the capability to actively respond to suspected threats.

• Signature matching: Monitors all traffic traversing a network and then matches each packet or series of packets with known attack patterns (signatures). The IDS then responds either passively or actively to that event. The response can vary from generating an SNMP alarm, crafting an email alert, or actively stopping the attacker from completing the attack (also referred to as intrusion prevention).

• Anomaly detection: Enables an IDS to establish a baseline of normal traffic patterns and information flows, and then respond whenever the normal thresholds are exceeded (for example, if a new protocol is detected on a network). Anomaly detection becomes most effective when it’s coupled with protocol decoding, whereby the IDS knows what normal behavior is expected within certain protocols and responds if abnormal commands or requests are detected.

Despite a common misconception, an IDS cannot monitor everything. Having an IDS as a layer in your overall security plan is a good idea; however, depending on it as an end-all, be-all security solution is a bad idea. It is part of your network security model, one layer among what should be several layers.

You can deploy an IDS in a variety of locations within a network to further increase an organization’s security and protection. There are four flavors of IDSs:

• Host-based: Monitors the characteristics of a single host and the events occurring within that host for suspicious activity (for example, network traffic, system logs, running processes, application activity, file access and modification, and system or application configuration changes). These systems are most often deployed on critical hosts such as publicly accessible servers and servers containing sensitive information. Use specialized software applications, called agents, that are installed on a computer (typically a server) to watch all inbound and outbound communication traffic to and from that server and to monitor the file system to identify unauthorized, illicit, and anomalous behavior. The installed agent uses a combination of signatures, rules, and heuristics to identify unauthorized activity. Host-based intrusion detection systems (HIDS) are extremely effective on mission-critical, Internet-accessible application servers, such as web or email servers, because they can watch the applications at its source to protect them. Just like the network-based IDS (NIDS), the role of a HIDS is passive.

• Network-based: Monitors network traffic for particular network segments or devices and analyzes the network and application protocol activity to identify suspicious activity. It can identify different types of events of interest. It is most commonly deployed at a boundary between networks (that is, border firewalls, routers, VPN servers, remote access servers, and wireless networks). NIDS deal with information passing on the wire between hosts. They are typically referred to as packet sniffers and reside directly on the network and watch all the traffic that traverses the wire. NIDS are effective at both watching for inbound or outbound traffic flows and traffic between hosts on or between local network segments. NIDSs are typically deployed in front of and behind firewalls and VPN gateways to measure the effectiveness of those security devices and interact with them to add more depth to the network’s security. NIDS use a network tap, port span, or hub to collect packets traveling over a given network. As stated before, an IDS does not actively block network traffic; its role is passive: gathering, identifying, logging, and alerting.

• Wireless: Monitors wireless network traffic and analyzes it to identify suspicious activity involving the wireless networking protocols. It cannot identify suspicious activity in the application or higher-layer network protocols (TCP and UDP) that the wireless network traffic is transferring. It is most commonly deployed within range of an organization’s wireless network to monitor it.

• Network Behavior Analysis (NBA): Examines network traffic to identify threats that generate unusual traffic flows, such as distributed denial-of-service (DDoS) attacks, certain forms of malware, and policy violations. These systems are most often deployed to monitor flows on an organization’s internal network and are also sometimes deployed where they can monitor flows between an organization’s network and the external network.

All four classifications of IDS implementation offer different techniques for detecting and deferring malicious activity, and you can deploy combinations of these four to provide the most effective enhancement to a layered defense strategy. Table 11-1 lists some examples of the various types of IDS by type, what they are commonly used for, their strengths, and some suggestions as to available products.

Table 11-1 Types of Intrusion Detection Systems

These various types of IDs solutions should be deployed together to provide a truly effective layered defense with visibility into, and control of, an organization’s communications. IDSs also provide organizations a check and balance on the effectiveness of their security systems and the overall effectiveness of their security dollars. The next section discusses the overall capabilities of an IDS.

Host Intrusion Detection System

Host Intrusion Detection Systems (HIDS) monitor, detect, and respond to user and system activity and attacks on a given host. In contrast to NIDSs, HIDSs are installed on the host (for example, the web or email server) to be monitored. HIDS monitors the host’s audit and event logs, whereas a NIDS monitors packets. Rather than trying to identify packets’ contents versus attack signatures, the HIDS approach attempts to identify known patterns of local or remote users doing things they should not be doing.

HIDSs act much like antivirus software (however, they are not a replacement for it) with extended capabilities that greatly increase the level of security that can be provided. HIDSs are best suited to combat security threats against hosts because of their capability to monitor and respond to specific user actions and file accesses on the server. The majority of computer threats come from within organizations, from many different sources such as disgruntled employees or corporate spies. HIDSs monitor servers by providing information about the following:

• Intrusion attempts or successes and suspicious behavior by authorized users.

• Scans of the host to ensure that they conform to accepted security practices such as having all the latest patches and not having unnecessary services running.

• Audit policy management and centralization, supply of host-based forensics, statistical analysis and evidentiary support, and, in certain instances, some measure of access control. More robust tools typically provide these functions.

The deployment of HIDS is fairly straightforward; it is an application that resides on a server that watches for file system changes, registry changes, open ports, running applications, and all traffic originating to and from the host on which it resides. Server farms are often placed on their own network, and application servers are strong candidates for HIDSs.

Where multiple hosts are concerned, HIDSs should be configured to report to a centralized management console to provide event correlation and enterprisewide reporting. Typical candidates for HIDSs deployments are web servers, file servers, or any application server that provides network communication resources to the public Internet.

To get complete coverage at your site using a host-based intrusion detection system, you need to load the IDS software on every computer. Following are two primary classes of host-based intrusion detection software:

• TCP, or host, wrappers/personal firewalls: A TCP wrapper is an access control list (ACL) system used in host-based networking. It is used to filter network access to Internet protocol servers on Linux or BSD operating systems. It enables host, or subnetwork, IP addresses, names, and indent query replies to be used as tokens to filter for access control purposes. Host wrappers or personal firewalls can be configured to look at all network packets, connection attempts, or login attempts to the monitored machine. This can include, but is not limited to, dial-in attempts or other nonnetwork-related communications ports.

• Agent-based software: Host-based software agents can monitor accesses and changes to critical files and changes in user privilege.

Either approach is more effective in detecting trusted-insider attacks than a network-based IDS, and both are more effective for detecting attacks from the outside. Table 11-2 lists the more popular wrapper packages and agent-based software.

Table 11-2 HIDS Detection Software

Network Intrusion Detection System

Network intrusion detection systems (NIDS) sit and “capture” all the packets on the network segment to which they are connected. This reading is similar to a packet sniffer; however, the differences appear after the packets are captured or sniffed. NIDSs are built on the wiretap concept and can be implemented in a couple different ways. These methods have been developed to deal with the prevalence of LAN switches and how they operate to isolate traffic. An IDS must see as much of the network traffic as possible to be effective. The different NIDS implementation methods are as follows:

• Inline wiretap: This method of capturing packets places a physical tap in between (that is, inline between) two network devices. The NIDS would be plugged in to this tap.

• Port mirroring: Depending on the switch you use, port mirroring, also known as port spanning, is perhaps a more flexible solution. This technique tells the switch to send to another port copies of every packet that, for example, is to be sent to the port your firewall is plugged into. The NIDS connects to this mirrored port.

Some NIDSs look for a fingerprint match by comparing the packet to the attack signatures it has in its database, whereas others look for unusual packet signatures indicating an attack is in progress. The NIDS inspects the packets as they pass through a sensor. The sensor can see only the packets that happen to be carried on the network segment it’s attached to. Packets are considered to be of interest if they match a certain signature. Depending on the NID you implement, the packets are analyzed against a variety of signatures. Overall, there are three primary types of signatures:

• String signatures: Look for a text string that indicates a possible attack. This has a tendency to produce false positives. You can refine the string signature to reduce the number of false positives by using a compound string signature. A compound string signature for a common UNIX-based web server attack might be “cgi-bin” AND “aglimpse” AND “IFS”.

• Port signatures: Watch for connection attempts to well-known, frequently attacked ports. Examples would be telnet (port 23), ftp (port 21/20), SUNRPC (port 111), and IMAP (port 143). If any of these ports aren’t used by your site, incoming packets to these ports should be considered suspicious activity.

• Header condition signatures: Watch for dangerous or illogical combinations in packet headers. The most famous example is Winnuke, where a packet is destined for a NetBIOS port and the Urgent pointer or Out-of-Band pointer is set. This results in the dreaded Blue Screen of Death (BSoD) for Windows-based systems. Another well-known header signature is a TCP packet with both the SYN and FIN flags set, signifying that the requestor wants to start and stop a connection at the same time.

Some issues relate to scalability and timeliness that the IDS industry is still trying to overcome. NIDS have had some trouble scaling as network speeds have increased, and with Gigabit Ethernet making inroads to networks of all sizes, it will not be long before 10-Gigabit speeds will be used. Of course, NIDS want to capture every packet and analyze its contents; this makes these new speeds a bottleneck that has not yet been completely solved. In addition, the updating of attack signatures is not yet close to being where it should be to detect the latest attacks. It is clear that IDS vendors and how they update signatures are still a far cry from the timeliness the antivirus community has achieved.

NIDS deployment is entirely based on the existing network design and architecture in place at each location. The more network segments a network has usually determines the number and placement of NIDSs.

Traditional NIDS placement enables them to be the most effective on the network perimeter, such as on both sides of the firewall (internal and external), near the VPN appliance server, and on links to business partner networks. This placement enables an organization to measure the real effectiveness of its prescreening routers and firewalls. These links tend to be low bandwidth (T1 speeds) such that an IDS can keep up with the traffic. This provides a good measure of checks and balances and is ideal for the security-aware organization, where application servers behind the firewall are accessible to the public Internet. Another high-value point is the corporate WAN backbone. A frequent problem is hacking from “remote” areas of the network into the main corporate network. Because WAN links tend to be low bandwidth, NIDS can be extremely beneficial.

Security best practice says that, when considering an IDS solution, both internal and external NIDSs should be used. This enables the NIDSs to monitor attacks from the Internet and internal threats. It might seem a bit odd to have two NIDSs; however, remember that statistically, the majority of attacks come from internal sources. Neglecting either location reduces the effectiveness of the IDS solution and greatly decreases your network’s security.

Wireless IDS

Like the dragon Smaug, you’re vulnerable in your underbelly. And unless you protect your weakest spot, you, too, will be felled with a carefully aimed black arrow. With the advent of wireless technologies and the saturation of wireless devices on the commercial market, it is more important than ever before to secure your wireless network and to ensure no one can break through this oh-so-big chink in your armor. A wireless IDS helps protect this spot by monitoring wireless network traffic and analyzing its wireless networking protocols to identify suspicious activity involving the protocols.

A wireless IDS is similar to a network-based IDS in many ways; they share the same components and those components have essentially the same functionality; the difference being the sensors. Wireless IDS sensors perform the same basic role as network IDS sensors, but they function differently because of the complexities of wireless communications. A major difference, however, is what the wireless IDS can monitor. A wireless IDS works by sampling traffic within two frequency ranges (2.4 GHz and 5 GHz) and each band is separated into channels. (802.11b and g support 14 channels and 802.11a supports 12 channels.) Currently a sensor cannot simultaneously monitor all traffic on a band; a sensor must monitor a single channel at a time.

Wireless sensors are available in three different ways:

• Dedicated: A device that performs wireless IDS functions but does not pass network traffic from source to destination. A dedicated sensor is often completely passive, functioning in a radio frequency (RF) monitoring state to “listen” to, or sniff, wireless traffic. Dedicated sensors can focus on detection and do not need to carry wireless traffic; they offer stronger detection capabilities than wireless sensors bundled with APs or wireless switches. That being said, the dedicated sensors may be cost-prohibitive to your organization; purchasing, installing, and maintaining a dedicated switch is much more than implementing a set of bundled sensors that can be installed on existing hardware.

• Bundled with an access point (AP): Typically provide less rigorous detection capability than a dedicated sensor. This is because the AP must split its time between doing its job and monitoring traffic on multiple channels or bands.

• Bundled with a wireless switch: Typically used to help administrators manage and monitor their wireless devices. Those wireless switches that do offer an IDS function typically do not offer detection capabilities as strong as bundled APs or dedicated sensors.

Wireless IDSs have several security capabilities that traditional network-based IDSs provide, such as information gathering, logging, detection, and prevention. Because wireless IDS technology is relatively new, these capabilities vary greatly between vendors.

Network Behavior Analysis

A network behavior analysis (NBA) tool examines network traffic to identify threats that generate unusual traffic flows, such as distributed denial-of-service (DDoS) attacks, certain forms of malware, and policy violations. These systems are most often deployed to monitor flows on an organization’s internal network and are also sometimes deployed where they can monitor flows between an organization’s network and the external network. An NBA solution typically uses both sensors and consoles; some products on the market also offer an NBA solution with a management server, called an analyzer. NBA sensors are typically only available as hardware appliances. These sensors are similar to network-based IDS sensors in that that “listen,” or sniff, packets to monitor network activity on a single segment. Other NBA sensors have the capability to monitor the entire network, but they rely on network flow information gathered from routers and other networking devices. Flow consists of the following components:

• Source and destination address

• Source and destination ports (TCP or UDP)

• Number of packets and bytes transmitted

• Session time stamps

NBAs have many of the same security capabilities as the other IDS classifications:

• Information gathering

• Logging

• Detection

The major difference lies in what the NBA detects. NBA technologies have the capability to detect malicious activity using, primarily, anomaly-based detection with some stateful protocol analysis. Using this unique, nonsignature-based detection array, an NBA can detect DoS attacks, scanning, worms, tunneled protocols, backdoors, and policy violations.

NBAs also provide limited intrusion prevention capabilities depending on sensor type. A passive sensor can attempt to end an existing TCP session by sending a TCP reset (RST) flag to both the source and the destination address. Inline NBA sensors offer firewall capabilities used to drop, or reject, suspicious network activity. The passive and inline NBA sensors share a couple of IPS characteristics. The first characteristic is the capability to instruct network security devices (firewalls and routers) to reconfigure themselves to block certain types of suspicious activity or route that suspicious activity elsewhere. The second such characteristic is the shared ability to run an administrator-specified script or program when certain suspicious activity is detected.