Intrusion Detection Techniques

An intrusion detection system (IDS) gathers and analyzes information from a computer or a network to identify intrusions and misuse. An IDS requires continuous monitoring in order to play an effective role in network security. An intrusion detection system uses signature recognition that identifies events that may indicate the abuse of a system. It relies heavily on a predefined set of attack and traffic patterns called signatures.

Anomaly detection is based on heuristics or behavioral rules, which can be called a baseline. Baselines are established during normal network operations as it monitors activity and attempts to classify it as either “normal” or “anomalous.” Protocol anomaly detection is based on the anomalies specific to a protocol and identifies TCP/IP protocol-specific flaws. Nowadays, machine learning (ML) algorithms are utilized for anomaly detection.

IDS Types

An IDS can be implemented in a variety of different forms, from a stand-alone appliance to a feature built into the operating system of a switch or a router. It can also be host-based as an application or a feature of an operating system or database. When categorizing an IDS, we typically identify two types: host-based and network-based. To detect possible attacks or suspicious behavior, the host-based systems analyze signatures and anomalies on the native host. A network-based IDS (NIDS) resides on border routers or appliances and identifies unusual network traffic or signatures of a network-based attack. A System Integrity Verifier (SIV) manages system files and tracks main system objects for changes. A Log File Monitor (LFM) monitors the log files that network services create.

IDS Placement

The placement of an IDS system is critical to its effectiveness and ability to interpret intrusions. An IDS system can be placed on the outside of a firewall as an early warning system in the DMZ or in the private network. When placed outside of the firewall, it generates a large number of alarms. An IDS system can also reside on any host within the network, which allows it to see and analyze traffic passed into the corporate network. When placed after the firewall, it results in fewer alarms. Typically a host-based IDS resides on the most critical systems including database servers, critical application servers, and network administration systems.

Indications of Intrusion

After an IDS Detects an Attack

IDS Attacks

There are several attacks that can be launched against an IDS. An insertion attack confuses the IDS by forcing it to read invalid packets. An evasion attack occurs when the IDS discards a packet, but the host that was to receive the packet accepts it. Many types of DoS attacks can be used against an IDS. Desynchronization uses SYN packets postconnection and preconnection. An IDS may not be able to detect a malicious program that was run through an obfuscator because the obfuscator makes the program harder to understand. An attacker may be able to direct an attack around an IDS by passing it. They may also use fragmentation methods or session splicing to evade the IDS by dividing a string across several packets.

Intrusion Prevention Systems

Intrusion prevention systems can be configured to control router operations, switch operations, firewall operations, VPN establishment, and wireless access. Based on detection of signatures and anomalies, an IPS can take corrective actions to stop an intrusion or attack. A warning IPS is also vulnerable to false positives, so operator knowledge and the ability to identify a false positive is critical in preventing an IPS from arbitrarily denying authorized activity.

Information Flow

Firewalls

Firewalls have become part of standard operations in most organizations. Firewalls can be hardware- or software-based, or a combination of the two. A firewall is designed to examine traffic and then allow or block that traffic based on the organization’s policies.

Types of Firewalls

There are a number of firewall alternatives available. Firewalls have their limitations, though. For instance, a firewall cannot prevent users who have modems from dialing into or out of the network. Firewalls do not protect against social engineering. Finally, a firewall cannot secure against tunneling attempts. Several firewalls you could use are hardware firewalls, software firewalls, packet filtering firewalls, circuit-level gateways, application-level firewalls, and stateful multilayer inspection firewalls.

Firewall Identification

There are several methods that attackers use to identify firewalls. They can scan ports using Nmap. They can perform firewalking, which has a network discovery phase and a scanning phase. An attacker may also use banner grabbing, which is sending messages out from network services.

Breeching Firewalls

When a firewall protects a network, an attacker can use various methods to hack their way through it. They can use an insider accomplice, find vulnerable services, access a vulnerable external server, hijack, bypass a firewall (HTTPTunnel), place backdoors through firewalls (rwwwshell), hide behind a covert channel (Loki), and use ACK tunneling.

Honeypots and Honeynets

Many organizations deploy honeypots and honeynets as early warning systems against potential attacks. Both of these systems are placed on the network and entice potential attackers to target them as easy victims within the organization. These devices may purposefully be configured with known vulnerabilities and weak security. The devices are designed to send alarms and messages that they have been attacked or breached. This enables network administrators to identify the source of an attack and close the gates to prevent the attack from spreading to critical devices and systems within the organization’s private network.

Types of Honeypots

Open Source Honeypots

There are many honeypots available as commercial products or in open source formats. A few commercially available honeypots include KFSensor, NetBait, ManTrap, and SPECTER. You have numerous choices if you wish to go the open source route.

Responding to Attacks

Not only is it important to detect intrusions, but an organization should have good defensive policies in place. An incident response team should include members from various departments throughout the organization. The company should have response procedures, communications, logging procedures, and training and rehearsals in place for such an event.

Intrusion Detection Tools

Tools to Evade an IDS

An administrator needs to be aware of the tools that are available to help an attacker evade an IDS. Real-time IDS systems can be fooled if they are not set up and configured correctly. SideStep, Mendax, Stick, Fragrouter, and ADMutate are a few of these tools an administrator should know.

Packet Generators

Tools to Breach a Firewall

There are several tools available for disguising communication between two servers and successfully breaching a firewall. A few of them are 007 Shell, ICMP Shell (ISH), AckCmd, and Covert_TCP.

Tools for Testing

Summary

In this chapter, you learned about various efforts and processes that can be implemented to protect against attacks on internal networks. You reviewed intrusion detection techniques, various types of firewalls, and how to identify when an attack is occurring through monitoring.