Most protocol discussions begin with the Open Systems Interconnection (OSI) Reference Model. The OSI model is a standard conceptual tool used to discuss protocols and their functions. The OSI model has seven layers (FIGURE 1-4). Each layer communicates with its peer layer on the other end of a communication session. Although the OSI model is helpful in understanding protocols, most protocols are not in full compliance with it.

Each layer of the OSI model has unique responsibilities, functions, and features. The OSI model defines what needs to take place at each layer and leaves the actual process of accomplishing those tasks to the protocols and, ultimately, to the protocol programmers.

• Application Layer (Layer 7)—The Application Layer (Layer 7) enables communications with the host software, including the operating system. The Application Layer is the interface between the host software and the network protocol stack. The sub-protocols of this layer support specific applications or types of data.
• Presentation Layer (Layer 6)—The Presentation Layer (Layer 6) translates the data received from the host software into a format acceptable to the network. This layer also performs this task in reverse for data going from the network to the host software.
• Session Layer (Layer 5)—The Session Layer (Layer 5) manages the communication channel, known as a session, between the endpoints of the network communication. A single Transport Layer connection between two systems can support multiple, simultaneous sessions.
• Transport Layer (Layer 4)—The Transport Layer (Layer 4) formats and handles data transportation. This transportation is independent of and transparent to the application.
• Network Layer (Layer 3)—The Network layer (Layer 3) handles logical addressing (IP addresses) and routing traffic.
• Data Link Layer (Layer 2)—The Data Link Layer (Layer 2) manages physical addressing (MAC addresses) and supports the network topology, such as Ethernet.
• Physical Layer (Layer 1)—The Physical Layer (Layer 1) converts data into transmitted bits over the physical network medium (network wiring or the air in the case of wireless communication).

As data moves from a software application for transmission over the network, it traverses the layers of the protocol stack from top to bottom. As each layer receives data from the layer above it, that data becomes the payload with a layer-specific header. At the Data Link Layer (Layer 2), where Ethernet resides, the data receives a footer as well. This process is known as encapsulation. The inverse, known as de-encapsulation, occurs when a network communication is received. As this process takes place, the data set being manipulated receives unique names, depending on the layer it traverses. The steps in the process are:

• Application Layer (Layer 7)—Payload data unit (PDU), that is, data obtained from the host software application.
• Presentation Layer (Layer 6)—PDU
• Session Layer (Layer 5)—PDU
• Transport Layer (Layer 4)—Segment
• Network Layer (Layer 3)—Packet
• Data Link Layer (Layer 2)—Frame
• Physical Layer (Layer 1)—Bits of data

The encapsulation process of adding headers (and a footer at the Data Link Layer) enables data exchange between layers on different systems. This is known as peer-to-peer communications. The content of a header includes information to be processed by the corresponding layer on the receiving end of a network link.

The content of the headers, mainly from Layers 2–4, are the greatest concern and focus of a firewall. Application proxy firewalls and stateful inspection firewalls can also examine the headers and the payload content of Layers 5–7.

TCP/IP is not a single protocol, but a collection of protocols. Often referred to as the TCP/IP suite, this collection includes several core protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP), as well as several commonly used protocols, such as Address Resolution Protocol (ARP), Internet Control Messaging Protocol (ICMP), Hypertext Transfer Protocol (HTTP), and Transport Layer Security (TLS). In addition to these, there are tens of thousands of proposed Request for Comments (RFCs) introducing potential protocols, but there are only a handful of protocols that typically operate within the network infrastructure created by TCP/IP.

A firm knowledge of the sub-protocols of TCP/IP is assumed but not essential to grasping the scope of this text.

Firewalls, specifically packet-filtering firewalls, inspect the contents of headers to allow or deny frames, packets, or segments. Depending on the type of filtering and the layer or protocol focus of the filtering, the item examined and the header it comes from can vary.

Packet filtering often focuses on four main headers:

• The Ethernet header of the frame from the Data Link Layer
• The IP header of the packet from the Network Layer
• The TCP header of the segment from the Transport Layer
• The UDP header of the segment from the Transport Layer

Each of these four headers has numerous details that affect filtering action. These include MAC addresses, IP addresses, TCP header flags, port numbers, and more.

With regard to the basic differences in IPv4 and IPv6 addressing, the issues to focus on are not subnet masks or the length of an address—both of which are important—but rather address length, available address capacity to meet the need for unique addresses, and subnet masking.

Many forms of filtering focus on the IP address and/or port number to make an allow or deny decision. Such decisions or rules can focus on either the source address or the destination address found in the header of a segment, packet, or frame. In other words, port number, IP address, or MAC address can support filtering as a source and/or destination concern.

Firewall filtering can focus on whether an address is public or private in both the source or destination position in the IP packet header. Generally, private addresses function on the private network and do not reach the outside. NAT will translate source-private addresses into a public address if the packet is heading toward an external destination. On the Internet, both routers and firewalls drop any IP packet that has an RFC 1918 address in the header. Firewalls can filter on whether an address is known or unknown. This function filters against addresses used for the source and/or the destination. Known addresses are usually trusted addresses. Only packets with trusted and known addresses are allowed to reach their destination. Unknown addresses are not likely to be trusted. Packets with unknown addresses can be stopped in every case or further inspected before an allow or deny decision is made.

Firewalls can filter on whether an address is known to be benign or commonly sends malicious content, a variation of known and unknown filtering. Instead of known addresses being trusted and unknowns not being trusted, this method twists the idea. A benign address is a known, trusted address and a malicious address is a known, not trusted address. In both cases, the addresses are known. Any unknown address encountered using this form of filtering will require additional subsequent filtering that looks at other aspects of the traffic.

Firewalls can also filter on whether an address is real or spoofed. This form of filtering is not as clear and distinct as the previous types. Whether a segment, packet, or frame’s address is the real correct address or a spoofed falsified address can be difficult to determine in every situation. When an address is a real address, in the sense that it’s within the subnet ranges of a network, the methods for determining whether it’s a spoofed address are few.

One method is to compare the address against a use table, such as that maintained by the Dynamic Host Configuration Protocol (DHCP) server, to see if the address is not currently assigned to any authorized system. Another method is to check the route, communication path, receipt vector, or receiving interface of the source address against what that aspect should be for a given address. For example, if a source address typically arrives on port 4 but it shows up on port 7, it is likely to be a spoofed source address.

Spoofed addresses can also be detected at border sentry points. When a source address comes from the opposite side of the firewall, then it is obviously a spoofed address. One clear example is when an internal LAN address appears as a source address in a packet on its way into a network from outside. This form of spoof filtering can be part of ingress filtering.

Likewise, the same process can be used for packets leaving a network. If a packet’s source address is from the outside, such as an Internet address, but the packet is received by a firewall from an interface in the private LAN, it must be from a spoofed address. This form of spoof filtering can be part of egress filtering.