Overview of IPsec VPNs

IPsec has become the de facto standard for creating VPNs in the networking industry providing excellent security. Several vendors have implemented it and, because the Internet Engineering Task Force (IETF) has defined IPsec in an RFC, interoperability between vendors makes IPsec the best option for building VPNs. IPsec offers a standard means of establishing authentication and encryption services between peers. IPsec is an IETF standard; furthermore, it is FIPS-compliant when used with AES encryption making it the best option for deploying VPNs. IPsec acts at the network layer of the OSI model, protecting and authenticating IP packets between participating IPsec devices (peers), such as Cisco routers or firewalls. IPsec provides the following network security services:

• Data confidentiality: The IPsec sender can encrypt packets before transmitting them across a network. If hackers cannot read the encrypted data, it is of no use to them.

• Data integrity: The IPsec receiving endpoint will authenticate all packets sent by the IPsec sending endpoint to ensure that the data has not been altered during transmission.

• Data origin authentication: The IPsec receiver can authenticate the source of the IPsec packets sent. This service is dependent upon the data integrity service.

• Anti-replay: The IPsec receiver can detect and reject replayed packets.

IPsec protects sensitive data that travels across unprotected networks, and IPsec security services are provided at the network layer; therefore, you do not need to configure individual workstations, PCs, or applications. This benefit can provide a great cost savings. Rather than providing the security services that you do not need to deploy and coordinate security on a per-application, per-computer basis, you can simply change the network infrastructure to provide the needed security services. For example, you can load updated VPN client software on a Cisco ASA, which can cause an outdated client to download and install the latest version before connecting. This is a huge time and cost savings if the alternative is manually installing updates on user laptops, plus it enables you to maintain excellent security.

IPsec provides enhanced security features, such as better encryption algorithms and more comprehensive authentication. Corporate networks connected to the Internet can enable flexible and secure VPN access with IPsec. With IPsec technology, customers can now build VPNs over the Internet with the security of encryption protection against wire tapping, eavesdropping, or other attacks that intrude on private communications.

IPsec provides authentication and encryption services to protect unauthorized viewing or modification of data within your network or as it is transferred over an unprotected network, such as the public Internet. IPsec can encrypt data between various devices, such as the following:

• Router to router

• Firewall to router

• Firewall to firewall

• User to router

• User to firewall

• Mobile device to firewall

Mobility is increasing exponentially as smartphones and tablet devices gain more and more adoption by the industry. In many cases, users are driving IT departments to give them access from wherever, whenever, and however necessary for them to access the network. Consider the practical applications of an IPsec-based VPN client configuration:

• The VPN client can be preconfigured for mass deployments.

• Requires little user intervention for initial logins, can also be tied in to current internal authentication methods for a single sign-on service for users.

• Supports Cisco Easy VPN capabilities, decreasing network security policy configuration at the remote location; also enables ease of management by your IT staff.

• All current operating systems are supported to include 64 bit.

• Security policies can be centralized and customized as needed to meet your security posture.

IPsec is a framework of open standards defined by the IETF. IPsec provides security for transmission of sensitive information over unprotected networks, such as the Internet. Figure 9-3 shows the three most common types of VPNs.

Figure 9-3 VPN Connectivity Overview

Authentication and Data Integrity

To establish trust, authentication verifies the identity of the two VPN endpoints and the users sending traffic through the VPN. An endpoint could be a VPN client, firewall, mobile device, or router. Authentication is a process of IPsec that occurs after data encryption and before decryption on the receiving end. It is a necessary function within IPsec to ensure that both the sending and receiving parties are who they claim to be. With IPsec, each peer must be manually configured with a preshared key (usually agreed upon before a connection attempt is made).

Data integrity is another function within IPsec. Integrity means that the packet that the receiving party received has not been altered during transmission. This is achieved via the use of a one-way hash algorithm. A one-way hash is the equivalent of an encrypted checksum. After the sending party encrypts and authenticates a packet, a one-way hash is run on the value of the entire packet. A hash is interesting in that its result will always be a fixed size, regardless of the input. This is another security mechanism so that hackers cannot know the input field size. The one-way hash creates an encrypted field appended to the message. On the receiving end, the one-way hash value is pulled from the packet, and the receiving end runs its own one-way hash. Because the hash is run on variables within the packet such as time sent, number of bytes, and so on, both ends’ hash value must be the same—meaning that the packet has not been tampered with. If the values are different, the packet is discarded and IPsec renegotiates its security parameters.

Tunneling Data

Tunneling is what VPNs rely on to create a private network over the Internet. Basically, this is the process of taking an entire packet of data and encapsulating it within another packet before sending it over a network. This act of encapsulating one packet of data into another is what happens when a packet is encrypted and transmitted through a tunnel. The network must understand the outer packet’s protocol for the packet to be routed across the network. Tunneling data within a VPN requires three different protocols to work:

• Data: The original data packet, usually an IP packet, which is to be encrypted and transmitted through the VPN. Perhaps the data to be sent through the tunnel is an FTP file transfer.

• Encapsulating protocol: The protocol (GRE, IPsec, PPTP, and L2TP) wrapped around the original data (that is, encapsulated). IPsec is the de facto VPN standard used as the encapsulating protocol at this stage, and it enables the data packet to be encrypted and protected.

• Carrier protocol: The protocol the network uses over which the encrypted VPN information travels. The original packet (data) is encapsulated inside the encrypting protocol, which is then put inside the carrier protocol’s header (usually IP) for transmission over the public network through the use of a routing protocol such as OSPF.

Tunneling works well with VPNs. It enables you to use protocols not supported on the Internet inside an IP packet, and it can still be safely sent. At the beginning of a VPN tunneled transmission, a data packet from the source LAN is encapsulated with new header information that enables intermediary networks to recognize and deliver it. After this is done and the transmission is complete, the tunneling protocol “header” is stripped off at the other end of the tunnel, and the original packet is transferred to the destination LAN for delivery.

Although tunneling enables data to be carried over public networks, tunneling alone does not ensure privacy. To secure a tunneled transmission against any interception and tampering, all traffic over the VPN is encrypted. In addition, VPNs typically include additional features, such as firewalls at the perimeters.

In site-to-site VPNs, the encapsulating protocol is usually IPsec or generic routing encapsulation (GRE). GRE includes information about what type of packet you are encapsulating and about the connection between the client and server. The difference depends on the level of security needed for the connection, with IPsec being more secure and GRE having greater functionality. IPsec can tunnel and encrypt IP packets, whereas GRE can tunnel IP and non-IP packets. When you need to send non-IP packets (such as IPX) through a tunnel, use IPsec and GRE together.

VPN Deployment with Layered Security

Security in depth is critical, as we have mentioned time and time again, when looking at putting all the various solutions together with regard to VPN. Figure 9-4 demonstrates layering your security.

Figure 9-4 VPNs with Layered Security

This figure shows several devices; consider their placement and role in the defense of your network:

• Internet: All visitors to your websites, email, and VPNs come from the Internet, which is where the dragons live.

• Internet edge router or firewall: This device can be many things. Its primary role is to provide the IP connection to the Internet via your service provider. Then it can prescreen traffic and even act as an initial firewall. The key point is to have this device do something to protect your network; it’s there, so use it to make another layer of defense.

• VPN gateway: This device is what all the VPN clients are terminating against. Its role is to provide the processing power to do all the necessary encryption and decryption. This is also the device that builds permanent VPN tunnels to remote sites and business partners.

• Firewall for decrypted traffic: When traffic is trying to enter your network, by the time it gets to this point, it is either regular Internet traffic (unencrypted, of course) or it is traffic sent via a VPN that is unencrypted. Regardless of the type, it must be subjected to the rules of your firewall and stateful packet inspection. This is the device that perform both functions.

• IPS for decrypted traffic: Inbound traffic can now be subjected to inspection by an intrusion prevention or detection system. You know that traffic has passed the firewall rules to reach this, so now look into the packets to ensure there are no embedded attacks within the packets.

• Network antivirus detection: This device can perform more than just antivirus. It might also be checking for botnets and redirect host to determine whether it meets the company security policy

IPsec Encryption Modes

IPsec has two encryption modes: tunnel and transport. Each mode differs in its application and in the amount of overhead added to the passenger packet. These different modes of operation are summarized briefly in that tunnel encrypts the packet header and the payload of each packet, whereas transport encrypts only the payload.

IPsec Tunnel Mode

This is the normal way in which an IPsec VPN is implemented between two ASA firewalls (or other security gateways) connected over an untrusted network, such as the public Internet. All discussions involving IPsec are about the tunnel mode because this is the most secure method and is the industry standard. Tunnel mode enables IPsec to encrypt, which then encapsulates the entire IP packet securing the data through the encryption. Because it encapsulates or hides the packets to be successfully forwarded, the encrypting routers themselves own the IP addresses used in these new headers because they are the next hop routing addresses needed. Using tunnel mode results in additional packet expansion of approximately 20 bytes per associated IP header; a new IP header must be added to the original packet, as shown in Figure 9-5.

Figure 9-5 Tunnel Mode

In tunnel mode, IPsec encrypts the entire packet and writes a new IP header onto the new encrypted packet, which masks the original source and destination IP address information. This information will be used when the packet is decrypted at the other VPN tunnel endpoint.

Transport Mode

This method of implementing IPsec is typically done with L2TP to enable the authentication of Windows VPN clients. Chapter 6, “Security Protocols,” covers this concept, so this chapter focuses on IPsec and tunnel mode. In transport mode, only the data portion of a packet is encrypted, making it less secure than tunnel mode. Tunnel mode is inherently more secure than transport mode (because the entire original packet is encrypted, not just the payload as in transport mode), as shown in Figure 9-6.

Figure 9-6 Transport Mode

IPsec Family of Protocols

IPsec works on the network layer of the OSI model—securing all data that travels between the two endpoints without an association to any specific application. IPsec accomplishes these goals through the use of three main protocols that combined form a cohesive and secure standards-based framework ideally suited for VPNs. The three protocols described in the IPsec standards have various functions within them, as detailed in the list that follows:

• Internet Security Association Key Management Protocol (ISAKMP): This protocol is used during the initial phase of negotiating the IPsec connection to establish the VPN between VPN endpoints or peers; Oakley defines the method to establish an authenticated key exchange. This method can take various modes of operation and can also derive keying material via algorithms such as Diffie-Hellman. Within ISAKMP is Internet Key Exchange (IKE), which provides a framework for negotiating security parameters (for example, SA lifetime, encryption type, and so on) and establishing the accuracy of the keys.

• Encapsulated Security Protocol (ESP): Provides data confidentiality and protection with optional authentication and replay-detection services. ESP completely encapsulates user data. ESP can be used either by itself or with AH. ESP runs using the TCP protocol on ports 50 and 51 and is documented in RFC 2406.

• Authentication Header (AH): Provides authentication and antireplay services (optional). AH provides services to limited portions of the IP header and extended header but does not provide for data encryption by applying a one-way hash to create a message digest of the packet. AH is embedded in the data to be protected and can be used either by itself or with Encryption Service Payload (ESP). (Refer to RFC 2402.) AH has largely been superseded by ESP and is considered deprecated.

Security Associations

Security associations (SA) establish trust between two devices in a peer-to-peer relationship and enable VPN endpoints to agree on a set of transmission rules by negotiating policies with a potential peer. Consider a security association such as a contract negotiated enabling the two VPN endpoints to agree to the various parameters for how the VPN tunnel is to be secured.

A security association is identified through an IP address, a security protocol identifier, and a unique security parameter index (SPI) value. The SPI value is a 32-bit number embedded in packet headers.

ISAKMP Overview

Internet Security Association and Key Management Protocol (ISAKMP) is a framework that defines the mechanics of implementing a key exchange protocol and negotiation of a security policy and threat mitigation (for example, DoS and replay attacks). ISAKMP is used for secure exchanges of both SA parameters and private keys between peers in an IPsec environment, and key creation and management, and is defined in RFC 2408.

ISAKMP provides for several methods of key management and provides secure transit of IPsec parameters between peers. It accomplishes this by using similar algorithms used by IPsec for the actual encryption of the data payload. Like IPsec, ISAKMP is not a protocol, but simply an interface to manage various ways of dynamic key exchange. ISAKMP defines various methods—such as digital signatures, certificates, and one-way hash algorithms—to ensure that negotiation of SAs between peers is securely handled.

Currently, the only supported protocol in ISAKMP is the Internet Key Exchange (IKE) protocol. When IKE is actively employed in the encryption process, many features become available to the IPsec communication process. Using public-key cryptography, IKE negotiates security parameters and key exchanges before the IPsec processing begins.

ISAKMP uses UDP port 500 to communicate. In addition, UDP port 4500 is used when NAT is present on a network and because 99 percent of today’s networks use NAT, make sure you allow this port through your firewall as well. This port is designated for a specific function referred to as NAT-T, as in Transparent, to account for operating IPsec over NAT.

Internet Key Exchange (IKE) Overview

IKE provides negotiation, peer authentication, key management, and key exchange. As a bidirectional protocol, IKE provides a secure communication channel between two devices that negotiates an encryption algorithm, a hash algorithm, an authentication method, and any relevant group information. It uses key exchange based on Diffie-Hellman algorithms, and network administrators can closely tie IKE with policy management systems. To prevent a man-in-the-middle attack—when an attacker sniffs packets from the network, modifies them, and inserts them back into the network.

IKE is the protocol that IPsec uses for completion of Phase 1 of negotiating the VPN tunnel. IKE negotiates and assigns SAs for each IPsec peer, which provides a secure channel for the negotiation of the IPsec SAs in Phase 2. IKE provides the following benefits:

• Eliminates the need to manually specify all the IPsec security parameters at both peers

• Enables you to specify a lifetime for the IPsec SAs

• Enables encryption keys to change during IPsec sessions

• Enables IPsec to provide antireplay services

• Enables CA support for a manageable, scalable IPsec implementation

• Enables dynamic authentication of peers

IKE negotiations must be protected, so each IKE negotiation begins by the peer agreeing on a common (shared) IKE policy. This policy states the security parameters used to protect subsequent IKE negotiations. After the two VPN peers agree on a policy on how to encrypt the tunnel, a security association established at each peer identifies the VPNs security parameters, and these SAs apply to all subsequent IKE traffic during the negotiation. Figure 9-7 illustrates the two modes of operation possible for IKE: main and aggressive. You can see that main is more involved and aggressive consolidates steps—personally, I like main better.

Figure 9-7 Modes of IKE

IKE Main Mode

An IKE session begins with the initiator sending a proposal or proposals to the responder. The proposals define what encryption and authentication protocols are acceptable, how long keys should remain active, and whether perfect forward secrecy should be enforced, for example. Multiple proposals can be sent in one offering. The first exchange between nodes establishes the basic security policy; the initiator proposes the encryption and authentication algorithms it is willing to use. The responder chooses the appropriate proposal (assume a proposal is chosen) and sends it to the initiator. The next exchange passes Diffie-Hellman public keys and other data. All further negotiation is encrypted within the IKE SA.

IKE Aggressive Mode

Aggressive mode squeezes the IKE SA negotiation into three packets, with all data required for the SA passed by the initiator. The responder sends the proposal, key material, and ID and authenticates the session in the next packet. The initiator replies by authenticating the session. Negotiation is quicker, and the initiator and responder ID pass in the clear.

IPsec Security Association (IPsec SA)

IPsec SA is unidirectional and thus requires that separate IPsec SAs be established in each direction. IPsec SA is a two-phase, three-mode procedure. In Phase 1, the basics of the security policy are exchanged. In phase 2, IKE’s two modes can be used: main mode and aggressive mode. In Phase 3, the only available mode is called quick mode. The third exchange authenticates the ISAKMP session. After the IKE SA is established, IPsec negotiation (quick mode) begins. IPsec negotiation, or quick mode, is similar to an aggressive mode IKE negotiation, except negotiation must be protected within an IKE SA. Quick mode negotiates the SA for the data encryption and manages the key exchange for that IPsec SA.

Both IKE and IPsec use SAs, although the SAs are independent of one another. The security associations define which protocols and algorithms should be applied to sensitive packets and specify the keying material to be used by the two peers. SAs are unidirectional and are established separately for different security protocols (AH and ESP). You can establish IPsec SAs in two ways:

• Manual SAs with preshared keys: The use of manual IPsec SAs requires a prior agreement between administrators of the ASA firewall and the IPsec peer. There is no negotiation of SAs, so the configuration information in both systems should be the same for IPsec to process traffic successfully. Manual is easy to configure; however, it is difficult to change preshared keys because the tunnel fails when you do, and the trouble is that preshared keys are usually never changed.

• IKE-established SAs: When IKE is used to establish IPsec SAs, the peers can negotiate the settings they will use for the new security associations. This means that you can specify lists (such as lists of acceptable transforms) within the crypto map entry, for example, when using a Cisco device.

IPsec Operational Overview

IPsec’s main task is to enable the exchange of private information over an insecure connection by negotiating the connection and providing the keys in a secure manner. IPsec uses encryption to protect information from interception or eavesdropping. However, to use encryption efficiently, both parties should share a secret key (password) used for both encrypting and decrypting the information as it enters and exits the VPN tunnel. IPsec uses IKE to establish the secure link, so the VPN forms connecting two endpoints, and data is encrypted securely flowing through the VPN. At a high level, the sequence of events for an IPsec tunnel creation are as follows:

1. One of the IPsec peers receives or generates interesting traffic on an interface that has been configured to initiate an IPsec tunnel when interesting traffic is received. Interesting traffic is defined in the endpoint as the type of data to be sent into the tunnel, thus its name.

2. IKE begins to negotiate the VPN security association. IKE either uses main mode or aggressive mode in the creation of an IKE SA between two IPsec peers. Upon success of this step, the ISAKMP SAs are created.

3. IKE now negotiates the IPsec SAs, and the IPsec SAs are created if successful.

4. Data starts passing through the encrypted VPN tunnel with all the encryption done per the parameters in the SAs.

These four seemingly simple steps require some additional explanation for Steps 2 and 3 because they are the critical aspects of the VPN tunnel creation. IPsec operates in two major phases to allow the confidential exchange of a shared secret key, as described in the following sections.

IKE Phase 1

IKE Phase 1 handles the negotiation of security parameters required to establish a secure channel between two IPsec peers. Phase 1 is implemented through the IKE protocol and is primarily concerned with establishing the protection for IKE messages. The sequence of events of IKE Phase 1 is as follows:

1. Phase 1 is the creation of the ISAKMP SA, where peers negotiate and agree upon policies, which are used by IPsec to negotiate and set up the SAs.

2. The key step here is that the VPN peers use their shared secret keys to authenticate with each other using Diffie-Hellman. If either the policies or the keys do not match, Phase 1 fails and the connection halts.

After Phase 1 is complete and a secure channel is established between peers, IKE moves into Phase 2. Figure 9-8 shows the negotiation of the Phase 1 parameters through the use of preshared keys.

Figure 9-8 IKE Phase 1 Operation

IKE’s Phase 1 operation has two modes of operation: aggressive and main mode. Aggressive mode eliminates several steps in the authentication of IKE, reducing it to just three steps, whereas main mode uses the full four steps to authenticate. Although it’s faster, aggressive mode is considered less secure than main mode, for obvious reasons. Cisco devices use main mode by default, but they respond for peers using aggressive mode if configured to do so.

IKE Phase 2

IKE Phase 2 advances the security of the connection by using the secure tunnel established in IKE Phase 1 to exchange the IPsec security parameters required to actually transmit user data (see Figure 9-9).

Figure 9-9 IKE Phase 2 Operation

In Phase 2, IKE negotiates SAs on behalf of IPsec, according to parameters configured in IPsec. The ISAKMP SA created in Phase 1 protects these exchanges. IKE phase 2 has one mode, called quick mode. Quick mode occurs after IKE has established the secure tunnel in Phase 1. It negotiates a shared IPsec policy, derives shared secret keys used by the IPsec security algorithms, and establishes IPsec SAs. Quick mode exchanges nonces that provide replay protection. The nonces are used to generate new shared secret key material and prevent replay attacks from generating bogus SAs.

Quick mode is also used to renegotiate a new IPsec SA when the IPsec SA lifetime expires. Base quick mode refreshes the keying material used to create the shared secret key based on the keying material derived from the Diffie-Hellman exchange in Phase 1.

Perfect Forward Secrecy

If perfect forward secrecy (PFS) is specified in the IPsec policy, a new Diffie-Hellman exchange is performed with each quick mode, providing keying material that has greater entropy (key material life) and thereby greater resistance to cryptographic attacks. Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use and exacting a performance cost. By default, Cisco devices do not have PFS configured.

The secure tunnels used in both phases of IPsec are based on SAs used at each IPsec endpoint. SAs describe the security parameters, such as the type of authentication and encryption that both endpoints agree to use.

Diffie-Hellman Algorithm

The Diffie-Hellman algorithm was the first public-key algorithm and is still considered one of the best. IKE uses public-key cryptography to negotiate security parameters and protect key exchanges. Specifically, the Diffie-Hellman algorithm is used in the IKE negotiations to enable the two peers to agree on a shared secret by generating the key for use. This is why you will see that the Diffie-Hellman algorithm is used several times throughout the process.

In general, here is how the algorithm works: Each peer contains a private key. The Diffie-Hellman algorithm takes that private key and generates a public key. The public key is a product of the private key, but is such that the private key cannot be deduced by knowing the public key. The peers then exchange public keys, as shown in Figure 9-10.

Figure 9-10 Diffie-Hellman Key Exchange

If Peer A wants to pass encrypted traffic to Peer B, Peer A encrypts the traffic going to Peer B with Peer B’s public key.

Peer B then uses its own private key to decrypt the message because its public key is derived from its private key. This ensures that only Peer B can decrypt the message because only Peer B knows its own private key.

This method enables a secure communications channel to be established (ISAKMP SA) so that subsequent IPsec SAs can securely exchange key information in privacy without having to use a public key algorithm to exchange their own keys every time encrypted traffic is passed. Figure 9-11 shows the various steps in ISAKMP Phase 1 and Phase 2 negotiations.

Figure 9-11 VPN Connection Establishment

Figure 9-11 illustrates that traffic is already encrypted before the end of IKE Phase 1. This provides for a secure exchange of the IPsec proposals and keys performed on behalf of IPsec in IKE Phase 2.

In addition to providing a secure mechanism for key exchange and managing IPsec SAs, ISAKMP also provides several other important functions. ISAKMP can be configured to set IPsec SA lifetimes, which enables more control over how often keys are exchanged. It also enables keys to change during communication without removing and re-creating the IPsec SAs. With standalone IPsec, if keys are to change during communication, existing SAs are “torn down” and rebuilt with the new keys. Because ISAKMP negotiates SAs for IPsec and protects them with its own SA, keys can be changed on-the-fly without re-creating SA negotiations. This provides a substantial advantage over IPsec alone. ISAKMP also enables dynamic authentication of peers and data integrity checks via the use of one-way hash algorithms.