CHAPTER 46. Virtual Private Networks (VPNs) and Tunneling

SOME OF THE MAIN TOPICS IN THIS CHAPTER ARE

---

What Is a VPN? 894

Protocols, Protocols, and More Protocols! 895

IPSec Protocols 895

The Point-to-Point Tunneling Protocol (PPTP) 899

Layer Two Tunneling Protocol (L2TP) 900

Creating a VPN Connection with Windows XP Professional 901

Selecting a Router with VPN Support 904

Security is a big issue both in the corporate network and on the Internet at large. As has been discussed in previous chapters, the basic TCP/IP protocols weren’t particularly designed with security in mind. However, over the years many new developments have built on the current TCP/IP base that provide for more secure connections over wide area networks (WANs), such as the Internet. This chapter looks at Virtual Private Networks (VPNs) and the methods they employ to create a private tunnel through a WAN so that you can communicate securely with another computer or network.

What Is a VPN?

Basically, a VPN is nothing more than a secure path through a shared network or WAN that connects two computers, or two networks, so that from the point of view of each endpoint of the connection, they are on the same network. The connection is private because some means have been taken to secure the payload information of the data carried through this virtual tunnel.

A VPN can be a good solution for security issues in many scenarios:

Employees who work from home and use the Internet to communicate with the company network

Mobile employees who travel and can dial in to the Internet using a national ISP

Branch offices using the Internet

Business partners, customers, or even technical support staff who need access

As this list demonstrates, two basic types of VPNs are used:

Remote access VPN—A connection between a remote computer and the Internet.

Site-to-site VPN—A connection between two networks, which usually is done between two routers, or in some cases firewall/router combinations.

The Mobile Workforce

Many people are on the move in the business world today, and many companies are allowing some employees to work from home. A technique still used today, but which is declining, is to set up a bank of modems and give dial-in access to certain people, such as salesmen, who are always on the move. For a business that needs data connections to branch offices but can’t justify the cost of leased lines, modem banks provide the necessary remote connection. You can host a bank of modems under many different operating systems, from Unix (with its efficient kernel and support for large numbers of serial devices) to Windows 2000/2003 (using the remote access service [RAS]). You even can install servers that are basically appliances that act as a front end to provide a bank of modems for dial-in services.

However, maintaining a bank of modems can be expensive because each modem needs a telephone line, which is an ongoing cost. There are several security issues to be considered. For example, what happens if someone discovers the telephone number of your dial-up access? It would then be easy to use a password dictionary attack to break into the network.

However, sometimes a simple dial-in modem is not the best solution. With Internet access available almost anywhere in the United States, Europe, Japan, and many other countries, the Internet can be a good solution to this problem. You can use a single, high-bandwidth connection (buy what you need) to allow multiple home workers, traveling salesmen, and other mobile workers to connect to your network just as if they were sitting at a desk at the office.

The only problem with this access method is the fact that the Internet is not exactly the most secure place in the world. As a matter of fact, just connecting your company’s network to the Internet is a serious task that should be accompanied by careful consideration of how you will control that connection (such as using a good firewall strategy), and how you will segment portions of your network to make sure that intrusions or other security breaches can be minimized.

In a typical LAN (local area network) setting, computers, servers, and other resources are connected using switches (or hubs in older networks that have not yet upgraded to newer hardware). Routers are used to connect LANs so that a logical addressing scheme can be used. The problem with security is that when the IP protocol is used, for example, the payload section of the IP packet carries some higher-level protocol message without any way of encrypting the data. If you can intercept the IP packet, you can easily determine which protocol is being used and get to the information very quickly.

As you can guess, VPNs are made up of two basic components: a tunnel, which is a virtual path through a WAN, and some form of encryption to render the contents of the payload (and possibly the header information of the upper-level protocol) unusable if intercepted.

Protocols, Protocols, and More Protocols!

Because the functions provided by a VPN include tunneling, data integrity, and authentication, it makes sense that a VPN is not created using a single protocol. Instead, several protocols can be used to create a VPN, each performing a particular function. In this section the following protocols are briefly examined:

Internet Protocol Security (IPSec)

Point-to-Point Tunneling Protocol (PPTP)

Layer Two Tunneling Protocol (L2TP)

For the most part, only IPSec should be a major factor in VPNs in the coming years. PPTP was used by Windows NT 4.0 as part of its VPN package, and L2TP has replaced it in Windows 2000 and Windows XP VPNs. L2TP is basically just the PPTP protocol combined with the L2F protocol developed by Cisco. However, most VPN vendors are using the IPSec protocols instead, which are described in greater detail than PPTP and L2TP in this chapter. The IPSec protocols incorporate some of the security mechanisms that were originally designed to be included in IPv6 but have been adapted for use in the existing IPv4 network.

IPSec Protocols

As noted previously, IPSec is the emerging standard being adopted by more and more VPN vendors. IPSec was derived from concepts that were originally designed to provide for secure communications in the next generation of the IP protocol, IPv6, which is gradually being developed.

Although Microsoft chooses to use L2TP and IPSec in combination as its VPN solution for Windows 2000 and Windows XP, many hardware and software vendors are sticking with a simple IPSec solution.

The good news is that if you decide on an all-IPSec solution, you can be virtually assured that equipment (or software) from one vendor to another will work together. If you have an all-Windows server environment, this might be of no concern. For those who operate multiprotocol networks, IPSec might be the best choice. As noted previously, IPSec is also the most widely supported VPN protocol on handheld devices.

IPSec is a standard defined in several Request for Comments (RFC) documents. IPSec is transparent to the end user and can traverse the Internet using standard IPv4 routers and other equipment without requiring any modification because it operates at the Network layer. IPSec is also flexible, allowing for the negotiation and use of many different encryption and authentication techniques.

The three main components of IPSec are the following:

Internet Key Exchange (IKE)—This is the protocol defined in RFC 2048, “Internet Security Association and Key Management Protocol (ISAKMP),” which defines a method for the secure exchange of the initial encryption keys between the two endpoints of the VPN link.

Authentication Header (AH)—This protocol, defined in RFC 1826, “The Authentication Header,” provides for inserting a standard IPv4 header into an additional header that can be used to ensure the integrity of the header information and payload as the packet makes its way through the Internet. AH does not encrypt the actual IP payload data, but instead provides a mechanism to determine whether the payload or header has been tampered with.

Encapsulating Security Payload (ESP)—This protocol performs the actual encryption of the data carried in the IP packet so that it cannot be understood by anyone who might intercept your data stream.

Internet Key Exchange (IKE)

IKE defines the mechanism used by the endpoints of the VPN to establish a secure connection and exchange encryption keys and other information pertinent to a secure connection. IKE uses public-key techniques that were discussed in the preceding chapter. If you recall, the public key half of a key pair can be known by anyone, as long as the private-key half of the key pair remains a secret. Thus, each end of the connection can use the other end’s public key to encrypt data, which can then be read only by the other end of the connection that holds the private key that can unlock the data.

IKE provides for the establishment of a security association (SA), which is the set of data that governs the particular connection. SAs are unidirectional; that is, each side negotiates an SA with the opposite end of the link. Think of it as a contact between the endpoints. The items that are negotiated by IKE for an SA include these:

The encryption algorithm to be used on the link—This can be DES (Data Encryption Standard), triple-DES, and so on.

The hash algorithm—Message Digest 5 (MD5) or Secure Hash Algorithm (SHA) is used to ensure the integrity of data transferred.

An authentication method—Not surprisingly, this is the method that will be used for authentication.

A Diffie-Hellman group—Diffie-Hellman takes its name from the inventors of public-key cryptography. A Diffie-Hellman group is basically a specification in which each group defines the length of the base prime numbers that are used for the key exchange. Group 1 is considered to be easier to break than Group 2, and so on. Both sides of the exchange must use the same Diffie-Hellman group, of course.

Encryption is detailed in Chapter 47, “Encryption Technology.”

Diffie-Hellman uses a public and private key to form a pair of keys. The public key is used to encrypt data, whereas the private (secret) key of the pair is used by the receiver to decrypt the data. Anyone can discover the public key because it can be used only to encrypt data, and not to perform the reverse process.

Using this process, a master secret key is exchanged so that further encryption can use symmetric encryption, which is much faster than public-key encryption, to protect data on the link.

After both sides have authenticated themselves to the other side, negotiations take place to determine whether AH or ESP will be used, what hashing algorithm will be used, and what encryption algorithm will be used (if ESP is used).

The actual mechanics of this exchange are a little more complicated. The Oakley protocol (defined in RFC 2412) is used by IKE to define such things as the prime number groups that are used for the public-key generation, and to decide whether certificate-based authentication will be used. A security parameters index (SPI) value is used, along with an IP address and the security protocol, to uniquely identify a specific SA. Using IKE, the value for the SPI is a pseudorandomly generated number.

The Authentication Header (AH)

IPSec consists of the two basic AH and ESP protocols that are used after IKE has established an SA. AH provides a mechanism to ensure the integrity of the IP header and the payload of the IP packet that will be transported across an untrusted link, such as the Internet. When used by itself, AH cannot provide a total guarantee of the entire IP header because some of the fields in the IP header are changed by routers as the packet passes through the network.

For more information about fields that make up an IP header, see Chapter 24, “Overview of the TCP/IP Protocol Suite.”

The AH is inserted directly after the IP header in an IPv4 packet and is composed of several important fields:

Next Header—This 8-bit field is used to identify the protocol that follows the header. If only AH is being used without ESP, typically this field contains the protocol number for TCP because TCP is the standard packet type used to carry most Internet traffic.

Length—This 8-bit field is used to specify the total length of the AH, and represents the number of 32-bit words that make up the AH.

Reserved—This field is not used at this time, but should instead be zero-filled according to the standard.

Security Parameters Index (SPI)—This 32-bit field contains a number used to identify the SA. A value of 0 indicates that no SA exists, whereas the numbers 1–255 are reserved by the IANA (Internet Assigned Numbers Authority).

Sequence Number—This 32-bit field is used as a counter to keep track of packets that belong to a particular SPI. The counter is incremented once for each packet sent. This is useful for preventing a man-in-the-middle sort of attack.

Authentication Data—This is a variable-length field that contains data used for authentication purposes, such as a digital certificate. If this field does not end on a 32-bit boundary, it’s padded to adjust its length.

As mentioned earlier, the AH is used to provide an integrity check to determine whether the actual header or payload has been tampered with during transit. It does this by using a hashing algorithm to provide a digital signature for the packet. AH does not encrypt the payload data. If a packet is received and the AH indicates that the packet has been tampered with, the packet is discarded. MD5 and SHA are the two basic hashing algorithms typically used. It is beyond the scope of this book to discuss the details of these algorithms, but rest assured that they are complex formulas that take a variable amount of information and reduce it to a fixed-length unit of data. The hash value can be calculated at each end of the connection to determine whether anything in the packet has changed. Thus, AH provides a method for ensuring the integrity of the packet, but not for keeping its contents secret.

AH can also be used in a Windows environment to ensure that only computers that have certificates administered by the administrator can communicate within the network. The administrator can control the distribution of certificates so that rogue computers (those connected to the network without permission of the administrator) won’t be able to use AH as long as certificates are used by computers to authenticate themselves to each other.

For a truly secure VPN connection, ESP must be used.

Encapsulation Security Payload (ESP)

ESP is used to encrypt the payload, or the actual IP packet that is carried in the data portion of the packet. It operates in two modes: transport and tunnel.

In transport mode, ESP provides protection for the payload and for headers created by upper-level protocols, such as TCP, that ride inside the IP packet. In this mode, nothing is done to protect the header information of the IP packet that serves as the workhorse to get the data from here to there. This is an efficient method for encrypting the contents of the IP packet in which bandwidth constraints are important.

When operating in tunnel mode, ESP is used between two IPSec gateways (such as a set of routers or firewalls) and it protects the IP header information. The entire IP datagram, including the IP header and its payload—usually an upper-level protocol such as TCP or UDP (User Datagram Protocol)—is encrypted and encapsulated by the ESP protocol. New header information is added to the resulting packet that identifies the endpoints of the transfer (the two gateways), but the true source address, destination address, and other packet information carried inside the ESP packet is protected. At the destination gateway, this outer wrapper of information is removed, the contents of the packet are decrypted, and the original IP packet is sent out onto the network to which the gateway is attached.

When in tunnel mode, the ESP header information is inserted directly before the IP or other protocol datagram that is to be protected. The datagram being protected is encrypted (according to methods set up by the SA), and additional headers are added in clear text format so that the new IP datagram can be transported to the appropriate gateway. In other words, the original protocol datagram is encrypted, the ESP header is added, and, finally, a new IP datagram is created to transport this conglomeration to its destination gateway point.

At the receiving gateway, this outer IP header information is stripped off, and according to the parameters defined by the SA, the protected payload of the original datagram is decrypted.

When in transport mode, the ESP header information follows the other header information of an IP datagram. Usually this is an authentication header that has been inserted to protect the integrity of the packet. The upper-level (Transport layer) header information follows the ESP header information. Any information following the ESP header, including the Transport layer headers, is encrypted according to the method described by the SA, and the packet is sent on its way. Note that this method does not use a gateway, so the clear text IP header at the front of the packet contains the actual destination address of the encapsulated datagram. This is the main difference between transport mode and tunnel mode. However, ESP can be used, as just mentioned, in conjunction with AH to protect the integrity of the IP header information.

At the receiving end of the communication path, this clear text header information is saved, the contents of the encrypted packet are decrypted and reassembled with the correct IP header information, and the packet is sent on its way onto the network.

ESP uses both a header and a trailer to encapsulate datagrams that it protects. The header consists of an SPI, such as the one used by AH, to identify the security association, and a sequence number to identify packets, ensure that they arrive in the correct order, and ensure that no duplicate packets are received. The trailer consists of padding from 0 to 255 bytes to make sure that the datagram ends on a 32-bit boundary. This is followed by a field that specifies the length of the padding that was attached so that it can be removed by the receiver. Following this field is a Next Header field, which is used to identify the protocol that is enveloped as the payload.

Additionally, ESP can include an authentication trailer that contains data used to verify the identity of the sender and the integrity of the message. This Integrity Check Value (ICV) is calculated based on the ESP header information, as well as the payload and the ESP trailer. The layout of an ESP datagram is shown in Figure 46.1.

Figure 46.1. The format of an ESP datagram.

As you can see, the ICV attached to the end of the packet is not encrypted. Instead, it is a value calculated on the contents of the rest of the ESP-encapsulated packet. The receiving end of the VPN can recalculate this value to determine whether the contents of the ESP header, or its payload, have been compromised during transit.

The Point-to-Point Tunneling Protocol (PPTP)

The Point-to-Point Protocol (PPP) is an error-checking protocol used for dial-up connections to the Internet. PPP allows for the encapsulation of data packets from multiple protocols for simple transmission across a dedicated link, such as a phone line when you dial in to the Internet. PPP performs no routing functions, but merely encapsulates the protocol packets it receives by attaching its own header and sends them to the other endpoint of the connection.

For more information about PPP, see Chapter 14, “Dial-Up Connections.”

PPTP extends the capabilities of PPP so that a tunnel can be created through a packet-switched network, such as the Internet, instead of across a serial link. The concepts are similar. PPTP encapsulates another protocol packet and the PPTP packet is then routed through the network. The endpoints that use the PPTP connection don’t have to be aware that they are at opposite ends of a large packet-switched network. Instead, it is as if both computers are on the same network.

Another difference between PPP and PPTP is that PPTP allows for the encryption of the payload portion of the packet so that IP (or other protocol) datagrams can be protected from prying eyes as they travel.

For example, a home user who wants to connect to a server on his company’s network first makes a dial-up connection to an Internet service provider (ISP). After the connection is set up, another set of protocol negotiations begins to set up the PPTP tunnel, over the PPP link. The endpoints for the PPP link are the home user and the ISP. The endpoints for the PPTP link are the home user’s computer and the company’s remote access service (RAS) that uses the Internet. At the RAS endpoint, the PPTP packets are unpackaged and the contents decrypted to reveal the original IP (or other protocol) packet, which can then be sent onto the company network. Thus, the home user can operate as if he were directly connected to the company network.

This protocol was used in Windows NT 4.0, and although it’s still supported in Windows 2000 and Windows XP, the newer L2TP is the preferred method for newer implementations for Windows clients.

Layer Two Tunneling Protocol (L2TP)

L2TP is one of two protocols with built-in support in Windows XP and Windows 2000. Users running either version of Windows can use the built-in VPN clients to connect via the Internet and create a secure connection to the home corporate network.

L2TP is an enhancement of PPTP that uses technology from a Cisco protocol called Layer 2 Forwarding (L2F). The combination of these two protocols is documented in RFC 2662, “Layer Two Tunneling Protocol ‘L2TP.’” L2TP uses UDP for sending user data packets as well as for maintenance messages used to manage the VPN connection. Because L2TP itself is only a tunneling protocol, the IPSec protocol, discussed previously in this chapter, is used for the actual encryption that protects the contents of the data traversing the tunnel.

Because UDP packets—rather than TCP packets—are used by L2TP, a session does not exist. Instead, L2TP uses sequence numbers for each message to make sure that packets are ordered correctly from the origination point to the destination.

L2TP Encapsulation

L2TP relies on the PPP protocol. The PPP datagram is encapsulated by L2TP by attaching an L2TP header directly in front of the PPP header. Because L2TP uses UDP, as you can probably guess, the UDP header is prefixed to the result. In Figure 46.2, you can see an overview of how the packet looks at this point.

Figure 46.2. The L2TP protocol transfers PPP datagrams using UDP as a transport protocol.

If you just want to create a tunnel, this level of encapsulation is all you need because the UDP packet will make a best-effort attempt to deliver the packet by passing it to the IP protocol for transmission on the routed network.

However, because a VPN needs to provide some level of security for the payload, the IPSec protocol comes into play. The packet shown earlier in Figure 46.1 is encapsulated by IPSec by attaching the IPSec header and trailer to the packet before it is sent to the IP protocol. In Figure 46.3, you can see the format for the resulting datagram.

Figure 46.3. IPSec provides the encryption necessary to create a true VPN when used with L2TP.

Finally, UDP passes the resulting packet to IP for transmission on the network, just like any other IP packet. The source and destination addresses used by IP are the addresses of the VPN client and server.

Creating a VPN Connection with Windows XP Professional

Windows XP Professional, like Windows 2000, contains integrated support for VPN clients using with L2TP or IPSec protocols. VPN connections, like other types of network connections, are created and configured through the Network Connections folder.

Before you can set up a connection to a VPN server, you need to know the following information:

The IP address or host name of the server

Your username and password

The security settings used by the remote server

The PPP settings used by the remote server

Setting Up and Using a VPN Connection in Windows XP

In Windows XP, VPN connections are created using the same New Connection Wizard used for other types of network connections. To set up a VPN connection in Windows XP, follow this procedure:

1. Click Create a New Connection from the Network Tasks menu in the Network Connections folder.

2. From the opening dialog, click Next to continue.

3. Select Connect to the Network at My Workplace, and click Next.

4. Select Virtual Private Network Connection, and click Next.

5. Enter a descriptive name for the connection. This is the name that is displayed in Network Connections. Click Next.

6. If you use a dial-up, ISDN, or PPPoE connection, select Automatically Dial This Initial Connection and select the connection to use from the pull-down menu. If you use an always-on connection, select Do Not Dial the Initial Connection, and click Next.

7. Enter the host name or IP address for the VPN server. Click Next.

8. To add a shortcut to the desktop, click the Add a Shortcut to This Connection to My Desktop check box. Click Finish to complete the setup.

When you click Finish, a Connect dialog appears. Enter your username and password, and click Connect to try your connection. If you want to save your credentials, click the Save This User Name and Password... check box. Specify Me Only if you want to restrict the use of this VPN connection to the current user only. Click Anyone... to enable any user of this computer to use the connection.

To configure the VPN connection, click Properties on the Connect dialog, or right-click the connection icon in Network Connections and select Properties.

VPN Connection—General Tab

Use this tab to change the host name or IP address of the remote VPN server. If you use a dial-up or other connection that requires you to log in first, such as an ISDN or PPPoE connection, use the First Connect portion of the dialog to select the connection to open before starting the VPN connection. You need to do this if you change to a different dial-up or other provider that requires a manual login after you created your VPN connection.

By default, an icon appears in the notification area (system tray) of your desktop when you are connected. Clear the Show Icon... check box if you prefer not to see a notification icon.

VPN Connection—Options Tab

Use this tab to adjust dialing and redialing options, such as if the line is busy or the connection is dropped. Keep in mind, however, that this dialog is also used to specify whether you are prompted for username and password, security certifications (enabled), and whether to include the Windows logon domain (disabled). If necessary, adjust the defaults to match the requirements of your VPN server.

VPN Connection—Security Tab

Use this tab to adjust the VPN security defaults used by Windows XP. This is necessary only if you receive an error message such as “The remote computer does not support the required data encryption type” when you attempt to connect.

The default (Typical) settings used by Windows XP’s VPN client include Require Secured Password and Require Data Encryption. If you use a smart card, select Use Smart Card instead of Require Secured Passport.

If you need to adjust data encryption, logon security, or allowed protocols, select Advanced, and click the Settings button. Before making any changes, be sure you know the settings required by the remote VPN server. Table 46.1 lists the default and optional settings for the Advanced Security Settings dialog.

Table 46.1. Advanced Security Settings for Windows XP VPN Client

Click the IPSec Settings button if you need to provide a preshared key for authentication of your connection.

VPN Connection—Networking Tab

Use the Networking tab if you need to specify the type of VPN connection you are making or to adjust the clients, protocols, or drivers used by the connection.

The default VPN connection type, Automatic, is normally satisfactory for either PPTP or L2TP IPSec connections. Click Settings to adjust the following PPP options:

Enable LCP Extensions (enabled)

Enable Software Compression (enabled)

Negotiate Multi-link for Single-link Connections (disabled)

The network clients, protocols, and drivers for the current connection are also listed. If you need to install a new client, protocol, or driver, click Install. To disable a particular client, protocol, or driver, clear the check box. Items with a grey check box are required for VPN and cannot be configured through the Properties button or disabled.

VPN Connection—Advanced Tab

The VPN connection Advanced tab is used to configure the Windows Firewall and Internet Connection Sharing.

Troubleshooting a VPN Connection

If you are unable to connect to a VPN server, check the following:

Check basic settings such as username, password, and host name or IP address. These settings are easy to change, and a single digit or letter error is enough to prevent a successful connection.

Compare the settings used by your VPN client to those expected by the VPN server. Adjust client settings to match those used by the server.

If you use a router to connect your computer to the Internet, make sure the router is configured to provide IPSec and PPTP pass through. With a Linksys router, check the Filters dialog to check these settings. For other routers, check your documentation. If either or both pass through settings are disabled, you are not able to connect to a VPN server.

If you use a router to connect your computer to the Internet and another user has a VPN connection running, but you can’t connect at the same time, this is normal. Most low-cost routers for home and small office support IPSec and PPTP pass through for only one user at a time.

If you need support for multiple VPN connections, see the next section, “Selecting a Router with VPN Support.”

For additional VPN troubleshooting suggestions, check with the help desk for the VPN server.

Selecting a Router with VPN Support

As noted in the previous section, typical SOHO and small-business wired or wireless routers support a single VPN tunnel at a time through a feature known as pass through. This feature enables VPN data to pass through the router without interference.

This is sufficient if you need to create only one VPN client connection to a remote server. If two or more people on the network, however, need simultaneous VPN access to a VPN server, or if you need to set up a VPN server for incoming VPN connections, this type of router is not sufficient.

SOHO and small-business router vendors take two approaches in providing additional VPN tunnels. Some models incorporate pass through for multiple VPN tunnels. Typically, routers that offer multiple VPN pass throughs can handle two to eight simultaneous tunnels (users). This type of router supports multiple outgoing VPN connections, but is not designed to host incoming VPN connections.

Routers with integrated VPN support include the encryption and authentication standards used by VPN clients such as IKE, MD-5 authentication, DES, 3DES, and others. Some of these routers are designed to support only outgoing VPN connections, while others support incoming VPN connections. Routers that support incoming VPN connections might be identified as supporting VPN endpoints or IPSec client/server. See the router’s specifications or documentation for details.

SOHO and small-office router vendors such as D-Link, Linksys, and NETGEAR manufacture wired and wireless routers that support as little as 2 or as many as 50 or more simultaneous VPN connections. Depending upon the vendor, routers with support for multiple VPN connections or VPN endpoints might be identified as firewall routers, VPN routers, or by other names.