IPsec, as the name implies, is a protocol suite that operates on the Internet layer of the four-layer network model (and the Network layer of the OSI model). It is the only protocol of the three discussed here that operates on this layer. Because it operates on the Internet/Network layer, it is capable of encrypting and authenticating the entire IP packet, thus not only ensuring privacy for our data, but also ensuring that the packet's final destination is kept private as well. Thus it differs from both OpenVPN (which offers encryption, but operates on the Application layer) and the Layer 2 Tunneling Protocol (which does not encrypt data at all).

As a protocol suite, IPsec is actually a group of protocols that in combination provide the functionality we require. These protocols can be divided into three groups:

SAs are established through the Internet Security and Key Management Protocol (ISAKMP). Key exchange is typically done through Internet Key Exchange (IKE) versions 1 or 2, but other protocols are available, such as Kerberized Internet Negotiation of Keys (KINK), which uses the Kerberos protocol for key negotiation. Currently, the only methods supported by pfSense are IKE and IKEv2.

There are two different modes for establishing an IPsec connection:

IPsec supports a number of encryption algorithms. Advanced Encryption Standard with a key size of 256 bits (AES-256) is the most commonly used option, but other options are available. Since some systems only support DES, 3DES is offered as an option. If you need a bigger key, SHA-2 (with a 512-bit key size) is available. For more information about the cryptographic options available with IPsec, see RFC 7321 (https://tools.ietf.org/html/rfc7321).

L2TP, or Layer 2 Tunneling Protocol, is a tunneling protocol that (not surprisingly) operates on the Link layer of the seven-layer OSI model. Unlike IPsec, it does not provide any encryption or confidentiality by itself, but instead relies on whatever encryption protocol is passing through its tunnel. As a result, it is often used in conjunction with IPsec.

The client end of an L2TP tunnel is known as the L2TP Access Concentrator (LAC). The server end is known as the L2TP Network Server (LNS). The LNS end, once configured, waits for new connections. A connection is established through the exchange of several control packets, for which L2TP provides reliability. No reliability is provided for data packets, although reliability may be provided by protocols running within the L2TP tunnel.

Because L2TP does not have any confidentiality or encryption, it is often implemented in conjunction with IPsec. This combination is known as L2TP/IPsec. Establishing an L2TP/IPsec connection involves negotiation of an IPsec security association (usually with IKE or IKEv2), establishment of ESP communication in transport mode, and negotiation of an L2TP tunnel. L2TP uses UDP; the default port is 1701.

L2TP, without any other protocols nested within it, is generally referred to as native L2TP. You are unlikely to implement L2TP in native mode; however, L2TP/IPsec is a common option for VPN tunnels, and L2TP can be combined with other protocols to provide confidentiality and encryption (for example, L2TP/PPP).

OpenVPN is an open source VPN protocol (published under the GNU General Public License, or GPL). Since it is based on SSL, and SSL operates just above the Transport layer, we can say that it exists between the Application and Transport layers in the four layer network model, and between the Session and Transport layers in the OSI model. OpenVPN client software is now available for Linux, Windows, and Mac OS.

OpenVPN uses the OpenSSL library for encryption. Thus, all the cryptographic algorithms available in OpenSSL are available in OpenVPN. It can also use HMAC packet authentication, and support for the mbed TLS (formerly PolarSSL) SSL library has been added.

Authentication in OpenVPN can be accomplished through certificates, a user/password combination, or both. Another one of the advantages of using OpenVPN is that it can be implemented with TCP or UDP. In either case, the official IANA assigned port number for OpenVPN is port 1194. Version 2.0 and later of OpenVPN support several simultaneous tunnels per process.

OpenVPN uses the universal TUN/TAP driver for networking. Thus, it can create either a layer 3 (the Network layer of the OSI model) TUN tunnel, or a layer 2 (Data link layer of the OSI model) TAP connection.

Although OpenVPN seems to be most heavily supported with Linux (not to mention UNIX variants such as FreeBSD), there are clients available for Windows (XP and later), as well as Mac OS X. Mobile devices are also supported, with clients available for iOS and Android.

Which VPN protocol you choose will likely be based on a number of factors. Interoperability is one factor to consider. If you need a VPN solution that is interoperable with another firewall or router, especially one from another vendor, then IPsec may be the ideal protocol to use, since it is included with every VPN-capable device. Using IPsec will also prevent you from being locked into a particular product or vendor. OpenVPN is gaining acceptance, but its usage is not quite as widespread as IPsec.

Another consideration is what type of authentication the protocol uses. IPsec allows you to use a pre-shared key or certificates, as well as username/password combinations. L2TP does not provide for any authentication, while OpenVPN supports pre-shared keys and certificates.

Ease of configuration is another consideration. All of the VPN protocol options available under the current version of pfSense (IPsec, L2TP, and OpenVPN) are fairly easy to configure, but some are easier than others. OpenVPN requires the use of certificates for remote access in many environments, but is otherwise relatively easy to configure. IPsec, on the other hand, can be somewhat difficult for the uninitiated, although IPsec may be preferable because of its near-universal acceptance.

If you have a multi-WAN setup, this may prove to be a factor in your choice. In this case, both IPsec and OpenVPN can be used.

More often than not, your choice will be dictated by what operating systems you will be supporting and what clients are available for these operating systems. If your network is Windows-centric, you may consider using IPsec. Support for IPsec is built directly into Windows, and has been since Windows Vista. As a result, connecting to a VPN with IPsec under Windows can be as easy as navigating to Settings | Control Panel, clicking on Network and Sharing Center, clicking on Set up a new connection or network, and using the wizard to set up an IPsec/L2TP connection. You can also use third-party VPN clients, such as the Shrew Soft VPN Client.

On the other hand, most Linux distributions do not have built-in VPN support. Third-party clients are available, and in some cases can be downloaded from repositories, and these clients involve varying degrees of configuration. If your network is Linux-centric, you should be able to support IPsec, although OpenVPN is probably a better option in such cases.

Mac OS X has had IPsec support for years, and now even has a user-friendly interface for IPsec. OS X 10.6 (Snow Leopard) and later have a built-in Cisco IPsec VPN client that provides an easy-to-use graphical interface for connecting to a network that supports IPsec. Earlier versions of Mac OS X do not have the Cisco built-in VPN client, but you can install the Cisco Remote Access IPsec client on them. You can also use the Cisco AnyConnect Secure Mobility Client on earlier versions, although you should be aware that support for Mac OS X 10.5 (Leopard) was dropped with version 3.1 of AnyConnect.

For a network that is likely to have a variety of platforms, L2TP is a good choice. Because of the inherent lack of encryption and confidentiality in L2TP, it is usually implemented in conjunction with IPsec. Still, there are several clients on different platforms that support L2TP without IPsec. Beginning with Windows Vista, Windows has built-in support for L2TP without IPsec. One of the utilities provided for L2TP configuration is a Microsoft Management Console (MMC) snap-in called Windows Firewall with Advanced Security (WFwAS) and can be found in Control Panel | Administrative Tools. The other is a command-line tool called netshadvfirewall.

Support for L2TP is not built in to Linux, but there are third-party clients available. They are available for most popular distributions such as Arch Linux and Ubuntu, and configuration for most of these clients is relatively easy.

The Cisco IPsec client for Mac OS X supports L2TP, but it appears that it supports only L2TP over IPsec. As of this writing, there does not appear to be a third-party client for Mac OS that supports native L2TP without IPsec. Thus L2TP is a poor choice if your network must support computers running Mac OS.

OpenVPN has been ported to several operating systems. Windows does not have built-in support for OpenVPN, but there are several third-party clients for Windows. In fact, the OpenVPN project has a client for Windows that works on XP or later, and is easy to install and configure.

Linux not only supports OpenVPN, but OpenVPN support is built into many popular Linux distributions. OpenVPN configuration through the Network Connections applet in Ubuntu and its variants is rather easy, and it supports authentication with both certificates and with a pre-shared key. This makes OpenVPN an excellent choice if you are mainly supporting Linux clients.

sudo apt-get install openvpn

Mac OS X does not have built-in support for OpenVPN. The OpenVPN project does not provide a Mac OS version of their client, and, to my knowledge, no one has successfully compiled the source code of the client under Mac OS. There is an open source project called Tunnelblick, which provides the necessary drivers for implementing OpenVPN under OS X. It has a graphical interface that provides a way to control either server or client connections. It can be used on its own or in conjunction with commercial software such as Viscosity. For more information, see the Tunnelblick website at http://tunnelblick.net.

If your network setup is fairly complex, your choice of protocol may be dictated at least in part by how well the protocol works behind multiple firewalls. Some of these firewalls may be beyond your control, and their configurations and capabilities may differ substantially.

IPsec uses both UDP port 500 (for IKE) and the ESP protocol. Not all firewalls handle ESP traffic well when NAT is used, because the ESP protocol does not have port numbers that make it easily trackable by NAT devices. IPsec clients behind firewalls may require NAT-T to function; it encapsulates ESP traffic over port 4500 using the UDP protocol. Versions 2.0 and later of pfSense support NAT-T, so you should be able to utilize NAT traversal with IPsec if necessary.

OpenVPN is generally more firewall-friendly than IPsec. It uses TCP or UDP and thus is not affected by NAT behavior such as the rewriting of source ports. As a result, it is rare that a firewall won't work with OpenVPN. One possible issue is that the protocol and port may be blocked. OpenVPN uses port 1194 by default; if that port is blocked, you may want to switch to a port commonly used for something else to evade egress filtering. For example, ports 80 and 443 are assigned for HTTP and HTTPS respectively, but any TCP traffic should pass through these ports, so you could use them.

Since L2TP uses UDP, it shouldn't create any especially challenging issues with firewalls. It is often used with IPsec, however, so all of the issues related to IPsec come into play when you are using L2TP/IPsec.

One of the justifications for using VPNs is cryptographic security, so this is another factor to consider. Point-to-Point Tunneling Protocol (PPTP), which has been removed from the current version of pfSense, has numerous security vulnerabilities, and thus became a poor choice for security-conscious administrator long ago. L2TP has no encryption capability; if you want encryption, you'll have to use it in combination with another protocol (usually IPsec). Therefore the choice essentially comes down to either IPsec or OpenVPN.

OpenVPN uses the SSL encryption library, which provides a number of different ciphers. To find out which ciphers the version of OpenVPN installed with pfSense supports, execute the following command, either at the pfSense console's command prompt or from Diagnostics | Command Prompt:

openvpn –-show-ciphers

OpenVPN's default encryption algorithm is BF-CBC, or Blowfish, block cipher, with a 128-bit (variable) key size. While this is not a terrible cipher, it may be beneficial to choose a stronger cipher, such as AES-256-CBC.

OpenVPN also offers a number of different digests for message authentication, including many of the digests supported by IPsec (for example, SHA512). To see a list of digests supported by OpenVPN, use the following command:

openvpn --show-digests

One factor working against OpenVPN is that it seems that OpenVPN developers have generally given priority to backward-compatibility over security. This, and the fact that IPsec operates at Layer 3 of the OSI model and therefore provides encryption on the IP level, would seem to give IPsec a slight advantage over OpenVPN in cryptographic security.

It might prove useful to provide a summary of some of the features of each VPN protocol currently supported by pfSense, so with that in mind, here it is: