IPsec, as you may have guessed, operates on the network layer of the seven-layer OSI model. It also resides on the internet layer of the four-layer network model. It is the only protocol (of the ones discussed in this section) that resides on this layer. The advantage of using this protocol is precisely the fact that it operates on the internet/network layer. Because of this, it is capable of authenticating the entire IP packet. As a result, not only is the privacy of our data ensured, but our packet's final destination is kept private, as well. It differs significantly from the two other VPN protocols supported by pfSense. OpenVPN offers encryption, but does so on the application layer of the OSI model. The Layer 2 Tunneling Protocol (L2TP) does not encrypt data at all.
IPsec is actually a protocol suite; therefore, it is a group of protocols that combine to provide the functionality we need in an encryption protocol. We can divide this suite into the following groups:
- Authentication Header (AH): This is a 32-bit header; it provides both authentication and connection-less data integrity.
- Encapsulating Security Payload (ESP): A protocol within the IPsec protocol, ESP provides authentication, integrity, and confidentiality. ESP exists in both authentication-only and encryption-only mode. It is responsible for encrypting the payload (in transport mode, which is the more common use of ESP), and, in some cases, the entire payload (in what is known as tunnel mode).
- Security Association (SA): This is the set of security attributes used in a connection. This can include things such as the encryption algorithm, encryption key, and other attributes.
Security associations are established through the Internet Security and Key Management Protocol (ISAKMP), a protocol that is defined in RFC 2408 and provides a framework for authentication and key exchange. Key exchange is usually done via Internet Key Exchange (IKE) versions 1 or 2. This protocol was developed by the IETF in November 1998, and is defined in RFCs 2407, 2408, and 2409. Other protocols are available for key exchange. For example, Kerberized Internet Negotiation of Keys (KINK) uses the Kerberos protocol for key negotiation. However, the only methods currently supported by pfSense are IKE and IKEv2.
We have already briefly mentioned the modes for establishing an IPsec connection. They are:
- Transport mode: The payload of the packet is encrypted, but the header is not encrypted. This mode does not support NAT traversal. As a result, it is a poor choice for IPsec connections that must traverse more than one router.
- Tunnel mode: The entire packet is encrypted. As you may have guessed, this mode does support NAT traversal.
If your main criteria for selecting a VPN protocol is the number of encryption algorithms supported, then IPsec is a good choice, since it supports several such algorithms. Advanced Encryption Standard (with a key size of 256 bits) is the most commonly used option, and 256-bit keys are generally considered secure. Other options are available, however. 3DES is offered as an option; this is useful, as many systems only support DES. For those who need a bigger key than AES offers, there's SHA-2, with a 512-bit key. More information about the cryptographic options available with IPsec can be found in RFC 7321 (http://tools.ietf.org/html/rfc7321).