There are several factors you will likely want to consider when choosing which VPN protocol to use. Some of the more crucial factors include the following:

• Interoperability with other routers and firewalls
• The type of authentication used
• Operating systems supported
• Security needs

Interoperability is something that comes into play when configuring a peer-to-peer VPN tunnel (in other words, one that connects to another firewall). In many cases, there may be a pfSense firewall at both ends of the tunnel, but in other cases, you may want to connect to a non-pfSense firewall. In such cases, IPsec is probably the best choice, since it is included with virtually every VPN-capable device. OpenVPN, being an open source protocol, is gaining acceptance, but it is not nearly as ubiquitous as IPsec.

The type of authentication used may also be a factor to consider. IPsec works with both username/password combinations, pre-shared keys, and certificates. L2TP does not have any built-in authentication. Therefore, if you require authentication, you either won’t be using L2TP, or you’ll be using it in combination with another protocol, such as IPsec. OpenVPN supports both pre-shared keys and certificates.

Another relevant consideration is what operating systems you will be supporting. If you will be primarily supporting Windows clients, IPsec is a good choice, as support for IPsec has been built into every version of Windows starting with Windows Vista. There are also third-party IPsec clients available for Windows, such as the Shrew Soft VPN client.

If you will be primarily supporting Linux clients, the choice is not quite as obvious, as built-in support for VPN protocols under Linux is limited. Ubuntu has built-in support for Point-to-Point Tunneling Protocol (PPTP), which is no longer supported by pfSense. OpenVPN has a client for Linux, and is probably your best choice. There are also IPsec clients available for Linux, which are of varying degrees of reliability and ease of use.

If you are supporting macOS clients, IPsec is likely the best option, as macOS has had built-in support for IPsec for many years. Snow Leopard (10.6) and later even has a built-in Cisco VPN client that provides a graphical interface. Although earlier versions of macOS do not come with this client pre-installed, you can install the Cisco Remote Access IPsec client on them.

If you must support a mixture of OSes, your choice of which protocol to use becomes more complicated. IPsec is the one protocol for which all three major OSes—Windows, Linux, and macOS—have clients available. L2TP is also a possible choice, although it is often implemented in combination with another protocol, Windows supports L2TP, either in conjunction with IPsec or L2TP alone. OpenVPN is a possibility if you are supporting both Windows and Linux; unfortunately, as of this writing, there is no OpenVPN client for macOS.

Finally, you likely want to consider cryptographic security. As L2TP has no encryption, this is essentially a choice between OpenVPN and IPsec. OpenVPN uses the SSL encryption library, which provides several different cryptographic ciphers. One disadvantage of OpenVPN is that it seems to favor backward compatibility over security. In addition, OpenVPN operates on the application layer of the seven-layer OSI model, whereas IPsec operates at the network layer, giving it a slight edge over OpenVPN.