High availability and uninterrupted service have been both marketing buzzwords and coveted goals for IT professionals and network administrators as long as most of us can remember. To meet this perceived need and solve a few related problems, CARP and pfsync were added as two highly anticipated features in OpenBSD 3.5.
The Common Address Redundancy Protocol (CARP) was developed as a non-patent-encumbered alternative to the Virtual Router Redundancy Protocol (VRRP), which was quite far along the track to becoming an IETF-sanctioned standard, even though possible patent issues have not been resolved.[39]
One of the main purposes of CARP is to ensure that the network will keep functioning as usual, even when a firewall or other service goes down due to errors or planned maintenance activities such as upgrades.
Complementing CARP, the pfsync protocol is designed to handle synchronization of PF states between redundant packet-filtering nodes or gateways. Both protocols are intended to ensure redundancy for essential network features with automatic failover.
CARP is based on setting up a group of machines as one master and one or more redundant backups, which are all equipped to handle a common IP address. If the master goes down, one of the backups will inherit the IP address. The handover from one CARP host to another may be authenticated, essentially by setting a shared secret (in practice, much like a password).
In the case of PF firewalls, pfsync can be set up to handle the synchronization, and if the synchronization via pfsync has been properly set up, active connections will be handed over without noticeable interruption. In essence, pfsync is a type of virtual network interface specially designed to synchronize state information between PF firewalls. Its interfaces are assigned to physical interfaces with ifconfig. It is strongly recommended to set up pfsync on a separate network (or even VLAN), even if it is technically possible to lump in pfsync traffic with other traffic on a regular interface. The main reason for this recommendation is that pfsync itself does not do any authentication on its synchronization partners, and you can guarantee correct synchronization only if you are using dedicated interfaces for your pfsync traffic.
To illustrate a useful failover setup with CARP and pfsync, we’ll walk through an example, beginning with a network that currently has one gateway to the world. The following are our goals for the reconfigured network:
The network work should keep functioning much the same way as earlier.
We should have better availability with no noticeable downtime.
The network should experience graceful failover with no interruption of active connections.
We start with the relatively simple network from Chapter 3, which looks something like Figure 7-2.
We replace the single gateway with a redundant pair of gateways that share a private network between them for state-information updates over pfsync. The result is something like Figure 7-3.
Note that the CARP addresses are virtual addresses. Unless you have console access to all machines in your CARP group, you will almost always want to assign an IP address to the physical interfaces. With a unique IP address for each of the physical interfaces, you will be able to communicate with the host and be absolutely sure with which machine you are interacting. Without IP addresses assigned to physical interfaces, you could find yourself with a setup where the backup gateways are unable to communicate (except with hosts in networks where the physical interfaces have addresses assigned), until they become the master in the redundancy group and take over the virtual IP addresses.
By convention, the IP address assigned to the the physical interface will belong in the same subnet as the virtual, shared IP address. By default, the kernel will try to assign the CARP address to a physical interface that is already configured with an address in the same subnet as the CARP address. You can make this interface selection explicit by specifying a different interface in the carpdev option in the ifconfig command string you use to set up the CARP interface.
Warning
When you are reconfiguring your network and the default gateway address goes from being fixed to a specific interface and host to a virtual address, it’s pretty much impossible to avoid temporary loss of connectivity.
Most of the CARP setup lies in cabling (according to the schematic for your network), setting sysctl values, and issuing ifconfig commands. Also, on some systems, you will need to check that your kernel is set up with the required devices compiled in.
On OpenBSD, both the CARP and pfsync devices are in the default GENERIC and GENERIC.MP kernel configurations. Unless you are running with a custom kernel where you removed these options, no kernel reconfiguration is necessary.
FreeBSD users need to check that your kernel has the CARP and pfsync devices compiled in. The GENERIC kernel does not contain these options by default. See the FreeBSD Handbook for information about how to compile and install a custom kernel with these options.
NetBSD users need to check that your kernel has pseudo-device CARP compiled in. NetBSD’s default GENERIC kernel configuration does not have CARP compiled in. However, you will find the relevant line commented out in the GENERIC configuration file for easy inclusion. NetBSD does not yet support pfsync, due to claimed protocol-numbering issues that were unresolved at the time this chapter was written.
On all CARP-capable systems, the basic functions are governed by a handful of sysctl variables. The main one, net.inet.carp.allow, is enabled by default. On a typical OpenBSD system, you will see this:
$ sysctl net.inet.carp.allow
net.inet.carp.allow=1This means that your system comes equipped for CARP.
If your kernel is not configured with a CARP device, this command will instead produce something like sysctl: unknown oid 'net.inet.carp.allow' on FreeBSD, or sysctl: third level name 'carp' in 'net.inet.carp.allow' is invalid on NetBSD.
Use this sysctl command to view all CARP-related variables:
$ sysctl net.inet.carp
net.inet.carp.allow=1
net.inet.carp.preempt=0
net.inet.carp.log=2Note
On FreeBSD, you will also encounter the variable net.inet.carp.suppress_preempt, which is a read-only status variable indicating whether or not preemption is possible. On systems with CARP code based on OpenBSD 4.2 or earlier, you will also see net.inet.carp.arpbalance, which is used to enable CARP ARP balancing to offer some limited load balancing for hosts on a local network.
To enable the graceful failover between the gateways in the setup we are planning, we need to set the net.inet.carp.preempt variable:
$ sudo sysctl net.inet.carp.preempt=1Setting the net.inet.carp.preempt variable means that on hosts with more than one network interface, such as our gateways, all CARP interfaces will move between master and backup status together. This setting needs to be identical on all hosts in the CARP group. When setting up, you need to repeat it on all hosts.
The net.inet.carp.log variable sets the debug level for CARP logging. The range of possible values is 0 through 7. The default is 2, which means only CARP state changes are logged.
Looking at the network diagram, we see that the local network uses addresses in the 192.168.12.0 network, while the external, Internet-facing interface is in the 192.0.2.0 network. With these address ranges and the CARP interface’s default behavior in mind, the commands for setting up the virtual interfaces come naturally.
In addition to the usual network parameters, CARP interfaces require one extra parameter: the virtual host ID (vhid) that uniquely identifies the interfaces that are to share the virtual IP address.
Warning
The vhid is an 8-bit value that must be set to a value that is unique within the network’s broadcast domain. Setting the vhid to the wrong value can lead to several different types of network problems that can be hard to debug. There is even anecdotal evidence that redundancy and load-balancing systems based on VRRP use a virtual node identification scheme similar enough to CARP’s that ID collisions with otherwise unrelated systems can occur and cause disruption.
On the machine you want to set up as the initial master for the group, use these commands:
$sudo ifconfig carp0 192.0.2.19 vhid 1$sudo ifconfig carp1 192.168.1.1 vhid 2
Note that we do not need to set the physical interface explicitly. The carp0 and carp1 virtual interfaces here will bind themselves to the physical interfaces that are already configured with addresses in the same subnets as the assigned CARP address.
Note
In contexts where the choice of physical network device for the CARP interface is not obvious from the existing network configuration, you can add a carpdev interface string to the ifconfig command. This can be useful in some nonintuitive configurations, as well as in scenarios where the number of free IP addresses in the relevant network is severely limited. At the time this chapter was written, the FreeBSD port of CARP did not offer the carpdev option.
With ifconfig, you should be able to check that each CARP interface was properly configured:
$ ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
carp: MASTER carpdev ep0 vhid 1 advbase 1 advskew 0
groups: carp
inet 192.0.2.19 netmask 0xffffff00 broadcast 192.0.2.255
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x5Note the carp: line, which indicates MASTER status.
On the backup, the setup is almost identical, except that you add the advskew parameter:
$sudo ifconfig carp0 192.0.2.19 vhid 1 advskew 100$ sudo ifconfig carp1 192.168.1.1 vhid 2 advskew 100
The advskew parameter indicates how much less preferred it is for the specified machine to take over than the current master. advskew and its companion value advbase are used to calculate the interval between the current host’s announcements of its master status when it has taken over. The default value for advbase is 1; for advskew, the default value is 0. In our example, the master would announce every second (1 + 0/256), while the backup would wait for 1 + 100/256 seconds. With net.inet.carp.preempt=1, when the master stops announcing or announces it is not available, the backups will take over, and the new master will start announcing at its configured rate.
Smaller advskew values mean shorter announcement intervals and higher likelihood for becoming the new master. If more hosts have the same advskew, the one that is already master will keep its master status.
From OpenBSD 4.1 onward, there is one more factor to the equation that determines which hosts takes over CARP master duty. The demotion counter is a value each CARP host announces for its interface group as a measure of readiness for its CARP interfaces. When the demotion counter value is 0, the host is in complete readiness; higher values indicate measures of degradation. You can set the demotion counter from the command line using ifconfig -g, but the value is usually set by the system itself, with higher values typically during the boot process. All other things equal, the host with the lowest demotion counter will win the contest to take over as the CARP master.
Note
At the time this chapter was written, the FreeBSD CARP port did not support setting the demotion counter.
On the backup, use ifconfig once again to check that each CARP interface is properly configured:
$ ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
carp: BACKUP carpdev ep0 vhid 1 advbase 1 advskew 100
groups: carp
inet 192.0.2.19 netmask 0xffffff00 broadcast 192.0.2.255
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x5Here, the output is only slightly different. Notice that the carp: line indicates BACKUP status along with the advbase and advskew parameters.
For actual production use, you will want to add a measure of security against unauthorized CARP activity by configuring the members of the CARP group with a shared, secret passphrase, such as the following:[40]
$sudo ifconfig carp0 pass mekmitasdigoat 192.0.2.19 vhid 1$sudo ifconfig carp1 pass mekmitasdigoat 192.168.1.1 vhid 2
Warning
Just like any other password, the passphrase will then be a required ingredient in all CARP traffic in your setup. Take care to configure all CARP interfaces in a failover group with the same passphrase (or none).
After you have figured out the appropriate settings, you will want to preserve them through future system reboots by putting them in the proper files in /etc:
On OpenBSD, put the proper
ifconfigparameters into hostname.carp0 and hostname.carp1.On FreeBSD and NetBSD, put the relevant lines in your rc.conf file as contents of the
ifconfig_carp0=andifconfig_carp1=variables.
The final piece of the configuration is to set up state-table synchronization between the hosts in your redundancy group. With synchronized state tables on the redundant firewalls, in almost all cases, there will be no apparent traffic disruption during failover. This is accomplished through a set of properly configured pfsync interfaces. (As noted earlier, at the time of writing, NetBSD does not support pfsync.)
Configuring pfsync interfaces is a matter of some planning and a few fairly straightforward ifconfig commands. It is possible to set up pfsync on any configured network interface, but it is generally a better idea to set up a separate network for the synchronization.
In our sample configuration (Figure 7-3), we have set aside a tiny network for the purpose. Here, a crossover cable connects the two Ethernet interfaces, but in configurations with more than two hosts in the failover group, you may want to set up with a separate switch, hub, or VLAN.
In our sample configuration, the interfaces we are planning to use for the synchronization have already been assigned the IP addresses 10.0.12.16 and 10.0.12.17, respectively. With the basic TCP/IP configuration in place, the complete pfsync setup for each of the two synchronization partner interfaces is as follows:
$ sudo ifconfig pfsync0 syncdev ep2This illustrates the advantage of identical hardware configurations, as well as keeping pfsync traffic on a physically separate network.
The pfsync protocol itself offers little in the way of security features. It has no authentication mechanism and, by default, communicates via IP multicast traffic. However, for the cases where a physically separate network is not a feasible option, you can tighten up your pfsync security by setting up pfsync to synchronize only with a specified syncpeer:
$ sudo ifconfig pfsync0 syncpeer 10.0.12.16 syncdev ep2This would produce a configured interface that shows up in your ifconfig output like this:
pfsync0: flags=41<UP,RUNNING> mtu 1500
priority: 0
pfsync: syncdev: ep2 syncpeer: 10.0.12.16 maxupd: 128 defer: off
groups: carp pfsyncOne other option would be to set up an IPsec tunnel and use that to protect the sync traffic. The ifconfig command would then be as follows:
$ sudo ifconfig pfsync0 syncpeer 10.0.12.16 syncdev enc0This means that the syncdev device becomes the enc0 encapsulating interface instead of the physical interface.
Note
If possible, set up your synchronization to happen across a physically separate, dedicated network. Any lost pfsync updates could lead to a less than clean failover.
A very useful way to check that your PF state synchronization is running properly is to watch the state table on both (or all) synchronized hosts using sysat states on each machine. The command gives you a live display of states, where you can see updates happening in bulk on the sync targets. In between the synchronizations, states should display identically on all hosts, except that traffic counters (such as the number of packets and bytes passed) will display updates only on the host that handles the actual connection.
This takes us to the end of the basic network configuration for CARP-based failover. In the next section, we consider what you need to keep in mind when writing rule sets for redundant configurations.
After all the contortions we’ve been through in order to get the basic networking configured, you are probably wondering what it will take to migrate the rules you have put in your current pf.conf to the new setup. You’ll be glad to know that it won’t take very much. The main change you have introduced is essentially invisible to the rest of the world, and a well-designed rule set for a single gateway configuration will generally work well for a redundant setup, too.
However, you have introduced two additional protocols: CARP and pfsync. In all likelihood, you will need to make some relatively minor changes to your rule set in order to let the failover work properly. Basically, you need to pass the CARP and pfsync traffic to the appropriate interfaces.
The most readable way to handle the CARP traffic is to introduce a macro definition for your carpdevs and an accompanying pass rule, such as the following:
pass on $carpdevs proto carp
You need to pass pfsync traffic on the appropriate interfaces.
Similary, for pfsync traffic, you can introduce a macro definition for your syncdev and an accompanying pass rule:
pass on $syncdev proto pfsync
If you want to take the pfsync device out of the filtering equation altogether, use this rule:
set skip on $syncdev
Skipping the pfsync interfaces entirely for filtering is cheaper performance-wise than filtering and passing.
Also, you should consider the roles of the virtual CARP interface and its address versus the physical interface. As far as PF is concerned, all traffic will pass through the physical interfaces, but the traffic may have the CARP interface’s IP addresses as source or destination addresses.
You may have rules in your configuration that you don’t want to bother to synchronize in case of a failover, such as connections to services that run on the gateway itself. One prime example would be the typical rule to allow SSH in for the administrator:
pass in on $int_if from $ssh_allowed to self
For those rules, you could use the state option no-sync to prevent synchronizing state changes for connections that are really not relevant after the failover has happened:
pass in on $int_if from $ssh_allowed to self keep state (no-sync)
With this configuration in hand, you will be able to schedule operating system upgrades and similar former downtime-producing activities on members of your CARPed group of systems at times when they are convenient to the system administrator, with no measurable or noticeable downtime for your services’ users.
Redundancy by failover is nice, but in some contexts, it is less attractive to have hardware just sitting around in case of failure, and you may prefer to create a configuration that spreads the network load over several hosts.
In addition to ARP balancing, which works by calculating hashes based on the source MAC address on incoming connections, CARP in OpenBSD 4.3 onward supports several varieties of IP-based load balancing, where traffic is allocated based on hashes calculated from the connections’ source and destination IP addresses. Since ARP balancing is based on the source MAC address, it will work only for hosts in the directly connected network segment. The IP-based methods are appropriate for load-balancing connections to and from the Internet at large.
Which mode is the wise choice for your application depends on the exact specifications of the rest of the network equipment you need to work with. The basic ip balancing mode uses a multicast MAC address to make the directly connected switch forward traffic to all hosts in the load-balancing cluster. The combination of a unicast IP address and a multicast MAC address is not supported by some systems. In those cases, you may need to configure your load balancing in ip-unicast mode (which uses a unicast MAC address) and configure your switch to forward to the appropriate hosts, or in ip-stealth mode, which does not use the multicast MAC address at all. As usual, the devil is in the details, and the answers are found in man pages and other documentation, most likely with a bit of experimentation thrown in.
Note
Traditionally, relayd has been used to do intelligent load balancing as the front end for servers that offer services to the rest of the world. In OpenBSD 4.7, relayd acquired the ability to keep track of available uplinks and alter the system’s routing tables based on link health, with the functionality wrapped in a bundle with the router keyword. For setups with several possible uplinks or variant routing tables, you can set up relayd to choose your uplink or, with a little help from the sysctl variables net.inet.ip.multipath and net.inet6.ip6.multipath, perform load balancing across available routes and uplinks. The specifics will vary with your networking environment. The relayd.conf man page contains a complete example to get you started.
In load-balancing mode, the CARP concept is extended by letting each CARP interface be a member of not only one failover group, but as many load-balancing groups as there are physical hosts that will share the virtual address. In contrast with the failover case where there can be only one master, each node in a load balancing cluster must be the master of its own group so it can receive traffic. Which group, and by extension which physical host, ends up handling a given connection is determined by CARP via a hash value calculation based on the connection’s source MAC address in the ARP balancing case, and source and destination IP address in the IP balancing case, in addition to actual availability.
The downside of this is that each of these groups consumes one virtual host ID, so you will run out of these IDs quite a bit quicker in a load-balancing configuration than with failover only. In fact, there is a hard upper limit to the number of CARP-based load balancing clusters at 32 virtual host IDs.
The advskew parameter plays a similar role in load-balancing configurations as in the failover ones, but the ifconfig (and hostname.carpN) syntax for CARP load balancing is slightly different from the failover case.
Changing the CARP failover group we built up over the previous sections to a load-balancing cluster is as easy as editing the configuration files and reloading. In this example, we go for an IP load-balancing scheme. If you choose a different load-balancing scheme, the configuration itself differs only in the keyword for mode selection.
On the first host, we change /etc/hostname.carp0 to this:
pass mekmitasdigoat 192.0.2.19 balancing ip carpnodes 5:100,6:0
This means that this on this host, the carp0 interface is a member of the group with vhid 5, where it has an advskew of 100 as well as the one with vhid 6, where it is the prime candidate for becoming initial master, with an advskew set to 0.
Next, we change /etc/hostname.carp1 to read as follows:
pass mekmitasdigoat 192.168.12.1 balancing ip carpnodes 3:100,4:0
For carp1, the memberships are vhids 3 and 4, with advskew values of 100 and 0, respectively.
For the other host, the advskew values are reversed, but the configuration is otherwise predictably similar. Here, /etc/hostname.carp0 reads as follows:
pass mekmitasdigoat 192.0.2.19 balancing ip carpnodes 5:0,6:100
It is a member of vhid 5 with advskew 0, and a member of vhid 6 with advskew 100. Complementing this is the /etc/hostname.carp1 file that reads like this:
pass mekmitasdigoat 192.168.12.1 balancing ip carpnodes 3:0,4:100
Again, carp1 is a member of vhid 3 and 4, with advskew 0 in the first and 100 in the other.
The ifconfig output for the carp interface group on the first host looks like this:
$ ifconfig carp
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:05
priority: 0
carp: carpdev vr0 advbase 1 balancing ip
state MASTER vhid 5 advskew 0
state BACKUP vhid 6 advskew 100
groups: carp
inet 192.0.2.19 netmask 0xffffff00 broadcast 192.0.2.255
inet6 fe80::200:24ff:fecb:1c10%carp0 prefixlen 64 scopeid 0x7carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:03
priority: 0
carp: carpdev vr1 advbase 1 balancing ip
state MASTER vhid 3 advskew 0
state BACKUP vhid 4 advskew 100
groups: carp
inet 192.168.12.1 netmask 0xffffff00 broadcast 192.168.12.255
inet6 fe80::200:24ff:fecb:1c10%carp1 prefixlen 64 scopeid 0x8
pfsync0: flags=41<UP,RUNNING> mtu 1500
priority: 0
pfsync: syncdev: vr2 syncpeer: 10.0.12.17 maxupd: 128 defer: off
groups: carp pfsyncThe other host has the following ifconfig output:
$ ifconfig carp
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:05
priority: 0
carp: carpdev vr0 advbase 1 balancing ip
state BACKUP vhid 5 advskew 100
state MASTER vhid 6 advskew 0
groups: carp
inet 192.0.2.19 netmask 0xffffff00 broadcast 192.0.2.255
inet6 fe80::200:24ff:fecb:1c18%carp0 prefixlen 64 scopeid 0x7
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 01:00:5e:00:01:03
priority: 0
carp: carpdev vr1 advbase 1 balancing ip
state BACKUP vhid 3 advskew 100
state MASTER vhid 4 advskew 0
groups: carp
inet 192.168.12.1 netmask 0xffffff00 broadcast 192.168.12.255
inet6 fe80::200:24ff:fecb:1c18%carp1 prefixlen 64 scopeid 0x8
pfsync0: flags=41<UP,RUNNING> mtu 1500
priority: 0
pfsync: syncdev: vr2 syncpeer: 10.0.12.16 maxupd: 128 defer: off
groups: carp pfsyncIf we had three nodes in our load-balancing scheme, each carp interface would need to be a member of an additional group, for a total of three groups. In short, for each physical host you introduce in the load-balancing group, each carp interface becomes the member of an additional group.
Once you have set up the load-balancing cluster, you can check the actual flow of connections by running systat states on each of the hosts in your load-balancing cluster. I recommend doing just that for a few minutes to make sure that the system works as expected and to see that all the effort you put in has been worth it.
[39] VRRP is described in RFC 2281 and RFC 3768. The patents involved are held by Cisco, IBM, and Nokia. See the RFCs for details.
[40] This particular passphrase has a very specific meaning. A web search will reveal its significance and why it is de rigeur for modern networking documentation. The definitive answer can be found via the openbsd-misc mailing list archives.