Chapter 17: Firewall High Availability

In this chapter, we will learn how to configure high availability by connecting two firewalls to sync configuration, connect states, and preserve network connectivity if something goes wrong with one of our firewalls. By the end of this chapter, you will understand the concepts surrounding high availability and be able to implement them using OPNsense.

In this chapter, we will cover the following topics:

• High availability concepts
• Configuring high availability
• Testing the HA configuration

Technical requirements

You will need two running OPNsense installations on the same network to follow the steps in this chapter. Good knowledge of how to configure OPNsense networking and firewall rules is mandatory.

High availability concepts

Let's introduce this topic with an aviation example. At the beginning of heavier-than-air history, airplanes had just one engine to fly from one location to another. As aviation grew, the demand for long-range flights increased, and new projects that used two or more engines began. Nowadays, it is possible to cross the oceans with a twin-engine plane thanks to reliable engine technology and the Extended-range Twin-engine Operations Performance Standards (ETOPS). But even with all this technology, two engines are required to keep a long-haul flight within safety standards. Developments similar to the aviation industry also happened in the IT world – redundancy standards/protocols were created to keep the availability of the systems at acceptable levels for the business.

In OPNsense, the Common Address Redundancy Protocol (CARP) is a protocol that ensures that the network interfaces of two or more firewalls keep operating in case of a hardware failure. As in a twin-engine plane, a firewall cluster must have at least two firewalls working together to keep everything online. Think about a firewall cluster as the airplane and each firewall as the plane's engines. If one fails, you have the other one to keep the network safe and available! The key difference between a twin-engine plane and an OPNsense firewall cluster is that, in the plane, both engines are in use during normal flight conditions, while with a firewall cluster, the second firewall only comes into play if the first one fails. This is called an active-passive cluster. An active-active cluster is when both firewalls are actively working in the network, but OPNsense doesn't support this operation mode.

Active-active and active-passive modes

Active-active mode will load-balance workloads to all cluster nodes, which can appear as an advantage at first glance. But let's suppose that a two-node cluster has an average CPU usage of 60%. What will happen if one of the nodes fails? The second one will probably be overloaded with 100% CPU usage since, in theory, the workload has a 20% deficit compared to the last cluster state (60% + 60% = 120%). So, it isn't a good idea to have a firewall cluster running in this mode. Do you agree?

Active-passive mode takes a more straightforward approach, and it is the OPNsense cluster's default mode. In this mode, considering the two-node cluster, it will have one node as the master and the other one as the backup node, so if the master node fails, the backup will jump in automatically. In OPNsense, the firewall states are synchronized between the nodes using the pfsync protocol. This ensures that network connections won't be dropped and recreated in case of a node failure. Users won't even notice that a failure happened in the network.

Important Note

Both CARP and pfsync were forked from OpenBSD to FreeBSD. To learn more about those two protocols, please refer to https://www.openbsd.org/faq/pf/carp.html.

To ensure both nodes have the same configuration, OPNsense uses the Extensible Markup Language Remote Procedure Call (XMLRPC) to transfer the OPNsense configuration from the master node to the backup node.

Important Note

Notice that only the master (primary) node transfers configurations to the backup (secondary) node. The reverse isn't supported.

We will assume that a firewall cluster has two nodes from this point on. Before we look at the configuration, let's dive into some CARP concepts.

CARP – how it works

The master node uses multicasting to frequently advertise in networks that have CARP virtual IPs configured, to let the backup node know that it is alive. Two parameters are configured to set the advertising frequency: base and skew. base specifies how often the advertisements (in seconds) will be multicast to the 224.0.0.18 address (IPv4), while skew defines which node is the preferable master. For example, the node with the lesser number will be elected as the master. For instance, considering that 0 is the minimum possible value, this will represent the master node in the cluster.

CARP uses a virtual host ID (VHID), which identifies the redundancy group to be shared between cluster nodes. Each CARP interface will have the following virtual MAC address format: 00:00:5e:00:<VHID number>. The VHID uses a password to authenticate the communication between cluster nodes.

Next, we'll learn how to configure CARP and high availability in OPNsense.

The preempt behavior

Even in experienced OPNsense teams, it is common to have a quick discussion about the preempt CARP configuration before defining it as enabled or disabled in OPNsense as a high availability option, as we will explore in the following section. But the key point here is to clarify what it does. When preempt is enabled, the preempt CARP configuration (net.inet.carp.preempt = 1) is enabled; in the case of a single network interface failure, the cluster node will assume a failure. All other CARP configured network interfaces will assume a failure condition, turning the secondary node (backup) into a master on all the networks. Otherwise, while the CARP configuration is disabled (the default in OPNsense), each CARP configured network interface will act individually, and if a failure occurs, just the failed network will have the BACKUP status.

Now that we've learned about the theory behind high availability and how it works in OPNsense, let's learn how to configure it in an OPNsense firewall.

Configuring high availability

First, let's look at the high availability scenario topology shown in the following diagram:

Figure 17.1 – A high availability topology example

In the preceding diagram, we have two OPNsense firewalls connected using a dedicated network interface. This interface, which we will call the HA Interface, is the network interface that will keep the firewall states synchronized using the pfsync protocol. We also use it to run XMLRPC to sync OPNsense 1's (master node) configuration to OPNsense 2 (backup node).

Proposed scenario

You can quickly reproduce this scenario using VirtualBox while configuring the network and the virtual machines using the steps we learned about in the previous chapters of this book (modifying the steps according to your lab environment).

First, configure two OPNsense firewalls, add a network interface (both from OPNsense), and connect them directly using a network interface that we will use for firewall states sync.

You can use a /30 CIDR, which is the address that was suggested in the topology (10.254.254.0/30), with two available IPs for host addressing:

ipcalc 10.254.254.0/30

Address:   10.254.254.0         00001010.11111110.11111110.000000 00

Netmask:   255.255.255.252 = 30 11111111.11111111.11111111.111111 00

Wildcard:  0.0.0.3              00000000.00000000.00000000.000000 11

=>

Network:   10.254.254.0/30      00001010.11111110.11111110.000000 00

HostMin:   10.254.254.1         00001010.11111110.11111110.000000 01

HostMax:   10.254.254.2         00001010.11111110.11111110.000000 10

Broadcast: 10.254.254.3         00001010.11111110.11111110.000000 11

Hosts/Net: 2                     Class A, Private Internet

If you need help configuring a new network interface, please go back to Chapter 3, Configuring an OPNsense Network, where we explored all the necessary steps for configuring a network interface in OPNsense.

Next, we have the virtual IPs for the networks that OPNsense is serving – LAN and WAN. Each network interface has a virtual IP that's been configured using CARP. Follow these steps to configure CARP virtual IPs on OPNsense 1:

1. Go to Interfaces | Virtual IPs | Settings and add a new IP:

Figure 17.2 – CARP virtual IP configuration

Here, you can make the following configurations:

• Mode: Select CARP.
• Interface: Select the network interface you want to configure the CARP virtual IP on (for example, LAN).
• Address: The virtual IP address.
• Virtual IP Password: This password will be used to help CARP authenticate between the cluster nodes. Once XMLRPC syncs the configuration, you don't need to save this password.
• VHID Group: Select an available VHID group. To help configure an available VHID, click the Select an unassigned VHID button.
• Advertising Frequency: The Base option, or advertisements' frequency, keeps the default value of 1 second. For example, in the Skew option, you can set 0 for the master node and 100 for the backup nodes.

  Important Note

  OPNsense automatically creates the CARP virtual IPs in the backup nodes by setting the appropriate skew value (100 by default).

• Description: Type in a description for this virtual IP.
• Click the Save button to add the virtual IP.

As a suggested scenario, on the master node, create a CARP virtual IP for the LAN interfaces. You can pick the better IP addresses for your network. This chapter will use the scenario presented previously in the topology diagram.

In our example scenario, the following LAN network address configuration will be used:

• OPNsense-1 (master) interface address: 192.168.0.2/24
• OPNsense-2 (master) interface address: 192.168.0.3/24
• CARP virtual IP address: 192.168.0.1/24

  Important Note

  Suppose you need help configuring a new virtual IP address, please refer to Chapter 3, Configuring an OPNsense Network.

After the configuration, the virtual IP address should look like the following screenshot:

Figure 17.3 – LAN's CARP virtual IP address example

For the WAN configuration, the following settings will be used:

• OPNsense-1 (master) interface address: 10.0.2.252/24
• OPNsense-2 (master) interface address: 10.0.2.253/24
• CARP virtual IP address: 10.0.2.1/24

2. To check the configured virtual IPs, go to Interfaces | Virtual IPs | Status:

Figure 17.4 – Configured CARP virtual IPs

Notice that the Status column for both virtual IP addresses is MASTER. You can also check that the VHID group is represented after the @ character – for example, LAN@1, which means VHID 1.

3. Next, you must add the proper firewall rules to set up the high availability configuration. For the OPNsense (master and backup nodes) firewalls, add the following rules to the HA network interface:

Figure 17.5 – Adding firewall rules

The preceding screenshot shows the following rules that were added while using the HA network interface IP address (HA net) as Source and the Destination address:

• The first rule allows traffic in the webGUI port (433 (HTTPS)). It is required for XMLRPC to sync the firewall configurations from the master node to the backup node.
• The second rule will allow the firewall states to be synced between the nodes cluster using the pfsync protocol.
• The third rule was created just for testing purposes – to run a ping command (ICMP protocol) and confirm the communication between the nodes using the HA interface, for example.

4. To confirm that the HA network interface is working properly, try running a ping command from the master node. You can use the webGUI by going to Interfaces | Diagnostic | Ping and try to reach the backup node IP address or use the CLI to run the ping command:

   # ping -c '3' '10.254.254.2'

   PING 10.254.254.2 (10.254.254.2): 56 data bytes

   64 bytes from 10.254.254.2: icmp_seq=0 ttl=64 time=0.706 ms

   64 bytes from 10.254.254.2: icmp_seq=1 ttl=64 time=0.794 ms

   64 bytes from 10.254.254.2: icmp_seq=2 ttl=64 time=0.690 ms

   --- 10.254.254.2 ping statistics ---

   3 packets transmitted, 3 packets received, 0.0% packet loss

   round-trip min/avg/max/stddev = 0.690/0.730/0.794/0.046 ms

To ensure that the outgoing WAN traffic keeps the same state as the source IP if the master node fails, we need to create an outbound NAT rule. Otherwise, the connections states will use the WAN interface IP address of the active node (the master, in this example), and if this master fails, the backup node won't maintain the same states once the source IP comes from the master node. To solve this issue, the CARP virtual IP must be used as the source IP.

Important Note

Before creating the outbound NAT rule, ensure that NAT outbound mode has been configured as Hybrid outbound NAT rule generation on the Firewall | NAT | Outbound page.

To address this issue, we need to create an outbound NAT rule:

Figure 17.6 – NAT outbound rule example

This rule was created by specifying a Source LAN net and the translation address set to the WAN CARP virtual IP address.

Great! We have the HA interface and the basic rules configured.

Now, we have to configure the HA settings to sync the configuration and the firewall states table.

Go to System | High Availability | Settings and set as it follows:

• Synchronize States: Check this option.
• Disable preempt: Leave unchecked.
• Synchronize Interface: Select the HA interface.
• Synchronize Peer IP: Specify the OPNsense-2 HA interface IP address – for example, 10.254.254.2.

The next section of this page configures XMLRPC OPNsense's configurations sync. Set it as follows:

• Synchronize Config to IP: Set it to the OPNsense-2 HA interface IP address.
• Remote System Username: Set the webGUI username: root.
• Remote System Password: Set the webGUI username's password.

On the same page, there are settings we must synchronize to the backup node. You can select all the available options to sync all the settings from the master node to the backup node. Click the Save button to save all the configurations.

Go to System | High Availability | Status to perform the configuration sync and click the Synchronize config to backup button:

Figure 17.7 – Synchronize config to backup

After the configuration sync has been completed, you can check the OPNsense-2 (backup) configurations to compare them to OPNsense-1's (master):

Figure 17.8 – Virtual IPs created after syncing on the backup node

Notice that the Virtual IPs were created after syncing with VHID group 100 on OPNsense-2 (backup node).

Important Note

OPNsense requires you to manually synchronize the master to the backup node settings. Some firewalls solutions (as OPNsense used to) automatically sync the settings after any change has been made to the master. Requiring human intervention to sync the changes can prevent some accidental changes from replicating to the backup node, which could lead to a possible crash or unavailability of the cluster. I recommend that you always check the changes on the master node in terms of side effects before replicating to the backup node.

Finally, we can proceed with the high availability testing. We'll learn how to do that in the next section.

Testing the HA configuration

Now that we have both OPNsense firewalls up and running and configured with high availability, let's do some testing!

First, check if the virtual IPs are working accordingly on both OPNsense firewalls, as shown in the following screenshot:

Figure 17.9 – OPNsense-1 (master) running the CARP virtual IP

On both firewalls, you can check the virtual IP's status in the webGUI. Go to Interfaces | Virtual IPs | Status:

Figure 17.10 – OPNsense-2 (backup) running the CARP virtual IP

If you want to use a DHCP server on the LAN interface, you will need to make some configuration adjustments, as follows:

• Go to Services | DHCPv4 | LAN and set the following settings on the master node:
  • Set both the DNS servers and Gateway fields to the CARP LAN virtual IP; that is, 192.168.0.1.

Remember that every time you change the configuration on the master node, you must sync to secondary node.

Important Note

You can skip the DNS server configuration if you want to use external DNS servers.

To avoid both firewalls sending DHCP offers to the network, we must set Failover peer IP to the backup node's LAN interface address of 192.168.0.3. Once the configurations have been synced, OPNsense automatically fills the Failover peer IP field of the backup node with the master node's LAN interface address; that is, 192.168.0.2.

Once you've checked that the CARP virtual IPs are working as expected, you can go to the host that's connected to the LAN and run a ping command. Here, OPNsense automatically fills the Failover peer IP field of the backup node with the master node's LAN interface address.

Now, let's perform a functional test. We will run a ping command from the host connected to the OPNsense cluster's LAN and turn off the master node to see what happens with the backup node:

1. Go to the host (Linux) and run the following command:

   ping 192.168.0.1

2. Power off OPNsense-1 (master node) or disconnect it from the LAN network.
3. Notice that the CARP virtual IP (192.168.0.1) continues replying to packets to the host.
4. Go to Interfaces | Virtual IPs | Status for OPNsense-2 and notice that it became the master node.
5. Take OPNsense-1 back online and wait for the boot process to finish.
6. Repeat Step 4 and check that the CARP virtual IP on OPNsense-2 changed back to the backup status.

Well done – high availability is working!

Caveats

To get a high availability cluster working correctly, you must use an identical hardware (or virtual machine) configuration.

In the webGUI, on the Interfaces | Virtual IPs | Status page, it is possible to simulate node failure using both the Temporarily Disable CARP and Enter Persistent CARP Maintenance Mode buttons. The difference between them is that the former doesn't survive a firewall reboot.

If the primary node fails and you want to keep the secondary as the master node, even after the primary is back online, you can check the Disable preempt box on the System | High Availability | Settings page.

Marking this option will also make the cluster CARP virtual IPs act like a group. So, if a CARP virtual IP is demoted to backup, all other CARP virtual IPs will also assume the backup status.

To prevent a firewall rule configuration from being synced from the master node to the backup nodes, you can mark the No XMLRPC Sync option inside the firewall rule editing page (Firewall | Rules | <Interface Name>). It is also possible to prevent the firewall states from syncing by checking the State Type | NO pfsync box inside the Advanced options of each rule.

Troubleshooting

High availability clusters can sometimes present some troubleshooting challenges as a complex configuration.

Let's look at some examples of issues where a high availability configuration can't work correctly.

Same virtual IP status on both cluster nodes

Sometimes, you may find one or more CARP virtual IPs set as master nodes on both firewalls. This can happen when the secondary node can't find the primary. Network connectivity issues often cause this. So, if you are using a physical hardware platform, debugging the switch logs is a good idea. Otherwise, start looking at the virtualization host logs if you are running it on a virtualization platform.

Configuration sync problems

A common issue is when you set the network interfaces differently on the primary and secondary nodes. This kind of misconfiguration can lead to the HA configuration not working correctly. Before configuring the other options, configure the network interfaces on the primary node and export its XML config file to the secondary. The network configuration is not synced between the nodes, so doing so can prevent a network configuration from happening. Other misconfigurations can occur when the System | High Availability | Settings page's options are not set correctly.

Summary

In this chapter, you learned how OPNsense implements high availability to guarantee enterprise-grade availability as a firewall solution. We explored how the CARP protocol provides a cluster virtual IP operation and that when combined with pfsync, XMLRPC brings a complete stack for a firewall high availability cluster. Now, you can build solid and highly available firewall clusters using OPNsense that can survive a hardware failure and keep the networks safe and protected. In the next chapter, we will learn how to protect websites and web applications using OPNsense.