This chapter will explore some multi-Wide Area Network (WAN) strategies such as load balancing and failover using the policy-based routing concept. You will also explore some common issues and how to solve them.
By the end of this chapter, you will be able to understand and configure the following multi-WAN related topics:
You will need a running OPNsense and a host to practice this chapter's steps. Knowledge of how to create and change network settings on VirtualBox is required. A good understanding of how to create/edit firewall rules on OPNsense is essential for this chapter.
In the past, we used to need a dedicated network appliance to deal with multiple internet connections and guarantee good availability of internet access. One of the best features of a modern firewall is working with multiple WAN connections. OPNsense has incredible features that we'll explore in this chapter so that it can be configured with various internet connections.
The first scenario we'll explore is the failover configuration; with two or more WAN connections, it is possible to configure OPNsense to change the active internet connection to a backup one automatically. An example of an OPNsense configured with two internet connections is shown in the following topology:
Figure 9.1 – Multi-WAN example scenario
As we can see in the preceding diagram, two WAN connections, A and B, are both connected to the internet and configured on OPNsense. WAN-A is configured in OPNsense as the primary connection and WAN-B as the secondary. If the primary fails, OPNsense will automatically change the Local Area Network's (LAN) outgoing connections to the secondary WAN in a failover manner.
Important Note
For simulating another WAN connection in a lab environment, you can add a new network interface and configure it on your OPNsense virtual machine (VM).
To configure the OPNsense VM to work in a failover manner, as proposed in the previous diagram, we need to adjust the VirtualBox network settings of our OPNsense VM:
Figure 9.2 – Additional WAN interface, VirtualBox's example configuration
As you can see in the preceding screenshot, I have configured an additional interface using the Bridged Adapter option and selected my internet-connected network interface (en0). These network options can be accessed by clicking on the Settings button of your VirtualBox's VM. The configuration in your OPNsense will depend on the network's settings connected to this new WAN interface. If you are not sure, try to configure it as Dynamic Host Configuration Protocol (DHCP) first.
With your OPNsense configured with two WAN interfaces, it is time to go through the steps to set the failover configuration:
Figure 9.3 – System | Gateways | Single page
In the preceding screenshot, you'll note that the primary WAN (WAN_A) has the gateway address 10.0.2.2 (VirtualBox's gateway) and the secondary WAN (WAN_B) has the gateway configured as 192.168.1.1. As I mentioned previously, using VirtualBox, you can set one WAN using a Network Address Translation (NAT) adapter (WAN_A) and another one using a bridged adapter (WAN_B).
Let's check the options available on the gateway configuration page by clicking on the edit button (pencil icon):
On the gateway editing page (System | Gateways | Single), you'll find the following options:
To save the gateway configuration, click on the Save button, then click on the Apply changes button.
After saving the gateway, you might notice two significant changes on the System | Gateways | Single page:
Figure 9.4 – Gateways with gateway monitoring enabled
As you can see in the preceding screenshot, the monitoring daemon will start to measure the gateway's latency, represented on the page by the round-trip time (RTT) column and the standard deviation, RTTd. The Loss column is how many packets (in percent) are being lost.
Remember to uncheck the Mark Gateway Monitoring option in the other gateway (WAN_B_DHCP).
Important Note
The RTTd, or RTT standard deviation is, in a simple manner of speaking, how much it varies over time.
To learn more about ping/ICMP standard deviation, check this link: https://newbedev.com/what-does-mdev-mean-in-ping-8.
The dpinger monitor daemon can be noted as running in the top-right corner of the page. You can also check whether it is running on Lobby | Dashboard:
Figure 9.5 – The dpinger service running on the services widget (dashboard)
The preceding screenshot shows the dpinger service running on the service widget. You can check all the running services in this widget. On the Command Line Interface (CLI), you can check it using the pluginctl -s command.
A gateway group can contain several system gateways inside it and the way we set each will define how this group will behave when some predefined condition is triggered.
Now that you have learned how to add a system gateway, we must create a group to work with it in failover and load-balance configurations later. Moving on with the configuration steps, let's now configure a group of gateways that will define how our configuration will behave, in this case, in a failover manner:
Important Note
It is a good practice to name your gateways group by writing the primary gateway before the secondary (as in the preceding example). Thus, it will be easier to understand how the group is organized (which is the primary and secondary gateway).
To finish, click on the Save button and then on the Apply changes button.
So far, we have configured an additional WAN interface, configured the basics of each WAN gateway, and created a gateway group to work in a failover manner. Now, it's time to learn about how policy routing works on OPNsense.
Unlike the static routes added to the system, policy-based routes will be created through firewall rules on OPNsense. In Chapter 5, Firewall, we explored firewall concepts and rules, but nothing related to using a gateway on rules, so now it's time to learn how to do that.
Before starting, to follow these steps, we'll need a host connected to OPNsense's LAN. If you are using VirtualBox as your lab platform, with an additional VM installed, follow these steps to connect it to the OPNsense LAN:
Figure 9.6 – Changing VM network settings to connect on OPNsense's VM LAN
Important Note
I'll use an Ubuntu VM host; feel free to choose your preferred operating system but pay attention to the commands demonstrated, as they will only work on Ubuntu (and probably most Linux distributions)!
sudo ip route del default
sudo ip route add default via <OPNsense LAN address>
Test whether the host is reaching the internet using OPNsense as the default gateway:
ping 8.8.8.8
With ping running on the host, check on OPNsense to ensure that the traffic is passing from the LAN to the internet; go to Firewall | Diagnostics | States and filter the destination address, 8.8.8.8:
Figure 9.7 – Firewall | Diagnostics | States page showing the traffic from our LAN's host
With this, we finished the required steps to connect an additional VM as a LAN host to OPNsense. Now, it is time to move on with the policy-based routing rule to make the failover work!
To enable the previously configured gateways settings, we need to add a firewall rule specifying the gateway group to enable the failover configuration. To do this, follow the given steps:
opnsense@ubuntu:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 0.817 ms 0.720 ms 0.806 ms
2 10.0.2.2 (10.0.2.2) 6.692 ms 6.660 ms 6.631 ms
3 * * *
4 192.168.15.1 (192.168.15.1) 12.545 ms 12.644 ms 13.841 ms
Important Note
In my lab, WAN_A_DHCP has the IP address 10.0.2.2, which is the OPNsense default gateway.
If your Ubuntu host doesn't have traceroute installed, you can install it by running apt install traceroute.
Important Note
Policy-based routing is only supported by rules with Direction set to in (inbound rules). For this reason, you need a LAN host connected to the OPNsense LAN interface to see things working.
opnsense@ubuntu:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 0.784 ms 0.997 ms 0.974 ms
2 192.168.1.1 (192.168.1.1) 2.408 ms 2.388 ms 2.378 ms
3 192.168.15.1 (192.168.15.1) 8.664 ms 8.638 ms 8.453 ms
4 * * *
opnsense@ubuntu:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 2.238 ms 2.972 ms 4.418 ms
2 10.0.2.2 (10.0.2.2) 5.921 ms 10.674 ms 10.626 ms
3 * * *
Figure 9.8 – Disconnecting a network adapter in VirtualBox
opnsense@ubuntu:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 2.450 ms 2.203 ms 2.151 ms
2 192.168.1.1 (192.168.1.1) 4.363 ms 4.308 ms 3.954 ms
3 192.168.15.1 (192.168.15.1) 8.288 ms 9.896 ms 9.852 ms
4 * * *
Even with this simple example, you can see that OPNsense works very well in a failover configuration. Users barely notice when a WAN link is down in a production environment, even with dozens of different protocols and thousands of hosts. I have excellent experience with failover scenarios (CloudFence's customers) that have five or more WANs with thousands of users; OPNsense is fantastic with that!
Now we have explored the failover configuration, let's look at the outbound load balance.
The load balance configuration differs slightly from the failover, and it can also act as a failover when a gateway goes offline. The main idea of a load balance configuration is to send packets through the gateways in an alternate manner. How the packets will alternate the gateway will depend on some configurations:
To configure load balance, follow the given steps:
opnsense@ubuntu:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 2.277 ms 4.733 ms 4.707 ms
2 10.0.2.2 (10.0.2.2) 4.685 ms 4.548 ms 4.512 ms
3 * * *
4 192.168.15.1 (192.168.15.1) 13.798 ms 14.349 ms 14.316 ms
opnsense@ubuntu:~$ traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
1 _gateway (192.168.56.3) 7.422 ms 7.342 ms 7.305 ms
2 192.168.1.1 (192.168.1.1) 16.747 ms 16.715 ms 10.0.2.2 (10.0.2.2) 16.682 ms
3 192.168.15.1 (192.168.15.1) 54.729 ms 58.178 ms *
Notice that the path changed! Our load balance configuration is working!
These were quite simple examples, but in a production environment, a lot of complexity might be added, and some issues can appear. Let's now see some examples of how to troubleshoot them.
Let's look at some of the common issues while configuring load balance and failover configurations:
These are some examples of failover/load balance issues we face daily while working with OPNsense. Sometimes a problem can be a combination of other ones. It will depend on the complexity of the OPNsense configuration. You can always count on the community's support in the forum to help you!
In this chapter, we have explored the failover, load balance, and policy-based concepts. Now, you can understand, create, and manage gateways, groups of gateways, and firewall rules using them. You also learned how to troubleshoot common issues involving failover, load balance, and gateways on OPNsense. In the next chapter, we will go through the reporting features available on OPNsense!
3.142.197.212