Multi-WAN gateway groups are not difficult to add and configure, but you may nevertheless have to troubleshoot such a setup. If you do, there are several problems you are likely to encounter:

• You may encounter a connectivity issue with your primary or secondary connection
• You may encounter a misconfiguration in your setup
• You may encounter a situtation where pfSense thinks a gateway is up when it is actually down, or vice-versa

The first point seems fairly obvious—that connectivity issues may cause problems with multi-WAN setups—but you will want to make sure that both (or all, if you have more than two connections) connections are working, independent of the gateway group. If the secondary connection represents a low percentage of our overall bandwidth, or if it is part of a failure group and therefore is inactive as long as the primary connection is up, then we might not notice that it is not working until the primary connection fails. Thus, ensuring that both connections work when we first set up multiple WAN connections will potentially save us some grief. If we are troubleshooting, ensuring that both connections work first can easily save us a significant amount of troubleshooting time.

Testing your multi-WAN setup before declaring it to be functional is a good idea. First, simulate a WAN interface going down. In the first case, you should disconnect the cable between pfSense and your primary connection and confirm that you are still connected to the internet. Confirm that the DNS resolution is still working. Then, reconnect the primary connection, disconnect the secondary connection, and repeat this process.

You may have a cable, DSL, or some other connection with a modem. Try unplugging the modem and also unplugging the connection between the modem and the demarcation point and see what happens. For a T1 connection, try unplugging the internet connection from the router—then, either unplug or power off the router. If you have more than two WAN interfaces in your setup, you may want to perform this test with different combinations of interfaces being up and down—as long as one interface in a gateway group is up, you should still have some level of internet connectivity and also be able to perform DNS resolution.

This testing is often invaluable; you will often uncover a configuration error that you would not have uncovered under normal circumstances. For example, what if the secondary WAN interface uses the public IP address of the primary WAN connection as a monitor IP? If this is the case, when you power off the modem connected to the primary WAN connection, the secondary WAN connection will go down, even though the secondary connection is solid and should still be up.

Misconfiguration can be a problem. Although creating a gateway group is easy enough, there are many steps needed to make sure that everything works. There must be NAT rules for each new WAN interface and there must be firewall rules to direct traffic to the appropriate gateway group. Check to make sure that any policy-based rules work as well. By default, the rules will direct traffic to the default gateway—so when the primary WAN interface is still up, everything might seem to be working. But when the primary WAN interface goes down, internet connectivity will be lost, even though there are still functional interfaces within the gateway group. You should check the firewall and NAT rules to make sure that they are doing what they are intended to do.

If the flow of traffic seems to work as it should, but DNS resolution does not work, then the issue may be that pfSense does not apply its internal routing tables to internal traffic. For that reason, you may have to configure a static route for the secondary WAN's DNS server.

There may be a failure with gateway load balancing in which one of the WAN connections in a gateway group no longer has internet connectivity, but remains up. The reason for this may be that the monitor IP is still responding—this would be the case if you set the monitor IP to an IP on your internal network. As a result, pfSense thinks the connection is still good. If this is the case, make sure that the monitor IP is correct and specifies an external IP address.

But alas, even if the monitor IP is an external IP, our problems may not be over. The network administrator at the external site we are pinging may see pfSense's pings to the site. The admin might suspect that these pings are the beginning of a denial-of-service attack, and might then block your pings. When the monitor pings fail, your gateway will go down, even though the network interface is still functioning and has internet connectivity. This could happen whenever you set a monitor IP you don't control. The best solution is to use a site for the monitor IP that is external to your network, but is a site that you control. If you cannot do this, the next best solution is to use a DNS server as your monitor IP, since a DNS server normally will not block your pings.

In many cases, when you connect to certain websites, they will store session information such as your public IP address. This includes such sites as e-commerce sites. If you subsequently connect to the same site through a different public IP address, then the site might not function as it should, because it will see a different IP address and assume that a new session should be started. The Use Sticky Connections option is designed to eliminate this problem. This option directs traffic from such websites to the same WAN address, as long as there are states that refer to the connection in the state table.

If you have implemented load balancing with your gateway group, you should verify that it works. One way to do this is to visit a website that tells you what your public IP address is. If you don't know the URL of any of these sites, just do a web search for what is my IP address. If you refresh the page several times, you should see your public IP address change. You may have to reload this page several times before you see a change. There may be other traffic on your network, and you may have set the weights on your gateway group to favor one gateway heavily.

If your IP address never changes, you should make sure that you are really reloading the page and not accessing the page from a cache. This might be the case if you have a browser cache or are using a web proxy. In addition, make sure that you do not have sticky connections enabled or some other option that enables persistent connections. If your IP address does not change, try deleting your web cache, visiting different what is my IP-type sites, and trying different web browsers. If you still can't get your IP address to change, you might have to troubleshoot further.

Another way to test load balancing is with command-line utilities such as traceroute. The traceroute utility is a command that displays the route a packet trackets to a site. It also displays the transit delay at each step, and is available on Windows, Linux/BSD, and macOS. We will cover traceroute in greater detail in Chapter 11, Diagnostics and Troubleshooting.

Finally, there is the possibility that your packet loss and latency settings are generating false positives. The opposite could also be true, and one or more of your connections could be down without pfSnese detecting it. If one of your gateways is generating false positives, then navigate to System | Routing and on the Gateways tab, edit the Advanced Settings for the gateway that is generating the false positives. Sometimes, increasing the time period over which results are averaged can eliminate this problem.