If you are implementing or maintaining a network, it is almost certain that, at some point, something will go wrong. This could be the result of human error (for example, a misconfiguration issue), hardware failure, or a software problem. In such circumstances, your troubleshooting skills will be put to the test. The aim of this chapter is to help you develop your troubleshooting skills.
The following topics will be covered in this chapter:
- Troubleshooting basics
- pfSense troubleshooting tools
- Troubleshooting scenarios
The first section will outline a procedure for troubleshooting networks in general before delving into pfSense troubleshooting. If you are already familiar with network troubleshooting, feel free to skip the section.
Implementing effective network troubleshooting involves a multi-step approach. These steps both provide a framework for troubleshooting and help reduce the amount of time spent resolving problems:
- Identify the problem: This seems obvious, but we often assume that we know the exact scope of the problem, when we might be better served gathering information, identifying symptoms, and, when applicable, questioning users. If there is more than one problem, we should recognize it as such so we can approach each problem individually. Sometimes end users are good sources of information. For example, you can ask a user how the system behaves during normal operation and compare it to how the system currently behaves. Recreate the problem, if possible, and try to isolate the location of the problem.
- Formulate a theory of probable cause: A single problem can have many causes, but if you have done your homework with information gathering and if you apply a modicum of common sense, you can eliminate many of these causes. Often the most obvious solution is the correct one, and looking at the easiest solution first is a reasonable approach. Keep in mind, however, that your initial theory may be incorrect and you may have to consider other theories.
- Test the theory: Once you have established a theory of probable cause, you should attempt to confirm the theory. If the theory can be confirmed, then you can move on to the next step. If not, you need to formulate another theory.
- Establish a plan: Once you think you have identified the cause, you need to establish a plan of action. This becomes more important when troubleshooting in enterprise-level environments. Implementing a solution may involve taking systems offline, and you have to determine when they will be taken offline and for how long. In many cases, your organization may have formal or informal procedures for taking the system offline. This often includes scheduling a time – often during non-working hours – when the work will be done. Nonetheless, once you have a plan in place, you should be able to implement a solution.
- Implement the solution: Once the corrective change has been made to your network, you still need to test the solution. You can't assume that the solution has worked without testing it, and often you need to be mindful that early results may be deceptive, and you may need to test it again to make sure the solution has worked.
- Verify system functionality: Sometimes a solution that fixes one problem creates another. This is why it is important to verify full system functionality before you decide the solution was successful. In fact, it might be best to assume that the changes you make will affect the network in one way or another and determine how it will affect it.
- Document the problem and solution: Documenting the solution involves keeping a record of all the steps taken while solving the problem. Documenting both failures and successes can save you time in the future, and in large organizations keeping a record of the person who implemented the solution can be helpful if someone in the organization has a question about it.
If the problem was initially reported by an end user, you might also consider providing feedback to the user. Such feedback might not only encourage users to report problems in the future, but you might be able to provide information as to how the problem could have been avoided in the first place.
One of the ways we can evaluate problems is to use the seven-layer OSI model and try to determine what layer or layers the problem is on:
- Physical layer: This covers such problems as damaged or dirty cabling, or terminations, high levels of signal attenuation, and insufficient cable bandwidth. It also covers such problems as wireless interference or access points malfunctioning.
- Data link layer: This covers such problems as MAC address misconfiguration and VLAN misconfiguration or sub-optimal VLAN performance. It also encompasses some protocol issues such as improper L2TP or OSPF configuration.
- Network layer: This covers such problems as damaged or defective networking devices, misconfigured devices, sub-optimal device configuration, authentication issues, and lack of sufficient network bandwidth.
- Transport layer: This covers issues with the TCP and UDP protocols.
- Session/Presentation/Application layers: This covers problems related to applications and application layer protocols (for example, FTP and SMTP).
There are also some issues that come up so often in the networking world that we would be remiss if we did not take note of them. You can consider the following problems as obvious possible causes that, in many scenarios, should be considered before other, less obvious possible causes are considered.
This one is very simple: if the subnet mask specified on the host does not match the subnet mask for the network, then communicating with the network will not be possible. Finding the subnet mask is usually quite easy. In recent versions of Windows, such information can be obtained by navigating to Settings | Network Connections, right clicking on the network adapter you are using, and selecting Properties. There should be a listbox displaying the installed protocols; scroll down to Internet Protocol Version 6 or Internet Protocol Version 4 depending on what version you are using, and click on the Properties button. This should show you what the subnet mask is.
In Linux, finding the subnet mask is just as easy. For example, in Ubuntu or Mint Linux, click on the networking icon in the tray on the right end of the taskbar (the icon should look like two interconnected cables if you have a wired connection or a series of arcs if you have a wireless connection). This should launch the Network Connections dialog box. Find your connection (for example, Wired Connection 1) and double-click on it. This will launch the Editing dialog box, where you can change settings for the connection. Click on either the IPv4 Settings or IPv6 Settings tab to find the subnet mask.
Setting the gateway in Mint Linux.
Correct configuration of the subnet mask should allow intra-network communications, but a missing or misconfigured gateway setting will prevent a host from communicating with other networks. You should confirm that the gateway is set correctly; since the gateway is often specified in the same place as the subnet mask, this should not be difficult to do.
The Domain Name System (DNS) provides us with a means of translating hostnames into IP addresses. If a DNS server is not specified at all, then the host will not be able to take advantage of DNS services. If the correct DNS server is not specified (for example, if the primary server is incorrect), then DNS resolution can take longer than necessary, thus giving the impression that Internet access is slower than it actually is.
A good indicator that DNS configuration is incorrect is if you can ping a site when you specify the IP address, but the ping fails when you specify the hostname (and ping returns unknown hostname or a similar error). If this happens, you should definitely check the DNS configuration, first on the host and then on the firewall.
All IP addresses on a network must be unique. This includes network cards, routers and access points. If a network device is on the LAN, then its IP address must be unique on the LAN. If it is connected to the Internet, then it must be unique on the Internet. Problems with duplicate IP addresses can range from receiving error messages informing you of the existence of duplicate IP addresses to not being able to connect to the network from the device with the duplicate address.
Obviously, the use of DHCP, in which assignment of IP addresses is managed by the DHCP server, and IPv6, which has a greater number of private addresses to assign, greatly cuts down on the possibility of duplicate IP addresses. Duplicate addresses are most likely to happen on IPv4 networks in which the IP addresses are statically assigned.
As mentioned in Chapter 8, Routing and Bridging, there can only be a single path between two network devices. If there is more than one path, then the resultant looping can generate a broadcast storm that brings your network to its knees. It is especially a concern if you have bridged one or more interfaces on your network. One way to prevent looping is to manually configure network ports to ensure there is only one path to each device. The more likely scenario, however, is that you utilize a protocol such as Spanning Tree Protocol (STP), or its successor, Rapid Spanning Tree Protocol (RSTP).
Another situation where looping can occur is when the information in the routing table is incorrect, either through a manual misconfiguration or a failure in automatic route detection. Such errors are often easy to detect because, as with physical loops, they will quickly bog down a network.
In Chapter 8, Routing and Bridging, we described how to set up a static route in pfSense. Static routes can easily cause problems on a network, since a change in network topology can render a static route incorrect. Therefore, if you make changes to your network and you have static routes, you should consider how the changes impact these static routes and make changes to them accordingly.
By default, pfSense will block all ports on the WAN side of the router. Therefore, if a remote user tries to connect to a port on a local host, the user will be blocked from doing so. In order to connect to a port on a local host, there must be a port-forwarding rule forwarding the traffic to the host, and there must be a rule on the network to which the local host is connected permitting such traffic (in pfSense, NAT port forwarding has an option for auto-generating firewall rules that correspond to port forwarding rules, thus ensuring that both steps can be completed at the same time).
Sometimes, network traffic is dropped without the source ever being informed that the traffic never reached its intended target. The error can only be detected by monitoring network traffic. Such a situation is referred to as a black hole.
One such scenario is when a host tries to connect to an IP address that was assigned to a host that is down or to an IP address that was never assigned to a host. Although TCP has mechanisms for communicating a failure to connect back to the original host, often the packets are just dropped. Moreover, if you are using a protocol such as UDP that is both connectionless and unreliable, then there is no means of communicating back to the original host that the IP address is dead.
Another common situation is with Maximum Transmission Unit (MTU) black holes. This happens when an MTU packet is larger than the maximum MTU size allowed on a network, and the Don't Fragment (DF) flag is set in the IP header. If this happens, any device whose MTU is smaller than the packet's size will drop the packet. The solution here is to make sure Path MTU Discovery (PMTUD) is running on all network devices. PMTUD solves this problem by sending back a Fragmentation Needed ICMP message back, thus causing the offending device to reduce its MTU size. Some network devices block ICMP messages for security reasons, however, and if this is the case on your network you could end up with black hole connections: the TCP three-way handshake will be completed, but when data is transferred, the connection will hang because of the MTU size mismatch.
One possible solution is to use the RFC 4821 version of PMTUD. This version uses TCP or another protocol to probe the patch with progressively larger packets. Another solution is to change the maximum segment size (MSS) of all TCP connections lower than the Ethernet default of 1500.
You should also be aware that there are many issues related to cabling that can cause problems. The most common form of network cabling used in homes and offices is unshielded twisted pair (UTP), with fiber-optic cabling being the more expensive alternative. UTP cabling is susceptible to various forms of interference. One such form is crosstalk, when the signal from one cable bleeds into another cable. This often happens when two cables are run too close to each other. Near end crosstalk (NEXT) is when an outgoing data transmission leaks into an incoming transmission. Far end crosstalk (FEXT) is when a transmitting station at the other end of a transmission line leaks into the receiving line. One of the ways you can minimize crosstalk is to purchase high-quality UTP cable, in which the twisted pairs are twisted more tightly; the greater the number of twists, the less crosstalk.
EMI can also reduce signal strength. Computer monitors and fluorescent lighting both create an electromagnetic field, and can cause problems with UTP network cabling. Radio frequency interference (RFI) from objects such as cell phones can also be an issue. The solution to this problem is to run network cabling away from such devices.
The signal in a UTP cable is susceptible to attenuation if the cable is too long. Keep in mind that the maximum length for Cat 5 and Cat 6 UTP cabling is 100 meters. If there are intermittent network problems and you notice that the cables are too long, this may be the problem. If you can't shorten the cable run, then installing a repeater will solve this problem.
One way to avoid all these problems is to install fiber-optic cabling. Because fiber-based media uses light transmissions instead of electronic signals, the issues discussed in this section, such as crosstalk, EMI, and attenuation, become nonissues. Fiber-optic cabling is also a secure medium, as accessing the data signals requires physically tapping into the media, which is difficult to do.
Unfortunately, the high cost of fiber-opting cabling precludes a lot of organizations from implementing it in their networks. Moreover, fiber-optic media is incompatible with most electronic equipment, requiring you to purchase fiber-compatible network equipment. Thus, while fiber-optic cabling will continue to play a role in networking, particularly in serving as the media for the Internet, WANs, and metropolitan area networks (MANs), its impact on smaller networks will likely be limited for the foreseeable future.