One of the most troublesome problems in communication networks is the broadcast and error storms. These problems can happen because of layer 2 loops, layer-2-based attacks, a problematic network adapter, or a service that sends packets to the network.
In this chapter we will provide some basic recipes on how to find, isolate, and solve these types of problems.
In these types of problems, you will usually be called on to solve the network is very slow or network has stopped working problems.
Several important facts to remember are:
- Broadcasts are not forwarded by routers.
- Broadcasts are not forwarded between VLANs (this is why VLANs are called broadcast domains), so every VLAN is a single broadcast domain.
- Error packets are not forwarded by LAN switches (at least not through the good ones).
- Multicasts are forwarded through switches, unless configured otherwise.
- Multicasts are forwarded through routers only if the routers are configured to do so.
- A reasonable number of broadcasts are transmitted in every network. This is how networks work. Too many broadcasts could be a problem.
To find out where the problem comes from, go through the following steps:
- Since "slow network" is a problem sensed by users, start with asking the following questions:
- Is this problem in the HQ?
- In a single branch?
- All over the network or a specific VLAN?
Don't ask the users about VLANs, of course; users are not networking experts. Ask them about applications running on their group, on their department, and so on.
Tip
In an organization network, VLAN will usually be configured per department (or several departments) and per geographical area (or several areas) or even per organization functionality; for example, HR VLAN, finance VLAN, users of a specific software VLAN, and so on. By asking if the problem is as per one of these characters, you will be able to narrow the area in which you need to look for the problem.
- The next question should be a trivial one: "Is the network still working?" In a broadcast storm, the network will become very slow; in most cases, to the point that applications will stop functioning. In this case, you have the following typical problems:
- Spanning Tree Problems
- A device that generates broadcasts
- Routing loops (will be discussed in Chapter 8, ARP and IP Analysis)
Tip
The question I'm always asked is: "How many broadcasts are too many?" Well, there are, of course, several answers for this. It depends on what the network devices are doing and the protocols that are running on them.
A reasonable number of broadcasts should be from 1 to 2 up to 4 to 5 per device per minute. For example, if your network is built from 100 devices on a single VLAN, you should expect no more then 5-10 broadcasts per second (5 broadcasts x 100 devices gives 500 broadcasts per minute, that is, around 9-10 per second). More than these is also reasonable, as long as they are not coming in thousands and you know what they are.
In Spanning Tree Problem, you will get thousands and even tens of thousands of broadcasts per second (refer to the How it works... section in this recipe to know why). In this case, your Wireshark, and probably your laptop, will freeze. Close Wireshark, disconnect cables to isolate the problem, and check the STP configuration in the switches.
A typical broadcast storm generated from a specific device will have the following characteristics:
We can see how we find a broadcast storm according to the parameters mentioned in the preceding list in the next three screenshots.
In the following screenshot we see a large number of broadcast packets sent from the source MAC (HP network adapter) to ff:ff:ff:ff:ff:ff:
In the preceding screenshot we just saw that the time column is configured in seconds since the previous displayed packet. You can configure it by navigating to View | Time Display Format.
The following screenshot shows the traffic on an IO Graph; we see that the total number of packets/second is 5,000:
In the following screenshot we see what we will get in the Conversations window. Here, we will also get an enormous number of broadcasts that can be viewed in the Ethernet and the IPv4 statistics (I've captured data for 18 seconds).
In the preceding case, the problem was a service called SMB Mailslot Protocol. Simple trial and error to find what this service is, and disabling it on the station that caused it solved the problem.
You can also have broadcasts in fixed patterns, for example, every fixed amount of time, as shown in the following screenshot:
The graph is configured for Tick interval: (under X Axis) of 1 min, and for the following filters:
- The red filter for all broadcasts in the network (
eth.addr == ff:ff:ff:ff:ff:ff) - The green filter for broadcasts that are ARP requests (
arp.opcode ==1)
What we see here is that around every five minutes, there is a burst of ARP requests (the green dots). If we click on one of the dots in the graph, it will take us to the packet in the capture pane. In the following screenshot, we see the scan pattern that happens every five minutes:
In the preceding screenshot, we can see that it is the D-Link router that scans the internal network. This can be good or bad, and we will get to the details later in Chapter 14, Understanding Network Security. In any case, it's good to check what is running in our network.
Broadcasts in IPv4 networks are quite common, and these layer 3 broadcasts will be sent over layer 2 broadcasts. Every time a layer 3 device sends a broadcast to the network (an IP address that ends with all 1s, refer to Chapter 8, ARP and IP Analysis), it will be converted to layer 2's all fs destination address.
There are several families of broadcasts that you will see in IP-based networks. Some of them are as follows:
- TCP/IP-based network protocols such as ARP requests, DHCP requests, and others
- Network protocols such as NetBIOS Name Service (NBNS) queries, NetBIOS Server Message Block (SMB) announcements, Network Time Protocol (NTP), and others
- Applications that send broadcasts such as Dropbox, Microsoft Network Load Balancing, and others
In IPv6, we don't have broadcasts, but we have unicasts, multicasts, and anycasts. Since the protocol works with multicasts for discovery mechanisms, announcements, and other mechanisms, we will see a lot of them.
A problem I come across in many cases is how to use the broadcast and multicast storm control definitions in LAN switches (the storm-control broadcast level [high level] [lower level] command in Cisco devices).
The problem is that in many cases, I see configurations that limit the number of broadcasts to 50, 100, or 200 broadcasts per second, and this is not enough. In a network, you could happen to install a software that sends broadcasts or multicasts to the network that crosses these values. Then, according to what you have configured in the switch, it will start sending traps to the management system or even disconnecting ports (the storm-control action {shutdown | trap} command in Cisco devices), depending on what you have configured.
The solution for this is simply to configure high levels of broadcasts as the threshold. When a broadcast storm happens, you will get thousands of broadcasts; so configuring a threshold level of 1,000 to 2,000 broadcasts or multicasts per second provides you with the same protection level, without any disturbances to the regular network operation.
- For more information about IPv4 refer to Chapter 8, ARP and IP Analysis