Introduction to network traffic analysis
Overview of Wireshark
Installing Wireshark
Setting up port mirroring
Introduction to Network Traffic Analysis
Simple: A network design should be simple. Most practitioners in the network field are familiar with the KISS principle: Keep It Simple, Stupid. A network is dependent on various technologies, protocols, hardware and software resources, and so on. Even though each of these components might be simple individually, their combination in a network will always add to the complexity. Identifying problems in large-scale networks is often like finding a needle in a haystack. It thus becomes more important that the network design and architecture are kept as simple as possible.
Highly available: Almost every network is designed to carry traffic for critical business applications, and a small network event could have a massive impact on the services provided by an organization. Thus, it is important to build redundancy into the network such that in the case of a failure event, the availability of services is maintained. Although redundancy in a network is vital, it is equally important to understand and define how much redundancy is acceptable. More redundant paths in the network result in higher costs.
Robust: As stated in the IEEE document “Robust Network Design,” robustness is defined as minimizing variations in network performance, such as average delay and throughput, due to perturbations in the network like topology, demand, and community of interest. If a network design as well as its requirements are not thought through in depth, the network will be affected with increased average delays and throughput or performance issues over a period of time.
Scalable: Because the scale of the application and user traffic are constantly growing, the ability to scale the existing network infrastructure holds great importance. While designing computer networks, it is important to choose designs that will allow you to scale the network horizontally as well as vertically as and when required. In other words, the network design should allow the organization to scale the network for east–west as well as north–south traffic on an on-demand basis. One such example is the Clos network design, which primarily focuses on the spine–leaf architecture and has been widely adopted by large-scale datacenter networks to help them cater to the increased traffic demand over the years.
Futuristic: In designing networks, it is also important to choose the right set of hardware and software. The right hardware and software choices will allow you to leverage the latest technologies and current innovations in the network industry and will enable you to upgrade the firmware or the network operating system to access the latest available features.
Even when all of these factors are taken into consideration while designing a network, every network still has to undergo changes to overcome dynamic application and resource demands. In addition, every network, irrespective of how carefully it has been designed, is prone to network issues. As the network grows, its complexity also grows. It is only a matter of time before a network problem arises and network engineers are called on to solve the problem. Network problems are usually difficult to manage, and the complexity is even greater when we have multiple features and encapsulations being used in the network.
Shutting down a bad or faulty link
Rebooting or shutting down a network device (e.g., router, switch, firewall, etc.)
Diverting traffic by adjusting the routing metrics
Flapping a routing adjacency
Some issues could take longer time to resolve, especially when it comes to software or hardware defects. Among the different network-related issues faced by network engineers, issues such as continuous or intermittent packet loss, latency, or routing or switching issues require a deeper analysis. Other issues can be quickly mitigated by replacing the hardware or cable or a particular port, for example, but issues related to packet loss or routing problems involve multiple elements in the network that cannot be identified and mitigated quickly. Such issues sometimes require more visibility at the packet level as to what data are being transmitted on the wire. That said, it is now time to understand what network analysis is.
Understanding network characteristics
Analyzing protocol behaviors
Troubleshooting slowness in the network
Troubleshooting packet forwarding issues
Identifying vulnerabilities in the network protocols and ciphers
Identifying malicious activities in the network
Collecting real-time information on activities between different network elements
Better visibility into the network allows the network administrators to optimize performance, enhance the security posture of the network, minimize the blast radius of a network attack, and better analyze the utilization of network resources. The packets collected in the network also give network administrators a better understanding of how the network users are implementing their applications. Techniques such as deep packet inspection (DPI) allow complete network visibility by transforming the raw packet data as well as metadata into a readable format.
Tcpdump: This is a powerful CLI-based packet analyzer tool that is freely available and runs on Linux or most UNIX-like operating systems.
Omnipeek: This is a GUI-based commercial packet analyzer tool from Savvius, a LiveAction company.
Wireshark: Wireshark is a free, open source, GUI-based packet analyzer available for download on various operating systems.
In this book, we primarily focus on Wireshark. Covering different network analyzer tools is outside the scope of the book.
Network Sniffing
- 1.
Placement of the sniffer in the network
- 2.
The number of sniffer placements
We discuss both these points in detail.
Sniffer Placement
For a capture device to be able to capture packets, the network interface card (NIC) should support promiscuous mode. A promiscuous mode driver allows a NIC to view all packets crossing the wire. When tools such as Wireshark are installed on the capture device, they also install a libpcap or WinPcap driver on the device. These drivers allow the NIC to switch to promiscuous mode and capture packets across the network.
When did the problem start?
What is the problematic source and destination?
What is the working source and destination?
What is the relevant topology between the source and destination?
How many users and services are affected?
What was the trigger of the problem?
Verify Address Resolution Protocol (ARP) information: Verify if the ARP entry is complete for host H4 or H5 versus host H6.
Traffic pattern: Verify if the issue is sending broadcast traffic or unicast traffic. If the ARP table on host H1 shows that the ARP entry is present or is getting completed for host H4 and H5 even after clearing the ARP, then it might be a problem just with the unicast traffic instead of all traffic (broadcast, unicast, or multicast traffic) between H1 and H4 or H5.
Path information: Perform traceroutes between working and nonworking hosts to identify any difference in paths taken by each of them. If the network has equal cost multiple paths (ECMP), then most routing and switching platforms perform flow-based hashing to send traffic out on one of the interfaces. If the traceroute fails at one of the hops in the path, that would indicate the problem might be isolated to that segment of the network. Note that it is important to perform traceroutes from both endpoints so that any possibility of asymmetrical routing could be detected.
Access control lists (ACLs): Leverage ACLs whenever possible to isolate where the traffic loss is happening. Users can configure both Layer 3 ACLs (standard and extended) and Layer 2 ACLs (media access control [MAC] ACLs) to match Layer 3 as well as Layer 2 traffic at different segments of the network. However, there could be instances where ACLs might not be of any help. For instance, ACLs do not allow users to capture Multiprotocol Label Switching (MPLS) packets. Thus, it becomes important to identify the kind of packets being investigated during troubleshooting.
Hop-by-hop ping tests: If a traceroute fails at a segment of the network, it might make sense to check the reachability of the source device to that segment of the network. It is possible that only the transit traffic might be affected and not the traffic destined for those devices in the segment. This usually happens if there is an ACL blocking the traffic in the path or due to a software misprogramming (software defect). In such instances, ping tests should be performed before performing deep-dive troubleshooting.
Platform troubleshooting tools: Most routing and switching platforms come with troubleshooting tools as part of the network operating system (OS). These platform troubleshooting tools can help you understand if a packet is being dropped on the device itself or not and why. These tools are primarily helpful when the issue has been isolated to a particular device or a network segment. Note that some network OS’s come with platform-level packet capture tools. These tools can be very useful to perform packet captures to understand if the packet is being received on the device or not and if what action is being taken on the packet by the network OS.
Debugs: It might sometimes help to run debugs on the network devices. The debug logs allow you to gain more insight on what is happening on the network device. For instance, if a BGP prefix is not being received, you could run a BGP protocol debug to understand if the prefix is being received or not.
When dealing with corrupted packets
Gathering more information about the packet headers, as they might be affecting forwarding decision
Troubleshooting encapsulated packets
Troubleshooting packet loss or retransmission issues
Voice or video traffic-related issues
Protocol issues such as OSPF not forming adjacency due to wrong information being exchanged or BGP not establishing peering due to TCP or wrong or missing information in BGP packets
Access switch connected to the source and destination host
Core switch(es) at each site
WAN routers (if they support enabling port spanning)
The captures taken at each site can easily help determine where the packet loss is happening. Even though the sniffer captures help a lot in investigating the issue, that is not the final step of the troubleshooting process. There are a few more steps that are involved in mitigating and remediating the problem, which we will see in the coming chapters based on different problem scenarios.
Number of Sniffer Placements
Placing sniffers is not always easy. Each organization, be it an enterprise, service provider, or datacenter, has its own set of policies for managing and troubleshooting in its network environment. Most organizations require scheduling a change and maintenance window to perform troubleshooting, let alone performing sniffer captures. Also, when the whole network environment that is under investigation is geographically displaced or made up of remote unmanned sites, it takes a while to get field engineers on site to help with sniffer captures. Further, when the sniffer captures are to be performed at multiple places in the network, the complexity is compounded. When the troubleshooting requires sniffer captures in the network, it is important that the points of placement should be carefully considered before actually enabling port spanning.
Some network environments are also set up in a way that supports remote spanning with specific hosts configured to collect mirrored traffic. Such network deployments allow users to perform port spanning at almost any given node in the network without having to wait for any human to be present on site. The only limitation of such deployments is either the support for the remote span feature on all network devices or the host or the switch performance with higher throughput interfaces.
In the example discussed in Figure 1-2, the ideal process would be to isolate the segment where the problem is and then place sniffer captures in that segment. If there are issues related to voice traffic such as users facing choppy voice or even TCP retransmission issues, it would require sniffer placements at multiple points across the network to determine where the issue is actually happening. For such a huge span of segments to troubleshoot, the approach for performing packet captures should be to isolate between the internal network versus the ISP network. For instance, the sniffer placements between the access and core or WAN layer at each site will allow us to identify if the issue is local to any of the two sites. If the packet sent from one site is not received by the WAN router on the remote site, that means the issue would be isolated to the ISP network instead of the site local networks.
Network Tap
Aggregated: The aggregated network taps allow bundling of multiple streams of data across multiple ports to one monitoring port. This type of network tap is useful when it is required to monitor bidirectional streams of traffic but only one NIC for monitoring.
Nonaggregated: The nonaggregated network taps provide additional flexibility for capturing traffic but also add to complexity when compared to aggregated network taps. In nonaggregated network taps, two ports are required for monitoring purposes, each of them capturing traffic in only one direction.
Based on the monitoring requirements, the choice can be made between aggregated and nonaggregated network taps.
So far, we have learned about the port spanning and network taps that can be used to enable and perform packet captures in the network. Next, we learn about the Wireshark tool that will be used for analyzing the captured traffic.
Overview of Wireshark
Wireshark is a widely used open source network protocol analyzer. The first version of the application was called Ethereal and was developed and released by Gerald Combs in 1998 under the GNU Public License (GPL). After some conflicts over the Ethereal brand rights with his employer, Combs, along with the rest of the development team, rebranded the project as Wireshark in mid-2006. Wireshark is freely available for personal, educational, and commercial purposes and is supported and maintained by a community of more than 1,800 developers.
It is the go-to tool for almost every network administrator or network engineer to analyze network traffic patterns, troubleshoot network protocol issues, and perform in-depth analysis of network security loopholes. Wireshark comes with tons of features, supports the most common and uncommon set of protocols and encapsulations, and is supported on all the well-known OSs. It provides an easy-to-use and easy-to-understand GUI and advanced filtering capabilities to search through millions of packets to allow network administrators to quickly analyze the events in the network.
Installing Wireshark
At the time of writing, the latest and stable version of Wireshark is 3.4.4. Wireshark installer is available in both 32-bit and 64-bit versions and have builds available for Windows, Mac, and various Linux OSs. Wireshark installer can be downloaded from https://www.wireshark.org/download.html. Installation of Wireshark is fairly simple. In the section, we cover the installation of Wireshark on different OSs.
Installing Wireshark on Windows
Download the installer (.exe file) from https://www.wireshark.org.
Double-click the installer to begin the installation process.
Click Next to begin the installation.
Acknowledge the License Agreement by clicking Noted.
Select the components that you want to install as shown in Figure 1-3 and click Next.
In the component selection you can see an option to install TShark. TShark is a CLI version of Wireshark, which is designed to capture and analyze network traffic. It supports the same options as Wireshark. To view all the options of TShark, use the command man tshark or tshark --help option.
Select the different shortcuts that you want to place on your PC and click Next.
Select the installation directory for Wireshark and click Next.
Select the Npcap or WinPcap version that is currently installed or is available to install and click Next. Note that Npcap or WinPcap is required by Wireshark to capture live network packets.
Optionally, you can install USBPcap to capture USB traffic and click Install to begin the installation process.
During the installation, another installer window will open for Npcap or WinPcap software. Select the necessary installation options and begin the installation process by clicking Install.
Once the installation completes, click Finish.
The Wireshark installation will continue further.
Once the installation process is completed, click Finish. At this point, Wireshark is now ready to perform packet captures on your system.
Installing Wireshark on Mac
Download the installer (.dmg file) from https://www.wireshark.org.
Double-click the installer file to begin the extraction process.
The extraction process will create a volume with all the necessary files on the desktop.
Once the extraction is completed, a pop-up window gives you an option to move the Wireshark app into the Applications directory
Drag the Wireshark app into the Applications directory to make Wireshark accessible from the launch pad.
Installing Wireshark on Ubuntu
Update the repository on the Ubuntu machine using the command apt update.
Install Wireshark using the command apt install wireshark.
Installing tshark on Ubuntu
Once set up, the GUI-based Wireshark app or CLI-based tshark app can be used to capture traffic traversing the network.
It is important to learn how to capture the packets and analyze the network traffic, but it is equally important to know the tools available with different network devices that you can use to set up packet captures.
Setting Up Port Mirroring
At the transmitting device or the device that initiated the packet
At the receiving device or the device for which the packet is destined
At the transit device
In the transmission media
The packet-level issues require the network engineers to perform packet captures and investigate the issues by analyzing the captured traffic. In most cases, switching devices have the capability of mirroring network traffic and sending it to a mirroring port that is connected to a PC. Let’s see how port mirroring can be enabled on different vendor devices.
SPAN on Cisco IOS/IOS-XE
Configuring SPAN
As per the Cisco.com documentation, you cannot have two SPAN sessions using the same destination port.
SPAN Session Verification
Refer to the Cisco online product documentation to verify how many SPAN sessions can be configured. The supported number of SPAN sessions varies from platform to platform and from vendor to vendor.
SPAN on Cisco Nexus Switches
Configuring SPAN Session on Cisco NX-OS
Verifying SPAN Session on Cisco NX-OS
SPAN Sessions with Filtering
Enabling Port Mirroring on Arista EOS
Configuring Port Mirroring on Arista
Enabling Port Mirroring on JunOS
Packets entering or exiting a port
Packets entering or exiting a VLAN or a bridge domain
Policy-based sample packets
For policy-based sample packets, a firewall filter with a policy is configured to mirror the packets. The sample traffic based on the firewall filter can be sent to the port-mirroring instance for further analysis.
Get into forwarding-options configuration mode.
Define a name for the analyzer and specify the input interface along with the direction of the traffic you wish to capture.
Choose the destination interface.
Commit the configuration.
Configuring Port Mirroring on JunOS
So far, we have seen how to configure local port mirroring on various vendor devices running their respective network OS. Similarly, you can also set up remote port mirroring. In remote port mirroring, the configuration is pretty much the same as local port mirroring with the minute difference that the destination interface does not reside on the local device, but is multiple hops away. Each vendor has its own method of implementing remote port mirroring. Unless necessary, it is not required to set up remote port mirroring.
Summary
In this chapter, we learned what NTA is and why it is important in the network. We also learned the factors that should be considered when implementing port mirroring and how we can set up the minimum number of capture points in the network to isolate a problem in the network. Unless it is necessary, one should avoid enabling port mirroring on network devices. Further, we learned what Wireshark is and how to install it on various OSs. Finally, we concluded the chapter by seeing how port mirroring can be enabled on network devices from different vendors.
References in This Chapter
Wireshark: https://www.wireshark.org
Network Management Configuration Guide: https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/16-10/configuration_guide/nmgmt/b_1610_nmgmt_9300_cg/configuring_span_and_rspan.html
Cisco Nexus 9000 Series NX-OS System Management Configuration Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x_chapter_010000.html
Arista EOS Introduction to Port Mirroring: https://eos.arista.com/introduction-to-port-mirroring/
Juniper Port Mirroring and Analyzers: https://www.juniper.net/documentation/us/en/software/junos/network-mgmt/topics/topic-map/port-mirroring-and-analyzers.html