Chapter 6. Command Line Packet Analysis Tools
In Chapter 3 and Chapter 4 we installed the SO software in several configurations, and we discussed housekeeping functions in Chapter 5. Now that you have this powerful NSM platform collecting data, in this chapter I’ll introduce the first set of command line tools used to present information to analysts. Some of these tools will be running all the time, while others will be invoked on demand. Each has its particular strengths and weaknesses. I’ll discuss how I use key features, though I won’t cover all tools in exhaustive detail here.
Because I’ve written this book for new analysts, my discussion of SO tools in this part will concentrate on data presentation. In this chapter I will look at data presentation tools that use a command line interface. In Chapter 7 I’ll address data presentation tools that use a graphical interface, and in Chapter 8 I’ll examine specialized forms of data presentation tools—the NSM consoles. For now, let’s step back and understand how all the NSM tools packaged with SO relate to one another.
SO Tool Categories
SO ships with a variety of tools, as listed on the SO wiki (http://code.google.com/p/security-onion/wiki/Tools). Some tools present data to analysts, some collect data directly from the network or via messages from other computers, and a third category sits between the others as middleware, delivering data or providing other essential capabilities. Let’s take a brief look at each category of tools: data presentation, data collection, and data delivery.
SO Data Presentation Tools
Data presentation tools expose NSM information to analysts. Two sorts of data presentation tools for packet analysis are available in SO. One relies on a command line interface, and the other offers analysts a graphical interface. SO also provides NSM consoles for data presentation.
Packet Analysis Tools
Packet analysis tools read network traffic from a live interface, or from a file containing traffic saved in pcap format. Analysts use packet analysis tools to better interpret network traffic, but not necessarily to implement an NSM-specific investigation or workflow. Some of these tools help analysts better understand individual packets, others group packets into sessions, and still others examine application data. The authors of these tools generally did not build them with NSM in mind, but nevertheless, they are key to understanding network traffic.
Two sorts of data presentation tools for packet analysis are available with SO. One relies on a command line interface. These tools include Tcpdump, Tshark, and the Argus Ra client, all examined in this chapter. Because certain uses of Tshark depend on a related data collection tool, Dumpcap, I’ll present it along with Tshark. The second sort of tool for packet analysis offers analysts a graphical interface. Wireshark, Xplico, and NetworkMiner are examples of this sort of software, and I discuss them in Chapter 7.
NSM Consoles
NSM consoles were built with NSM-specific investigation and workflows in mind. The console authors began with the core NSM principles and implemented them in software. These tools also function as data presentation applications, but they act more as gateways to NSM data. Software in this category includes Sguil, Squert, Snorby, and ELSA. I’ll explain how to use these NSM consoles in Chapter 8.
SO Data Collection Tools
Once NSM analysts become comfortable with the data presentation tools, they turn to data collection tools. Software in this category includes the Argus server, Netsniff-ng, Passive Real-Time Asset Detection System (PRADS), Snort, Suricata, and Bro. (Dumpcap belongs in this category as well, but SO does not enable it by default.) These applications collect and generate the NSM data available to the presentation tools.
The Argus server and PRADS create and store their own forms of session data. Argus data is stored in a proprietary binary format suited for rapid command line mining, whereas PRADS data is best read through an NSM console. Analysts can choose which form of data suits them best.
Netsniff-ng simply writes full content data to disk in pcap format. Snort and Suricata are network intrusion detection systems, inspecting traffic and writing alerts according to the signatures deployed with each tool. Bro observes and interprets traffic that has been generated and logged as a variety of NSM datatypes.
In the default configuration enabled by the SO platform, all of these applications provide a wealth of NSM data to the presentation tools discussed in this chapter and the next two.
SO Data Delivery Tools
Finally, between the data presentation and data collection tools sits a suite of data delivery applications. Broadly speaking, this middleware enables the functionality of the other categories of software on the SO platform. Tools like PulledPork, Barnyard2, and CapMe manage IDS rules, alert processing, and pcap access, respectively.
A suite of “agents” associated with Sguil—such as pcap_agent, snort_agent, and the like—shuttle data from the collection tools to the presentation software. This includes the Apache web server, the MySQL database, and the Sphinx index application, which may already be familiar to you.
Finally, SO includes tools for integrating certain host-centric analysis features. These include the OSSEC host IDS and Syslog-ng for transport and aggregation of log messages. Because this book concentrates on network-centric data, we won’t examine data from OSSEC and Syslog-ng, but you should know that those components are running on SO platforms.
Figure 6-1 shows the core SO tools in relation to one another. This chapter covers the tools Tcpdump, Tshark, Dumpcap, and the Argus Ra client. Chapter 7 covers Wireshark, Xplico, and NetworkMiner. Chapter 8 discusses the NSM consoles Sguil, Snorby, Squert, and ELSA. We’ll begin our look at data presentation tools with Tcpdump.
Running Tcpdump
Tcpdump (http://www.tcpdump.org/) is a command line network traffic analyzer. Tcpdump is available on SO, but it is not running by default. Analysts can invoke it on demand, most often to view data stored in /nsm/sensor_data/<sensorname>/dailylogs.
Note
Bill Fenner, David Young, Fulvio Risso, Guy Harris, Hannes Gredler, and Michael Richardson are the current Tcpdump maintainers, and they code under a three-clause BSD license. (See the Tcpdump CREDITS file at http://svnweb.freebsd.org/base/vendor/tcpdump/4.3.0/CREDITS?revision=241212&view=markup for all contributors.) They also develop the libpcap traffic capture library under the same license. Van Jacobson, Craig Leres, and Steven McCanne wrote the original version in 1987 while working at the Lawrence Berkeley Laboratory Network Research Group.
Tcpdump works against a live network interface or a saved trace file. It can display results in real time or write output to a file.
Tcpdump is a protocol analyzer because it can depict multiple layers of detail for any traffic it understands. As a protocol analyzer, its rendition of network traffic depends on its ability to decode the data it sees. Without knowledge of the underlying protocols, Tcpdump could produce only a byte stream that analysts would need to decode manually.
Displaying, Writing, and Reading Traffic with Tcpdump
Tcpdump runs in a command terminal. To display live traffic in real time, run it with these options:
$ tcpdump -n -i <interface> -s <snaplen> -c <count>The -n switch tells Tcpdump to not resolve IP addresses to hostnames via DNS queries. I always run Tcpdump with the -n switch to avoid waiting while the tool resolves IP addresses to hostnames via DNS. The -i switch tells it which interface to monitor. The -s switch tells it how many bytes to capture from each packet. By default Tcpdump captures 68 bytes for IPv4 packets and 96 bytes for IPv6 packets. (Use -s 0 to capture the entire packet, or specify a value appropriate for the medium from which you are capturing.) Finally, -c tells Tcpdump how many packets to capture. (If you forget this switch, Tcpdump will run until you stop it with ctrl-C.)
Example 6-1 shows some example output. Tcpdump requires elevated privileges to sniff traffic in promiscuous mode, so preface the command with sudo.
$sudo tcpdump -n -i eth1 -c 5tcpdump: WARNING: eth1: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes19:48:51.723139 IP 192.168.2.120.55060 > 205.233.0.226.443: UDP, length 461
19:48:51.886312 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [P.], seq 928328861:928329246, ack 1080949825, win 39, length 385
19:48:51.898576 IP 192.168.2.104.49608 > 69.171.246.17.443: Flags [P.], seq 1:978, ack 385, win 4220, length 977
19:48:51.914324 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [.], ack 978, win 45, length 0
19:48:51.915284 IP 69.171.246.17.443 > 192.168.2.104.49608: Flags [P.], seq 385:823, ack 978, win 45, length 438 5 packets captured 5 packets received by filter 0 packets dropped by kernel
This traffic includes one User Datagram Protocol (UDP) packet , followed by four Transmission Control Protocol (TCP) packets (, , , and ). The UDP traffic has the following format:
timestamp / layer 3 protocol / source IP address.source port > destination IP address.destination port: layer 4 protocol / data length
The format for the TCP traffic is similar:
timestamp / layer 3 protocol / source IP address.source port > destination IP address.destination port: layer 4 protocol / TCP flags, TCP sequence numbers, TCP acknowledgement numbers, TCP window size, data length
Note
The time in this trace is UTC. When you configure SO, it sets the local clock to use UTC, so expect to see UTC timestamps in network evidence. In files saved in libpcap format, time is stored as the number of seconds and microseconds since the Unix “epoch time” of January 1, 1970. The local system then translates this value into the time displayed by a network tool.
To save traffic to disk while watching a live interface, add the -w switch followed by the target filename. Example 6-2 shows how to accomplish this task.
$ sudo tcpdump -n -i eth1 -c 5 -w demo1.pcap
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
5 packets captured
5 packets received by filter
0 packets dropped by kernelTo read the traffic, use the -r switch. (The sudo command isn’t needed because you’re reading from a trace, not eth1.) Example 6-3 shows the results of reading five captured packets.
$ tcpdump -n -r demo1.pcap
reading from file demol.pcap, link-type EN10MB (Ethernet)
20:23:44.858470 IP 74.125.228.54.443 > 192.168.2.104.49945:
Flags [P.], seq 1145489012:1145489069, ack 1920080636, win 4132, length 57
20:23:44.859134 IP 74.125.228.54.443 > 192.168.2.104.49945:
Flags [P.], seq 57:1407, ack 1, win 4132, length 1350
20:23:44.859154 IP 74.125.228.54.443 > 192.168.2.104.49945:
Flags [P.], seq 1407:2757, ack 1, win 4132, length 1350
20:23:44.859505 IP 74.125.228.54.443 > 192.168.2.104.49945:
Flags [P.], seq 2757:4107, ack 1, win 4132, length 1350
20:23:44.860006 IP 74.125.228.54.443 > 192.168.2.104.49945:
Flags [P.], seq 4107:4261, ack 1, win 4132, length 154Using Filters with Tcpdump
Along with displaying, writing, and reading traffic, the other core usage for Tcpdump involves applying filters. Filters are a mechanism to limit the traffic shown or captured by Tcpdump and other tools. The popular term for filters is BPF, a nod to the Berkeley Packet Filter virtual machine, which translates the human-readable filter syntax into a code syntax suitable for machine consumption.
Applying Filters
You apply a BPF by appending it to the Tcpdump command line. For example, to capture only ICMP traffic, add icmp to the syntax, as shown in Example 6-4 ().
To read the trace, use Tcpdump again, as shown in Example 6-5.
$ tcpdump -n -r icmp.pcap
reading from file icmp.pcap, link-type EN10MB (Ethernet)
20:30:28.203723 IP 172.16.2.1 > 172.16.2.2: ICMP echo request,
id 20822, seq 44313, length 44
20:30:28.204282 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id
20822, seq 44313, length 44
20:30:28.844237 IP 192.168.2.108 > 173.194.75.104: ICMP echo
request, id 1, seq 5, length 40
20:30:28.871534 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply,
id 1, seq 5, length 40
20:30:29.213917 IP 172.16.2.1 > 172.16.2.2: ICMP echo request, id
20822, seq 44569, length 44
20:30:29.214475 IP 172.16.2.2 > 172.16.2.1: ICMP echo reply, id
20822, seq 44569, length 44
20:30:29.850913 IP 192.168.2.108 > 173.194.75.104: ICMP echo request,
id 1, seq 6, length 40
20:30:29.875103 IP 173.194.75.104 > 192.168.2.108: ICMP echo reply,
id 1, seq 6, length 40
20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request,
id 47441, seq 1, length 64
20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply,
id 47441, seq 1, length 64Instead of using icmp, you can capture other specific traffic by using options like tcp, udp, and so on. For example, you can collect traffic for a specified TCP or UDP port, like port 53, as shown in Example 6-6.
$ sudo tcpdump -n -i eth1 -s 0 port 53
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
20:53:42.685078 IP 192.168.2.106.33348 > 172.16.2.1.53: 55862+ A?
daisy.ubuntu.com. (34)
20:53:42.701421 IP 172.16.2.1.53 > 192.168.2.106.33348: 55862 2/0/0 A 91.189.95.54, A
91.189.95.55 (66)Example 6-6 captures UDP or TCP traffic on port 53. To capture port 53 and TCP traffic only, modify the filter as shown in Example 6-7.
$ sudo tcpdump -n -i eth1 -s 0 port 53 and tcp
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
21:02:06.430169 IP 192.168.2.126.44334 > 8.8.8.8.53: Flags [S],
seq 1330246822, win 42340,
options [mss 1460,sackOK,TS val 157066547 ecr 0,nop,wscale 11], length 0The manual page for pcap-filter included with SO shows all available options. View it by entering man pcap-filter at a command terminal.
Some Common Filters
Now let’s look at some of the more common filters for showing traffic to or from particular hosts and even networks.
To show traffic to or from a specific computer, use the host BPF, as shown in Example 6-8.
$ tcpdump -n -r icmp.pcap host 192.168.2.127
reading from file icmp.pcap, link-type EN10MB (Ethernet)
20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request,
id 47441, seq 1, length 64
20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id
47441, seq 1, length 64To show traffic from a certain source computer, use the src host BPF, as shown in Example 6-9.
$ tcpdump -n -r icmp.pcap src host 192.168.2.127
reading from file icmp.pcap, link-type EN10MB (Ethernet)
20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request,
id 47441, seq 1, length 64The dst host BPF works the same way as the src host version, as shown in Example 6-10.
$ tcpdump -n -r icmp.pcap dst host 192.168.2.127
reading from file icmp.pcap, link-type EN10MB (Ethernet)
20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply,
id 47441, seq 1, length 64You can specify networks instead of hosts with the net BPF, as shown in Example 6-11.
$ tcpdump -n -r icmp.pcap dst net 192.168.2.0
reading from file icmp.pcap, link-type EN10MB (Ethernet)
20:30:28.844237 IP 192.168.2.108 > 173.194.75.104: ICMP echo request,
id 1, seq 5, length 40
20:30:29.850913 IP 192.168.2.108 > 173.194.75.104: ICMP echo request,
id 1, seq 6, length 40
20:30:29.987013 IP 192.168.2.127 > 173.194.75.99: ICMP echo request,
id 47441, seq 1, length 64Many protocols offer BPF primitives that allow you to look at specific aspects of the traffic, and you can also combine elements of the previous examples to limit what you see. For example, Example 6-12 shows only ICMP echo replies from IP address 192.168.2.127.
$tcpdump -n -r icmp.pcap 'icmp[icmptype] =icmp-echoreply' and dst host 192.168.2.127reading from file icmp.pcap, link-type EN10MB (Ethernet) 20:30:30.013728 IP 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64
Extracting Details from Tcpdump Output
In addition to displaying traffic more specifically, with Tcpdump, you can also extract more details from the results. For example, Example 6-13 tells Tcpdump to show timestamps as YYYY-MM-DD HH:MM:SS.milliseconds via -tttt, adds layer 2 headers with -e, and tells Tcpdump to show all headers and data in hex and ASCII format with -XX.
$tcpdump -n -tttt -e -XX -r icmp.pcap 'icmp[icmptype] = icmp-echoreply' and dst host 192.168.2.127reading from file icmp.pcap, link-type EN10MB (Ethernet) 2013-02-16 20:30:30.013728 00:0d:b9:27:f1:48 > 00:13:10:65:2f:ac, ethertype IPv4 (0x0800), length 98: 173.194.75.99 > 192.168.2.127: ICMP echo reply, id 47441, seq 1, length 64 0x0000: 0013 1065 2fac 000d b927 f148 0800 4500 ...e/....'.H..E. 0x0010: 0054 0000 0000 fb01 035c adc2 4b63 c0a8 .T.........Kc.. 0x0020: 027f 0000 2092 b951 0001 65ec 1f51 0000 .......Q..e..Q.. 0x0030: 0000 d30a 0f00 0000 0000 1011 1213 1415 ................ 0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$% 0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345 0x0060: 3637 67
Note
Tcpdump offers other matching and storage options. For more information, see the Tcpdump manual page on SO. Type man tcpdump at a command prompt to read the manual.
Examining Full Content Data with Tcpdump
Because Tcpdump also works on saved traces, you can use it to examine the full content data saved on SO stand-alone or sensor platforms in the /nsm/sensor_data/<sensorname>/dailylogs directory. When searching for indicators of compromise in network traffic, you may want to search every file in these directories. You can use Tcpdump and a BPF modifier to hone your output.
For example, Example 6-14 looks through all files for traffic involving host 8.8.8.8 and TCP thanks to a for loop and the find command. Note the backticks (on the same key as the tilde symbol) in front of the find and after -type f.
$for i in `find /nsm/sensor_data/sademo-eth1/dailylogs/ -type f`; do tcpdump -n -c 1 -r $ihost 8.8.8.8 and tcp; donereading from file /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-16/ snort.log.1361019690, link-type EN10MB (Ethernet)snip--
Example 6-14 shows that the first trace did not contain any traffic matching the BPF. The second trace contains a matching SYN packet . The third trace at did not contain any matching packets.
With a repository of full content data at your disposal, you give greater context to your NSM analysis. While most NSM analysts use many tools to access full content data, I often use Tcpdump to take a quick look at specific network activity, applying a BPF for a certain port or host of interest.
Using Dumpcap and Tshark
The Dumpcap and Tshark tools are shipped with the Wireshark (http://www.wireshark.org/) suite. Dumpcap is a simple traffic collection tool, and Tshark is the command line version of the Wireshark network traffic analyzer. Dumpcap, and by extension Tshark, depend on the libpcap traffic capture library to access packets. Both Dumpcap and Tshark are available on SO, but they are not running by default. Analysts can invoke each on demand, most often to access full content data in /nsm/sensor_data/<sensorname>/dailylogs.
Note
Gerald Combs is the original author of Dumpcap, and he and the Wireshark team code under the GNU General Public License version 2 (http://www.wireshark.org/faq.html).
Tshark’s strength lies in protocol analysis, thanks to the hundreds of protocols it understands, and, unlike Tcpdump, it allows you access just about any aspect of a protocol using fairly human-friendly syntax. For this reason, if I need to decode a specific protocol in a command line environment, I choose Tshark over Tcpdump.
Running Tshark
You can run Tshark from a command terminal, although if you start it with sudo, it will likely report the following error and warning as shown in Example 6-15.
$ sudo tshark -i eth1
tshark: Lua: Error during loading:
[string "/usr/share/wireshark/init.lua"]:45: dofile has been disabled
Running as user "root" and group "root". This could be dangerous.
Capturing on eth1The protocol dissectors shipped with Wireshark and Tshark may contain vulnerabilities. Clever intruders could exploit those vulnerabilities by sending specially crafted network traffic past a sensor. If malicious packets exploit Wireshark or Tshark while it is sniffing traffic, an intruder could gain control of the sensor. If Wireshark or Tshark is running with root privileges when exploitation occurs, the intruder could gain total control of the sensor.
To partially mitigate the risk of granting intruders unauthorized access, the Wireshark developers recommend that users not run either program with root privileges. Instead, they suggest capturing traffic with Dumpcap first, and then analyzing saved packets with Wireshark or Tshark.
Running Dumpcap
Dumpcap uses the same BPF syntax as Tcpdump, as shown in Example 6-16.
$ sudo dumpcap -i eth1 -c 2 -w /tmp/tshark-icmp.pcap -f "icmp and host 192.168.2.108"
File: /tmp/tshark-icmp.pcap
Packets captured: 2
Packets Received/Dropped on Interface eth1: 2/0The command in Example 6-16 tells Dumpcap to listen to the eth1 interface, save two packets, write to the /tmp/tshark-icmp.pcap file, and limit capture to ICMP traffic involving the computer at IP address 192.168.2.108.
As you can see in the listing, you don’t need to specify a snaplength via -s as you do with Tcpdump, because Dumpcap uses a default maximum value. Example 6-15 writes to the /tmp directory because the operating system won’t let me write to my home directory as root through sudo. I must write to a directory that the root user can also write to, which doesn’t include my user’s home directory.
Besides using sudo and writing to a directory writable by root, you can reconfigure Wireshark on SO to create a wireshark group, and then add your user account to that group. Doing so will allow your users to capture packets with Dumpcap without invoking sudo to elevate privileges. To accomplish this goal, run the following command:
$ sudo dpkg-reconfigure wireshark-commonIf you run this command within an OpenSSH session, the screen should look like Example 6-17.
wireshark-common via OpenSSH sessionâââââââââââââââââââââââg Configuring wireshark-common âââââââââââââââââââââââ â â â Dumpcap can be installed in a way that allows members of the "wireshark" â â system group to capture packets. This is recommended over the â â alternative of running Wireshark/Tshark directly as root, because less â â of the code will run with elevated privileges. â â â â For more detailed information please see â â /usr/share/doc/wireshark-common/README.Debian. â â â â Enabling this feature may be a security risk, so it is disabled by â â default. If in doubt, it is suggested to leave it disabled. â â â â Should non-superusers be able to capture packets? â â â â <Yes> <No> â â â âââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ
Use the tab or arrow keys to select Yes, and then press enter. The script will add a wireshark user to the /etc/group file. Next, add your user to the wireshark group. Here, the username is sademo:
$ sudo usermod -a -G wireshark sademoNow log out of the system and log back in. (If you try to capture traffic without logging in again, you will get an error.) Try capturing traffic as a normal user, as shown in Example 6-18.
$ dumpcap -i eth1 -c 2 -w tshark-icmp.pcap -f "icmp and host 192.168.2.108"
File: tshark-icmp.pcap
Packets captured: 2
Packets received/dropped on interface eth1: 2/0Running Tshark on Dumpcap’s Traffic
Once Dumpcap has captured traffic, analyze it with Tshark. To run Tshark in its most basic mode, use the -r switch, as shown in Example 6-19.
$ tshark -r tshark-icmp.pcap
1 0.000000 192.168.2.108 -> 8.8.8.8 ICMP 74 Echo (ping) request
id=0x0001, seq=17/4352, ttl=127
2 0.022643 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply
id=0x0001, seq=17/4352, ttl=251This output should be fairly easy to understand, although the time field may be unfamiliar. Specifically, host 192.168.2.108 issues an ICMP echo request to host 8.8.8.8 in packet 1, and host 8.8.8.8 responds with an ICMP echo reply in packet 2. By default, Tshark shows an initial time of 0, followed by time elapsed since the first packet. You can change that to show a more readable format with the -t ad switch, as shown in Example 6-20.
-t ad switch in Tshark$ tshark -t ad -r tshark-icmp.pcap
1 2013-02-17 13:37:45.922462 192.168.2.108 -> 8.8.8.8 ICMP 74 Echo
(ping) request id=0x0001, seq=17/4352, ttl=127
2 2013-02-17 13:37:45.945105 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo
(ping) reply id=0x0001, seq=17/4352, ttl=251Using Display Filters with Tshark
Tshark provides a robust language to show packets that match display filters. Tshark and Wireshark use display filters to control what traffic is shown, but display filters do not affect packet capture. Use BPF syntax if you want to influence what Tshark (or Dumpcap, for that matter) collects and stores. For example, Example 6-21 invokes a display filter to show only ICMP echo replies (ICMP type 0 messages).
$ tshark -t ad -r tshark-icmp.pcap -R "icmp.type == 0"
2 2013-02-17 13:37:45.945105 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo
(ping) reply id=0x0001, seq=17/4352, ttl=251This output may not seem very different from that of the Tcpdump filter shown in Example 6-20, but the power of Tshark (and Wireshark) comes from the extensive catalog of available display filters. The ICMP protocol has 64 display filters available as of this writing, as listed at http://www.wireshark.org/docs/dfref/i/icmp.html. All of these can be used to define specific values to be matched with a display filter.
Tshark reveals its depth of knowledge for protocols when you pass it the -V switch, which tells Tshark to produce a verbose protocol decode for the specified traffic. Add -x to display a hex and ASCII listing of the packet. Both options are shown in Example 6-22.
$tshark -t ad -r tshark-icmp.pcap -R "icmp.type == 0" -x -V
The full decode for this packet is broken into five main sections:
Section
displays frame information, with metadata on time, frame size, and other details.Section
shows details found in the Ethernet header such as source, destination, and Media Access Control (MAC) addresses.Section
offers information from the IP header, like source and destination IP addresses and other IP protocol data.Section
is a hexadecimal and ASCII representation of the entire frame.
Tools like Tshark are helpful because they expose every detail of a protocol. For example, you may find that it is important to know an ICMP sequence number, if that element may have been used for suspicious or malicious purposes.
Tshark Display Filters in Action
In this section, we’ll look at some display filter examples that demonstrate the power of Tshark.
Imagine you want to search traffic for Simple Mail Transport Protocol (SMTP) commands. You could use the smtp.req.command display filter, as shown in Example 6-23.
$ tshark -t ad -r smtp.pcap -R 'smtp.req.command'
4 2014-02-17 14:09:14.659043 192.168.2.127 -> 68.87.26.155 SMTP 76 C: helo test
10 2014-02-17 14:09:19.090208 192.168.2.127 -> 68.87.26.155 SMTP 71 C: quitTo look for user agents in HTTP GET request traffic generated by curl, you could use two filters together. Example 6-24 uses a for loop to search an entire directory. The echo statement shows the trace in question as Tshark searches it.
$for i in `find /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/ -type f`; do echo $i;tshark -t ad -r $i -R 'http.user_agent contains "curl" andhttp.request.method == GET'; done/nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/snort.log.1361107364 143841 2014-02-17 14:26:43.875022 192.168.2.127 -> 217.160.51.31 HTTP 223 GET / HTTP/1.1
Tshark display filters also make it easy to search for traffic to or from a range of IP addresses. For example, Example 6-25 looks for traffic with IP addresses between 192.168.2.100 and 192.168.2.110 inclusive that is not TCP or UDP.
$tshark -t ad -r /nsm/sensor_data/sademo-eth1/dailylogs/2013-02-17/snort.log.1361107364 -R'ip.dst >= 192.168.2.100 and ip.dst <= 192.168.2.110 and not tcp and not udp'10327 2014-02-17 13:33:01.775757 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply id=0x0001, seq=16/4096, ttl=251 12519 2014-02-17 13:37:45.945105 8.8.8.8 -> 192.168.2.108 ICMP 74 Echo (ping) reply id=0x0001, seq=17/4352, ttl=251
For more detail, add the -V and/or -x switch.
As you can see, I like to use Tshark to review saved traces for specific elements. It would be difficult to create the equivalent BPF syntax for many of these display filters. While technically possible, the BPF syntax can be horribly complex.
Running Argus and the Ra Client
Our final command line tool is Argus (http://www.qosient.com/argus/), a session data generation and analysis suite, and its client for reading data, Ra. The Argus server is running by default on SO, but analysts must use the Argus client tools to access the data stored in the /nsm/sensor_data/<sensorname>/argus directory.
Note
Carter Bullard first started writing Argus at Carnegie Mellon’s Software Engineering Institute (SEI) in 1993, and released the code publicly as Argus 1.5 in early 1996. Today, the code exists as a server component and multiple client components, licensed under the GNU General Public License version 3.
You can validate the status of the Argus server by running the nsm_sensor_ps-status script with the --only-argus switch, as shown in Example 6-26.
$ sudo nsm_sensor_ps-status --only-argus
Status: sademo-eth1 [ OK ]
* argusStopping and Starting Argus
If Argus is not running, you can restart it. Let’s stop it, and then restart it, as shown in Example 6-27.
$sudo nsm_sensor_ps-stop --only-argusStopping: sademo-eth1 * stopping: argus [ OK ] $sudo nsm_sensor_ps-start --only-argusStarting: sademo-eth1 * starting: argus [ OK ] * disk space currently at 21%
The Argus data stored in the /nsm/sensor_data/<sensorname>/argus directory appears as individual files, one for each day, named YYYY-MM-DD.log. Stopping and starting the Argus server will not destroy the previous file, only append to it.
The Argus File Format
The files in the Argus directory are binary files readable only by the Argus client tools. The binary format keeps the files compact. In comparison, a sample sensor with 48 days of NSM data shows the following directory usage for full content and Argus session data. Example 6-28 has the details.
$sudo du -csh /nsm/sensor_data/soe-eth0/argus/1.8G /nsm/sensor_data/soe-eth0/argus/ 1.8G total $sudo du -csh /nsm/sensor_data/soe-eth0/dailylogs/83G /nsm/sensor_data/soe-eth0/dailylogs/ 83G total
As you can see, 48 days of full content data in pcap format on this sensor occupies 83GB, but Argus session data for the same period occupies only 1.8GB, or 1/46 of the space. This ratio is likely to be quite different depending on the nature of each network, but you can see the space advantage associated with session data compared to full content data.
This comparison demonstrates the power of session data. If you just need to know the IP address, protocol, and/or ports associated with a connection, you can acquire all of that information from session data. You don’t need to capture or search through piles of full content data to get it.
Examining Argus Data
Analysts who enjoy parsing data using command line tools are likely to find Argus data particularly useful. I’ll show a few ways to examine this data for interesting results. You might take this approach if you want to look for specific information or script searches of session data for anomalous activity.
First, we’ll compare reading session data using two Argus clients, Ra and Racluster. Example 6-29 shows an example of using Ra to look for session records with destination port 21, which is used by many FTP servers.
$ra -n -r 2014-02-10.log - tcp and dst port 21 -sstime saddr sport daddr dport sbytes dbytesStartTime SrcAddr Sport DstAddr Dport SrcBytes DstBytes
The -n switch tells Ra to not resolve port numbers to names. The BPF syntax filter tcp and dst port 21 specifies a protocol and port of interest. The -s switch tells Ra which fields to display. (The Ra man page lists all output fields controlled by the -s switch.) The SrcBytes and DstBytes columns in the results count transaction data bytes, which include packet headers. (To get application layer bytes, use sappbytes and dappbytes instead of sbytes and dbytes on the command line.)
Notice that there are several session records for certain conversations. The Argus server wrote these records as it saw the connection stay active. That’s fine for a short result like the one in Example 6-29, but not for connections that stay open longer. To collapse these records, use Racluster, as shown in Example 6-30.
$racluster -n -r 2013-02-10.log - tcp and dstport 21 -s stime saddr sport daddr dport sbytesdbytesStartTime SrcAddr Sport DstAddr Dport SrcBytes DstBytes
Notice that the first three records (, , and ) from the Ra record in Example 6-29 have been collapsed into one record in Example 6-30, though when you add the byte counts from the same sessions in the Ra output, you’ll find that they match the total byte count in the Racluster output. For example, the SrcBytes count for the session to 202.12.29.205 in the Ra output is 140 + 769 + 204 = 1113, which is the same value as the SrcBytes field for the session to 202.12.29.205 in the Racluster output.
I often use Argus with Racluster to quickly search a large collection of session data via the command line, especially for unexpected entries. Rather than searching for specific data, I tell Argus what to omit, and then I review what’s left.
As an example, we’ll walk through building a fairly complicated Racluster search. It will tell Racluster to search three Argus archives for UDP traffic, but to exclude ports 53 (DNS), 123 (Network Time Protocol, or NTP), or host 192.168.2.120.
This will require the use of the -m saddr daddr switch, which instructs Ra to group records by source and destination IP address, and the -s switch, which specifies the desired output fields. Two additional elements add the year, month, and day to the timestamps in this report. To add these, first create the /tmp/ra.conf file, as shown in Example 6-31, with a variable telling Ra how to display the time. (To learn more about this format, see the manual page for the date command.)
Next, add the stime element of the -s switch that tells Ra to provide enough room in the print buffer to show the entire date and timestamp. Example 6-32 assembles all these components and shows the output.
$racluster -F /tmp/ra.conf -n -r 2014-02-10.log2013-02-16.log 2014-02-17.log - udp and not(port 53 or port 123 or host 192.168.2.120) -m saddr daddr-s stime:20 saddr sport daddr dportsbytes dbytesStartTime SrcAddr Sport DstAddr Dport SrcBytes DstBytes 2013-02-17 13:26:49 192.168.2.114.16403 17.173.254.222.0157.56.149.60.3544 33372 48169 2013-02-16 13:37:19 192.168.2.117.0
157.56.149.61.3544 515 755
In Example 6-32, you see entries where the destination port is 0 at , , , , and , and where the source port is 0 at and . When the destination port shows 0, Racluster has aggregated multiple destination ports into one record. For example, Example 6-33 shows a similar Racluster search that looks at Argus records involving 192.168.2.117 as the source IP address and 157.56.149.0/24 (meaning any fourth octet is acceptable) as the destination net block.
$racluster -F /tmp/ra.conf -n -r 2014-02-10.log2013-02-16.log 2014-02-17.log - src host192.168.2.117 and dst net 157.56.149.0/24 and udp and not(port 53 or port 123 or host192.168.2.120) -s stime:20 saddr sport daddr dport sbytes dbytesStartTime SrcAddr Sport DstAddr Dport SrcBytes DstBytes 2013-02-16 13:37:19 192.168.2.117.64412 157.56.149.60.3544
Notice that this output represents four distinct connections: two to 157.56.149.60 at and , and two to 157.56.149.61 at and . When you aggregate results using the source IP address, as in Example 6-32, you lose this granularity.
I mentioned earlier that I like to use Argus and its Ra or Racluster client to omit certain traffic, and then review what’s left for anomalies. Example 6-32 contains some data that I could review for suspicious or malicious entries. Doing this sort of review requires some ability to recognize net blocks and protocols, but it can yield interesting results.
Taking a net block approach means determining the source or destination of traffic. Tools like the Robtex website (http://www.robtex.com/) can help identify network owners. For example, traffic in Example 6-32 to the 17.0.0.0/8 traffic is likely related to Apple protocols, because Apple owns that entire Class A net block. Doing similar analysis shows Microsoft owns the 157.56.0.0/14 net block, Amazon owns 23.20.0.0/14, and Verizon owns 96.224.0.0/11.
Taking a protocol approach requires looking at the protocols involved, often by deciphering which applications use certain TCP or UDP ports. Online resources like the SANS Internet Storm Center (ISC) Port Report (https://isc.sans.edu/portreport.html) provide clues concerning the functions of various TCP and UDP ports. For example, Apple uses port 3544 UDP for its push notification service, and port 16386 UDP for its FaceTime service. Many systems run UDP-based Traceroute using port 33434. Based on this knowledge, I can determine that the applications depicted in Example 6-32 are likely all benign, and that they’re associated with Apple traffic and network path discovery using Traceroute. Of course, in order to firmly identify these sessions, I would need access to full content data or logs from other sources. Still, this approach provides a way to identify interesting activity with a minimum amount of effort.
Conclusion
This chapter began by explaining the three types of tools available in SO: software for data collection, presentation, and delivery. Within the presentation category, we find tools for packet analysis, and applications that work best as NSM consoles. Some of the packet analysis tools rely on command line interfaces, and others use graphical interfaces. This chapter discussed several packet analysis data presentation tools that are used from the command line: Tcpdump, Tshark, and the Argus Ra client. You also saw how to use Dumpcap in concert with Tshark.
In Chapter 7, we’ll look at the graphical interface packet analysis tools: Wireshark, Xplico, and NetworkMiner. You’ll see that GUI access to packets offers several distinct advantages, including the availability of more forms of NSM data.