6. Additional Data Analysis

This chapter supplements the core tools presented in Chapter 5. All of them work with full content data. They provide additional ways to examine and manipulate that data, beyond the capabilities of Tcpdump and related applications.

Editcap and Mergecap

Purpose: Packet capture assistance

Author: Originally Gerald Combs, with many contributors

Internet site: http://www.ethereal.com

FreeBSD installation: Installed via /usr/ports/net/ethereal

Version demonstrated: Versions shipped with Ethereal 0.10.0a

Editcap and Mergecap are two utilities packaged with Tethereal and Ethereal. Editcap allows users to make certain adjustments to capture files, while Mergecap allows users to combine two or more libpcap traces into a single file. Editcap is particularly useful for transforming trace files from one format to another. For example, we can create a new Snoop file out of an existing libpcap file by using the following command.

bourque# editcap -F snoop sf1.lpc sf1.snoop
bourque# file sf1.*
sf1.lpc:   tcpdump capture file (little-endian) - version 2.4
  (Ethernet, capture length 1515)
sf1.snoop: Snoop capture file - version 2 (Ethernet)

Editcap is also very useful for adjusting timestamps in packets. This is most frequently done to account for systems with incorrect clocks. From a forensic standpoint it's better to leave the “best evidence” copy in its original time format. In less sensitive scenarios, adjusting time to be more accurate is conceivable. Remember to document any changes made to the original captures.

This command makes the timestamps in sf1.lpc be one hour (3,600 seconds) later. Here the original appears first, followed by the new version for comparison.

bourque# tethereal -n -t ad -r sf1.lpc icmp

  1 2004-01-01 15:20:04.783092  172.27.20.4 -> 192.168.60.3
  ICMP Echo (ping) request

  2 2004-01-01 15:20:04.783217  172.27.20.4 -> 192.168.60.5
  ICMP Echo (ping) request

...edited...

bourque# editcap -t 3600 sf1.lpc sf1_plus_one_hour.lpc

bourque# tethereal -n -t ad -r sf1_plus_one_hour.lpc icmp

  1 2004-01-01 16:20:04.783092  172.27.20.4 -> 192.168.60.3
  ICMP Echo (ping) request

  2 2004-01-01 16:20:04.783217  172.27.20.4 -> 192.168.60.5
  ICMP Echo (ping) request

To move time backward, use a minus symbol in front of the time value, e.g., –3600.

Mergecap is most often used to concatenate two sequential libpcap traces into a single file. The syntax for this operation is simple:

mergecap –w <result.lpc> <input_1.lpc> <input_2.lpc>

By default, Mergecap will take two files whose times overlap and interleave them as necessary.

Tcpslice

Purpose: Packet capture manipulation

Author: Vern Paxson

Internet site: http://www.tcpdump.org

FreeBSD installation: Installed via /usr/ports/net/tcpslice; Tcpslice packaged with FreeBSD base (in /usr/sbin/tcpslice) does not work on files captured after the year 2000

Version demonstrated: Tcpslice for Tcpdump 3.7

Tcpslice allows analysts to break large libpcap files into smaller ones. If you're running Tcpslice on FreeBSD, do not use the version included with the base system. Install Tcpslice from ports and then make the system use the new version. For example, the first command shown here renames the original Tcpslice, and the second makes a link from the version installed by the ports tree to the expected location in /usr/sbin.

bourque# mv /usr/sbin/tcpslice /usr/sbin/tcpslice.old
bourque# ln -s /usr/local/sbin/tcpslice /usr/sbin/tcpslice

Tcpslice breaks up traces by time values. When using Tcpslice, it's best to begin by understanding the time frame of the packets in a trace file. The first set of switches shows how Tcpslice reports the time span of a given trace file.

The –r switch reports the timestamps of the first and last packets in a human-readable format.

bourque# tcpslice -r sf1.lpc
sf1.lpc Thu Jan  1 15:20:04 2004        Thu Jan  1 16:32:14 2004

The –R switch reports the timestamps of the first and last packets in UNIX format (i.e., seconds.microseconds since the UNIX epoch).

bourque# tcpslice -R sf1.lpc
sf1.lpc 1072988404.783092       1072992734.170640

Let's verify what 1072988404 and 1072992734 mean by using the date command and the –r switch. Note that the date tool shipped with many Linux distributions does not support the –r switch.

bourque# date -r 1072988404
Thu Jan  1 15:20:04 EST 2004

bourque# date -r 1072992734
Thu Jan  1 16:32:14 EST 2004

These values match the timestamp provided by the tcpslice –r results. We can look directly at the trace file and see its timestamp as well.

bourque# tcpdump -n -r sf1.lpc -c 1
15:20:04.783092 172.27.20.4 > 192.168.60.3: icmp: echo request

bourque# tcpdump -n -r sf1.lpc -c 1 -tt
1072988404.783092 172.27.20.4 > 192.168.60.3: icmp: echo request

The 15:20:04.783092 and 1072988404 values are as we expected.

The –t switch reports the timestamps of the first and last packets in a year-month-day-hour-minute-second-microsecond format, as shown here.

bourque# tcpslice -t sf1.lpc

sf1.lpc 2004y01m01d15h20m04s783092u   2004y01m01d16h32m14s170640u

With an idea of the times of the packets in the trace, we can begin cutting. Start by checking the time frame you want against Tcpslice's interpretation of your command. The following syntax tells Tcpslice we are interested in the first 10 minutes of the trace sf1.lpc.

bourque# tcpslice -d -r +0 +10m sf1.lpc
sf1.lpc Thu Jan  1 15:20:04 2004        Thu Jan  1 16:32:14 2004
start   Thu Jan  1 15:20:04 2004
stop    Thu Jan  1 15:30:04 2004

The first line reports the times of the first and last packets, as we've come to expect. The next line reports the start time indicated by our syntax, and the last line reports the end time. We used the –d switch to “dump” the start and end times indicated by the –r +0 +10m syntax. Remember that the –r flag specifies human-readable format. We could have just as easily specified the number of seconds in a 10-minute period by using –R +0 +600 syntax.

bourque# tcpslice -d -R +0 +600 sf1.lpc
sf1.lpc 1072988404.783092       1072992734.170640
start   1072988404.783092
stop    1072989004.783092

bourque# date -r 1072988404
Thu Jan  1 15:20:04 EST 2004

bourque# date -r 1072989004
Thu Jan  1 15:30:04 EST 2004

To now write the results to a file, we use the following command. We verify the time frame by using the tcpslice –r command.

bourque# tcpslice -w first_10.lpc +0 +10m sf1.lpc

bourque# tcpslice -r first_10.lpc
first_10.lpc   Thu Jan  1 15:20:04 2004  Thu Jan  1 15:30:03 2004

You may be wondering how Tcpslice differentiates between the m used for “month” versus the same m used for “minutes.” If Tcpslice sees a d value following the m, such as 1m1d to represent January 1, the m refers to “month.” Otherwise, Tcpslice treats m to mean “minutes.”

For example, the following command validates a time span starting at 00:00 on January 1 of the year specified in the trace file and continuing for the next 1,200 minutes (20 hours).

bourque# tcpslice -d -r 1m1d +1200m sf1.lpc
sf1.lpc Thu Jan  1 15:20:04 2004        Thu Jan  1 16:32:14 2004
start   Thu Jan  1 00:00:00 2004
stop    Thu Jan  1 20:00:00 2004

To start 5 minutes into the trace and record the next 30 minutes, use this syntax.

bourque# tcpslice -w pl5-30.lpc +5m +30m sf1.lpc
bourque# tcpslice -r pl5-30.lpc
pl5-30.lpc     Thu Jan  1 15:25:06 2004  Thu Jan  1 15:52:08 2004

That looks odd. Why is the last timestamp 15:52:08 and not 15:55:06? After all, we told Tcpslice to write results starting 5 minutes into the trace and then for 30 minutes from that point. Here's where we can use Tcpslice with Tcpdump to see the packets during a certain time frame without writing results to a file. Let's look at packets starting 30 minutes into the trace and continuing for another 6 minutes. Notice that the command ends with a hyphen, indicating that Tcpdump will read from standard input. In this case, that information will be provided by Tcpslice.

bourque# tcpslice +30m +6m sf1.lpc | tcpdump -n -r -
15:50:08.105829 172.27.20.105.32820 > 192.168.60.5.22:
  P 779421602:779421642(40) ack 2657410081 win 24820 (DF)
  [tos 0x10]
15:50:08.107777 192.168.60.5.661 > 192.168.60.3.22:
  P 2764320758:2764320778(20)ack 327421799 win 32120
  <nop,nop,timestamp 24817698 24839254> (DF) [tos 0x10]
...edited...

15:52:08.295603 192.168.50.2.19383 > 192.168.60.5.3389:
  S 858437626:858437626(0) win 16384 <mss 1260,nop,nop,
  sackOK> (DF)
15:52:08.296217 192.168.60.5.3389 > 192.168.50.2.19383:
  R 0:0(0) ack 1 win 0
15:55:12.900842 172.27.20.105.32820 > 192.168.60.5.22:
  P 1680:1720(40) ack 2185 win 24820 (DF) [tos 0x10]
15:55:12.903106 192.168.60.5.661 > 192.168.60.3.22:
  P 840:860(20) ack 1305 win 32120
  <nop,nop,timestamp 24848177 24840752> (DF) [tos 0x10]

Note that there's a 3-minute gap between the packet at 15:52:08 and the next at 15:55:12. When we asked earlier for packets starting 5 minutes into the trace and continuing for another 30 minutes, the trace could only offer a packet at 15:52:08 and no more until 15:55:12.

Incidentally, by piping our results to Tcpdump via | tcpdump –n –r -, we demonstrated how to use a time filter to see packets within a specified time span without writing them to a file.

Tcpreplay

Purpose: Packet replay utility

Authors: Aaron Turner and Matt Bing

Internet site: http://tcpreplay.sourceforge.net/

FreeBSD installation: FreeBSD 4.9 RELEASE package

Version demonstrated: 1.4.4

Tcpreplay is a tool used to replay packets captured in libpcap format. It works very well with applications that do not have the capability to read in libpcap-formatted traces. Using Tcpreplay, you can replay a trace file and have the non-libpcap-friendly application listen for packets.

Tcpreplay is often used to send traffic from one system while a monitoring platform listens for that traffic. In this sample setup, host janney uses Tcpreplay to send the contents of the sf1.lpc trace onto a network segment where sensor bourque listens. Host janney will send its traffic out its em0 interface. Note that if the interface that transmits the traffic doesn't have an IP address, Tcpreplay will complain. We assign the arbitrary address 192.168.1.1 to satisfy Tcpreplay. After a few minutes we interrupt Tcpreplay by hitting CTRL+C on the keyboard.

janney# tcpreplay -i em0 sf1.lpc
Can't open em0: libnet_select_device(): Can't find interface em0

janney# ifconfig em0 inet 192.168.1.1 netmask 255.255.255.0 up
janney# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
     options=3<rxcsum,txcsum>
     inet6 fe80::207:e9ff:fe11:a0a0%em0 prefixlen 64 scopeid 0x1
     inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
     ether 00:07:e9:11:a0:a0
     media: Ethernet autoselect (100baseTX <half-duplex>)
     status: active

janney# tcpreplay -i em0 sf1.lpc
sending on em0
^Cnanosleep error: Interrupted system call
  94 packets (10101 bytes) sent in 116.92 seconds
  86.4 bytes/sec 0.00 megabits/sec 0 packets/sec

Meanwhile, sensor bourque sees the traffic in sf1.lpc as it listens with Tcpdump.

bourque# tcpdump -n -s 1515 -i sf0
16:07:42.979590 172.27.20.4 > 192.168.60.3: icmp: echo request
16:07:42.994861 172.27.20.4 > 192.168.60.5: icmp: echo request
16:07:42.994903 192.168.60.3 > 172.27.20.4: icmp: echo reply

Be careful to collect the traffic you intend to collect when using Tcpreplay on a real network. For the cleanest results, use a separate LAN without production traffic. Three ways to use Tcpreplay on a hardware-based network involve connecting the two systems with a hub, a switch, or a crossover cable. Also be aware of the sorts of traffic your workstations emit as a result of normal operation. Host janney's em0 interface produced the following traffic after being initialized with the ifconfig command listed earlier.

16:03:35.627604 arp who-has 192.168.1.1 tell 192.168.1.1
16:03:35.627611 :: > ff02::1:ff11:a0a0:
  icmp6: neighbor sol: who has fe80::207:e9ff:fe11:a0a0
16:03:37.342562 fe80::207:e9ff:fe11:a0a0 > ff02::2:f2f0:bec6:
  HBH icmp6: multicast listener report max resp delay:
  0 addr: ff02::2:f2f0:bec6 [hlim 1]
16:03:41.742242 fe80::207:e9ff:fe11:a0a0 > ff02::1:ff11:a0a0:
  HBH icmp6: multicast listener report max resp delay:
  0 addr: ff02::1:ff11:a0a0 [hlim 1]

Apparently the em0 driver acts on the Intel gigabit NIC's desire to locate other systems using IP version 6 ICMP. This icmp6 traffic is not part of the sf1.lpc capture as transmitted by Tcpreplay; the icmp6 traffic is caused by the em0 NIC independently.

Sometimes it is unnecessary to put packets onto a real Ethernet wire. In addition to the three hardware-based options, a software-only, single-system method can be used to transmit and observe packets with Tcpreplay. FreeBSD systems support creating a virtual network interface called tap, which can be used to implement Tcpreplay on a single system.¹ By creating a tap pseudo-interface, we can have Tcpreplay send packets to this interface, while our listening application monitors the same tap interface.

We can create a tap interface by using the following sequence of commands on FreeBSD 4.9 STABLE. (This may not work on FreeBSD 4.9 RELEASE.) First we run ifconfig against the not-yet-created tap0 interface. This is a prerequisite for the next command. The dd command creates a “bit bucket,” which keeps the tap0 interface open as it receives packets. After executing the dd command, we rerun ifconfig to see that the tap0 interface now exists. In the last command we assign tap0 an arbitrary 10.1.1.1 IP address and check its status.

janney# ifconfig tap0
ifconfig: interface tap0 does not exist

janney# dd if=/dev/tap0 of=/dev/null bs=1500 &
[1] 193

janney# ifconfig tap0 inet 10.1.1.1 netmask 255.255.255.0 up
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet6 fe80::2bd:16ff:fe29:0%tap0 prefixlen 64 scopeid 0x8
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        ether 00:bd:16:29:00:00
        Opened by PID 193

Once tap0 is created, we can send packets out the tap0 interface and have another application listen on tap0.

janney# tcpreplay -i tap0 sf1.lpc
sending on tap0
^Cnanosleep error: Interrupted system call
  5 packets (300 bytes) sent in 2.51 seconds
  119.1 bytes/sec 0.00 megabits/sec 1 packets/sec

In a second terminal, Tcpdump sees the packets sent to the tap0 interface.

janney# tcpdump -n -s 1515 -i tap0
tcpdump: listening on tap0
16:13:23.235085 172.27.20.4 > 192.168.60.3: icmp: echo request
16:13:23.253563 172.27.20.4 > 192.168.60.5: icmp: echo request
16:13:23.253641 192.168.60.3 > 172.27.20.4: icmp: echo reply
16:13:23.253711 192.168.60.5 > 172.27.20.4: icmp: echo reply
16:13:25.754032 172.27.20.4.58173 > 192.168.60.3.21:
  S 2986655065:2986655065(0) win 2048

Figure 6.1 depicts these ways to use Tcpreplay.

Figure 6.1. Using Tcpreplay with a hub or switch, a crossover cable, and a tap0 interface

Tcpreplay depends on Libnet, which as of version 1.1.1 is reported to support sending packets to the loopback (lo0) device.² With this improvement, use of the tap0 interface should not be necessary. The alpha versions of Tcpreplay, in the 1.5.x series, offer a new tool called Flowreplay. To understand the importance of Flowreplay, remember that Tcpreplay transmits exactly the same packets that appear in a libpcap trace file. The same source and destination IP addresses, ports, and application content are sent on the wire using Tcpreplay. Flowreplay takes a different approach. It replays application content, constructing new network and transport headers to carry that application data. Flowreplay can be used to connect to a server and transmit content, while Tcpreplay can only send packets exactly as they occurred sometime in the past.³

In this book, I use Tcpreplay to place packets in the view of applications that can't read libpcap traces. Programs like Tcpdump and Ethereal read libpcap files with their –w switches. This is not the case with many other programs. By deploying Tcpreplay in one of the configurations described earlier, we can rerun sf1.lpc and em0.lpc against applications that were not active when the traffic was originally created.

Tcpflow

Purpose: TCP application data reconstruction

Author: Jeremy Elson

Internet site: http://www.circlemud.org/~jelson/software/tcpflow/

FreeBSD installation: FreeBSD 4.9 RELEASE package

Version demonstrated: 0.21

Tcpflow is a session data reconstruction program. I've found it to be one of the easiest ways to quickly display full content data captured via Tcpdump. Tcpflow also expands on the two themes of full content data: access to all header details and access to application data. Tcpflow can be run against a live interface or on a libpcap trace file. I heartily recommend running it against the latter.⁴ It's a lot easier to work with a known input like a trace file than to try your luck on a live interface.

Tcpflow is most commonly used to send results to standard output with its –c flag. Because it understands BPFs, we can tell it to show us the traffic to or from port 3389 with the following command.

bourque# tcpflow -r em0.lpc -c port 3389

192.168.060.003.34720-010.010.010.003.03389: ...'"......Cookie:
mstshash=administr

010.010.010.003.03389-192.168.060.003.34720: .........4.

192.168.060.003.34720-010.010.010.003.03389: ........e...........
.0..."......................0........................0...........
...................../....|...&........Duca.......... .X.........
(..C.A.I.N.E.....................................................
.............................................................5.5.
2.7.4.-.6.4.0.-.1.1.5.7.4.7.3.-.2.3.8.9.7........................
.......................,.....rdpdr.......cliprdr.....rdpsnd......

010.010.010.003.03389-192.168.060.003.34720: ...M....f..A.....0..
."...............................|..*.v......McDn................
...................... .......y..ioA|g.o......v.Z..a...,xV.......
............RSA1H.......?........v[...t...NrrM..U.)=..(.Pf2....B
.[N<5.s[.^M..Mv..._.g...#..y....u..........H..P.........1L)...7..
.0V.Gz,/L..7....s.,?`'.].T2.I.~&....N...f.3........

192.168.060.003.34720-010.010.010.003.03389: ............

192.168.060.003.34720-010.010.010.003.03389: .......(

010.010.010.003.03389-192.168.060.003.34720: ...........

This trace isn't especially appealing! Port 3389 TCP is used for Microsoft Terminal Services and Remote Desktop Protocol. It's a graphic display system not meant to be interpreted in ASCII. While we see some human-readable characters, the majority are unprintable, as designated by the dot (.) characters. ASCII-based protocols like that used on port 21 render better results.

bourque# tcpflow -r em0.lpc -c port 21

172.027.020.005.00021-010.010.010.003.01075: 220
  janney.taosecurity.com FTP server (Version 6.00LS) ready.

In both examples the output is straightforward as far as headers go. We see the source IP and port and the destination IP and port, followed by application data for each packet.

Omitting the –c flag causes Tcpflow to write what it finds into two files: One is named for the data sent by the source, and the second is named for the data sent by the destination. While this makes it easy to let Tcpflow tear through a large libpcap file, understanding the output is difficult. I prefer to find a specific flow of interest, define it by using the appropriate socket, and use Tcpflow with the –c flag to redirect its output to a file, as shown here.

tcpflow -r em0.lpc -c "( src port 21 and dst port 1075 )" or
  "( src port 1075 and dst port 21 )" > flowfile.txt

Sguil uses Tcpflow to create transcripts from full content data saved by Snort. When the application data is that of a file transfer data channel, it's far better to let Tcpflow run in its native mode. When it rebuilds the sessions it sees, one of the files (from the FTP server to the client) will contain the binary downloaded by the client, or perhaps the directory listing requested.

In the following example, other methods (such as perusing session data, discussed in Chapter 7) identify a likely candidate for rebuilding with Tcpflow. We see a session involving ports 20 and 1041 TCP; it is most likely an active FTP data channel. Knowing the server will send data from port 20 TCP to a client port, we use Tcpflow to recover the downloaded file.

bourque# tcpflow -r sf1.lpc src port 20 and dst port 1041

bourque# ls

172.027.020.005.00020-192.168.060.005.01041

bourque# file 172.027.020.005.00020-192.168.060.005.01041


172.027.020.005.00020-192.168.060.005.01041: ELF 32-bit LSB
executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5,
statically linked, not stripped

The file created by Tcpflow, awkwardly named 172.027.020.005.00020-192.168. 060.005.01041 after the socket from which it was born, is a Linux executable. Assuming we didn't drop any packets, we have a pristine copy of a tool downloaded by the intruder. This technique applies to the tools transferred by Ardala in Chapter 4 and to any files downloaded by users on networks you monitor. Passing the program through the strings command, we see the usage statement and identify the program as a reconnaissance tool.

strings 172.027.020.005.00020-192.168.060.005.01041

Usage: portscan [-b start] [-e end] [-bm start] [-em end]
  [-v[v]] [-a] [-s] [-i]
  [-? | h] <address>
-b start: Specify the port at which we begin scanning
-e end: Specify the port at which we stop scanning
-bm: Subnet machine number at which to start scanning [dfl=1]
-em: Subnet machine number at which to stop scanning [dfl=254]
-v: Level 1 of verbosity, basic information
-vv: Level 2 of verbosity, full information report
-a: if we want subnet scanning (start at .1, end at .254)
-s: strobe scanning: scan only ports found in /etc/services,
  much faster
-? or -h: print this help text
-i: Copyright and version information
portscanner v%s was written by Tennessee Carmel-Veilleux
12:42:46
Jan  1 2004
This version compiled on %s at %sEST
This program is licensed under the GPL. See www.gnu.org for more
  info on the license

Tcpflow is the fastest way I know to reconstruct application data from the command line. To rapidly inspect application data for strings of interest, let's turn to Ngrep.

Ngrep

Purpose: String matching of packet contents

Author: Jordan Ritter

Internet site: http://ngrep.sourceforge.net/

FreeBSD installation: Installed via /usr/ports/net/ngrep

Version demonstrated: 1.40.1

We've talked about ways to find items of interest in packet headers, so the next logical step is finding items of interest in application data. Snort can provide this sort of deep inspection, but Ngrep is a simpler way to achieve the same goal.

Ngrep works by examining application data and reporting matches it finds. The program understands regular expressions and BPFs. Consider the following example, which uses these switches.

• –I specifies the libpcap trace to read as input.

• –x instructs Ngrep to report hexadecimal and ASCII output.

• –q tells Ngrep to be quiet, that is, to not print hashes for nonmatching packets.

• –i specifies case-insensitive matching.

• p.ng is a regular expression meaning “match p, any character, n, and g.”

• udp is a BPF telling Ngrep to inspect only UDP datagrams.

bourque# ngrep -I sf1.lpc -x -q -i 'p.ng' udp
input: sf1.lpc

U 172.27.20.5:3605 -> 192.168.60.5:7983
  70 69 6e 67                                         ping

U 172.27.20.5:3605 -> 192.168.60.3:7983
  70 69 6e 67                                         ping

U 192.168.60.3:32774 -> 172.27.20.5:9325
  70 6f 6e 67                                         pong

U 192.168.60.5:1048 -> 172.27.20.5:9325
  70 6f 6e 67                                         pong

The results show four UDP packets. The first two are commands sent by the Mstream DDoS tool handler on 172.27.20.5 to the agents on 192.168.60.3 and 192.168.60.5.⁵ The last two are replies showing that the DDoS agents from Chapter 4 are alive.

The the next example uses * to mean “match zero or more appearances of the letter n between pe and y.”

bourque# ngrep -I sf1.lpc -x -q -i 'pen*y'
input: sf1.lpc

T 172.27.20.3:3307 -> 192.168.60.5:21 [AP]
  70 65 6e 6e 79 0a                                 penny.

T 192.168.60.5:1032 -> 172.27.20.5:21 [AP]
  50 41 53 53 20 70 65 6e    6e 79 0d 0a            PASS penny..

T 192.168.60.5:1039 -> 172.27.20.5:21 [AP]
  50 41 53 53 20 70 65 6e    6e 79 0d 0a            PASS penny..

The following example uses the ^ character to say “match at the beginning of the line.” By searching for “2.0” we look for all “2x0” status codes on an FTP control channel. We also use the –n 1 switch to report only the first match. The highlighted portion caused the regular expression to match.

bourque# ngrep -I sf1.lpc -x -q –n 1 -i '^2.0' port 21
input: sf1.lpc

T 192.168.60.5:21 -> 172.27.20.3:3307 [AP]
32 32 30 20 6f 61 74 65 73 2e 74 61 6f 73 65 63  220 oates.taosec
75 72 69 74 79 2e 63 6f 6d 20 46 54 50 20 73 65  urity.com FTP se
72 76 65 72 20 28 56 65 72 73 69 6f 6e 20 77 75  rver (Version wu
2d 32 2e 36 2e 30 28 31 29 20 4d 6f 6e 20 46 65  -2.6.0(1) Mon Fe
62 20 32 38 20 31 30 3a 33 30 3a 33 36 20 45 53  b 28 10:30:36 ES
54 20 32 30 30 30 29 20 72 65 61 64 79 2e 0d 0a  T 2000) ready...

Here's an example of using a regular expression to search for evidence of an attempt to execute /bin/sh through a buffer overflow against the FTP server on 192.168.60.5.

bourque# ngrep -I sf1.lpc -x -q -i 'bin.sh'
input: sf1.lpc

T 172.27.20.3:3307 -> 192.168.60.5:21 [AP]
50 41 53 53 20 90 90 90 90 90 90 90 90 90 90 90  PASS ...........
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90  ................
...20 lines like the preceding omitted...
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90  ................
31 c0 31 db 31 c9 b0 46 cd 80 31 c0 31 db 43 89  1.1.1..F..1.1.C.
d9 41 b0 3f cd 80 eb 6b 5e 31 c0 31 c9 8d 5e 01  .A.?...k^1.1..^.
88 46 04 66 b9 ff ff 01 b0 27 cd 80 31 c0 8d 5e  .F.f.....'..1..^
01 b0 3d cd 80 31 c0 31 db 8d 5e 08 89 43 02 31  ..=..1.1..^..C.1
c9 fe c9 31 c0 8d 5e 08 b0 0c cd 80 fe c9 75 f3  ...1..^.......u.
31 c0 88 46 09 8d 5e 08 b0 3d cd 80 fe 0e b0 30  1..F..^..=.....0
fe c8 88 46 04 31 c0 88 46 07 89 76 08 89 46 0c  ...F.1..F..v..F.
89 f3 8d 4e 08 8d 56 0c b0 0b cd 80 31 c0 31 db  ...N..V.....1.1.
b0 01 cd 80 e8 90 ff ff ff ff ff ff 30 62 69 6e  ............0bin
30 73 68 31 2e 2e 31 31 0d 0a                    0sh1..11...

Another way to find an indication of this sort of activity involves hexadecimal matching, specified by using the –X switch.

bourque# ngrep -I sf1.lpc -x -q -X '909090' | less
input: sf1.lpc

T 172.27.20.3:3307 -> 192.168.60.5:21 [AP]
50 41 53 53 20 90 90 90 90 90 90 90 90 90 90 90  PASS ...........
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90  ................
90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90  ................
...truncated...

This is a very primitive way to detect a buffer overflow, due to the use of polymorphic shell code generated by ADMutate and other anti-IDS techniques.⁶

Ngrep offers a few other helpful switches.

• –O <outfile.lpc> sends any matching packets to an output file in libpcap format.

• –v specifies an inverted matching, meaning anything that doesn't match the specified string will be shown.

• –d tells Ngrep to watch a live interface, rather than read a libpcap file using –I.

The following example uses the –d switch. Notice how the combination of a regular expression and live capture detected Web activity on a nonstandard port (10000 TCP, used by default by the Webmin application).

bourque# ngrep -d em0 -q -i '^get'

T 192.168.50.2:19514 -> 172.27.20.5:10000 [AP]
GET /unauthenticated/nav/bottom_shadow.jpg HTTP/1.1..Accept:
*/*..Referer: http://janney.taosecurity.com:10000..
Accept-Language: en-us..Accept-Encoding: gzip, deflate..
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
..Host: janney.taosecurity.com:10000..Connection: Keep-Alive....

I use Ngrep to search trace files for content of interest. It's quicker than writing a Snort rule to match application content, and it's much easier than writing fancy BPFs for Tcpdump.

We've looked at a variety of tools to inspect full content data. Now let's check out a tool that can summarize packet details, possibly for use by other tools.

IPsumdump

Purpose: Command-line packet summarization application

Author: Eddie Kohler

Internet site: http://www.icir.org/kohler/ipsumdump/

FreeBSD installation: Installed via source code

Version demonstrated: 1.33

IPsumdump is a program for reading libpcap data and producing user-customizable single-line output. This is useful for analysts who want to parse traces for certain fields and output those fields into another application. Consider the following example.

-bash-2.05b$ ipsumdump -tpsSdD -r sf1.lpc
!IPSummaryDump 1.1
!creator "ipsumdump -tpsSdD -r sf1.lpc"
!host bourque.taosecurity.com
!runtime 1073490693.320906 (Wed Jan  7 10:51:33 2004)
!data timestamp ip_proto ip_src sport ip_dst dport
1072988404.783092 I 172.27.20.4 - 192.168.60.3 -
1072988404.783217 I 172.27.20.4 - 192.168.60.5 -
1072988404.783322 I 192.168.60.3 - 172.27.20.4 -
1072988404.785244 I 192.168.60.5 - 172.27.20.4 -
1072988407.945253 T 172.27.20.4 58173 192.168.60.3 21
1072988407.945315 T 172.27.20.4 58173 192.168.60.3 22
1072988407.945337 T 192.168.60.3 21 172.27.20.4 58173
1072988407.945429 T 192.168.60.3 22 172.27.20.4 58173

All of the action takes place in the series of flags passed after IPsumdump is called. They control the fields of the packet to be displayed:

• t: timestamp

• p: protocol; I for ICMP, T for TCP, U for UDP, or a number for others

• s: source IP

• S: source port

• d: destination IP

• D: destination port

Arranging these fields in a different order in the command line changes the order that the fields are displayed in each output line. Because the first four packets were ICMP (denoted by I), they have dashes (-) in the fields for port numbers.

IPsumdump accepts a slew of flags. Here is a group of them targeted at collecting information from TCP segments. (Although this output is broken into separate lines to accommodate page widths, in reality it is produced as a single line.)

-bash-2.05b$ ipsumdump -r sf1.lpc -tsSdD --tcp-seq --tcp-ack
  --tcp-flags --tcp-opt --filter "tcp"

!IPSummaryDump 1.1
!creator "ipsumdump -r sf1.lpc -tsSdD --tcp-seq --tcp-ack
  --tcp-flags --tcp-opt --filter tcp"

!host bourque.taosecurity.com
!runtime 1073490943.275335 (Wed Jan  7 10:55:43 2004)

!data timestamp ip_src sport ip_dst dport
  tcp_seq tcp_ack tcp_flags tcp_opt

1072988407.945253 172.27.20.4 58173 192.168.60.3 21
  2986655065 0 S .

1072988407.945315 172.27.20.4 58173 192.168.60.3 22
  2986655065 0 S .

1072988407.945337 192.168.60.3 21 172.27.20.4 58173
  0 2986655066 RA .

1072988407.945429 192.168.60.3 22 172.27.20.4 58173
  2769807338 2986655066 SA mss 1460

IPsumdump can listen on an interface in promiscuous mode, interact with NetFlow records, and write output in a binary format. In short, IPsumdump gives us a different way to look at full content data. The next tool, Etherape, gives analysts a graphical way to examine interactions between systems.

Etherape

Purpose: Graphical traffic display

Authors: Juan Toledo and Riccardo Ghetta

Internet site: http://etherape.sourceforge.net/

FreeBSD installation: Installed via FreeBSD 4.9 RELEASE package

Version demonstrated: 0.9.0

Etherape allows analysts to visually inspect traffic patterns. The program may either listen on a live interface or read a libpcap trace. Etherape can help analysts trace the flow of an incident if the full content data has been purged of nonrelevant information. Figure 6.2 shows Etherape interpreting the sf1.lpc trace from Chapter 4. I started it with the following command line to disable name resolution.

etherape –n –r sf1.lpc

Figure 6.2. Etherape interprets sf1.lpc

Screen captures don't really do Etherape justice because the program replays the events graphically in real time. The image changes as hosts talk to each other, with evidence of each session fading as the time since the last observed packet increases. The image in Figure 6.2 is a snapshot of an interesting portion of Etherape's representation of the trace. As traffic appears, Etherape tries to understand it and colors the links according to the protocols it recognizes. It lists the color mapping in the left-hand pane of the display.

Etherape knows the connection from 172.27.20.3 to 192.168.60.5 is an FTP session because Etherape recognizes the traffic to the FTP control channel (port 21 TCP) on 192.168.60.5. This was the exploit against the FTP server on 192.168.60.5 described as step 3 in Figure 4.3.

Etherape interprets the link between 192.168.60.5 and 172.27.20.5 as an unidentified TCP session, probably because the majority of the conversation is an FTP date channel session. This represents the intruder downloading tools from 172.27.20.5 to 192.168.60.5, depicted as step 5 in Figure 4.5.

The final link of interest involves 192.168.60.3 and 172.27.20.5. This is another FTP session, seen by Etherape as an unidentified TCP session due to the fact it is mostly FTP data traffic. This corresponds to step 8 in Figure 4.6, where the intruder downloads tools from 172.27.20.5 to 192.168.60.3.

You may notice that 172.27.20.4 is shown on the screen in Figure 6.2, but it has no links to any of the systems. Earlier in the Etherape replay, it connected to both 192.168.60.3 and 192.168.60.5. As was shown in step 1 of Figure 4.3, 172.27.20.4 performed reconnaissance before 172.27.20.3 exploited port 21 TCP on 192.168.60.5.

Besides graphical information, Etherape keeps a running tally of the traffic it sees in a separate window. Enable this feature by clicking the Protocol button (see Figure 6.3).

Figure 6.3. Etherape protocol list

Like the visual display, this window updates as the traffic moves along. I recommend using Etherape to take a fresh look at traces but not to replace session-level traffic analysis. I find session data (discussed in Chapter 7) to be the easiest way to understand conversations between hosts.

Netdude

Purpose: Graphical packet manipulator and editor

Author: Christian Kreibich

Internet site: http://netdude.sourceforge.net/

FreeBSD installation: Installed via source code

Version demonstrated: 0.4.3

Everyone who sees Netdude thinks its one of the coolest programs around. Netdude is a visual packet editor and manipulator built around three components.

1. Libpcapnav wraps the libpcap library with an application programming interface (API). This API allows user to move to arbitrary locations in a libpcap trace, using timestamps or fractional offsets.
2. Libnetdude provides data structures and APIs to create and manipulate arbitrarily large traces, packets, trace parts, protocols, Tcpdump output, and packet filters.
3. The Netdude application is GUI front end to libnetdude and libpcapnav. Users interact with Netdude via this interface.

Netdude features one of the most thorough sets of documentation I've ever seen in an open source project, so I'll only highlight some of its features in the next subsection.⁷

Using Netdude

Netdude makes it easy to work with large trace files. It loads a user-definable portion of the trace into memory; the default is 500 packets. Users can open more than one trace at a time. Netdude shows the trace in Tcpdump format, making it recognizable to people accustomed to command-line packet interpretation. In Figure 6.4, I've selected one of the packets from an FTP control channel session in trace sf1.lpc, then clicked the TCP tab to show TCP fields.

Figure 6.4. Original TCP header

Let's say I want to change the destination port of this packet from 3307 to 60000. By highlighting the field “Dst. port (3307)”, and entering 60000 in the window that appears, I alter the trace. I don't like the PSH flag (depicted by P in the figure) being set either, so I click on that field. Finally I set all of the remaining TCP options to be NOP (decimal value 1). Figure 6.5 shows the result.

Figure 6.5. Altered TCP header

Unfortunately for TCP purists, my alteration has rendered the original 0xcc5e checksum to be incorrect for this new destination port. This is no problem for Netdude. After selecting the Plugins→Checksum Fixer menu item, Netdude calculates a new checksum of 0xf5f9 and inserts it in the correct location.

We're not done with this packet, however. Besides altering headers, Netdude can alter application data. The packet in question has the FTP content shown in Figure 6.6.

Figure 6.6. Original application data

I don't like the name of the system, oates, so I edit it with Netdude. I also change the version number of the FTP server (see Figure 6.7).

Figure 6.7. Altered application data

To commit the changes to disk, use the File→Save As feature. By using the shift or control keys, analysts can highlight groups of packets and then use Netdude's Copy, Cut, and Paste capabilities to move packets around. Netdude is any forensic analyst's nightmare because it makes it easy to alter almost any aspect of a packet. The best defense is to compute hashes of trace files once they are created by using a utility like md5.

janney# md5 em0.lpc sf1.lpc
MD5 (em0.lpc) = 4995cbe06aa8618f0aceef95a362d056
MD5 (sf1.lpc) = f51c445a95541bda06229221de0aad80

Imagine looking for the IP ID field in the first packet in the sf1.lpc trace, shown here in bold.

janney# tcpdump -n -c 1 -v -r sf1.lpc

15:20:04.783092 172.27.20.4 > 192.168.60.3:
  icmp: echo request (ttl 53, id 4134, len 28)

We use Netdude to change the IP ID from 4134 to 4135. We don't recompute the checksum, so only a single bit of the packet has changed: 34 in binary is 00100010, and 35 in binary is 00100011. Had we changed the checksum using Netdude's Checksum Fixer feature, more than a bit in the whole packet would have changed, but the checksum would match the new IP header. In the modified trace that follows, Tcpdump reports a bad checksum because we modified the IP ID but did not let Netdude correct the checksum as well.

janney# tcpdump -n -c 1 -v -r sf1_mod.lpc
  icmp: echo request (ttl 53, id 4135, len 28, bad cksum b8f0!)
15:20:04.783092 172.27.20.4 > 192.168.60.3:

If we run md5 against the new sf1_mod.lpc file, we see an entirely different result.

janney# md5 sf1_mod.lpc
MD5 (sf1_mod.1pc) = b87d00765491d03481bc83e49da9a718

Changing just one bit in the IP header causes a completely different MD5 hash.

What Do Raw Trace Files Look Like?

This book revolves around capturing files in various formats and analyzing them with open source tools. Exactly what does a raw trace file look like? To find out, I sent a single ICMP echo packet to a Sun Ultra 30 machine running Solaris 8, listening with Snoop and Tcpdump. First, here's the setup for the Snoop session.

solaris# snoop -o bigend.snoop src host 192.168.50.2 and icmp
Using device /dev/hme (promiscuous mode)
1 ^C
solaris# ls -al bigend.snoop
-rw-r--r--  1 richard  richard  120 Jan 22 12:46 bigend.snoop

For analysis I brought the capture file back to a FreeBSD system and checked the trace with the file command.

freebsd# file bigend.snoop
bigend.snoop: Snoop capture file - version 2 (Ethernet)

Here's how I set up Tcpdump.

freebsd# tcpdump -n -i hme0 -s 1515 -w bigend.lpc -c 1
  icmp and src host 192.168.50.2 and dst host 10.10.10.5
tcpdump: listening on hme0
357 packets received by filter
0 packets dropped by kernel
freebsd# ls -al bigend.lpc
-rw-r--r--  1 richard  richard  114 Jan 22 12:45 bigend.lpc

When I brought the trace to my FreeBSD system, the file command yielded interesting but not unexpected results.

freebsd# file bigend.lpc
bigend.lpc: tcpdump capture file (big-endian) - version 2.4
  (Ethernet, capture length 1515)

While collecting the trace files on the Solaris system, I also collected the same packet on an Intel box running FreeBSD.

freebsd# tcpdump -n -i em0 -s 1515 -w litend.lpc -c 1 icmp and
  src host 192.168.50.2 and dst host 10.10.10.5
tcpdump: WARNING: em0: no IPv4 address assigned
tcpdump: listening on em0
1949 packets received by filter
0 packets dropped by kernel
freebsd# ls -al litend.lpc
-rw-r--r--  1 richard  richard  114 Jan 22 12:45 litend.lpc

Here's how the file command saw that trace.

freebsd# file litend.lpc
litend.lpc: tcpdump capture file (little-endian) - version 2.4
  (Ethernet, capture length 1515)

What do the terms big-endian and little-endian mean? They refer to the order in which a sequence of bytes is stored in a computer's memory. Computers following big-endian conventions, like SPARC systems, store the most significant byte of a multibyte sequence in the lowest memory address. Computers following little-endian conventions, like Intel systems, store the least significant byte of a multibyte sequence in the lowest memory address.⁸

Consider a sequence of bytes like these in hexadecimal form: 0x00 02 23 e5. This value is 140261 in decimal. In binary it's the following:

00000000 00000010 00100011 11100101

Table 6.1 shows how this value would be stored in memory on big- and little-endian systems.

Table 6.1. Big-endian and little-endian forms of a binary value

In addition to computer memory storing information in either format, programs can store data in big- or little-endian form. The Solaris on SPARC version of Tcpdump stored the trace in big-endian form because the SPARC processor follows the big-endian convention. The FreeBSD on Intel version of Tcpdump stored the trace file in little-endian format because the Intel processor follows the little-endian convention. Keep these two forms in mind when we look at raw traces.⁹

Now that we have our traces, let's look at the information stored in each. The format of Snoop records are standardized by RFC 1761. Figure 6.8 shows the contents of the raw bigend.snoop record, with each field explained.

Figure 6.8. Snoop record

Now that we're familiar with the internals of a Snoop record, let's look at the Tcpdump record in big-endian format from the Solaris system (see Figure 6.9).

Figure 6.9. Tcpdump record in big-endian format

Besides a different header format, the only real difference between the two packets is the 17 microsecond gap between the two timestamps. (The slightly earlier Tcpdump microsecond timestamp is highlighted in the Tcpdump record trace.) The purpose of the “magic numbers” field is not intuitive, but its purpose won't be apparent until we compare this trace to the version collected on the little-endian Intel system, shown in Figure 6.10.

Figure 6.10. Tcpdump record in little-endian format

All of the highlighted fields differ from their counterparts from the big-endian Tcpdump trace. Evidence of byte swapping appears everywhere. The “magic numbers” field from the big-endian SPARC trace contains a1 b2 c3 d4, but this little-endian Intel trace shows d4 c3 b2 a1.

The timestamps are not only byte-swapped, they are very different. The little-endian Intel trace shows a time of 1074793483.627227 (seconds.microseconds), about 325 seconds different from the record seen in the SPARC big-endian trace. A check of the time on the SPARC box shows it is slow.

solaris# ntpdate clock.isc.org
22 Jan 13:40:12 ntpdate[3346]: step time server 204.152.184.72
  offset -325.798466 sec

The time on the Intel box is almost dead-on.

freebsd# ntpdate clock.isc.org
22 Jan 13:48:19 ntpdate[3675]: step time server 204.152.184.72
  offset -0.793575 sec

This is a good example of the necessity of checking time differences between systems when doing simultaneous monitoring.

Notice that I did not interpret the little-endian 0x0b0c1040 seconds timestamp as 185,339,968 seconds. That is obviously not the correct date.

-bash-2.05b$ date -r 185339968
Sat Nov 15 22:19:28 EST 1975

Instead, I converted the little-endian 0x0b0c1040 to the big-endian format of 0x40100c0b, which is 1,074,793,483 decimal or the following date.

-bash-2.05b$ date -r 1074793483
Thu Jan 22 12:44:43 EST 2004

Does Tcpdump know to swap bytes, to accommodate a big-endian trace file on a little-endian system, or vice versa? Let's compare the way Tcpdump interprets the timestamps to verify this assertion. First we have Tcpdump read the little- and big-endian traces on the little-endian Intel system. We pass the –tt flag to display the timestamp, which, as we've seen, varies in the raw traces.

freebsd# tcpdump -n -r litend.lpc -tt
1074793483.627227 192.168.50.2 > 10.10.10.5: icmp: echo request
freebsd# tcpdump -n -r bigend.lpc -tt
1074793808.538438 192.168.50.2 > 10.10.10.5: icmp: echo request

These are just as we expected. Let's run the same check on the big-endian Solaris system.

solaris# tcpdump -n -r litend.lpc -tt
1074793483.627227 192.168.50.2 > 10.10.10.5: icmp: echo request
solaris# tcpdump -n -r bigend.lpc -tt
1074793808.538438 192.168.50.2 > 10.10.10.5: icmp: echo request

How does Tcpdump know how to handle the different file formats? The “magic numbers” field helps Tcpdump differentiate the two file formats. “Magic number” refers to a value used by the file command to determine the type of data in a file. A check of the /usr/share/misc/magic file on the FreeBSD system shows four entries for Tcpdump.

freebsd# grep -i tcpdump /usr/share/misc/magic
0 ubelong 0xa1b2c3d4 tcpdump capture file (big-endian)
0 ulelong 0xa1b2c3d4 tcpdump capture file (little-endian)
0 ubelong 0xa1b2cd34 extended tcpdump capture file (big-endian)
0 ulelong 0xa1b2cd34 extended tcpdump capture file little-endian

The first column is the offset, meaning where in the file to check for the value represented by the third column. The second column is a data type. Here we see ubelong, which is an unsigned four-byte big-endian value, and ulelong, an unsigned four-byte little-endian value. This information helps the file command recognize the format of each trace, as shown earlier. Even the Snoop trace is recognized, thanks to the following entry in the magic file.

0 string   snoop      Snoop capture file

Because the Snoop data type is string, the file command recognizes the snoop identifier in the beginning of the trace as being a Snoop record.

You may recognize that the magic file helps the file program recognize saved libpcap traces. Tcpdump doesn't rely on the magic file when it saves packets. Instead, it incorporates a program called savefile.c, which offers the following clues.

  * savefile.c - supports offline use of tcpdump
  *      Extraction/creation by Jeffrey Mogul, DECWRL

  *      Modified by Steve McCanne, LBL.
  *
  * Used to save the received packet headers, after filtering, to
  * a file, and then read them later.
  * The first record in the file contains saved values for the
  * machine dependent values so we can print the dump file on any
  * architecture./
...edited...
#define TCPDUMP_MAGIC 0xa1b2c3d4
#define PATCHED_TCPDUMP_MAGIC 0xa1b2cd34

/*
  * We use the "receiver-makes-right" approach to byte order,
  * because time is at a premium when we are writing the file.
  * In other words, the pcap_file_header and pcap_pkthdr
  * records are written in host byte order.
  * Note that the packets are always written in network
  * byte order.
  *
  * ntoh[ls] aren't sufficient because we might need to swap
  * on a big-endian machine (if the file was written in
  * little-end order).
  */

We see the TCPDUMP_MAGIC value of 0xa1b2c3d4 matches the value found in our traces. We also see that libpcap headers are written in host byte order while packet contents are saved in network byte order. This accounts for the different byte orders for the libpcap headers on SPARC versus Intel and the consistent packet contents on both architectures.

Network byte order describes the format of data as it is sent on the wire. By convention, network byte order uses the big-endian format. That's why a value like the IP header checksum is 0x23b1 in both traces. Another example is the IP ID, which is 0x02cb in both traces.

Now that you know what raw Snoop and Tcpdump (libpcap) traces look like, keep in mind the issues with big- and little-endian byte ordering. While it won't make a difference for the packet contents, it does affect values added by the applications themselves. These include timestamps, lengths, and other meta-data calculated by Snoop or Tcpdump.

If you'd like to experiment with another packet alteration tool besides Netdude, try Netsed by Michal Zalewski, author of P0f.¹⁰ Netdude suits our generic packet mangling needs, but let's take a look at P0f next.

P0f

Purpose: Passive operating system identification system

Author: Michal Zalewski

Internet site: http://lcamtuf.coredump.cx/p0f.shtml

FreeBSD installation: Installed via /usr/ports/net/p0f

Version demonstrated: 2.0.3

P0f is a passive operating system identification tool. It tries to determine the operating system of hosts it observes communicating by using three tests.

1. The default uses a SYN packet test, where P0f watches inbound SYN packets destined for the local network.
2. Using –A enables the SYN+ACK test, where P0f makes decisions based on SYN ACK packets. These are responses from open remote ports.
3. Using –R activates the RST+ACK test, where P0f makes decisions based on RST ACK packets. These are responses from closed remote ports.

These tests are depicted visually in Figure 6.11, with P0f making decisions on the packets represented by solid lines. Although the diagram shows P0f running on a separate probe, P0f can also be run on the host at the bottom communicating with the three remote systems.

Figure 6.11. Three P0f passive operating system identification methods

Analysts can run P0f's three tests using two modes. The first, default mode, operates against a live interface and reports what it sees. First we run P0f in default mode and enable the SYN packet test. P0f will watch inbound SYN packets from remote hosts. Since P0f understands BPF syntax, we tell it to observe port 22 TCP.

bourque# p0f -i fxp0 'port 22'
p0f - passive os fingerprinting utility, version 2.0.3
(C) M. Zalewski <[email protected]>, W. Stearns
  <[email protected]>
p0f: listening (SYN) on 'fxp0', 206 sigs (12 generic), rule:
  'port 22'.
10.10.10.2:57868 - HP-UX 11.00-11.11
  -> 172.27.20.3:22 (distance 1, link: ethernet/modem)
10.10.10.3:1085 - Windows 2000 SP2+, XP SP1 (seldom 98 4.10.2222)
  -> 172.27.20.3:22 (distance 1, link: ethernet/modem)

10.10.10.5:32805 - Solaris 8 (1)
  -> 172.27.20.3:22 (distance 1, link: ethernet/modem)
172.27.20.11:44521 - OpenBSD 3.0-3.4 [high throughput]
  (up: 5613 hrs)
  -> 172.27.20.3:22 (distance 0, link: ethernet/modem)
192.168.60.3:34744 - Linux 2.4/2.6 [high throughput]
  (up: 532 hrs)
  -> 172.27.20.3:22 (distance 1, link: ethernet/modem)
172.27.20.5:1376 - FreeBSD 4.6-4.8 [high throughput]
  (up: 26 hrs)
  -> 172.27.20.3:22 (distance 0, link: ethernet/modem)

This default SYN test is P0f's most accurate. It is correct in all operating system brands and very close on all specific versions.

With the following command we tell P0f to use SYN+ACK test to watch the fxp0 interface. It looks for SYN ACK packets from port 22 or 139 TCP on remote systems.

bourque# p0f -i fxp0 -f p0fa.fp  -A 'port 22 or port 139'
p0f - passive os fingerprinting utility, version 2.0.3
(C) M. Zalewski <[email protected]>, W. Stearns
  <[email protected]>
p0f: listening (SYN+ACK) on 'fxp0', 57 sigs (1 generic), rule:
  'port 22 or port 139'.

10.10.10.2:22 - UNKNOWN [32768:63:1:60:M1460,W0,N,N,N,T:AT:?:?]
  (up: 446 hrs)
  -> 172.27.20.3:4492 (link: ethernet/modem)
10.10.10.3:139 - UNKNOWN [S12:127:1:60:M1460,N,W0,N,N,T0:A:?:?]
  -> 172.27.20.3:4493 (link: ethernet/modem)
10.10.10.5:22 - UNKNOWN [24616:63:1:60:N,N,T,N,W0,M1460:AT:?:?]
  (up: 446 hrs)
  -> 172.27.20.3:4494 (link: ethernet/modem)
172.27.20.5:22 - FreeBSD 4.6-4.8 (RFC1323) (up: 27 hrs)
  -> 172.27.20.3:4496 (distance 0, link: ethernet/modem)
172.27.20.11:22 - UNKNOWN [17376:64:1:60:M1460,N,W0,N,N,T:AT:?:?]
  (up: 5613 hrs)
  -> 172.27.20.3:4497 (link: ethernet/modem)
192.168.60.3:22 - UNKNOWN [5792:63:1:60:M1460,N,N,T,N,W0:ZAT:?:?]
  (up: 532 hrs)
  -> 172.27.20.3:4498 (link: ethernet/modem)

We see P0f had lots of trouble with these packets. This means you should be wary using P0f to identify systems based on the SYN ACK packets they send back to you.

Finally, we run P0f's RST+ACK test. The following command specifies to watch for RST ACK packets from port 81 TCP on remote systems.

bourque# p0f -i fxp0 –f p0fr.fp -R 'port 81'
p0f - passive os fingerprinting utility, version 2.0.3
(C) M. Zalewski <[email protected]>, W. Stearns
  <[email protected]>
p0f: listening (RST+) on 'fxp0', 46 sigs (3 generic), rule:
  'port 81'.
10.10.10.2:81 - UNKNOWN [0:63:1:51:.:K0AD:?:?] (refused)
  -> 172.27.20.3:4423 (link: unspecified)
10.10.10.3:81 - Windows XP/2000 (refused)
  -> 172.27.20.3:4424 (distance 1, link: unspecified)
10.10.10.5:81 - FreeBSD 4.8 (refused)
  -> 172.27.20.3:4425 (distance 1, link: unspecified)
172.27.20.5:81 - FreeBSD 4.8 (refused)
  -> 172.27.20.3:4429 (distance 0, link: unspecified)
172.27.20.11:81 - FreeBSD 4.8 (refused)
  -> 172.27.20.3:4427 (distance 0, link: unspecified)
192.168.60.3:81 - Linux recent 2.4 (refused)
  -> 172.27.20.3:4428 (distance 1, link: unspecified)

These results aren't bad, with P0f guessing half of the operating systems correctly. As a passive system using a single packet to make decisions, P0f does a good job with the data it has. When users contribute more signatures to its database, P0f will be able to identify operating systems more accurately.

The second mode for P0f involves reading a trace file. With the following command, we run the program against the em0.lpc trace. We specify the trace with the –s switch and dump full packet contents with the –x switch. Again, P0f is broadly accurate.

bourque# p0f -s /nsm/pt2/em0.lpc -x | less
p0f - passive os fingerprinting utility, version 2.0.3
(C) M. Zalewski <[email protected]>, W. Stearns
  <[email protected]>
p0f: listening (SYN) on '/nsm/pt2/em0.lpc', 206 sigs
  (12 generic),  rule: 'all'.
[+] End of input file.
192.168.60.3:34720 - Linux 2.4/2.6 (up: 69 hrs)
  -> 10.10.10.3:3389 (distance 1, link: ethernet/modem)
45 00 00 3c 65 91 40 00 3f 06 c5 72 c0 a8 3c 03 |E..<e.@.?..r..<.
0a 0a 0a 03 87 a0 0d 3d 8c 59 87 20 00 00 00 00 |.......=.Y. ....
a0 02 16 d0 93 f8 00 00 02 04 05 b4 04 02 08 0a |................
01 7d e2 b1 00 00 00 00 01 03 03 00             |.}..........
10.10.10.3:1075 - Windows 2000 SP2+, XP SP1 (seldom 98 4.10.2222)
  -> 172.27.20.5:21 (distance 0, link: ethernet/modem)
45 00 00 30 04 15 40 00 80 06 22 86 0a 0a 0a 03 |E..0..@...".....
ac 1b 14 05 04 33 00 15 8a 33 c3 13 00 00 00 00 |.....3...3......
70 02 40 00 1d 63 00 00 02 04 05 b4 01 01 04 02 |[email protected]..........

Other useful P0f flags include the following.

• –w writes SYN and SYN ACK packets used for analysis to a libpcap-formatted trace file for further analysis.

• –o sends P0f's judgments to an ASCII text file.

• –N tells P0f to report only source IP and operating system type.

• –D disables operating system version guessing, so “Linux 2.4/2.6” is reported as “Linux.”

• –p activates promiscuous status, forcing P0f to make decisions on all packets it sees when watching a live interface. Without this switch, P0f will only judge traffic to or from the local machine.

• –d enables daemon status and requires sending P0f's operating system guess to an outfile with –o.

• –f tells P0f the location of its fingerprint files. The default file is p0f.fp. SYN+ACK test uses p0fa.fp. RST+ACK test uses p0fr.fp.

The Sguil project incorporates P0f data to provide additional details on traffic from remote hosts. P0f is particularly helpful in situations where the operating system fingerprinting tool must be hidden. Since P0f emits no packets to make its identification decisions, remote hosts will not be aware of its presence. The exception to this rule involves telling P0f to resolve IP addresses to host names with the –r switch. If P0f resolves an intruder's IP address by contacting a name server the intruder controls, the intruder might assume the system requesting the host name is a monitoring platform of some type.

Conclusion

This chapter introduced a variety of tools that augment the core full content tools described in Chapter 5. Tools like Editcap, Mergecap, and Tcpslice prepare libpcap files for additional analysis. Tcpreplay gives applications that work only on live interfaces a chance to see old data. Tcpflow rebuilds TCP application streams without the GUI required by Ethereal's “follow TCP sessions” function. Ngrep allows easy string matching of application data using regular expressions. IPsumdump shows packet details on a single line, suitable for parsing by other tools. Etherape depicts traffic graphically. Netdude facilitates packet editing and manipulation. Finally, P0f reads full content data from live interfaces or a trace and identifies operating systems. Having taken a close look at packet details revealed by full content data, we step up one level of analysis. Leaving headers and application data behind, in Chapter 7 we turn to session data.