14. Discovering DNS

Analysis is the heart of NSM, which was developed to give analysts the tools and tactics they need to validate and escalate events. This chapter is the first of three case studies designed to help analysts learn how to investigate events using NSM principles.

One of the best ways to become a proficient NSM analyst is to handle traffic. Network activity can be classified as normal, suspicious, or malicious. Learning to differentiate among the three classes of traffic requires exposure to examples of each. This chapter only hints at some of the traffic an analyst will find in the wild, but it's a start.

Analysts practice NSM to detect and scope intrusions. Traffic caused by many intruders appears much different from run-of-the-mill network activity. Why even discuss normal traffic? By having a baseline for normalcy, analysts can judge traffic to be suspicious or perhaps malicious. This chapter does not show normal traffic for its own sake, however. If you want to know how normal HTTP or SMB/CIFS appears, refer to the books on TCP/IP mentioned in the Preface. When we do discuss normal traffic, it will be for the sake of comparison with suspicious and malicious activity.

This case study describes various services that use ports 53 UDP and TCP. I describe normal, suspicious, and malicious examples. Some of them are the result of Sguil detections, while others are examples of traffic that occurs normally. While researching this chapter I was amazed at the wide variety of traffic found on ports 53 UDP and TCP. Because DNS is so fundamental to the health of the modern Internet, it's important to understand what does and does not belong on those ports.

Normal Port 53 Traffic

Normal traffic is what analysts expect to see in the course of healthy, uncompromised network operations. Normal traffic is usually described in textbooks and RFCs, and it makes sense as soon as you see it. Let's discuss normal activity for UDP and TCP services using port 53.

Normal Port 53 UDP Traffic

We start with an alert deployed to detect unusually large UDP DNS packets. One day while I was monitoring Sguil, well over 1,300 LOCAL Large UDP DNS packet alerts appeared within a few minutes. This alert is highlighted in Figure 14.1.

Figure 14.1. Sguil displaying the LOCAL Large UDP DNS packet alert

I had deployed the following Snort rule to detect DNS-based covert channels.

alert udp any any -> any 53 (msg:"LOCAL Large UDP DNS packet";)
  dsize > 256; classtype:unknown; sid:1000001; rev:1;)

This rule fires whenever Snort sees a packet with destination port 53 UDP that exceeds 256 bytes. Why 256 bytes? Consider the trace of routine DNS traffic viewed with Ethereal, shown in Figure 14.2. Source 68.84.6.72 asks the name server at 68.48.0.5 for the A record for slashdot.org. The A record shows the IP address corresponding to the server slashdot.org.

Figure 14.2. Normal DNS query for slashdot.org

Ethereal displays packet contents in its bottom tier. Each line begins with a hexadecimal number representing the offset of the first byte of data shown on that line. For example, the first line in the Ethereal data window of Figure 14.2 begins with 0010. This means the packet byte value that follows, 00, is located at hexadecimal offset 0010, or decimal offset 16, in the packet. Combined with the 3a at offset 0x0011, we have 0x003a or decimal 58. This position in the IP header holds a two-byte value for the total length of the IP datagram; therefore, this packet has 58 bytes of IP information. A packet begins at offset 0x0000 (not shown in Figure 14.2) with a 14-byte Ethernet frame header. Adding 58 bytes of IP data (IP header, UDP header, and UDP data) to 14 bytes of Ethernet header yields a complete packet that's 72 bytes long.

The response, shown in Figure 14.3, is slightly more interesting because it contains four main sections:

1. A restatement of the query, namely, the A record for slashdot.org
2. An answer section, showing that slashdot.org has IP address 66.235.250.150 (highlighted in Figure 14.3)
3. An authoritative name servers section, displaying three DNS servers considered authoritative for the slashdot.org domain: ns3.vasoftware.com, ns2.vasoftware.com, and ns1.vasoftware.com
4. An additional records section, showing the IP addresses of each of the authoritative name servers

Figure 14.3. Normal DNS response for slashdot.org

For illustration purposes, the first authoritative name server record and the first additional record are expanded in the Ethereal screenshot. How big is this packet? You can tell by looking at the last line number Ethereal assigns to the packet in Figure 14.3. It's 0x00c0, which is 192 decimal. We subtract 4 bytes, however, because the last line is not a full 16-byte line. That means the entire packet is 188 bytes.

Now that we see how a normal DNS query and its response appear, let's look at the specifics of the LOCAL Large UDP DNS packet alert. Let's take a close look at the packet content shown in Sguil's lower-right window (see Figure 14.4).

Figure 14.4. Packet data for LOCAL Large UDP DNS packet alert

This is odd; the packet displayed doesn't appear to be larger than 256 bytes as required by the Snort rule. It's possible that Sguil is having trouble locating the proper packet. We do get a piece of useful information, however: The query involves www.yahoo.akadns.net. The next step is to go to the full content data and inspect it directly.

Sguil allows an analyst to directly retrieve full content data using Ethereal. The process may show related traffic, which requires scrolling through the packet results to locate the traffic of interest. The top window in Figure 14.5 shows three request-response pairs involving DNS requests for www.yahoo.akadns.net.

Figure 14.5. DNS query for www.yahoo.akadns.net

How can we tell which pair corresponds to the Sguil alert? Often we can use the source and destination ports, which are 1114 and 53 UDP. Here those ports are all the same, which means that information won't help us. (Remember port 1114 UDP for the next section, though.) For the purposes of determining whether the alert represents normal, suspicious, or malicious traffic, any ports will do. Nevertheless, the easiest way to identify the offending packet is to match the DNS application data in the Sguil packet with the values shown in Ethereal. Figure 14.4 shows that Sguil's packet data begins with values 0xc7f5. These are the first values for any DNS record, as defined by Internet Best Current Practice (BCP) 42 “Domain Name System (DNS) IANA Considerations,” also known as RFC 2929.¹ The first 16 bytes are the transaction ID, a value used to match DNS queries and responses. Figure 14.6 shows the packet format.

Figure 14.6. DNS query and response packet format

To get a better understanding of this DNS traffic, we told Sguil to launch Ethereal against the full content data associated with the alert in question. When Ethereal opened and analyzed the DNS traffic, it showed the first packet of interest in Figure 14.5. Ethereal identifies the highlighted packet as having DNS transaction ID 0x1b57, which doesn't match the Sguil packet. Scrolling through the second and third packets, we find that the DNS ID in the third packet in Figure 14.5 does match the DNS transaction ID of 0xc7f5 displayed in Figure 14.4. This means the DNS request and response pair of the third and fourth packets in Figure 14.5 are the ones we want to investigate.

In all respects except the transaction ID, the query packets in Figure 14.5 are identical. To find something more interesting, we turn from the query request to the query response. Figure 14.7 shows the query response for the query request bearing DNS transaction ID 0xc7f5.

Figure 14.7. DNS response for www.yahoo.akadns.net

The immediate impression given by this packet is its immense size: 510 bytes. Why is it so large? The reason involves the number of answers returned by the DNS server: eight A records are returned. How does one get such a response? Try the following. Using the nslookup command, query for www.yahoo.com, as shown in Figure 14.8.

Figure 14.8. DNS query for www.yahoo.com using Nslookup

The IP addresses returned in Figure 14.8 are similar to the eight seen in the Ethereal trace in Figure 14.7. The results are not the same because Yahoo operates dozens of IP addresses for its http://www.yahoo.com Web site. Beyond those eight IP addresses, the DNS server returns authoritative name servers and additional records, as shown in Figure 14.9. One of the most interesting aspects of DNS traffic is its concise representation of answers. Notice that the authoritative name server record for zh.akadns.net displays only the zh portion plus nonprintable characters, shown in the ASCII output as periods (.). These nonprintable characters are references so the DNS recipient can associate zh with the domain akadns.net.

Figure 14.9. Highlighting authoritative name server zh.akadns.net

When the additional record for zh.akadns.net is shown in Ethereal, there is no human-readable mention of the entire system name in the bottom window (see Figure 14.10). Rather, the DNS recipient associates the IP addresses with the system name using a pointer.

Figure 14.10. Highlighting additional record for zh.akadns.net

Ultimately, this Sguil alert for LOCAL Large UDP DNS packet was caused by the normal DNS response for a query associated with www.yahoo.com. This is an example of completely normal traffic. In light of this discovery, it may be appropriate to alter the Snort signature. Or perhaps not. By leaving this signature, we may discover traffic worth investigating.

Normal Port 53 TCP Traffic

Normal port 53 TCP traffic is rarer than port 53 UDP traffic, but it does occur naturally on networks. DNS uses TCP for two cases. First, the DNS standard states that queries that return large responses (exceeding 512 bytes of application data) must use TCP. Second, DNS servers in master and slave relationships share their files via a zone transfer mechanism. This transfer takes place over port 53 TCP.

While it's extremely rare to see TCP-based DNS query responses, we can force DNS to use TCP by using the dnsquery command, as shown here.

-bash-2.05b$ dnsquery -s www.amazon.com.
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59265
;; flags: qr rd ra; QUERY: 1, ANSWER: 5,AUTHORITY:4,ADDITIONAL:4
;;      www.amazon.com, type = ANY, class = IN
www.amazon.com.         56S IN A        207.171.184.16
www.amazon.com.         16h9m4s IN NS   ns-20.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-10.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-30.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-40.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-20.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-10.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-30.amazon.com.
www.amazon.com.         16h9m4s IN NS   ns-40.amazon.com.
ns-20.amazon.com.       15h46m30s IN A  207.171.189.20
ns-10.amazon.com.       15h46m30s IN A  207.171.178.146
ns-30.amazon.com.       14h55m12s IN A  207.171.167.7
ns-40.amazon.com.       15h46m30s IN A  207.171.169.7

This traffic takes place over port 53 TCP, as Figure 14.11 demonstrates. The highlighted packet is the query, and packet 15 contains the response.

Figure 14.11. DNS query using TCP

While it's odd to see TCP-based DNS queries and replies, there's nothing malicious about it. The second sort of TCP-based DNS, zone transfers, may be malicious if the party requesting the exchange is unauthorized. Users can prompt zone transfers by using the ls –d <domainname.com> command within nslookup, or by using the host command, as shown here.

-bash-2.05b$ host -l -v -t any taosecurity.com.
rcode = 0 (Success), ancount=1
Found 1 addresses for moog.taosecurity.com
Trying 172.27.20.1
taosecurity.com 3600 IN SOA     taosecurity.com
                        root.moog.taosecurity.com (
                        2003121600      ;serial (version)
                        3600    ;refresh period
                        900     ;retry refresh this often
                        3600000 ;expiration period
                        3600    ;minimum TTL
                        )
taosecurity.com          3600 IN NS    moog.taosecurity.com
sweeney.taosecurity.com  3600 IN A     192.168.2.1
wireless.taosecurity.com 3600 IN A     192.168.50.2
...edited...
janney.taosecurity.com   3600 IN A     172.27.20.5
1.taosecurity.com        3600 IN PTR   localhost.taosecurity.com
gruden.taosecurity.com   3600 IN A     10.10.10.10

...edited...
lemelin.taosecurity.com  3600 IN A     172.27.20.11
taosecurity.com 3600 IN SOA     taosecurity.com
                        root.moog.taosecurity.com (
                        2003121600      ;serial (version)
                        3600    ;refresh period
                        900     ;retry refresh this often
                        3600000 ;expiration period
                        3600    ;minimum TTL
                        )

The traffic looks like the following output in Tcpdump.

1. 16:32:23.097693 172.27.20.5.2748 > 172.27.20.1.53:
  S 2025854818:2025854818(0) win 57344
  <mss 1460,nop,wscale 0,nop,nop,timestamp 37170134 0> (DF)

2. 16:32:23.097697 172.27.20.1.53 > 172.27.20.5.2748:
  S 2044888453:2044888453(0) ack 2025854819 win 57344
  <mss 1460,nop,wscale 0,nop,nop,timestamp 5944388 37170134> (DF)

3. 16:32:23.097815 172.27.20.5.2748 > 172.27.20.1.53:
  . ack 1 win 57920 <nop,nop,timestamp 37170134 5944388> (DF)

4. 16:32:23.097939 172.27.20.5.2748 > 172.27.20.1.53:
  P 1:3(2) ack 1 win 57920 <nop,nop,timestamp 37170134
  5944388> (DF)

5. 16:32:23.192887 172.27.20.1.53 > 172.27.20.5.2748:
  . ack 3 win 57920 <nop,nop,timestamp 5944398 37170134> (DF)

6. 16:32:23.192895 172.27.20.5.2748 > 172.27.20.1.53:
  P 3:36(33) ack 1 win 57920 <nop,nop,timestamp 37170143
  5944398> 256 [b2&3=0x1] [0q] [2932au] (31) (DF)

7. 16:32:23.194375 172.27.20.1.53 > 172.27.20.5.2748:
  P 1:1231(1230) ack 36 win 57920
  <nop,nop,timestamp 5944398 37170143> 33717*- 1/0/0 SOA
  (1228) (DF)

8. 16:32:23.285064 172.27.20.5.2748 > 172.27.20.1.53:
  . ack 1231 win 57920 <nop,nop,timestamp 37170153 5944398> (DF)

9. 16:32:23.285193 172.27.20.1.53 > 172.27.20.5.2748:
  P 1231:1359(128) ack 36 win 57920
  <nop,nop,timestamp 5944407 37170153> 33717- [0q]
  1/0/0 A 172.27.20.11 (126) (DF)

10. 16:32:23.285564 172.27.20.5.2748 > 172.27.20.1.53:
  F 36:36(0) ack 1359 win 57920
  <nop,nop,timestamp 37170153 5944407> (DF)

11. 16:32:23.285690 172.27.20.1.53 > 172.27.20.5.2748:
  . ack 37 win 57920 <nop,nop,timestamp 5944407 37170153> (DF)

12. 16:32:23.285940 172.27.20.1.53 > 172.27.20.5.2748:
  F 1359:1359(0) ack 37 win 57920
  <nop,nop,timestamp 5944407 37170153> (DF)

13. 16:32:23.286063 172.27.20.5.2748 > 172.27.20.1.53:
  . ack 1360 win 57920 <nop,nop,timestamp 37170153 5944407> (DF)

This Tcpdump output is fairly cryptic; it's easier to make sense of the protocol using Ethereal. After launching Ethereal, we see the packets as shown in Figure 14.12.

Figure 14.12. DNS zone transfer

The highlighted packet in Figure 14.12 is the beginning of the response to the zone transfer query. The highlighted section of the middle pane shows the query was of type AXFR, indicating a zone transfer. We see the displayed answer section showing that moog.taosecurity.com is the authoritative name server for the taosecurity.com domain. If we were to scroll farther down in the packet, we would see additional information.

Remember that this activity is all occurring over port 53 TCP. When done between DNS servers, zone transfers keep slave servers in sync with the master's zone file. Unfortunately, intruders can make the same requests to download the contents of a name server's zone file. Best security practices dictate that administrators disallow zone transfer requests from anyone except authorized slave name servers. Nevertheless, many DNS servers on the Internet happily answer zone transfer queries. When performed by an unauthorized party, a zone transfer should be considered a malicious form of reconnaissance.

Suspicious Port 53 Traffic

Suspicious traffic involves packets caused by unrecognized applications and protocols. These are outside the scope of an analyst's daily investigations. In many cases their odd nature leads an analyst to consider them dangerous or indicative of compromise. Don't jump to conclusions without examining every possible angle; use alert, full content, session, or statistical data as necessary.

Suspicious Port 53 UDP Traffic

The following example continues the investigation of the LOCAL Large UDP DNS packet alerts. Remember that Sguil reported over 1,300 of them. So far we've investigated only a handful, caused by normal DNS queries for Yahoo-related sites. Now we turn our attention to the bulk of the alerts, which Sguil aggregated into a single line in its display earlier in Figure 14.1. If we query for the individual alerts themselves, we get a listing like the one shown in Figure 14.13. The very top alert was the result of the DNS query for www.yahoo.com; remember port 1114 UDP?

Figure 14.13. Individual Sguil LOCAL Large UDP DNS packet alerts

The first difference between the highlighted alert in Figure 14.13 and the Yahoo query alert is the source port—1114 UDP for Yahoo and 1205 UDP for these new alerts. The fact that all of the alerts after the first involve port 1205 UDP indicates they were generated by the same session on an unknown application. The next difference between the alerts is the odd content of the highlighted packet. The human-readable portion ends in “netrogenic.com”, which is a domain name, but what do the preceding characters mean? If we accept this as legitimate DNS traffic, the first two bytes (0x0588) in the highlighted packet of Figure 14.13 should be the DNS transaction ID. What about the rest?

At this point we might be tempted to visit www.netrogenic.com, but let's turn to Ethereal's interpretation of the traffic first to see if it accepts these packets as legitimate DNS traffic. We can tunnel anything over arbitrary ports. This could be a back door that happens to use port 53 UDP. Ethereal may shed some light on the matter in Figure 14.14.

Figure 14.14. Query involving netrogenic.com

The highlighted packet matches the one selected earlier in Sguil. Bytes 0x0588 are highlighted, showing that they appear to represent transaction ID 0x0588. This is where the action begins, but to satisfy your curiosity let's briefly check out the pertinent sections of the first four packets.

The very first packet caused the DNS server to reply with a “Format error” in packet 2. How did that first packet look? Figure 14.15 displays the section that caused the problem.

Figure 14.15. Packet 1: odd query for root

Without knowing how a normal query for the DNS root appears, it's tough to identify the quirks of this packet. We suspect the additional records section, but we can't be sure what's odd about this query. Packet 3 (see Figure 14.16), which resulted in a normal response, is a good candidate for comparison.

Figure 14.16. Packet 3: normal query for root

Packet 3 in Figure 14.16 shows no additional records, which is the reason why our DNS server responded in packet 4 with its list of the root name servers. This brings us back to packet 5, which is a request to the name server at 68.48.0.5 for a very long string ending in “netrogenic.com”, as shown in Figure 14.17.

Figure 14.17. Packet 5: DNS query for netrogenic.com

We don't know what this means by direct inspection, but perhaps the query response in packet 6 (see Figure 14.18) will help us understand packet 5.

Figure 14.18. Packet 6: DNS response for netrogenic.com

If you thought the query for the string ending in “netrogenic.com” was long, check out the response. Highlighted in the bottom window of Figure 14.18, it's 253 bytes long. The entire DNS response is 414 bytes.

If you look back at the entire list of packets in Figure 14.14, a pattern emerges. Rather than show every single packet detail, I've matched up the requests and responses for you here and summarized them.

• Packet 9 is a query for the TXT record 1_0_1197_56633ab0d90f43c68ed1b47358eccfe7. netrogenic.com, matched by a response in packet 12.

• Packet 13 is a query for the TXT record 2_1197_56633ab0d90f43c68ed1b47358eccfe7. netrogenic.com, matched by a response in packet 14.

• Packet 15 is a query for the TXT record 3_1197_56633ab0d90f43c68ed1b47358eccfe7. netrogenic.com, matched by a response in packet 16, and so on.

What about packets 7, 8, 10, and 11?

• Packet 7 is a query for ns.netrogenic.com, matched by a response in packet 10.

• Packet 8 is a query for ons.funpla.net, matched by a response in packet 11.

At this point you may think that this traffic is caused by some sort of covert channel or back door. That's a fair assumption, given that we appear to be seeing a transfer of information in an orderly, incrementing fashion.

If you're dying to visit www.netrogenic.com, now's the time to do so. Although most NSM practitioners believe in not “touching” the source of suspicious or malicious activity, guessing at an associated Web site is a logical investigative step. Luckily, we do find that www.netrogenic.com exists.² After passing through the introductory page, we find a news item, mentioning “a program that shares torrent files through DNS.” That sounds intriguing. Sure enough, the link leads to a program written in Python to distribute BitTorrent .torrent files using DNS. It was first mentioned in a Slashdot.org article on November 30, 2003 (http://slashdot.org/article.pl?sid=03/11/30/1533215).

The .torrent files are used by the BitTorrent system to help users download and share files, like archives or CD-ROM images of operating systems. A .torrent file is not the file to be distributed (i.e., not the .iso file); a .torrent file is small (less than a few megabytes) and contains information to assist in the retrieval of the much larger archive or .iso CD-ROM image. The author of this Python program figured out how to distribute .torrent files by using DNS.

Here's what using the program looks like from the user's point of view. Let's say a user wanted the .torrent file for Red Hat's new Fedora Core Linux distribution, also known as “yarrow.” If he knew the appropriate DNS TXT record to query, he could ask for it. The program doing the work is dnsTorrentToFileTorrent.py (see Figure 14.19).

Figure 14.19. Retrieving the Fedora Core .torrent file

Each of the Downloading statements corresponds to a DNS query. Since these query responses were over 256 bytes, they triggered our Sguil LOCAL Large UDP DNS packet alerts. The end result of the traffic, however, looks like the output shown in Figure 14.20. I used the hd program to display the first 25 lines of the .torrent file, which references a BitTorrent server at torrent.dulug.duke.edu:6969.

Figure 14.20. Yarrow's .torrent file

While it may be annoying for someone to use the DNS system to transfer .torrent files, this traffic is merely suspicious. It doesn't mean an intruder is trying to compromise a victim. For evidence of that scenario, see the examples later in this chapter.

Suspicious Port 53 TCP Traffic

This subsection presents two sorts of traffic to port 53 TCP that have annoyed NSM analysts for years. In the late 1990s and continuing through the early 2000s, analysts saw patterns resembling the following output.

09:22:56.960442 tester.newjersey.net.2100 > name.server.net.53:
  S 2070441966:2070442030(64) win 2048 (ttl 246, id 34960)

09:22:56.960555 tester.newjersey.net.2101 > name.server.net.53:
  S 1884680148:1884680212(64) win 2048 (ttl 246, id 8490)

09:22:56.960669 tester.newjersey.net.2102 > name.server.net.53:
  S 938156752:938156816(64) win 2048 (ttl 246, id 17966)


09:23:26.765186 tester.argentina.net.2100 > name.server.net.53:
  S 1616673589:1616673653(64) win 2048 (ttl 241, id 21017)

09:23:26.765744 tester.argentina.net.2101 > name.server.net.53:
  S 1351385345:1351385409(64) win 2048 (ttl 241, id 9204)

09:23:26.766781 tester.argentina.net.2102 > name.server.net.53:
  S 184647009:184647073(64) win 2048 (ttl 241, id 8397)

09:24:13.867591 tester.brazil.net.2100 > name.server.net.53:
  S 795939539:795939603(64) win 2048 (ttl 241, id 53652)

09:24:13.868783 tester.brazil.net.2101 > name.server.net.53:
  S 2049322111:2049322175(64) win 2048 (ttl 241, id 13883)

09:24:13.873062 tester.brazil.net.2102 > name.server.net.53:
  S 1779866028:1779866092(64) win 2048 (ttl 241, id 14298)

These SYN packets with 64 bytes of null data seemed to be unsolicited. Were they some sort of scan? Plenty of security researchers at the time thought so, with some calling them another form of distributed reconnaissance. After all, they are happening at relatively the same time from multiple, seemingly independent sites. A second pattern, shown next, confused even more analysts.

06:01:15.001304 mayfield.ohio.net.44132 > name1.server.net.53:
  S 10399587:10399587(0) ack 10399586 win 4128 <mss 556>
  (ttl 241, id 0)

06:01:16.999359 mayfield.ohio.net.44132 > name1.server.net.53:
  S 10399587:10399587(0) ack 10399586 win 4128 <mss 556>
  (ttl 241, id 0)

06:01:17.498365 mayfield.ohio.net.44133 > name2.server.net.53:
  S 10399588:10399588(0) ack 10399587 win 4128 <mss 556>
  (ttl 241, id 0)

06:01:14.967214 greenbelt.maryland.net.63604 >
  name1.server.net.53:
  S 34541003:34541003(0) ack 34541002 win 4128 <mss 556>
  (ttl 249, id 0)

06:01:17.461642 greenbelt.maryland.net.63607 >
  name2.server.net.53:
  S 34541006:34541006(0) ack 34541005 win 4128 <mss 556>
  (ttl 249, id 0)


06:01:18.503320 greenbelt.maryland.net.63609 >
  name1.server.net.53:
  S 34541008:34541008(0) ack 34541007 win 4128 <mss 556>

In this case, separate sites sent SYN ACK packets to port 53 TCP. These packets were spooky because they showed odd sequence numbers. For example, the last packet shows an initial response number of 34541008 while it acknowledges 34541007. If this SYN ACK were a true, unforged response to a legitimate SYN packet, that original SYN packet had to have an initial sequence number of 34541006. (Remember that ACK 34541007 means the receiver got a SYN of 34541006 and now expects the next byte of data from the sender to be number 34541007. So, there's a difference of two between the expected initial sequence number and the one shown in the last packet of this output.) Looking at this and the other packets, we wonder why disparate systems would consistently reply with their own sequence numbers supposedly different by two. Incidentally, these patterns did not match the criteria for third-party effects or back scatter, as a denial-of-service victim sending SYN ACK replies would not consistently pick ACK values so close to the supposed offending SYN segment.

After investigating the issue with my Air Force colleague Mark Shaw, we determined both sets of traces were caused by global load balancing systems.³ The SYN segments with 64 bytes of null data were created by F5's 3DNS product, while the SYN ACK segments with odd sequence numbers were the result of Cisco's Distributed Director. Each product used these probes to check latency between themselves and the DNS servers of Web visitors. Figure 14.21 depicts the process, and the following steps explain the traffic further.

1. A Web-browsing client in Chile wants to visit a major e-commerce Web site. She enters the URL in her browser. Her host contacts her local DNS to find the IP address associated with that host name.
2. The local DNS server does not have the IP address in its cache, so it begins querying DNS servers until it reaches the authoritative name server of the domain that owns the IP in question. This system, a load balancing manager (LBM), is either tied to or serves as the DNS for the domain.
3. The LBM checks its cache for any traffic management rules that declare how to handle requests from the client's IP address. At this stage the LBM may immediately return an IP address to the client's local DNS, or it may proceed to step 4.
4. Not finding any cached values and choosing not to deliver a less-than-optimal IP choice to the client, the LBM queries its load balancing systems (LBSs) at its three Web sites in New Jersey, Brazil, and Argentina.
5. The LBSs at the three sites conduct latency testing against the client's local DNS. These may include ICMP or TCP packets for which the round trip time (RTT) is calculated, based on responses from the client's DNS. The site whose tests result in the lowest RTT is deemed “closest” (in Internet space) to the client. The IP of the closest site, in this case, the Argentina site, is returned to the LBM. Remember that the closest IP could belong to a host with a responsive pipe, but very far away in terms of geographical distance.
6. The LBM provides the client's local DNS with the IP of the Argentina Web site.
7. The client's local DNS provides the IP of the Argentina Web site to her host.
8. Her host makes contact with the Web site in Argentina, displaying content.

Figure 14.21. Checking latency for Web visitors

Once the client has visited a Web enterprise that employs load balancing, her local DNS server may be subject to repeated and seemingly aggressive latency testing for extended periods of time. These are not malicious probes; the goal of the system is to provide the quickest response time to the client while efficiently managing activity on the Web server. While some in the security community view this activity as a malicious attempt to map the customer's network, I see it as a realistic attempt to serve the millions of customers who visit the more popular Web sites each day.

Some of these LBSs begin their tests by sending ICMP packets. If ICMP is denied by the client's routers or firewalls, the load balancer then attempts to connect to TCP port 53 on the client's name server. This explains the packets we were investigating. Sometimes a final, more aggressive latency test can be made, where the system essentially scans the client's name server for an open port. It uses any response to test latency.

Malicious Port 53 Traffic

This is the section most of you have been waiting for. After looking at evidence of normal and suspicious events, you're wondering what real attacks look like. Remember that we presented the zone transfer as a case of malicious activity if caused by an unauthorized party. Otherwise, zone transfers are normal aspects of healthy network life. In this section we move beyond the routine and “slightly odd” traffic to investigate verified intrusive activity.

Malicious Port 53 UDP Traffic

Let's look at a new set of LOCAL Large UDP DNS packet alerts (see Figure 14.22). Earlier we wondered if we should scrap that alert due to its susceptibility to triggering on normal DNS traffic.

Figure 14.22. New LOCAL Large UDP DNS packet alerts

Just by looking at the content of the packet data in the lower-right window for this new set of alerts, we see a problem. It appears to be information from a UNIX password file. At this point we turn to Snort's ASCII display mode (snort –dv –C) to see what else might be there.

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

12/23-15:29:20.819980 192.168.60.3:53 -> 192.168.60.5:53
UDP TTL:64 TOS:0x0 ID:1000 IpLen:20 DgmLen:31 DF
Len: 3
id.^@
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

12/23-15:29:20.855573 192.168.60.5:53 -> 192.168.60.3:53
UDP TTL:64 TOS:0x0 ID:1000 IpLen:20 DgmLen:116 DF
Len: 88
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4
(adm),6(disk),10(wheel).^@
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+


12/23-15:30:21.881247 192.168.60.3:53 -> 192.168.60.5:53
UDP TTL:64 TOS:0x0 ID:1000 IpLen:20 DgmLen:44 DF
Len: 16
cat /etc/passwd.^@
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

12/23-15:30:21.931092 192.168.60.5:53 -> 192.168.60.3:53
UDP TTL:64 TOS:0x0 ID:1000 IpLen:20 DgmLen:540 DF
Len: 512
root:x:0:0:root:/root:/bin/bash.bin:x:1:1:bin:/bin:.daemon:x:2:2
:daemon:/sbin:.adm:x:3:4:adm:/var/adm:.lp:x:4:7:lp:/var/spool/lp
d:.sync:x:5:0:sync:/sbin:/bin/sync.shutdown:x:6:0:shutdown:/sbin
:/sbin/shutdown.halt:x:7:0:halt:/sbin:/sbin/halt.mail:x:8:12:mai
l:/var/spool/mail:.news:x:9:13:news:/var/spool/news:.uucp:x:10:1
4:uucp:/var/spool/uucp:.operator:x:11:0:operator:/root:.games:x:
12:100:games:/usr/games:.gopher:x:13:30:gopher:/usr/lib/gopher-d
ata:.ftp:x:14:50:FTP User:/home/ftp:.nobody:x:99:99:Nobody:/:.xf
^@
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

12/23-15:30:21.933179 192.168.60.5:53 -> 192.168.60.3:53
UDP TTL:64 TOS:0x0 ID:1000 IpLen:20 DgmLen:215 DF
Len: 187
s:x:43:43:X Font Server:/etc/X11/fs:/bin/false.named:x:25:25:Nam
ed:/var/named:/bin/false.postgres:x:26:26:PostgreSQL Server:/var
/lib/pgsql:/bin/bash.lee:x:500:500:lee:/home/lee:/bin/bash.^@
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

It appears that someone using a client on 192.168.60.3 is issuing commands to a server on 192.168.60.5. The first command, id, displayed the user and group names and numeric IDs of the calling process. In this case, id showed that the user has root access on the server. The second command, cat /etc/passwd, caused the contents of the server's password file to be displayed on the screen.

The program that caused this traffic is called Tunnelshell, written by Fryxar.⁴ The program runs on Linux and can be used to create covert channels over fragmented traffic, ICMP, raw IP, or arbitrary TCP or UDP ports. An intruder who has already compromised a victim could install Tunnelshell during the consolidation exploit phase to facilitate future remote access. Tunnelshell's ability to run on raw IP could allow it to create tunnels over IP protocol 11, similar to the case demonstrated in Chapter 11.

To use Tunnelshell, the intruder first installs the tool on the victim. In the following example case the victim is 192.168.60.5, a Red Hat Linux 6.2 server named oates. He starts Tunnelshell by telling the server tunneld what protocol to use and what ports to transmit and receive traffic. When operating in UDP mode, Tunnelshell does not bind to a port. This means it will not appear in netstat output. It will appear in a process listing, unless the intruder hides it with a loadable kernel module like Knark or alters the ps executable.⁵ In Figures 14.23 and 14.24, I show how to start Tunnelshell, I demonstrate how it appears in a process list, and I show that it does not appear in netstat output as listening on port 53 UDP. (Only an unrelated socket containing the number “53” is seen.) I conclude with lsof output for process 9024, which is the process ID for the Tunnelshell server, tunneld.

Figure 14.23. Tunnelshell server tunneld on victim oates (192.168.60.5)

Figure 14.24. Tunnelshell client tunnel connects to victim oates

The client is a system named juneau with IP address 192.168.60.3. On juneau the intruder uses the tunnel client to connect to the victim oates. He then issues the cat /etc/passwd command (see Figure 14.24). Remember the id command and response shown earlier? Those were issued automatically by the Tunnelshell system, and the results were not displayed to the intruder. The id command served as a check to ensure that the Tunnelshell covert channel was operational.

Before concluding our discussion of this example of truly malicious traffic, let's examine how it appears in Ethereal (see Figure 14.25).

Figure 14.25. Tunnelshell traffic displayed in Ethereal

The packets displayed earlier in Snort are shown here as interpreted by Ethereal. Clearly they do not appear to be anything like normal DNS traffic. In fact, Ethereal consistently reports them as being “malformed” or as requesting “unknown operations.” While the passing of .torrent files via DNS appeared odd, those packets still conformed to DNS standards. Here, Tunnelshell is merely using port 53 UDP as a covert channel. If the intruder's commands and the victim's replies were encapsulated and obfuscated in DNS packets that appeared legitimate, it would be more difficult to identify them.

Remember that these packets were flagged by a Snort rule that checked for UDP application data exceeding 256 bytes. First the rule fired on normal traffic for queries about Yahoo because our DNS servers replied with eight possible IPs for www.yahoo.akadns.net. Next the rule fired because someone transferred .torrent files via an innovative method using the DNS system. Here the rule found evidence of an actual intrusion, although only because the packet payload exceeded 256 bytes. A quick look at the tunneld usage statement shows how an intruder could have evaded this signature.

usage: ./tunneld [options]

options:
  -h                        Display this screen
  -d <delay>                Set delay in sending packets
  -o <protocol>             Set protocol to use with type frag/ip
                            tunnels (tcp|udp|...)
  -p <cli_port,srv_port>    Set ports to use with type tcp/udp
                            tunnels (default: 80,2000)
  -m <cli_req,srv_req>      Set packet to use with icmp tunnels
                            (default: echo,reply)
  -i <id>                   Set session id (range: 0-65535)
  -t <type>                 Set tunnel type (frag|ip|tcp|udp|icmp)
  -s <packetsize>           Set data size
  -a                        Using ppp interface

By setting the data size via the –s switch to a conservative amount, say, 200 bytes, an intruder would easily bypass our LOCAL Snort rule. That is why NSM practitioners cannot rely on a single source of data for their intrusion detection efforts. A purely alert-based system could miss this traffic. Only packet 3 out of the first four packets of the original Tunnelshell session triggered our custom Snort rule.

We can't conclude our discussion of malicious port 53 UDP traffic without mentioning queries for the version of BIND running on name servers. BIND often reports its version when asked a certain query, as shown here.

-bash-2.05b$ dig @moog.taosecurity.com. version.bind txt chaos

; <<>> DiG 8.3 <<>> @moog.taosecurity.com. version.bind txt chaos
; (1 server found)
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45102
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0,
   ADDITIONAL: 0
;; QUERY SECTION:
;;      version.bind, type = TXT, class = CHAOS

;; ANSWER SECTION:
VERSION.BIND.           0S CHAOS TXT    "8.3.6-REL-p1"

;; Total query time: 1 msec
;; FROM: janney.taosecurity.com to SERVER: 172.27.20.1
;; WHEN: Fri Jan  2 17:05:06 2004
;; MSG SIZE  sent: 30  rcvd: 67

Here we see a person conducting a form of application reconnaissance against the name server running on moog.taosecurity.com. By specifying a query of class chaos and asking for the text record associated with version.bind, the intruder learned that the DNS server runs BIND version 8.3.6-REL-p1. The exchange took place over port 53 UDP.

17:04:03.778304 172.27.20.5.3848 > 172.27.20.1.53:
  34252+ AAAA? moog.taosecurity.com. (38)

17:04:03.778553 172.27.20.1.53 > 172.27.20.5.3848:
  34252* 0/1/0 (79)

17:04:03.778802 172.27.20.5.3849 > 172.27.20.1.53:
  34253+ A? moog.taosecurity.com. (38)


17:04:03.779052 172.27.20.1.53 > 172.27.20.5.3849:
  34253* 1/1/1 A 172.27.20.1 (84)

17:04:03.780053 172.27.20.5.3850 > 172.27.20.1.53:
  45102+ TXT CHAOS? version.bind. (30)

17:04:03.780176 172.27.20.1.53 > 172.27.20.5.3850:
  45102* 1/0/0 CHAOS TXT 8.3.6-REL-p1 (67)

Although I seem to remember various institutions querying for the version of BIND to support their research, most BIND version requests are malicious. One exception is a DNS-based load balancer that uses these queries as an alternative to sending ICMP and TCP packets to prompt replies and thereby measure latency.

Malicious Port 53 TCP and UDP Traffic

We can't conclude a discussion of port 53 traffic without looking at an exploit against BIND. In early 2001 CERT released an advisory for various versions of BIND that warned of a buffer overflow in BIND's transaction signature (TSIG) handling code.⁶ Last Stage of Delirium (LSD) coded an exploit for this vulnerability that combined the TSIG vulnerability with an earlier “infoleak” vulnerability. This information leakage flaw allowed intruders asking inverse queries to read environment variables from remote vulnerable systems.⁷ A DNS inverse query is the request for the domain name associated with a specific IP address.

I tested this exploit against a supposedly vulnerable BIND implementation running on Red Hat 6.2. I learned by reading a different exploit (not written by LSD) that default Red Hat 6.2 installations were not vulnerable, due to compiler options used to create the package shipped with the operating system.⁸ Nevertheless, the traffic caused by launching the LSD exploit is sufficient to demonstrate exploit traffic on port 53 UDP. Figure 14.26 shows the LSD code in action. The traffic for the exploit appears in Ethereal as shown in Figure 14.27.

Figure 14.26. Running the BIND TSIG/“infoleak” exploit against Red Hat 6.2

Figure 14.27. Inverse query for “infoleak” exploit

The exploit first sets up a TCP three-way handshake to port 53 TCP on the victim, shown in packets 1, 2, and 3 in Figure 14.27. In packet 4 the exploit makes a malformed DNS inverse query. Ethereal reports it as malformed because the query doesn't actually ask a proper question! The highlighted packet includes what Ethereal interprets as an answer record, which doesn't belong at all. One Snort rule uses the fields set in bold in the following output to identify this as an attack. (Notice that the Snort content in bold matches the highlighted data in the bottom window of Figure 14.27.)

alert tcp $EXTERNAL_NET any -> $HOME_NET 53
  (msg:"DNS EXPLOIT named tsig overflow attempt";
  flow:to_server,established; content:"|AB CD 09
  80 00 00 00 01 00 00 00 00 00 00 01 00 01 20
  20 20 20 02 61|"; reference:cve,CVE-2001-0010;
  reference:bugtraq,2302;
  reference:arachnids,482;
  classtype:attempted-admin;
  sid:303; rev:8;)

It's interesting to note that this Snort rule reports a named tsig overflow attempt, when really an information leak exploit is being attempted. While the LSD exploit combines the two, seeing this alert trigger indicates attempted exploitation of an “infoleak” and not a TSIG vulnerability.

For comparison's sake, Figure 14.28 shows a normal inverse query for 172.27.20.5.

Figure 14.28. Normal inverse query for 172.27.20.5

After the normal query we see the response come back in packet 2. That response, shown in Figure 14.29, is much tamer than what the exploit code solicited. The DNS server replies that 172.27.20.5 resolves to janney.taosecurity.com. Note that the entire length of this frame is 155 bytes.

Figure 14.29. Normal inverse query response for 172.27.20.5

In the earlier exploit example, however, the victim DNS server responded with information of potential use to the intruder. Figure 14.30 displays what the intruder saw when running the LSD exploit.

Figure 14.30. LSD exploit in action

Where did that data come from? It was sent in the response to the malformed DNS inverse query. We see this in Figure 14.31, which shows a frame of 674 bytes in length—far greater than the 155 bytes of the normal DNS inverse query response for 172.27.20.5 (janney.taosecurity.com) in Figure 14.29.

Figure 14.31. Response to malicious inverse query

In Figure 14.30, where the LSD exploit is shown from the attacer's perspective, you can see the beginning of the stack dump start with 0xd639 near the end of the first line of data in the packet content window. Because the victim was not vulnerable to the attack, the LSD exploit executed a graceful TCP close and concluded the TCP session with an exchange of FIN ACK packets. Remember that UDP has nothing to do with TCP, so the closing of the TCP session was an action taken as a result of the LSD exploit discovering the victim was not vulnerable. The exploit essentially worked in three stages.

1. Establish a session to port 53 TCP on the victim.
2. Use “infoleak” to check the victim's vulnerability.
3. Execute the TSIG exploit over port 53 TCP, or close the session if the victim is not vulnerable.

In our example, the session closed. Snort would still have reported this activity as a named tsig overflow attempt, even though that part of the exploit never happened.

Conclusion

This chapter explored the many faces of traffic using ports 53 UDP and TCP. Since almost every Internet site relies on DNS, it's important to understand the protocol's normal, suspicious, and malicious aspects. We started the chapter with a look at normal traffic passed on port 53 UDP and TCP. The normal traffic was run-of-the-mill UDP-based DNS, albeit with larger payloads than we expected. Using the dnsquery command, we forced DNS over port 53 TCP. Zone transfers were shown as a form of TCP-based DNS that is normal if done between master and slave DNS servers but malicious if done by an unauthorized party. We explored how global LBSs use odd TCP segments to measure latency by prompting responses from DNS servers. The suspicious UDP traffic was caused by a creative way to distribute BitTorrent files over DNS. The malicious traffic was the result of an intruder running Tunnelshell. The covert channel was easy to spot because it looked nothing like normal DNS traffic on port 53 UDP. We concluded with a look at a query for the version of BIND running on a target, and an examination of an unsuccessful attempt to exploit BIND.

This chapter relied heavily on alert and full content data to detect and validate events. In the next chapter, we turn to session data to understand an intrusion scenario.