Preface
Welcome to The Tao of Network Security Monitoring: Beyond Intrusion Detection. The goal of this book is to help you better prepare your enterprise for the intrusions it will suffer. Notice the term “will.” Once you accept that your organization will be compromised, you begin to look at your situation differently. If you've actually worked through an intrusion—a real compromise, not a simple Web page defacement—you'll realize the security principles and systems outlined here are both necessary and relevant.
This book is about preparation for compromise, but it's not a book about preventing compromise. Three words sum up my attitude toward stopping intruders: prevention eventually fails. Every single network can be compromised, either by an external attacker or by a rogue insider. Intruders exploit flawed software, misconfigured applications, and exposed services. For every corporate defender, there are thousands of attackers, enumerating millions of potential targets. While you might be able to prevent some intrusions by applying patches, managing configurations, and controlling access, you can't prevail forever. Believing only in prevention is like thinking you'll never experience an automobile accident. Of course you should drive defensively, but it makes sense to buy insurance and know how to deal with the consequences of a collision.
Once your security is breached, everyone will ask the same question: now what? Answering this question has cost companies hundreds of thousands of dollars in incident response and computer forensics fees. I hope this book will reduce the investigative workload of your computer security incident response team (CSIRT) by posturing your organization for incident response success. If you deploy the monitoring infrastructure advocated here, your CSIRT will be better equipped to scope the extent of an intrusion, assess its impact, and propose efficient, effective remediation steps. The intruder will spend less time stealing your secrets, damaging your reputation, and abusing your resources. If you're fortunate and collect the right information in a forensically sound manner, you might provide the evidence needed to put an intruder in jail.
Audience
This book is for security professionals of all skill levels and inclinations. The primary audience includes network security architects looking for ways to improve their understanding of their network security posture. My goal is to provide tools and techniques to increase visibility and comprehension of network traffic. If you feel let down by your network-based intrusion detection system (NIDS), this book is definitely for you. I explain why most NIDS deployments fail and how you can augment existing NIDS with open source tools.
Because this book focuses on open source tools, it is more likely to be accepted in smaller, less bureaucratic organizations that don't mandate the use of commercial software. Furthermore, large organizations with immense bandwidth usage might find some open source tools aren't built to handle outrageous traffic loads. I'm not convinced the majority of Internet-enabled organizations are using connections larger than T-3 lines, however.1 While every tool and technique hasn't been stress-tested on high-bandwidth links, I'm confident the material in this book applies to a great majority of users and networks.
If you're a network security analyst, this book is also for you. I wrote this book as an analyst, for other analysts. This means I concentrate on interpreting traffic, not explaining how to install and configure every single tool from source code. For example, many books on “intrusion detection” describe the Transmission Control Protocol/Internet Protocol (TCP/IP) suite and how to set up the Snort open source IDS engine with the Analysis Console for Intrusion Databases (ACID) interface. These books seldom go further because they soon encounter inherent investigative limitations that restrict the usefulness of their tools. Since my analytical techniques do not rely on a single product, I can take network-based analysis to the next level. I also limit discussion of odd packet header features, since real intrusions do not hinge on the presence of a weird TCP flag being set. The tools and techniques in this book concentrate on giving analysts the information they need to assess intrusions and make decisions, not just identify mildly entertaining reconnaissance patterns.
This book strives to not repeat material found elsewhere. You will not read how to install Snort or run Nmap. I suggest you refer to the recommended reading list in the next section if you hunger for that knowledge. I introduce tools and techniques overlooked by most authors, like the material on protocol anomaly detection by Brian Hernacki, and explain how you can use them to your advantage.
Technical managers will appreciate sections on best practices, training, and personnel issues. All the technology in the world is worthless if the staff manning it doesn't understand their roles, responsibilities, and escalation procedures. Managers will also develop an intuition for the sorts of information a monitoring process or product should provide. Many vendors sell services and products named with combinations of the terms “network,” “security,” and “monitoring.” This book creates a specific definition for network security monitoring (NSM), built on a historical and operational foundation.
Prerequisites
I've tried to avoid duplicating material presented elsewhere, so I hope readers lacking prerequisite knowledge take to heart the following reading suggestions. I highly recommend reading the following three books prior to this one. If you've got the necessary background, consider these titles as references.
• Internet Site Security, by Erik Schetina, Ken Green, and Jacob Carlson (Boston, MA: Addison-Wesley, 2002). This is an excellent “security 101” book. If you need to start from the ground floor, this book is a great beginning.
• Counter Hack: A Step-by-Step Guide to Computer Attacks and Effective Defenses, by Ed Skoudis (Upper Saddle River, NJ: Prentice Hall PTR, 2001). Counter Hack offers the best single-chapter introductions to TCP/IP, Microsoft Windows, UNIX, and security issues available.
• Hacking Exposed: Network Security Secrets and Solutions, 4th ed., by Stuart McClure, Joel Scambray, and George Kurtz (New York: McGraw-Hill, 2003). Hacking Exposed explores the capabilities and intentions of digital threats. By knowing how to compromise computers, you'll understand the sorts of attacks network security monitoring practitioners will encounter.
If you need an introduction to intrusion detection theory, I recommend the following book:
• Intrusion Detection, by Rebecca Gurley Bace (Indianapolis, IN: New Riders, 2000). While not strictly needed to understand the concepts in this book, Intrusion Detection provides the history and mental lineage of IDS technology. As The Tao of Network Security Monitoring focuses on network-based tactics, you can turn to Intrusion Detection for insight on host-based detection or the merits of signature- or anomaly-based IDS.
It helps to have a good understanding of TCP/IP beyond that presented in the aforementioned titles. The following are a few of my favorite books on TCP/IP.
• Internet Core Protocols: The Definitive Guide, by Eric A. Hall (Cambridge, MA: O'Reilly, 2000). Many people consider Richard Stevens' TCP/IP Illustrated Volume 1: The Protocols (Reading, MA: Addison-Wesley, 1994) to be the best explanation of TCP/IP. I think Eric Hall's more recent book is better suited for modern network traffic analysts.
• Network Analysis and Troubleshooting, by J. Scott Haugdahl (Boston, MA: Addison-Wesley, 2000). Troubleshooting books tend to offer the more interesting explanations of protocols in action. Scott Haugdahl works his way up the seven layers of the Open Systems Interconnect (OSI) model, using packet traces and case studies.
• Troubleshooting Campus Networks: Practical Analysis of Cisco and LAN Protocols, by Priscilla Oppenheimer and Joseph Bardwell (Indianapolis, IN: Wiley, 2002). This title is considerably broader in scope than Scott Haugdahl's work, with coverage of virtual local area networks (VLANs), routing protocols, and wide area network (WAN) protocols like Asynchronous Transfer Mode (ATM).
One other book deserves mention, but I request you forgive a small amount of self-promotion. The Tao of Network Security Monitoring is primarily about detecting incidents through network-based means. In some senses it is also an incident response book. Effective incident response, however, reaches far beyond network-based evidence. To learn more about host-based data, such as file systems and memory dumps, I recommend Real Digital Forensics (Boston, MA: Addison-Wesley, 2005). I wrote the network monitoring sections of the book, and coauthors Keith Jones and Curtis Rose did the host- and memory-level forensics. If you'd like to see the big picture for incident response, read Real Digital Forensics.
A Note on Operating Systems
All of the tools I discuss in this book run on the FreeBSD (http://www.freebsd.org) operating system. FreeBSD is a UNIX-like, open source environment well suited for building network security monitoring platforms.2 If you're familiar with Linux or any other Berkeley Software Distribution (OpenBSD or NetBSD), you'll have no trouble with FreeBSD. I strongly recommend running NSM tools on UNIX-like platforms like the BSDs and Linux.
You might consider trying a live CD-ROM FreeBSD distribution prior to committing a hard drive to installation. You may already know about Knoppix (http://www.knopper.net/knoppix/index-en.html), the most famous Linux-based live CD-ROM operating system. FreeBSD offers the FreeSBIE distribution (http://www.freesbie.org). FreeSBIE recently shipped version 1.0, based on the FreeBSD 5.2.1 RELEASE edition.
Live distributions boot from the CD-ROM and run all programs within memory. They can be configured to write to removable media like USB thumb drives or the hard drive of the host computer. Live distributions are a good way to test hardware compatibility before going through the time and effort to install a new operating system on a system's hard drive. For example, before upgrading a FreeBSD 4.9–based system to version 5.2.1, I booted a FreeBSD 5.2.1–based live distribution and checked whether it saw all of the hardware properly.
Figure 1 shows FreeSBIE 1.0 running several programs. Many security tools are included in the distribution, including Nessus, Nmap and NmapFE, Snort, and Ethereal. I am investigating building an NSM-minded FreeBSD-based live distribution to run the tools discussed in this book.
Figure 1. FreeSBIE 1.0 running Ethereal, NmapFE, Snort 2.1.0, and The Gimp
If you want to learn about FreeBSD, I suggest these books.
• FreeBSD: An Open-Source Operating System for Your Personal Computer, 2nd ed., by Annelise Anderson (Portola Valley, CA: Bit Tree Press, 2001). Absolute UNIX newbies will find Annelise Anderson's book the gentlest introduction to FreeBSD.
• Absolute BSD: The Ultimate Guide to FreeBSD, by Michael Lucas (San Francisco, CA: No Starch Press, 2002). Michael Lucas has an uncanny ability to answer the questions his readers are bound to ask. Keep in mind that Annelise Anderson's book and Absolute BSD focus on FreeBSD 4.x, so certain details might change with FreeBSD 5.x.
• The Complete Guide to FreeBSD, 4th ed., by Greg Lehey (Cambridge, MA: O'Reilly, 2003). Greg Lehey covers more than just FreeBSD; he addresses system and network administration issues as well. This is the first book explicitly written with FreeBSD 5.x in mind.
I'm often asked why I use FreeBSD and not OpenBSD. I use FreeBSD because I believe it is the best general-purpose operating system available. It has more applications in its ports tree, a larger development community, and better network and multiprocessor performance. I develop and test all of my applications and techniques on FreeBSD.
OpenBSD is more innovative in terms of security, with integrated defensive features like Systrace, the Pf firewall, increased use of privilege separation, and relentless removal of coding flaws. I believe OpenBSD may be a superior platform for building dedicated “security appliances.” Once the application is tested under a general-purpose operating system like FreeBSD, it can be deployed on a security-minded platform like OpenBSD.
As the TrustedBSD project (http://www.trustedbsd.org) brings additional security features into the FreeBSD 5.x tree, FreeBSD's security features are competing well with OpenBSD. FreeBSD is beginning to adopt security systems like mandatory access control that are found in commercial operating systems like Trusted Solaris. In reality all three major BSD projects feed security ideas into each other, so competition among the projects is not a huge concern.
Linux and Windows users might wonder where I stand on their operating systems. I believe Linux benefits from having a very large development community. Because so many coders run Linux, users are more likely to see patches introduced to improve Tcpdump's performance or implement other features useful to security professionals. I still prefer the BSDs to Linux because Linux is a kernel supplemented by tools selected by various distribution aggregators.3 There is also doubt about which Linux distribution is most likely to be used by the community. Prior to the arrival of Fedora Core, Red Hat Linux was more or less the de facto standard. Debian may be the heir to Red Hat's throne, but that situation remains in flux. This is not the best environment for developing security applications and standards.
Windows is an operating system for consumers. It was designed to “make life easy” at the expense of security and operational transparency. The underlying Windows design model has not withstood connectivity to the Internet very well. The operating system provides far too many services on single ports. How can one disable port 135 or 139 TCP, for example, without breaking a dozen built-in applications?
I believe the supposed ease of use of a Windows system, even if one accepted this feature to be true, is far outweighed by the risk of introducing the operating system in a critical security role. Those adding a security platform to a network should not violate the first rule of the Hippocratic Oath: do no harm. I have far more confidence in the reliability and resiliency of a FreeBSD or other UNIX system compared to a Windows system.
Scope
The book is broken into five major parts, followed by an epilogue and appendices. You can focus on the areas that interest you, as the sections were written in a modular manner. You may wonder why greater attention is not paid to popular tools like Nmap or Snort. With The Tao of Network Security Monitoring, I hope to break new ground by highlighting ideas and tools seldom seen elsewhere. If I don't address a widely popular product, it's because it has received plenty of coverage in another book.
Part I offers an introduction to NSM, an operational framework for the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions. Part I begins with an analysis of the terms and theory held by NSM practitioners. Chapter 1 discusses the security process and defines words like security, risk, and threat. It also makes assumptions about intruders and their prey that set the stage for NSM operations. Chapter 2 addresses NSM directly, explaining why NSM is not implemented by modern NIDSs alone. Chapter 3 focuses on deployment considerations, such as how to access traffic using hubs, taps, SPAN ports, and inline devices.
Part II begins an exploration of the NSM “product, process, and people” triad. Chapter 4 is a case study called the “reference intrusion model.” This is an incident explained from the point of view of an omniscient observer. During this intrusion, the victim collected full content data in two locations. We will use those two trace files while explaining the tools discussed in Part II. Following the reference intrusion model, I devote chapters to each of the four types of data that must be collected to perform NSM—full content, session, statistical, and alert data. Chapters 5 through 10 describe open source tools tested on the FreeBSD operating system and available on other UNIX derivatives. Part II also includes a look at tools to manipulate and modify traffic. Featured in Part II are little-discussed NIDSs like Bro and Prelude, and the first true open source NSM suite, Sguil.
Part III continues the NSM triad by discussing processes. If analysts don't know how to handle events, they're likely to ignore them. I provide best practices in Chapter 11 and follow with Chapter 12, written explicitly for technical managers. That material explains how to conduct emergency NSM in an incident response scenario, how to evaluate monitoring vendors, and how to deploy an NSM architecture.
Part IV, intended for analysts and their supervisors, completes the NSM triad. Entry-level and intermediate analysts frequently wonder how to move to the next level of their profession. In Chapter 13, I offer some guidance for the five topics with which a security professional should be proficient: weapons and tactics, telecommunications, system administration, scripting and programming, and management and policy. Chapters 14 through 16 offer case studies, showing analysts how to apply NSM principles to intrusions and related scenarios.
Part V is the offensive counterpart to the defensive aspects of Parts II, III, and IV. I discuss how to attack products, processes, and people. Chapter 17 examines tools to generate arbitrary packets, manipulate traffic, conduct reconnaissance, and exploit flaws in Cisco, Solaris, and Microsoft targets. In Chapter 18 I rely on my experience performing detection and response to show how intruders attack the mind-set and procedures on which analysts rely.
An epilogue on the future of NSM follows Part V. The appendices feature several TCP/IP protocol header charts and explanations. I also wrote an intellectual history of network security, with excerpts and commentary on the most important papers written during the last 25 years. Please take the time to at least skim that appendix; you'll see that many of the “revolutionary ideas” often heralded in the press were in some cases proposed decades ago.
Neither Part V nor other parts are designed as “hacking” references. You will not find “elite” tools to compromise servers; if so inclined, refer to the suggested reading list. The tools I profile were selected for the traffic they generate. By looking at packets created by readily available offensive tools, analysts learn to identify normal, suspicious, and malicious traffic.
Welcome Aboard
I hope you find this book useful and enjoyable. I welcome feedback on its contents, especially tips on better uses of tools and tactics. While doing research I was amazed at the amount of work done in the field of intrusion detection over the last 25 years. Intrusion detection is only one component of NSM, but it is the general community in which NSM practitioners feel most at home.
Much of what I present is the result of standing on the shoulders of giants.4 Our community is blessed by many dedicated and talented people who contribute code, ideas, and resources to Internet security issues. I hope my contribution is worthy of the time you dedicate to reading it.
Acknowledgments
I would first like to thank my wife Amy for her encouragement and understanding. Many nights I wrote until two or three o'clock in the morning. I appreciate the space and time she gave me to complete this book, as well as the unconditional love and support she has shown as my wife. Our dog Scout was also helpful, reminding me to stop writing every once in a while to play fetch with him.
I thank my parents and sisters for providing a nurturing childhood home and encouraging a desire to learn.
I owe a lot to the NSM gurus I met as a captain in the Air Force. These include Bamm Visscher, author of Sguil and the person who's been a great mentor and friend for the last five years. I enjoyed working with some real security professionals in the Air Force Computer Emergency Response Team (AFCERT) where I started my NSM journey: Sam Adams, Dave Bibighaus, Dustin Childs, Steve Chism, LeRoy Crooks, John Curry, DeWayne Duff, Ryan Gurr, Steve Heacox, Bill Kelly, Zeb King, Jason Mathews, Bruce McGilvery, Don Nelson, Will Patrick, Greg Patton, Chuck Port, Jason Potopa, Chad Renfro, Chris Rochester, Billy Rodriguez, Christi Ruiz, Marty Schlachter, Jay Schwitzgebel, Mark Shaw, Larry Shrader, Byron Thatcher, Ralph Toland, and Rich Zanni. I appreciate Cheryl Knecht's patience when I caught my first reconnaissance activity from Russia. I'd also like to recognize my former supervisors in the Air Intelligence Agency's plans division, Jesse Coultrap and J.J. Romano, who acted when they realized I would be happier in the AFCERT.
At Ball Aerospace & Technologies Corporation, Bamm, Dave Wheeler, and I built an NSM operation from scratch. When writing this book I kept in mind the needs of our first analysts, who in many ways were guinea pigs for the “new NSM” built on the ruins of the “good ol' days” of AFCERT NSM operations. I know some of them are watching your networks right now.
Working at Foundstone gave me the chance to work on the incident response side of the NSM experience. I learned from my former boss Kevin Mandia that “we win some, and we lose some.” Forensic gurus Keith Jones and Matt Pepe showed how to replace people with very small scripts, usually named “parser.” Julie Darmstadt was there to see me “pit out” in front of dozens of students and was ready to carry a class forward when we risked another “debacle.”
The Addison-Wesley team helped make this book a reality. Jessica Goldstein guided me through the writing process with skill and tact. Chrysta Meadowbrooke copyedited the text with incredible attention to detail. Heather Mullane, Chanda Leary-Coutu, and Joan Murray helped bring news of my work to readers worldwide. Talented reviewers, including Luca Deri, Ron Gula, Aaron Higbee, Kirby Kuehl, Paul Myrick, and Marcus Ranum, kept me on track. I appreciate the contributions to Chapter 9 by Bro expert Christopher Manders, Prelude-IDS founder Yoanne Vandoorselaere, and IT solution provider Dreamlab. Brian Hernacki wrote the great appendix on protocol anomaly detection. Amy Fisher of Net Optics gave expert advice on Chapter 3.
I've learned quite a bit while reviewing books for Amazon.com. I appreciate the review copies sent by Joan Murray at Pearson Education, Bettina Faltermeier at McGraw-Hill/Osborne, Amy Pedersen at Syngress, Eric Holmgren at Wiley, and my friends at O'Reilly. I was tempted to cover much more ground than what appears here, but I defer to subjects better covered by other authors like Ross Anderson and Ed Skoudis.
I would also like to thank the members of the FreeBSD community who devote themselves to the world's most capable operating system. Articles and books by Dru Lavigne, Greg Lehey, and Michael Lucas have been extremely helpful. I encourage anyone looking for a coherent, consistent, stable, feature-rich operating system to consider FreeBSD. I hope those of us who benefit from open source projects support them by purchasing distributions from vendors like FreeBSDMall.com and BSDMall.com.
In addition to the FreeBSD community, I tip my hat to all of the developers of the open source software profiled in this book. Open source software is proving to be the wave of the past and the future. I have yet to find a software requirement not met by open source software. The next time you need an application, search an archive like SourceForge.net. If you don't find what you need, consider hiring a developer to write the code and then release it to the world under a license approved by the Open Source Initiative (http://www.opensource.org).