Preface
Abstract: This chapter provides an introduction to the Applied Network Security Monitoring book. It describes the purpose of the book, the intended audience, and the suggested prerequisites. It also describes the content of every chapter included in the book, as well as the location of the book’s companion website and methods for contacting the authors.
Keywords: Network Security Monitoring, Collection, Detection, Analysis, Charity
I love catching bad guys. Ever since I was a little kid, I wanted to catch bad guys in some fashion or another. Whether it was adorning a cape made from the nearest towel I could find or running around the house playing cops and robbers with my childhood friends, I lived for the thrill of serving justice to wrongdoers of all sorts. As hard as I tried however, I was never quite able to channel my rage into an ability that would allow me to grow into a giant green monster, and no matter how many spider bites I received I never developed the ability to shoot web from my wrists. I also realized pretty quickly that I wasn’t quite cut out for law enforcement.
Once these realities set in and I realized that I was nowhere near rich enough to build a bunch of fancy gadgets and fly around at night in a bat suit, I ended up turning my attention to computers. Several years later, I’ve ended up in a position where I am able to live my childhood dreams of catching bad guys, but not in the sense that I had originally imagined.
I catch bad guys through the practice of network security monitoring (NSM). That’s what this book is about. NSM is based upon the concept that prevention eventually fails. This means that no matter how much time you invest in securing your network, eventually the bad guys win. When this happens, you must be organizationally and technologically positioned to be able to detect and respond to the intruder’s presence so that an incident may be declared and the intruder can be eradicated with minimal damage done.
The path to knowledge for the practice of NSM typically begins with that question. It’s because of that question that I refer to NSM as a practice, and someone who is a paid professional in this field as a practitioner of NSM.
Scientists are often referred to as practitioners because of the evolving state of the science. As recently as the 1980s, medical science believed that milk was a valid treatment for ulcers. As time progressed, scientists found that ulcers were caused by bacteria called Helicobacter pylori and that dairy products could actually further aggravate an ulcer.1 Perceived facts change because although we would like to believe most sciences are exact, they simply aren’t. All scientific knowledge is based upon educated guesses utilizing the best available data at the time. As more data becomes available over time, answers to old questions change, and this redefines things that were once considered facts. This is true for doctors as practitioners of medical science, and it is true for us as practitioners of NSM.
Unfortunately, when I started practicing NSM there weren’t a lot of reference materials available on the topic. Quite honestly, there still aren’t. Aside from the occasionally bloggings of industry pioneers and a few select books, most individuals seeking to learn more about this field are left to their own devices. I feel that it is pertinent to clear up one important misconception to eliminate potential confusion regarding my previous statement. There are menageries of books available on the topics TCP/IP, packet analysis, and various intrusion detection systems (IDSs). Although the concepts presented in those texts are important facets of NSM, they don’t constitute the practice of NSM as a whole. That would be like saying a book about wrenches teaches you how to diagnose a car that won’t start.
This book is dedicated to the practice of NSM. This means that rather than simply providing an overview of the tools or individuals components of NSM, I will speak to the process of NSM and how those tools and individual components support the practice.
Audience
Ultimately, this book is intended to be a guide on how to become a practicing NSM analyst. My day-to-day job responsibility includes the training of new analysts, so this book is not only to provide an education text for the masses, but also to provide a book that can serve as the supportive text of that training process. That being the case, my intent is that someone can read this book from cover to cover and have an introductory level grasp on the core concepts that make a good NSM analyst.
If you are already a practicing analyst, then my hope is that this book will provide a foundation that will allow you to grow your analytic technique to make you much more effective at the job you are already doing. I’ve worked with several good analysts who were able to become great analysts because they were able to enhance their effectiveness with some of the techniques and information I’ll present here.
The effective practice of NSM requires a certain level of adeptness with a variety of tools. As such, the book will discuss several of these tools, but only from the standpoint of the analyst. When I discuss the Snort IDS, the SiLK analysis tool set, or other tools, those tasked with the installation and maintenance of those tools will find that I don’t speak too thoroughly to those processes. When the time arises, I will reference other resources that will fill the gaps there.
Additionally, this book focuses entirely on free and open source tools. This is not only in effort to appeal to a larger group of individuals who may not have the budget to purchase commercial analytic tools such as NetWitness or Arcsight, but also to show the intrinsic benefit of using open source, analyst designed tools that provide more transparency in how they interact with data.
Prerequisites
The most successful NSM analysts are usually those who have experience in other areas of information technology prior to starting security-related work. This is because they will have often picked up other skills that are important to an analyst, such as an understanding of systems or network administration. In lieu of this experience, I’ve created a brief listing of books I really enjoy that I think provide insight into some important skills useful to the analyst. I’ve tried my best to write this book so that a significant amount of perquisite knowledge isn’t required, but if you have the means, I highly recommend reading some of these books in order to supplement information provided in Applied Network Security Monitoring.
TCP/IP Illustrated, Vol 1, Second Edition: The Protocols by Kevin Fall and Dr. Richard Stevens (Addison Wesley 2011)
A core understanding of TCP/IP is one of the more crucial skills required to practice NSM effectively. The classic text by the late Dr. Richard Stevens has been updated by Kevin Fall to include the latest protocols, standards, best practices, IPv6, security primers by protocol, and more.
Richard Bejtlich helped to define a lot of the concepts that underlie the practice of NSM. As a matter of fact, I will reference his book and blog quite often throughout Applied NSM. Although Richard’s book is nearly ten years old, a lot of the material in it continues to make it a relevant text in the scope of NSM.
I’m not above shameless self-promotion. Whereas Dr. Stevens book provides a thorough in-depth reference for TCP/IP, PPA discusses packet analysis at a practical level using Wireshark as a tool of choice. We will examine packets in this book, but if you’ve never looked at packets before then I recommend this as a primer.
I’ve always thought this book was one of the absolute best general security books available. It covers a bit of everything, and I recommend it to everyone regardless of their level of experience. If you’ve never done security-related work, then I’d say Counter Hack Reloaded is a must read.
Concepts and Approach
Applied NSM is broken down into three primary sections: Collection, Detection, and Analysis. I will devote individual chapters to the discussion of tools, techniques, and procedures related to these core areas. I’m a simple country boy from Kentucky, so I try my best to write in a simple tone without a lot of added fluff. I also try to take typically advanced concepts and break them down into a series of repeatable steps whenever possible. As with any book that addresses generalized concepts, please keep in mind that when a concept is presented it will not cover every potential scenario or edge case. Although I may cite something as a best practice, this book ultimately constitutes theories based upon the collective research, experience, and opinions of its contributors. As such, it may be the case that your research, experience, and opinions lead you to a different conclusion regarding the topic being presented. That’s perfectly fine; that’s why NSM is a practice.
Chapter 1: The Practice of Applied Network Security Monitoring The first chapter is devoted to defining network security monitoring and its relevance in the modern security landscape. It discusses a lot of the core terminology and assumptions that will be used and referenced throughout the book.
Part 1: Collection
Chapter 2: Planning Data Collection The first chapter in the Collection section of ANSM is an introduction to data collection and an overview of its importance. This chapter will introduce the Applied Collection Framework, which is used for making decisions regarding what data should be collected using a risk-based approach.
Chapter 3: The Sensor Platform This chapter introduces the most critical piece of hardware in an NSM deployment: the sensor. First, we will look at a brief overview of the various NSM data types, and the types of NSM sensors. This will lead us to discuss important considerations for purchasing and deploying sensors. Finally, we will cover the placement of NSM sensors on the network, including a primer on creating network visibility maps for analyst use.
Chapter 4: Session Data This chapter discusses the importance of session data, along with a detailed overview of the SiLK toolset for the collection of NetFlow data. We will also briefly examine the Argus toolset for session data collection and parsing.
Chapter 5: Full Packet Capture Data This chapter begins with an overview of the importance of full packet capture data. We will examine several tools that allow for full packet capture of PCAP data, including Netsniff-NG, Daemonlogger, and Dumpcap. This will lead to a discussion of different considerations for the planning of FPC data storage and maintenance of that data, including considerations for trimming down the amount of FPC data stored.
Chapter 6: Packet String Data This chapter provides an introduction to packet string (PSTR) data and its usefulness in the NSM analytic process. We will look at several methods for generating PSTR data with tools like Httpry and Justniffer. We will also look at tools that can be used to parse and view PSTR data, including Logstash and Kibana.
Part 2: Detection
Chapter 7: Detection Mechanisms and Indicators of Compromise, and Signatures This chapter examines the relationship between detection mechanisms and Indicators of Compromise (IOC). We will look at how IOCs can be logically organized, and how they can be effectively managed for incorporation into an NSM program. This will include a system for classifying indicators, as well as metrics for calculating and tracking the precision of indicators that are deployed to various detection mechanisms. We will also look at two different formats for IOC’s, OpenIOC and STIX.
Chapter 8: Reputation-Based Detection The first specific type of detection that will be discussed is reputation-based detection. We will discuss the fundamental philosophy of reputation-based detection, along with several resources for examining the reputation of devices. This discussion will lean towards solutions that can automate this process, and will demonstrate how to accomplish this with simple BASH scripts, or by using Snort, Suricata, CIF, or Bro.
Chapter 9: Signature-Based Detection with Snort and Suricata The most traditional form of intrusion detection is signature-based. This chapter will provide a primer on this type of detection and discuss the usage of the Snort and Suricata intrusion detection systems. This will include the usage of Snort and Suricata, and a detailed discussion on the creation of IDS signatures for both platforms.
Chapter 10: The Bro Platform This chapter will cover Bro, one of the more popular anomaly-based detection solutions. It will review of the Bro architecture, the Bro language, and several practical cases that demonstrate the truly awesome power of Bro as an IDS and network logging engine.
Chapter 11: Anomaly-Based Detection with Statistical Data This chapter will discuss the use of statistics for identifying anomalies on the network. This will focus on the use of various NetFlow tools like rwstats and rwcount. We will also discuss methods for visualizing statistics by using Gnuplot and the Google Charts API. This chapter will provide several practical examples of useful statistics that can be generated from NSM data.
Chapter 12: Using Canary Honeypots for Detection Previously only used for research purposes, canary honeypots are a form of operational honeypot that can be used as an effective detection tool. This chapter will provide an overview of the different types of honeypots, and how certain types can be used in an NSM environment. We will look at several popular honeypot applications that can be used for this purpose, including Honeyd, Kippo, and Tom’s Honeypot. We will also briefly discuss the concept of Honeydocs.
Part 3: Analysis
Chapter 13: Packet Analysis The most critical skill an NSM analyst can have is the ability to interpret and decipher packet data that represents network communication. To do this effectively requires a fundamental understanding of how packets are dissected. This chapter provides that fundamental backing and shows how to break down packet fields on a byte by byte basis. We demonstrate these concepts using tcpdump and Wireshark. This chapter will also cover basic to advanced packet filtering techniques using Berkeley Packet Filters and Wireshark Display Filters.
Chapter 14: Friendly and Threat Intelligence The ability to generate intelligence related to friendly and hostile systems can make or break an investigation. This chapter begins with an introduction to the traditional intelligence cycle and how it relates to NSM analysis intelligence. Following this, we look at methods for generating friendly intelligence by generating asset data from network scan and leveraging PRADS data. Finally, we examine the types of threat intelligence and discuss some basic methods for researching tactical threat intelligence related to hostile hosts.
Chapter 15: The Analysis Process The final chapter discusses the analysis process as a whole. This begins with a discussion of the analysis process, and then breaks down into examples of two different analysis processes; relational investigation and differential diagnosis. Following this, the lessons learned process of incident morbidity and mortality is discussed. Finally, we will look at several analysis best practices to conclude the book.
IP Address Disclaimer
Throughout this book, several examples are provided where IP addresses are mentioned, in both raw data and in screenshots. In most every case, and unless otherwise specified, these IP addresses were randomized using various tools. Because of this, any reference to any public IP address belonging to an organization is purely coincidental, and by no means represents actual traffic generated by those entities.
Companion Website
There were quite a few things we wanted to include in this book that we simply weren’t able to fit in or find a place for. As a result, we created a companion website that contains all kinds of additional thoughts on various NSM topics, along with code snippets and other tips and tricks. If you like what you read in the coming chapters, then consider checking out the companion website at http://www.appliednsm.com. While it wasn’t updated too much during the production of the book, we plan to provide regular contributions to this blog after the book’s release. Any errata for the book will also be located here.
Charitable Support
We are incredibly proud to donate 100% of the author royalties from this book to support five unique charitable causes.
Rural Technology Fund
Rural students, even those with excellent grades, often have fewer opportunities for exposure to technology than their urban or suburban counterparts. In 2008, Chris Sanders founded the Rural Technology Fund. The RTF seeks to reduce the digital divide between rural communities and their urban and suburban counterparts. This is done through targeted scholarship programs, community involvement, and the general promotion and advocacy of technology in rural areas.
Our scholarships are targeted at students living in rural communities who have a passion for computer technology and intend to pursue further education in that field. A portion of the royalties from this book will go to support these scholarship programs, and to provide Raspberry Pi computers to rural schools.
More Info: http://www.ruraltechfund.org
Hackers for Charity
Established by Johnny Long, HFC employs volunteer hackers (no questions asked) and engages their skills in short “microprojects” designed to help charities that cannot afford traditional technical resources. In addition to this, HFC is also on the ground in Uganda, East Africa to support aid organizations working to help some of the world’s poorest citizens. They provide free computer training, technical support, networking services, and more. They have supported many local schools with the addition of computers and training software. In addition, HFC also provides food to East African children through their food program.
More Info: http://www.hackersforcharity.org
Kiva
Kiva is the first online lending platform that allows people to donate money directly to people in developing countries through multiple field companies. Kiva includes personal stories of each person who requests a loan so that donors can connect with those individuals. Simply put, Kiva facilitates loans that change lives. Funds donated from the sale of Applied NSM will go to provide these loans.
More Info: http://www.kiva.org
Hope for the Warriors®
The mission of Hope For The Warriors® is to enhance the quality of life for post-9/11 service members, their families, and families of the fallen who have sustained physical and psychological wounds in the line of duty. Hope For The Warriors® is dedicated to restoring a sense of self, restoring the family unit, and restoring hope for our service members and our military families.
More Info: http://www.hopeforthewarriors.org
Autism Speaks
Autism is a very complex condition characterized in varying degrees by difficulties in social interaction, communication, and repetitive behaviors. The U.S. Centers for Disease Control Estimate that 1 in 88 American children have some form of autism. Autism Speaks is an organization dedicated to changing the future for all who struggle with autism spectrum disorders. They accomplish this through funding biomedical research related to the causes, prevention, treatment, and cure of autism. Autism Speaks also provides autism advocacy and support for families of autistic individuals.
More Info: http://autismspeaks.org
Contacting Us
My contributing authors and I put a lot of time and effort into our work, so we are always excited to hear from people who have read our writing and want to share their thoughts. If you would like to contact us for any reason, you can send all questions, comments, threats, and marriage proposals directly to us at the following locations:
E-mail: [email protected]
Blog: http://www.chrissanders.org; http://www.appliednsm.com
E-mail: [email protected]
Blog: http://www.appliednsm.com
David J. Bianco, Contributing Author and Tech Editor
E-mail: [email protected]
Blog: http://detect-respond.blogspot.com/; http://www.appliednsm.com
Liam Randall, Technical Contributor
E-mail: [email protected]
1Jay, C. (2008, November 03). Why it's called the practice of medicine. Retrieved from http://www.wellsphere.com/chronic-pain-article/why-it-s-called-the-practice-of-medicine/466361