Many unique devices exist.
These devices can be temporal, meaning that they may appear and disappear from networks.
They can operate on different wireless and wired networks.
They can communicate directly with each other, in many cases without supervision or use of an intermediary such as a switch or wireless access point.
New devices can be added by simply applying power to them as with zero configuration devices.
They can utilize one of several dynamic discover protocols.
Finally, if devices are compromised they may impact other local devices, leak information, or disrupt activities of other devices and networks.
- 1.
What passive observations data should we collect?
- 2.
How should we categorize collected observations as meaningful, redundant, or plain noise?
- 3.
How might we organize and store the observations?
- 4.What do we plan to do with the collected observations?
- a.
How will we define “normal” versus “abnormal” behavior?
- b.
Can the collected observations be used to train machine learning elements?
- c.
Can the observations be a viable source of forensic evidence?
- a.
- 5.
What networks should we passively monitor?
- 6.
What observations will be recorded based on this passive monitoring?
- 7.
Are we really going to use a Raspberry Pi to do this?
Clearly, we are not going to address these all of these questions at the same time. Rather, let’s develop a model for the basics and then we can build upon that model in a spiral fashion.
We will do this by starting with what we already know well, Ethernet-based IP wired networks. To monitor and collect data we can use a variety of off-the-shelf tools or we can roll our own. Because we are planning to deploy this technology on a Raspberry Pi, using Python (I know, sounds crazy, right?), we need to keep this as simple and as close to the metal as possible. Thus, for this first experiment, I will be only using the Python standard socket library to perform this collection, and I will be using the built-in Python dictionary type to store, categorize, and at the same time, reduce the observations.
What Should We Collect?
Simplified LAN diagram
- 1.
Ethernet Packets
- 2.
ARP (Address Resolution Protocol) Packets
- 3.
IP Packets
- 4.
TCP Packets
- 5.
UDP Packets
- 6.
ICMP (Internet Control Message Protocol) Packets
Ethernet Packet Format
Ethernet packet overview
Note
Network device MAC addresses can be modified in many cases. Thus, it is possible to modify such devices to report an inaccurate or spoofed organizationally unique identifier (OUI).
- 1.
Bit 0 defines whether the packet is set to all nodes individually (unicast) or if only one packet is sent (multicast) and individual NICs can decide to accept or reject the packet.
- 2.
Bit 1 defines whether the MAC address is defined globally through OUI registration or whether the MAC address set by the manufacturer is overridden by the local administrator.
OUI breakdown
Our objective in extracting the MAC address is to map the packet to a specific device. If the MAC address is not locally defined, then extract information about the device defined by the manufacturer.
ARP
The ARP is used to dynamically discover the mapping of devices operating on a network. This maps the MAC address (layer 2) with the IP address (layer 3).
Simplified ARP request/replay process
Mapping ARP behavior (request, replies, frequency, and time frame) can identify devices that are behaving normally, or devices that could be rogue, are new to the network, or are operating erratically or maliciously. Mapping such behaviors under “normal conditions” will help to identify aberrant conditions.
ARP Tables
In Ethernet, LAN, a table, also referred to as the ARP cache, is used to maintain a correlation between each MAC address and its corresponding IP address.
IP Packets
IP packets
Next, we will examine the specific data contained in the associated data (for example TCP, UDP, and ICMP) contents delivered using IP packets.
TCP Packets
- 1.
Ports 1–1023 are considered well-known ports.
- 2.
Ports 1024–49151 are considered “registered ports” that are assigned by the Internet Assigned Numbers Authority (IANA).
- 3.
Ports 49152–65535 are considered dynamic, private, or more commonly ephemeral (i.e., lasting for a brief time or transient). For example, ports in this range are commonly used by clients making a connection to a server. It should be noted that some of the ports in this range have been mapped to known malware usage.
TCP packet details
UDP Packet
UDP packet details
ICMP Packet
ICMP packet details
Passively Monitoring IoT Behavior
Compared to active probing, passive monitoring provides greater insight into the activities of the network being monitored. The difference can be likened to a movie versus a still photograph. Using tools like NMAP to identify devices operating on your network provides an instantaneous view of those devices that properly respond. In many cases IoT devices are transient and thus could and will be missed by active or probing-based methods. Mapping the behavior of these devices over an extended period of time is critical to understanding the potential threats that they pose, along with connections to other devices.
Modeling Normal Behavior
Now that we have defined several key elements from the Ethernet layer, IP layer, and transport layers, let’s take inventory of the key elements that we could observe and determine how we can store and categorize these observed values.
Ethernet Layer
IP Layer
Transport Layer
Because it is likely that we will encounter many packets with the same MAC, source IP, destination IP, protocol, and nonephemeral port values, we need to reduce the data that we store regarding these observations. We will also be choosing a data type that is built in. We could choose a Python list or set, but both have limitations that make them not the best choice. However, the built-in dictionary data type in Python provides the ideal solution for storing these observations.
Python dictionaries, much like traditional Webster-style dictionaries, have a key and a value, which is typically referred to as a key/value pair. In Python both the key and the value can be complex, the only rule being that the key must be a hashable type such as an integer, long, string, or tuple. The value part of the key/value pair can be a list or other nonhashable data type.
The question is how would we structure the key to help us reduce the observations that we need to store and begin to build and hold a model of normal behavior. To simplify the question: what combination of fields from the collected observations would be considered unique?
I’m going to use the following tuple as the key:
Notice I didn’t include SRC and DST port. The reason is that when a client makes a connection to a server, the port that is chosen is dynamic and normally comes from the ephemeral set of ports. Thus, the port that will be included in the key will be the nonephemeral port. If both ports are nonephemeral then two entries will be made in the dictionary, one using the SRC port and one using the DST port. If both ports are ephemeral, again both entries will be made.
So that takes care of the key. Now the question is what does the value portion of the key/value pair contain?
For this we are interested in keeping track of the number of occurrences of each unique combination. Furthermore, we would like to keep track of when and how often that combination occurred. Therefore, I will use a list to keep track of the number of occurrences of each unique key. Note, keeping track of the number of occurrences can be very fine-grained (down to the hour, day, day of week, etc.). To keep this simple, and use this data later for machine learning, I have decided to break the occurrences count down in the following way.
Therefore, the value list will be initialized with just five occurrence count values:
Each time a new observation is made with the same key, the number associated with that time will be incremented by one in the value argument associated with that key.
How Can This Be Accomplished on a Raspberry Pi with Python?
I promise to only say this once: It’s as easy as Pie.
Part I: Passively Capture Packets in Python on a Raspberry Pi
- 1.
As depicted in Figure 2-1 we need to attach the Pi to a monitoring or SPAN port of an Ethernet switch along with the WIFI 802.11 air waves.
- 2.
Next, we need to place the Pi Ethernet Port into promiscuous mode.
- 3.
Finally, we need to capture packets using the Python standard socket library.
Examine a Simple Code Snippet to Perform These Operations
- 1.
The code places the standard Ethernet port of the Pi into promiscuous mode. This allows us to view any traffic flowing over the network even if it is not destined or originating from the Pi itself.
- 2.
The code opens a socket associated with the Ethernet port to listen to traffic passing over the network.
- 3.
The code captures a single packet and displays the results in hexadecimal.
sniff.py Capture One Packet with Python
Sample Execution of the Script
When executing this script we need to have privilege. In other words, we need to be operating as superuser (sudo) in order to place the network interface card (NIC) into promiscuous mode. Next, since the script is written in Python we need to invoke the Python interpreter (python). Finally, we need to identify the script (sniff.py) we are executing. The script then performs as expected, setting the NIC into promiscuous mode, capturing a single packet and displaying the packet details in hex (see Listing 2-2).
Sample Hex Dump of a Received Packet
Part II: Identify and Extract the Key Packet Components
The next step in the process is to capture and then parse the packet contents. This includes extracting the Ethernet, IP, ARP, TCP, ICMP, IGMP, and UDP components in our first example.
You may notice a new entry in the list IGMP. The IGMP protocol is used to establish multicast group memberships. Multicast protocols are commonly used by IoT devices in order to discover nearby devices along with the services that they offer.
To handle this, I have created a new Python script called “PacketRecorder.py” which continually captures packets, extracts the key information, and records the occurrences of each unique combination in a Python dictionary.
Let’s take a deeper look at some of the key components of the script (see Listing 2-3). At the end of the chapter I will provide the complete source code for the script.
PacketRecorder.py Script Overview
Required Python Standard and Third-Party Libraries
For this script, I will be using almost exclusively standard Python libraries to perform the operations. I have imported one third-party library, PrettyTable, to provide tabular results of the recording (see Listing 2-4).
Required Libraries
The Script Main Loop
Taking a top-down look at the script, let’s first examine the main script entry point in Listing 2-5. Note that this script is completely contained in a single file (not including the importing of the standard and third-party libraries).
- 1.
Creates a PacketProcessor object that will be used to extract and record key information from each packet.
- 2.
Configures Ethernet port 0 on the Raspberry Pi in promiscuous mode.
- 3.
Creates a raw socket using this promiscuous port.
- 4.
Sets a signal timer to capture packets for 1 hour (3600 seconds).
- 5.
Creates a loop to receive packets.
- 6.
Each received packet is then passed to the PacketExtractor method of the PacketProcessing object.
- 7.
Finally, once the timer expires the PrintMap method of the PacketProcessing object is called to print out the results.
PacketRecorder Main Loop
PacketProcessor Class
- 1.
__Init__ or the constructor: This function is called when an object is instantiated from the class. It creates two lookup objects for converting Ethernet frame types and transport protocol numbers to readable values. It also creates an empty dictionary to hold the key/value pairs observed.
- 2.
PacketExtractor: This function processes the observed packet data. It extracts key information from the Ethernet frame, IP header, and transport protocols. Once the required information is collected, the key will be equal to SRC-MAC, DST-MAC, SRC-IP, DST-IP, protocol, and port, and the value will be equal to the observed occurrence times. A dictionary entry is created or updated based on the observed data from the packet.
- 3.
PrintMap: This function iterates through each of the entries in the dictionary of recorded observations and prints them in a table format (see Figure 2-9).
- 4.
SaveObservations: This function uses the Python pickle library to save the dictionary as a pickle file. We will be recalling this dictionary in later chapters to perform additional operations and analysis and to use as a key input to the machine learning process.
PacketProcessor Class
Sample output from the PacketRecorder.py script
Summary
This chapter provides a deep look at the collection, reduction, and mapping of network traffic. The basic methods of capturing and recording observations will be used in future chapters to create a baseline of “normal” operations within an IoT environment. These observations will be used to monitor and detect aberrant behavior and to train machine learning methods.