Chapter 4: Network Scanning

This chapter deals with the first phase of ethical hacking: information gathering and reconnaissance. Information gathering is one of the most important aspects of ethical hacking. Without having proper access to the required information, it is extremely hard to carry out a successful attack. We will learn what network scanning is and how it can be used to carry out attacks in a network. We will go through the following topics in this chapter:

• Introduction to networking
• Data encapsulation in TCP/IP
• Introduction to Scapy
• Introduction to ARP
• Network scanner using Scapy based on ARP

Introduction to networking

In Chapter 3, Reconnaissance and Information Gathering, we learned about the basics of networking from a very high perspective. We learned about the different components and devices present in a network and what the role of each component is. In this section, we will learn a bit more about the actual packets and data that are delivered over a network.

Data representation in digital systems

Let's first understand how your computer system manages to transmit data over a network. Every part of data in a computer system is defined by binary logic levels. These levels are defined as low or high. Every image, file, video, voice recording, or anything else that is stored in a modern-day computing system is represented by these logic levels. In physical hardware, these levels are mapped to either voltage levels or switch statuses. For example, a voltage of 5 V in a digital system might represent high logic and a voltage of 0 V will represent low logic. You might be wondering how these different types of data are represented by logic levels. Let's see how that works. Let's say you want to send the message Hello to a friend. For the sake of simplicity, let's consider that your friend is present in the same network. For now, we will assume that the underlying communication works. Now, in order to send this Hello message, we need to convert this message into a form that is understandable by the computers. We just learned that computers only understand the low and high logic levels, so we will have to encode our message into these logic levels. Now, as we can see, Hello contains five letters and we only have two logic levels, so it is not possible to encode the complete message with just one instance of just two levels. This instance is called a bit. In order to achieve this encoding, a system was developed called American Standard Code for Information Interchange (ASCII). Using this coding scheme, we can represent English letters easily along with a few other symbols and letters. Every single letter of the English alphabet is represented by a sequence of 8 bits called a byte. To represent the letter H of our Hello message, we can encode it as the following sequence of bits. H is 01001000 in ASCII format. This value is predefined in an ASCII code table; similarly, other characters are also defined in the same format:

• H = 01001000
• e = 01000101
• l = 01101100
• l = 01101100
• o = 01101111

Now we have a stream of data that can we send using any digital system. Note that this is a very simplified explanation of data representation. Real systems also use other encodings, such as Unicode and byte representation, to send complex data.

Data encapsulation

Now that we understand the data representation, let's turn our focus back to our original topic on how to send this data. We've already learned about different layers in a TCP/IP stack and how they are used to send data. In the preceding section, we said that we want to send a Hello message to someone in our local network. Let's call this message our data:

Figure 4.1 – Data representation

Now, in order for it to successfully reach the other computer, the packet must know its exact destination, similar to how a postal delivery system works. You have a country, city, postal code, street, and house number. In digital systems, you have IP addresses, MAC addresses, and source and destination ports. Let's say you write your message in your browser application and your friend is also waiting for your message in their browser. In order to successfully send the message to the exact same process in the destination computer, the IP protocol will add a new header to your message.

From the topmost layer, the application header is added. Similarly, each layer below the application layer adds its own header. The overall process looks as in Figure 4.2. Figure 4.2 shows how data is encapsulated in the TCP/IP stack before it is sent over the network. We will learn about what each of these segments contains and how this helps the packet to go to its destination:

Figure 4.2 – Data encapsulation

We will talk about these segments in detail in the following section.

The packet delivery process

The packet delivery process depends on whether the destination device is located in the same local network or not. If the device is located in the same subnet, we can directly use the Ethernet addresses to send the data. There is a lot of information present in these headers and for the scope of this book, you will not be concerned with most of them, I will only explain the fields that are relevant to this book.

TCP header

The TCP header has the fields shown in the following diagram:

Figure 4.3 – TCP header

In this header, we are only concerned with the source and destination ports. The source port relates to the process in your local machine associated with the message you want to send. The destination port is where the packet should go. The source port is usually randomly generated from the sending side while the receiving port is defined by the message. For example, when you request an HTTPS website, your PC generates request packets with the destination port number set to 443. Some services have fixed port numbers. For example, FTP works on port 21 and HTTP on port 80. In our case, if we are sending the Hello message to a browser application working on HTTPS, the source port field in the sending packet will be randomly selected (you can also set it manually as well; for example, the SSH default port is 22, but if we changed SSH to work on a different port and 22 has become available, it can be used as source port in packets) and the destination would be 443. Note that some ports are reserved, as seen previously, so your PC will assign a source port number between 10000 and 65355. Once the TCP header is added to the data, it is called the TCP segment.

IP header

Next, an IP header is added that looks as shown in Figure 4.4:

Figure 4.4 – IP header

Here, the fields that we are interested in are the source IP and destination IP. This defines where your packet will go and where it is originating from. Once the IP header is added, it is called an IP datagram.

Ethernet header

The Ethernet header helps the data to navigate in the local network. The most important fields here are the source and destination MAC addresses. As the name implies, the source MAC address will be your MAC address and the destination will be the MAC address of the recipient in the local network:

Figure 4.5 – Ethernet header

Once an Ethernet header is added, it is now called an Ethernet frame. We have learned about the most important fields in the packets being sent over the network. We have also learned how data is encapsulated and which fields it uses to route it over the network. In the next section, we will learn about creating our network scanner using the information we have just learned.

Introduction to Scapy

In order to create a network scanner, we will use a Python networking library called Scapy. This library is designed to send, sniff, dissect, and edit network packets. Scapy is a very powerful network packet manipulation tool. To read more about the tool, you can go to the following link: https://scapy.readthedocs.io/en/latest/introduction.html.

Installing Scapy

To install Scapy, first open the terminal. Let's understand a few things first. In Linux, there are two user privileges, user and root, and the environment for both users is different. Higher privileges are required for system-level commands. To send and receive packets, we will need to install Scapy as a root user as well as a normal user. We will write our program as a normal user and when we run it, we will run it as root as sending packets requires higher privileges in Linux (you can think of it as the Run as Administrator equivalent in Windows). You will see what I mean in a moment. To install Scapy as a root user, write the following command:

sudo pip3 install scapy

This will install Scapy with administrator privileges. Once this is done, open your Visual Studio Code and create a new folder for the new project. We will call this example2-introduction-scapy.

Note that if you face some problems, you will need to update your system's Python with the following commands:

sudo apt-get update && sudo apt-get upgrade && sudo apt-get install python3-virtualenv -y

Now, we will create a new virtual environment for this specific project. To create a new virtual environment, navigate to the folder directory just created and write the following command:

python3 -m venv venv

Now, if everything is done properly, you will see a folder created named venv.

Next, activate the virtual environment by running the following command:

source venv/bin/activate

Now, your virtual environment should be activated. Once it is activated, you can see the installed packages in the environment by typing the following command:

pip freeze

If you have any packages already installed in the environment, they will be listed down; otherwise, you will have nothing showing. Now, to install Scapy, write the following command:

pip3 install scapy

This should take some time to install. Once done, you can write the pip freeze command again to see the installed packages:

Figure 4.6 – Installed packages in a virtual environment

In this section, we have learned how we can install Scapy in our virtual environment and how to see whether it has been installed properly. Note that some functionalities of Scapy require the program to be run with higher privileges, otherwise they won't work. In the next section, we will learn more about how Scapy is used and how we can manipulate network packets with Scapy.

Understanding how Scapy works

In this part, we will learn about how Scapy works and how we can use it to create our own network manipulation tools. Let's create a new file called main.py and open it. Once the file is open, we can import any Scapy module inside the file. In this section, we will create a small ping request to any website. Ping requests are usually used to test whether a device is available or not. A ping request (also called an echo request) uses an underlying ICMP application layer protocol. To import a package inside your program, write the following code:

from scapy.all import scapy

Important note

Note that in the latest version of Kali Linux, some dependencies have been changed and you may see an error related to missing files. To correct this issue, you can write the following command:

cd /usr/lib/x86_64-linux-gnu/sudo ln -s -f libc.a liblibc.a

To send a ping request, you will need to create an IP layer packet, which will help you set the source and destination IP addresses. To import the IP layer, we can write the following command:

from scapy.all import IP

And lastly, to send and receive packets, we can use a function called sr. To import this function, use the following command:

from scapy.all import sr

This IP will be different for you depending on your system. You can find this IP using the sudo ifconfig command.

Then, we will define our source and destination IP:

src_ip = "192.168.74.128"

Then, we will define the destination IP. We want to create a ping request to a google.com server. You can either manually write the IP address of this server, which you can find by writing ping www.google.com in your terminal, or you can simply give www.google.com.

Scapy will automatically translate this address:

dest_ip = "www.google.com"

Now, we will create an ip_layer packet and print it out to see what it contains:

from scapy.all import ICMP

from scapy.all import IP

from scapy.all import sr

if __name__ == "__main__":

    src_ip = "192.168.74.128"

    dest_ip = "www.google.com"

    ip_layer = IP(

        src = src_ip,

        dst = dest_ip

    )

    print(ip_layer.show())

This will create an IP layer packet and display the content of the created packet. Note that the packet has not been sent yet.

The output of this program looks like this:

Figure 4.7 – IP layer packet creation

Take a look at the src and dst fields. The destination is an instance of Net, which means that Scapy will take care of translating it into an actual IP address. Now, if you check the fields displayed here and compare them with Figure 4.4, you will see that these are the same fields.

Next, to send an ICMP request, you can call the class to create an instance like this:

icmp_req = ICMP(id=100)

id=100 helps the protocol to trace packets. To see what fields are present inside this request, you can write the following command:

print(icmp_req.show())

The result will look something like this:

Figure 4.8 – ICMP packet contents

From here, you can see that the packet type is an echo request, which is used for testing the connection availability.

From our previous discussion, we know that the application layer resides on top of the IP layer, and we have created two layers up until now. Now, the next goal would be to combine these two layers into a single packet that can be sent over the network. To do this, we can write the following code:

packet = ip_layer / icmp_req

print(packet.show())

This will list out the combined packet. Note the / operator. This operator is used to combine different layers in Scapy. You start with the lower layer and keep on adding new layers with this / operator. The print result will show the result of the packet with the previous layers combined into one:

Figure 4.9 – Combined layers

Now, our request is ready to be sent. To send it, we can use the sr1 method we already imported:

    response = sr1(packet, iface="eth0")

    if response:

        print(response.show())

The response will look something like this:

Figure 4.10 – ICMP reply

You can see the type of response is echo-reply and the src field in the reply is the IP address of the server that replied to this ping request.

Now you have learned how to craft and send packets using Python. Theoretically, you can create any network application with Scapy.

The complete code mentioned previously to send a packet is shown next:

from scapy.all import ICMP

from scapy.all import IP

from scapy.all import sr,

if __name__ == "__main__":

    src_ip = "192.168.74.128"

    dest_ip = "www.google.com"

    ip_layer = IP(src = src_ip, dst = dest_ip)

    icmp_req = ICMP(id=100)

    packet = ip_layer / icmp_req

    response = sr(packet, iface="eth0")

// to see available interfaces, write ifconfig command in

// terminal

    if response:

        print(response.show())

The good thing about Scapy is that it lets you create raw_packets, which means that even packets with false information (malformed packets) can be created and there is no mechanism for checking whether the packet has correct values or not. You can change the src ip field from your computer and put the value of some other packet, and in some cases, the destination will have no way of knowing which PC actually generated these packets (idle scan). This way, you can spoof packets.

So, until now, we have learned about IP stack and header fields. We also learned about how to install Scapy and use it to create raw packets that can be sent over the network. Let's now take a look at few more helpful functions that will help us understand the workings of Scapy in a bit more detail.

If you want to see more details about a certain layer in Scapy and what options are available in the layer to modify, you can use the ls function in Scapy. To import this function, you can use this command:

from scapy.all import ls, IP

To get information about ip_layer, we can print ls like this:

dest_ip = "www.google.com"

ip_layer = IP(dst = dest_ip)

print(ls(ip_layer))

In the next screenshot, you will see the execution of the previously mentioned code. The screenshot shows the list of fields in the IP packet:

Figure 4.11 – The ls function

If you want to access the individual field of any layer, you can simply use the dot (.) operator. For example, if you want to print dst in ip_layer, you can write the following code:

ip_layer = IP(dst = dest_ip)

print("Destination  = ", ip_layer.dst)

The result is as follows:

Figure 4.12 – Accessing individual fields

If you want to see a quick summary of the layer, you can call the summary method on the layer:

print("Summary  = ",ip_layer.summary())

The summary result will be as follows:

Figure 4.13 – Layer summary

So now, we have familiarized ourselves with Scapy and how it works. We learned about creating basic packets and how to manipulate these packets. In the next section, we will move toward how to use Scapy for information gathering.

Network scanner using Scapy

In this section, we will create a simple scanner, scan hosts in our local network, and find their MAC addresses. In order to create the scanner, we need to first understand what the Address Resolution Protocol (ARP) is and how it can be used for creating a network scanner.

Address Resolution Protocol

ARP in its simplest form is a translation tool that helps us to translate IP addresses into MAC addresses. Whenever a device needs to communicate with a device within the same local network, it needs the device's MAC address. IP addresses are not used for local communication.

Let's say that device A wants to communicate with device B in a local network. In order to find the MAC address of device B, computer A will first look inside an internal list maintained by it called the ARP cache to see whether computer B's IP addresses are mapped to a physical MAC address inside its table. This is called an ARP table as well. You can check the ARP table on your PC by typing the arp -a command.

Here is the result of running the arp -a command on Kali Linux:

Figure 4.14 – ARP table

You can see that it lists out the IP addresses and corresponding MAC addresses associated with them. You can use the same command in Windows as well.

If the corresponding MAC address of the requested device is not present locally, device A will send out a broadcast request to the whole network to ask which device has the respective IP. In our case, it will be device B. Those devices that are not device B will ignore this request while device B will give out a reply with the corresponding MAC address of device B. This way, device A will get to know the MAC address of device B. Once both devices get to know each other, the communication between them can follow. Once device A gets the MAC address of device B, it will update its ARP table. Figure 4.15 shows how an ARP request is generated by the source device and how the destination device replies with the correct MAC address:

Figure 4.15 – ARP request

Now that we understand how ARP works, we can start working on creating our own ARP scanner with Scapy to find out the MAC address of these devices. You might be wondering why we need an ARP scanner. Well, knowing the MAC addresses of a device can help us perform a man-in-the-middle attack, which we will perform in Chapter 5, Man in the Middle Attacks.

ARP scanner using Scapy

The ARP protocol works on the Ethernet layer, so using Scapy, we will import the Ethernet layer. Let's import the layers and functions we will use:

from scapy.all import Ether, ARP, srp

If all the bits of a MAC address are set to 1, it means that the packet is a broadcast and it should go to every device in the network. Scapy uses hexadecimal representation, so we will create the following variable to denote the broadcast address:

broadcast = "FF:FF:FF:FF:FF:FF"

Then, we can create an Ethernet layer packet and put the destination as broadcast.

We will also need to define the ip range we want to scan. In my case, I want to scan my local network:

ip_range = "192.168.74.1/24"

This represents that we want to scan all the devices starting with IP address 192.168.74.1 up to 192.168.74.255. The last 8 bits are called a bitmask and represent the number of hosts we want to scan. Remember that an IP address is 32 bits, and we say here that we want to mask 24 bits, so the remaining 32-24 = 8 bits are addressable only, which means that we are only scanning the last 256 hosts in the network.

Now, to create an ARP layer packet, use these commands:

ip_range = "192.168.74.1/24"

arp_layer = ARP(pdst = ip_range)

Now we have created two layers, Ether and ARP. Next, we will create a packet with both these layers:

packet = ether_layer / arp_layer

Next, we will send this packet as a broadcast. To do this, we can use the following srp function:

ans, unans = srp(packet, iface = "eth0", timeout=2)

packet is the name of the packet we want to send, iface is the network interface card we want to use to send this packet, and timeout is to make sure that if we don't get a reply in 2 seconds this means that the device is most probably offline.

srp returns both answered and unanswered packets. We are interested in answered packets from online devices only. Now, to get the IP addresses and MAC addresses of the online devices, we can write the following code. We can iterate over the answer to see the IP and corresponding MAC addresses:

    for snd, rcv in ans:

        ip = rcv[ARP].psrc

        mac = rcv[Ether].src

        print("IP = ", ip, " MAC = ", mac)

rcv represents the packets that have been received by the sender. To get the IP address, we can use the ARP layer, and to get the MAC address, we can use the Ether layer. Remember the fields set in packets correspond to the respective layer.

The complete code will look something like this:

from scapy.all import Ether, ARP, srp

if __name__ == "__main__":

    broadcast = "FF:FF:FF:FF:FF:FF"

    ether_layer = Ether(dst = broadcast)

    ip_range = "192.168.74.1/24"

    arp_layer = ARP(pdst = ip_range)

    packet = ether_layer / arp_layer

    ans, unans = srp(packet, iface = "eth0", timeout=2)

    for snd, rcv in ans:

        ip = rcv[ARP].psrc

        mac = rcv[Ether].src

        print("IP = ", ip, " MAC = ", mac)

The output of the program looks like this:

Figure 4.16 – ARP scan result

Now you can see the MAC and IP addresses of all the devices available in the network. The third one, IP = 192.168.72.129 is my Windows machine, which I will use as a victim/target machine in later chapters. To verify that the result we obtained in our program is correct, we can check these fields manually from the network connection settings:

Figure 4.17 – Victim's IP configuration

Here, you can see that the results obtained in our scan match these values. Take a look at the MAC address as well. The other devices can also be seen.

192.168.74.254 represents the DHCP server. DHCP servers assign IP addresses to devices in a network when the devices are configured to automatic IP assignment. 192.168.72.2 represents the default gateway in the network.

Summary

In this chapter, we learned how data is sent from one device to another over the network. We learned about how data is encapsulated in the TCP/IP protocol and what fields are added to each header. Next, we learned about a very important network manipulation and packet crafting tool called Scapy. We also learned how to craft packets using Scapy and how these packets can be sent over the network. We then learned about the ARP protocol and finally, we created an ARP scanner to get the IP and mac addresses of live devices in a network. In the next chapter, we will learn how to use this scanner to create a man-in-the-middle attack to intercept network traffic from a victim machine.