The traceroute Command

When you send a network packet to a remote system, especially across the Internet, it often needs to go through several gateways before it reaches its destination. You can see the gateways that the packet passes through by executing the traceroute command, as shown in Example 21-1.

# traceroute onecoursesource.com
traceroute to onecoursesource.com (38.89.136.109), 30 hops max,
  60 byte packets
 1  10.0.2.2 (10.0.2.2)  0.606 ms  1.132 ms  1.087 ms
 2  b001649-3.jfk01.atlas.cogentco.com (38.104.71.201)  0.738 ms
    0.918 ms  0.838 ms
 3  154.24.42.205 (154.24.42.205)  0.952 ms  0.790 ms 0.906 ms
 4  be2629.ccr41.jfk02.atlas.cogentco.com (154.54.27.66)  1.699 ms
    1.643 ms 1.347 ms
 5  be2148.ccr41.dca01.atlas.cogentco.com (154.54.31.117)  8.053 ms
    7.719 ms  7.639 ms
 6  be2113.ccr42.atl01.atlas.cogentco.com (154.54.24.222)  18.276 ms
    18.418 ms 18.407 ms
 7  be2687.ccr21.iah01.atlas.cogentco.com (154.54.28.70)  32.861 ms
    32.917 ms  32.719 ms
 8  be2291.ccr21.sat01.atlas.cogentco.com (154.54.2.190)  38.087 ms
    38.025 ms  38.076 ms
 9  be2301.ccr21.elp01.atlas.cogentco.com (154.54.5.174)  48.811 ms
    48.952 ms  49.151 ms
10  be2254.ccr21.phx02.atlas.cogentco.com (154.54.7.33)  57.332 ms
    57.281 ms  56.896 ms
11  te2-1.mag02.phx02.atlas.cogentco.com (154.54.1.230)  56.666 ms
    65.279 ms  56.520 ms
12  154.24.18.26 (154.24.18.26)  57.924 ms 58.058 ms  58.032 ms
13  38.122.88.218 (38.122.88.218)  79.306 ms 57.740 ms 57.491 ms
14  onecoursesource.com (38.89.136.109)  58.112 57.884 ms 58.299 ms

This output might seem useful, but it is not going to help you solve networking issues as often as you would like. For example, suppose you try to connect to a remote system, such as test.onecoursesource.com, and you are not able to get any response. You are confident that that system is up and connected to the Internet, so you execute the traceroute command to troubleshoot:

Click here to view code image

# traceroute onecoursesource.com
traceroute to test.onecoursesource.com (38.89.136.120), 30 hops max,
60 byte packets
 1  10.0.2.2 (10.0.2.2)  0.606 ms  1.132 ms  1.087 ms
 2  b001649-3.jfk01.atlas.cogentco.com (38.104.71.201)  0.738 ms
0.918 ms  0.838 ms
 3  154.24.42.205 (154.24.42.205)  0.952 ms  0.790 ms 0.906 ms
 4  ***
 5  ***
#Output omitted
 30  ***

Well, there is the problem: Gateway 4—whatever that is—doesn’t seem to be responding! Well, not exactly. The traceroute command uses ICMP packets, and these packets can be (and often are) ignored by gateways. The *** that you see in this output could be a gateway functioning correctly but just not responding to ICMP packets.

Even if gateway 4 is the problem, you need to determine what to do next. The gateway listed at number 3 is most likely outside your scope of control. You could try to hunt down the person/company responsible for gateway 3, but by the time you find and contact that person or company, that gateway will likely already have adjusted to use a different gateway 4. Gateways are normally pretty smart, and the ones on the Internet are almost always configured to be able to switch to another gateway if the one they are communicating with becomes nonresponsive.

So, is the traceroute command completely worthless? Not for larger organizations that have many internal gateways. It is not uncommon in large organizations that a network packet travels through multiple gateways before heading out to the Internet. The traceroute command can be used to pinpoint errors due to nonresponsive internal gateways.

If you want a really cool variation of the traceroute command, install the mtr command. This command performs a “traceroute-like” operation every second, updating the display with statistics, as demonstrated in Figure 21-1.

For the Linux+ exam, you should also be aware of the tracepath command, which is similar to the traceroute command. In fact, the man page for the tracepath command does an excellent job of describing the differences: “It is similar to traceroute, only does not require superuser privileges and has no fancy options.”

The tcpdump Command

When troubleshooting network issues or performing network security audits, it can be helpful to view the network traffic, including traffic that isn’t related to the local machine. The tcpdump command is a “packet sniffer” that allows you to view local network traffic.

By default, the tcpdump command displays all network traffic to standard output until you terminate the command. This could result in a dizzying amount of data flying by on your screen. You can limit the output to a specific number of network packets by using the -c option, as shown here:

Click here to view code image

# tcpdump -c 5
tcpdump: verbose output suppressed, use -v or -vv for full protocol
  decode
listening on eth0, link-type EN10MB (Ethernet), capture size
  65535 bytes
11:32:59.630873 IP localhost.43066 > 192.168.1.1.domain: 16227+ A?
  onecoursesource.com. (37)
11:32:59.631272 IP localhost.59247 > 192.168.1.1.domain: 2117+ PTR?
  1.1.168.192.in-addr.arpa. (42)
11:32:59.631387 IP localhost.43066 > 192.168.1.1.domain: 19647+ AAAA?
  onecoursesource.com. (37)
11:32:59.647932 IP 192.168.1.1.domain > localhost.59247: 2117
  NXDomain* 0/1/0 (97)
11:32:59.717499 IP 192.168.1.1.domain > localhost.43066: 16227 1/0/0
  A 38.89.136.109 (53)
5 packets captured
5 packets received by filter
0 packets dropped by kernel

You can also use this command to capture output based on some sort of criteria. For example, you can have the tcpdump command capture only packets that are available on a specific interface by using the -i option:

# tcpdump -i eth0

To limit packets to only a specific protocol, indicate the protocol name as an argument:

# tcpdump -i eth0 tcp

To display only packets associated with a specific port, use the port argument:

# tcpdump -i eth0 port 80

You can also limit the packets based on the source or destination IP address:

Click here to view code image

# tcpdump -i src 192.168.1.100
# tcpdump -i dst 192.168.1.100

In many cases, you will likely want to leave the tcpdump command running for a short period of time and view the data at some later time. In this case, it is best to use the -c option to limit the output and place the data into a file by using the -w option:

Click here to view code image

# tcpdump -c 5000 -w output-tcpdump

This file contains binary data. To read its contents, use the -r option to the tcpdump command:

# tcpdump -r output-tcpdump

Wireshark

Wireshark is an amazing network sniffer tool. It is very useful in viewing network traffic and troubleshooting network issues. It is also a very robust tool, and entire books have been written on how to use it. The goal of this section is to cover the basics of Wireshark to prepare you for the Linux+ exam.

It is likely that Wireshark is not installed on your distribution by default. You will need to install Wireshark with the appropriate installation tool (yum, apt, and so on).

Wireshark provides both GUI-based and TUI-based tools. To start the GUI tool, execute the wireshark command. You should see the GUI shown in Figure 21-2.

To view network traffic, you need to start a capture, which you do by clicking Capture and then Start. You can also limit what is captured by setting filters and options (by clicking Capture and then Options). Figure 21-3 shows an example of captured packets.

To use the TUI-based form of Wireshark, execute the tstark command as the root user:

Click here to view code image

# tshark
Capturing on 'enp0s3'
    1 0.000000000    10.0.2.15 → 68.105.28.11 DNS 81 Standard query
0xeec4 A google.com OPT
    2 0.001031279    10.0.2.15 → 68.105.28.11 DNS 81 Standard query
0x3469 AAAAgoogle.com OPT
    3 0.017196416 68.105.28.11 → 10.0.2.15    DNS 109 Standard query
response 0x3469 AAAA google.com AAAA 2607:f8b0:4007:800::200e OPT
    4 0.017265061 68.105.28.11 → 10.0.2.15    DNS 97 Standard query
response 0xeec4 A google.com A 172.217.14.110 OPT
    5 0.018482388    10.0.2.15 → 172.217.14.110 ICMP 98 Echo (ping)
request  id=0x122c, seq=1/256, ttl=64
    6 0.036907577 172.217.14.110 → 10.0.2.15    ICMP 98 Echo (ping)
reply    id=0x122c, seq=1/256, ttl=251 (request in 5)
    7 1.021052811    10.0.2.15 → 172.217.14.110 ICMP 98 Echo (ping)
request  id=0x122c, seq=2/512, ttl=64
    8 1.039492225 172.217.14.110 → 10.0.2.15    ICMP 98 Echo (ping)
reply    id=0x122c, seq=2/512, ttl=251 (request in 7)

The arp Command

Most users and administrators use hostnames to communicate with remote systems. A hostname must be translated into an IP address because Internet Protocol (IP) uses IP addresses rather than hostnames. This function is provided by a resolver, such as a DNS server.

IP is part of a seven-layer networking model called the Open Systems Interconnection model (OSI model). At Layer 2 of this model, devices communicate using the network card’s MAC address. In most cases, for two hosts to communicate, not only do they need to know each other’s IP addresses, they also need to know each other’s MAC addresses.

Initially, the local system does not know the MAC addresses of any other host. When a remote IP address is first used, a broadcast request sent on the network matches that IP address. The machine with the matching IP address responds to the original request, reporting to the original host what its MAC address is. The original system then stores this MAC address and corresponding IP address in a memory address called the ARP table.

The arp command is used to view the ARP table or make changes to it. When executed with no arguments, the arp command displays the ARP table, as shown here:

Click here to view code image

# arp
Address          HWtype  HWaddress          Flags Mask       Iface
192.168.1.11     ether   30:3a:64:44:a5:02  C                eth0

In the event that a remote system has its network card replaced, it may be necessary to delete an entry from the ARP table. This can be accomplished by using the -d option with the arp command:

# arp -i eth0 -d 192.169.1.11

When the address has been removed from the ARP table, there should be no need to add the new address manually. The next time the local system uses this IP address, it sends a broadcast request on the appropriate network to determine the new MAC address.

The nmap Command

The nmap command is used to probe a remote system to determine which network ports are reachable from the local system. This is useful for many reasons, including

• Determining what services are available on the remote system

• Testing security features on the remote system, such as TCP wrappers

• If the nmap command is executed from a remote network, verifying the effectiveness of the network’s firewall

To use the nmap command, provide either the IP address or hostname of the system that you want to scan. For example, Example 21-2 shows the output of a scan performed on a router.

# nmap 192.168.1.1
Starting Nmap 5.51 ( http://nmap.org ) at 2015-10-31 23:22 PDT
Nmap scan report for 192.168.1.1
Host is up (2.9s latency).
Not shown: 987 closed ports
PORT     STATE SERVICE
23/tcp   open  telnet
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
119/tcp  open  nntp
143/tcp  open  imap
465/tcp  open  smtps
563/tcp  open  snews
587/tcp  open  submission
993/tcp  open  imaps
995/tcp  open  pop3s
5000/tcp open  upnp
Nmap done: 1 IP address (1 host up) scanned in 4.89 seconds

The lines that describe the open ports start with the port number/protocol (for example, 23/tcp) and end with the corresponding service (for example, telnet).

By default, only Transmission Control Protocol (TCP) ports are scanned. To scan User Datagram Protocol (UDP) ports, use the -sU option combination, as demonstrated in Example 21-3.

# nmap -sU 192.168.1.1
Starting Nmap 5.51 ( http://nmap.org ) at 2015-10-31 23:36 PDT
Nmap scan report for 192.168.1.1
Host is up (0.0011s latency).
Not shown: 999 open|filtered ports
PORT   STATE SERVICE
53/udp open  domain
Nmap done: 1 IP address (1 host up) scanned in 4.09 seconds

By default, only certain common ports (about 2,000 of them) are scanned. To scan all ports, use the command shown in Example 21-4 (and then take a coffee break because it could take a while to run).

# nmap -p 1-65535 192.168.1.1
Starting Nmap 5.51 ( http://nmap.org ) at 2015-11-01 00:26 PDT
Nmap scan report for 192.168.1.1
Host is up (1.0s latency).
Not shown: 65521 closed ports
PORT     STATE SERVICE
23/tcp   open  telnet
25/tcp   open  smtp
53/tcp   open  domain
80/tcp   open  http
110/tcp  open  pop3
119/tcp  open  nntp
143/tcp  open  imap
465/tcp  open  smtps
563/tcp  open  snews
587/tcp  open  submission
993/tcp  open  imaps
995/tcp  open  pop3s
1780/tcp open  unknown
5000/tcp open  upnp
Nmap done: 1 IP address (1 host up) scanned in 5731.44 seconds

Typically a port has an associated local service. The nmap command can also probe these services to determine what version of the service is available. This feature isn’t available for all services, but for those that it is available for, it can provide useful information. Use the -sV option combination to see service version information, as shown in Example 21-5.

# nmap -sV 192.168.1.1
Starting Nmap 5.51 ( http://nmap.org ) at 2015-11-01 09:41 PST
Nmap scan report for 192.168.1.1
Host is up (1.0s latency).
Not shown: 987 closed ports
PORT     STATE SERVICE     VERSION
23/tcp   open  telnet?
25/tcp   open  smtp?
53/tcp   open  domain      dnsmasq 2.15-OpenDNS-1
###Remaining output omitted

Say that you discover a machine with IP address 192.168.1.11 on your network, but you don’t even know what sort of system it is. One of the benefits of the nmap command is that it might provide a clue as to what sort of system this is by probing it. For example, executing the nmap command on that IP address can provide you with an idea of what sort of system a machine is, as demonstrated in Example 21-6.

# nmap -sU 192.168.1.11
Starting Nmap 5.51 ( http://nmap.org ) at 2015-10-31 23:38 PDT
Nmap scan report for 192.168.1.11
Host is up (0.00045s latency).
Not shown: 992 filtered ports
PORT     STATE         SERVICE
67/udp   open|filtered dhcps
137/udp  open          netbios-ns
138/udp  open|filtered netbios-dgm
443/udp  open|filtered https
1900/udp open|filtered upnp
4500/udp open|filtered nat-t-ike
5353/udp open|filtered zeroconf
5355/udp open|filtered llmnr
Nmap done: 1 IP address (1 host up) scanned in 52.23 seconds

Given that netbios-ns and some of the other services listed in this output are Microsoft Windows–based services, it is likely that Windows is the operating system of this unknown system.

In some cases, you might be able to use the -O option to determine the operating system type, but it isn’t always successful, as demonstrated in Example 21-7.

# nmap -O 192.168.1.11
###Output omitted
Aggressive OS guesses: QEMU user mode network gateway (91%),
  Bay Networks BayStack 450 switch (software version 3.1.0.22) (85%),
  Bay Networks BayStack 450 switch (software version 4.2.0.16) (85%),
  Cabletron ELS100-24TXM Switch or Icom IC-7800 radio transceiver (85%),
  Cisco Catalyst 1900 switch or RAD IPMUX-1 TDM-over-IP
  multiplexer (85%), Sanyo PLC-XU88 digital video projector (85%),
  3Com SuperStack 3 Switch 4300, Dell PowerEdge 2650 remote access
  controller, Samsung ML-2571N or 6555N printer, or Xerox Phaser 3125N
  printer (85%), Dell 1815dn printer (85%)
No exact OS matches for host (test conditions non-ideal).
OS detection performed. Please report any incorrect results at
  http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 15.37 seconds

So, how did you discover that 192.168.1.11 machine in the first place? Another useful feature of the nmap command is its capability to scan an entire network to determine which IP addresses are in use. To perform this operation, use the -sP option combination, as demonstrated in Example 21-8.

# nmap -sP 192.168.1.0/24
Starting Nmap 5.51 ( http://nmap.org ) at 2015-10-31 23:51 PDT
Nmap scan report for 192.168.1.0
Host is up (0.00043s latency).
Nmap scan report for 192.168.1.1
Host is up (0.0026s latency).
Nmap scan report for 192.168.1.2
Host is up (0.70s latency).
Nmap scan report for 192.168.1.3
Host is up (0.045s latency).
Nmap scan report for 192.168.1.4
Host is up (0.043s latency).
Nmap scan report for 192.168.1.7
Host is up (0.00011s latency).
Nmap scan report for 192.168.1.11
Host is up (0.0020s latency).
Nmap scan report for 192.168.1.12
Host is up (0.00013s latency).
Nmap scan report for 192.168.1.14
Host is up (3.7s latency).
Nmap scan report for 192.168.1.16
Host is up (0.00088s latency).

You can even see information about your own system, including a list of network interfaces and the routing table, by using the --iflist option, as shown in Example 21-9.

# nmap --iflist


Starting Nmap 5.51 ( http://nmap.org ) at 2015-11-01 09:39 PST
************************INTERFACES************************
DEV  (SHORT) IP/MASK      TYPE     UP MTU   MAC
lo   (lo)    127.0.0.1/8  loopback up 65536
eth0 (eth0)  10.0.2.15/24 ethernet up 1500  08:00:27:E0:E2:DE


**************************ROUTES**************************
DST/MASK    DEV  GATEWAY
10.0.2.0/24 eth0
0.0.0.0/0   eth0 10.0.2.2

The nc Command

The man page of the nc command (also referred to as the netcat command) provides an excellent summary of the nc command:

• The nc (or netcat) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet(1), nc scripts nicely and separates error messages onto standard error instead of sending them to standard output, as telnet(1) does with some.

There are quite a few uses for the nc command. For example, suppose you want to know whether a specific port is being blocked by your company firewall before you bring online a service that makes use of this port. On the internal server, you can have the nc command listen for connections on that port:

# nc -l 3333

You should end up with a blank line below the nc command. Next, on a remote system outside your network, you could run the following nc command to connect (replacing server with the resolvable hostname or IP address of the local system):

# nc server 3333

If the connection is established, you see a blank line under the nc command line. If you type something on this blank line and press the Enter key, then what you typed appears below the nc command on the server. Actually, the communications works both ways: What you type on the server below the nc command appears on the client as well.

The following are some useful options to the nc command:

• -w: This option is used on the client side to close a connection automatically after a timeout value is reached. For example, nc -w 30 server 333 closes the connection 30 seconds after it is established.

• -6: Use this option to enable IPv6 connections.

• -k: Use this option to keep the server process active, even after the client disconnects. The default behavior is to stop the server process when the client disconnects.

• -u: Use this option to use UDP connections rather than TCP connections (the default). This is important for correctly testing firewall configurations as a TCP port might be blocked, while the UDP port might not be blocked.

You can also use the nc command to display open ports, similar to the way you use the netstat command:

Click here to view code image

# nc -z localhost 1000-4000
Connection to localhost 3260 port [tcp/iscsi-target] succeeded!
Connection to localhost 3333 port [tcp/dec-notes] succeeded!

The -z option can also be used for port scanning on a remote host.

nc -l 3333 | cmd

cmd | nc server 3333

nc -l 333 | tar xvf -

tar cvf - /home | nc server 333

The whois Command

The whois command is useful for determining which company or person owns a domain. Often the output also contains information regarding how to contact this organization, but it might also be redacted for privacy reasons. The following is an example of using the whois command:

Click here to view code image

# whois onecoursesource.com | head
   Domain Name: ONECOURSESOURCE.COM
   Registry Domain ID: 116444640_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.tucows.com
   Registrar URL: http://www.tucows.com
   Updated Date: 2016-01-15T01:49:45Z
   Creation Date: 2004-04-07T19:45:31Z
   Registry Expiry Date: 2021-04-07T19:45:31Z
   Registrar: Tucows Domains Inc.
   Registrar IANA ID: 69
   Registrar Abuse Contact Email:

The iftop Command

The iftop command provides a good way to display the network connections that are using the most bandwidth. It is an interactive display that updates every 1 second by default to display the “top” network connections. Figure 21-4 shows an example.

The iperf and ipset Commands

This chapter covers the “Network Monitoring and Configuration” section of Objective 4.1, “Given a scenario, analyze system properties and remediate accordingly.” There are two commands in this section of Objective 4.1 that haven’t been covered so far:

• iperf: This command provides a means to create tests of the throughput between two systems. It must be installed and configured on two systems: a client and a server. Complete coverage of the iperf command is beyond the scope of the Linux+ exam. You should be aware of the purpose of the tool, though, and the fact that it requires both client and server setup.

• ipset: This tool, which creates sets of IP addresses and then uses these sets to apply rules to a collection of systems, is covered in Chapter 22, “Linux Firewalls.” As a result, it isn’t covered in this chapter.