Testing Longer Routes from Near the Source of the Problem

Most problems begin with some idea like “host X cannot communicate with host Y.” A great first troubleshooting step is to issue a ping command from X for host Y’s IP address. However, assuming the engineer does not have access to host X, the engineer can instead issue the ping from the router nearest X, typically the router acting as host X’s default gateway.

For instance, in Figure 18-1, imagine that the user of host A had called IT support with a problem related to sending packets to host B. A ping 172.16.2.101 command on host A would be a great first troubleshooting step, but the CSR cannot access host A or get in touch with the user of host A. So, the CSR telnets to Router R1 and pings host B from there, as shown in Example 18-2.

R1# ping 172.16.2.101
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.101, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/2/4 ms
R1# ping 172.16.2.101
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.101, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

First, take a moment to review the output of the first IOS ping command. By default, the Cisco IOS ping command sends five echo messages, with a timeout of 2 seconds. If the command does not receive an echo reply within 2 seconds, the command considers that message to be a failure, and the command lists a period. If a successful reply is received within 2 seconds, the command displays an exclamation point. So, in this first command, the first echo reply timed out, whereas the other four received a matching echo reply within 2 seconds.

As a quick aside, the example shows a common and normal behavior with ping commands: the first ping command shows one failure to start, but then the rest of the messages work. This usually happens because some device in the end-to-end route is missing an ARP table entry.

Now think about troubleshooting and what a working ping command tells us about the current behavior of this internetwork. First, focus on the big picture for a moment:

• R1 can send ICMP echo request messages to host B (172.16.2.101).

• R1 sends these messages from its outgoing interface’s IP address (by default), 172.16.4.1 in this case.

• Host B can send ICMP echo reply messages to R1’s 172.16.4.1 IP address (hosts send echo reply messages to the IP address from which the echo request was received).

Figure 18-2 shows the packet flow.

Next, think about IPv4 routing. In the forward direction, R1 must have a route that matches host B’s address (172.16.2.101); this route will be either a static route or one learned with a routing protocol. R2 also needs a route for host B’s address, in this case a connected route to B’s subnet (172.16.2.0/24), as shown in the top arrow lines in Figure 18-3.

The arrow lines on the bottom of Figure 18-3 show the routes needed to forward the ICMP echo reply message back to Router R1’s 172.16.4.1 interface. First, host B must have a valid default router setting because 172.16.4.1 sits in a different subnet than host B. R2 must also have a route that matches destination 172.16.4.1 (in this case, likely to be a connected route).

The working ping commands in Example 18-2 also require the data-link and physical layer details to be working. The WAN link must be working: The router interfaces must be up/up, which typically indicates that the link can pass data. On the LAN, R2’s LAN interface must be in an up/up state. In addition, everything discussed about Ethernet LANs must be working because the ping confirmed that the packets went all the way from R1 to host B and back. In particular

• The switch interfaces in use are in a connected (up/up) state.

• Port security (discussed in the CCNA 200-301 Official Cert Guide, Volume 2) does not filter frames sent by R2 or host B.

• STP has placed the right ports into a forwarding state.

The ping 172.16.2.101 command in Example 18-2 also confirms that IP access control lists (ACL) did not filter the ICMP messages. One ACL contains a set of matching rules and actions: some matched packets are filtered (discarded), while others can continue on their path as normal. ACLs can examine packets as they enter or exit a router interface, so Figure 18-4 shows the various locations on routers R1 and R2 where an ACL could have filtered (discarded) the ICMP messages. (Note that an outbound ACL on router R1 would not filter packets created on R1, so there is no rightward-facing arrow over R1.)

Finally, the working ping 172.16.2.101 command on R1 can also be used to reasonably predict that ARP worked and that switch SW2 learned MAC addresses for its MAC address table. R2 and host B need to know each other’s MAC addresses so that they can encapsulate the IP packet inside an Ethernet frame, which means both must have a matching ARP table entry. The switch learns the MAC address used by R2 and by host B when it sends the ARP messages or when it sends the frames that hold the IP packets. Figure 18-5 shows the type of information expected in those tables.

As you can see from the last few pages, a strategy of using a ping command from near the source of the problem can rule out a lot of possible root causes of any problems between two hosts—assuming the ping command succeeds. However, this ping command does not act exactly like the same ping command on the actual host. To overcome some of what is missing in the ping command from a nearby router, the next several examples show some strategies for testing other parts of the path between the two hosts that might have a current problem.

Using Extended Ping to Test the Reverse Route

Pinging from the default router, as discussed in the past few pages, misses an opportunity to test IP routes more fully. In particular, it does not test the reverse route back toward the original host.

For instance, referring to the internetwork in Figure 18-2 again, note that the reverse routes do not point to an address in host A’s subnet. When R1 processes the ping 172.16.2.101 command, R1 has to pick a source IP address to use for the echo request, and routers choose the IP address of the outgoing interface. The echo request from R1 to host B flows with source IP address 172.16.4.1 (R1’s G0/0/0 IP address). The echo reply flows back to that same address (172.16.4.1).

A standard ping often does not test the reverse route that you need to test. In this case, the standard ping 172.16.2.101 command on R1 does not test whether the routers can route back to subnet 172.16.1.0/24, instead testing their routes for subnet172.16.4.0. A better ping test would test the route back to host A’s subnet; an extended ping from R1 can cause that test to happen. Extended ping allows R1’s ping command to use R1’s LAN IP address from within subnet 172.16.1.0/24. Then, the echo reply messages would flow to host A’s subnet, as shown in Figure 18-6.

The extended ping command does allow the user to type all the parameters on a potentially long command, but it also allows users to simply issue the ping command, press Enter, with IOS then asking the user to answer questions to complete the command, as shown in Example 18-3. The example shows the ping command on R1 that matches the logic in Figure 18-6. This same command could have been issued from the command line as ping 172.16.2.101 source 172.16.1.1.

R1# ping
Protocol [ip]:
Target IP address: 172.16.2.101
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 172.16.1.1
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.2.101, timeout is 2 seconds:
Packet sent with a source address of 172.16.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms

This particular extended ping command tests the same routes for the echo request going to the right, but it forces a better test of routes pointing back to the left for the ICMP echo reply. For that direction, R2 needs a route that matches address 172.16.1.1, which is likely to be a route for subnet 172.16.1.0/24—the same subnet in which host A resides.

From a troubleshooting perspective, using both standard and extended ping commands can be useful. However, neither can exactly mimic a ping command created on the host itself because the routers cannot send packets with the host’s IP address. For instance, the extended ping in Example 18-3 uses source IP address 172.16.1.1, which is not host A’s IP address. As a result, neither the standard or extended ping commands in these two examples so far in this chapter can test for some kinds of problems, such as the following:

• IP ACLs that discard packets based on host A’s IP address but allow packets that match the router’s IP address

• LAN switch port security that filters A’s frames (based on A’s MAC address)

• IP routes on routers that happen to match host A’s 172.16.1.51 address, with different routes that match R1’s 172.16.1.1 address

• Problems with host A’s default router setting

Testing LAN Neighbors with Standard Ping

Testing using a ping of another device on the LAN can quickly confirm whether the LAN can pass packets and frames. Specifically, a working ping rules out many possible root causes of a problem. For instance, Figure 18-7 shows the ICMP messages that occur if R1 issues the command ping 172.16.1.51, pinging host A, which sits on the same VLAN as R1.

If the ping works, it confirms the following, which rules out some potential issues:

• The host with address 172.16.1.51 replied.

• The LAN can pass unicast frames from R1 to host 172.16.1.51 and vice versa.

• You can reasonably assume that the switches learned the MAC addresses of the router and the host, adding those to the MAC address tables.

• Host A and Router R1 completed the ARP process and list each other in their respective Address Resolution Protocol (ARP) tables.

The failure of a ping, even with two devices on the same subnet, can point to a variety of problems, like those mentioned in this list. For instance, if the ping 172.16.1.51 on R1 fails (Figure 18-7), that result points to this list of potential root causes:

• IP addressing problem: Host A could be statically configured with the wrong IP address.

• DHCP problems: If you are using Dynamic Host Configuration Protocol (DHCP), many problems could exist. Chapter 7, “Implementing DHCP” in CCNA 200-301 Official Cert Guide, Volume 2, discusses those possibilities in some depth.

• VLAN trunking problems: The router could be configured for 802.1Q trunking, when the switch is not (or vice versa).

• LAN problems: A wide variety of issues could exist with the Layer 2 switches, preventing any frames from flowing between host A and the router.

So, whether the ping works or fails, simply pinging a LAN host from a router can help further isolate the problem.

Testing LAN Neighbors with Extended Ping

A standard ping of a LAN host from a router does not test that host’s default router setting. However, an extended ping can test the host’s default router setting. Both tests can be useful, especially for problem isolation, because

• If a standard ping of a local LAN host works…

• But an extended ping of the same LAN host fails…

• The problem likely relates somehow to the host’s default router setting.

First, to understand why the standard and extended ping results have different effects, consider first the standard ping 172.16.1.51 command on R1, as shown previously in Figure 18-7. As a standard ping command, R1 used its LAN interface IP address (172.16.1.1) as the source of the ICMP Echo. So, when the host (A) sent back its ICMP echo reply, host A considered the destination of 172.16.1.1 as being on the same subnet. Host A’s ICMP echo reply message, sent back to 172.16.1.1, would work even if host A did not have a default router setting at all!

In comparison, Figure 18-8 shows the difference when using an extended ping on Router R1. An extended ping from local Router R1, using R1’s S0/0/0 IP address of 172.16.4.1 as the source of the ICMP echo request, means that host A’s ICMP echo reply will flow to an address in another subnet, which makes host A use its default router setting.

The comparison between the previous two figures shows one of the most classic mistakes when troubleshooting networks. Sometimes, the temptation is to connect to a router and ping the host on the attached LAN, and it works. So, the engineer moves on, thinking that the network layer issues between the router and host work fine, when the problem still exists with the host’s default router setting.

Testing WAN Neighbors with Standard Ping

As with a standard ping test across a LAN, a standard ping test between routers over a serial or Ethernet WAN link tests whether the link can pass IPv4 packets. With a properly designed IPv4 addressing plan, two routers on the same serial or Ethernet WAN link should have IP addresses in the same subnet. A ping from one router to the IP address of the other router confirms that an IP packet can be sent over the link and back, as shown in the ping 172.16.4.2 command on R1 in Figure 18-9.

A successful ping of the IP address on the other end of an Ethernet WAN link that sits between two routers confirms several specific facts, such as the following:

• Both routers’ WAN interfaces are in an up/up state.

• The Layer 1 and 2 features of the link work.

• The routers believe that the neighboring router’s IP address is in the same subnet.

• Inbound ACLs on both routers do not filter the incoming packets, respectively.

• The remote router is configured with the expected IP address (172.16.4.2 in this case).

Testing by pinging the other neighboring router does not test many other features. However, although the test is limited in scope, it does let you rule out WAN links as having a Layer 1 or 2 problem, and it rules out some basic Layer 3 addressing problems.

How the traceroute Command Works

The traceroute command gathers information by generating packets that trigger error messages from routers; these messages identify the routers, letting the traceroute command list the routers’ IP addresses in the output of the command. That error message is the ICMP Time-to-Live Exceeded (TTL Exceeded) message, originally meant to notify hosts when a packet had been looping around a network.

Ignoring traceroute for a moment and instead focusing on IP routing, IPv4 routers defeat routing loops in part by discarding looping IP packets. To do so, the IPv4 header holds a field called Time To Live (TTL). The original host that creates the packet sets an initial TTL value. Then each router that forwards the packet decrements the TTL value by 1. When a router decrements the TTL to 0, the router perceives the packet is looping, and the router discards the packet. The router also notifies the host that sent the discarded packet by sending an ICMP TTL Exceeded message.

Now back to traceroute. Traceroute sends messages with low TTL values to make the routers send back a TTL Exceeded message. Specifically, a traceroute command begins by sending several packets (usually three), each with the header TTL field equal to 1. When that packet arrives at the next router—host A’s default Router R1 in the example of Figure 18-12—the router decrements TTL to 0 and discards the packet. The router then sends host A the TTL Exceeded message, which identifies the router’s IP address to the traceroute command.

The traceroute command sends several TTL=1 packets, checking them to see whether the TTL Exceeded messages flow from the same router, based on the source IP address of the TTL Exceeded message. Assuming the messages come from the same router, the traceroute command lists that IP address as the next line of output on the command.

To find all the routers in the path, and finally confirm that packets flow all the way to the destination host, the traceroute command sends a small set of packets with TTL=1, then a small set with TTL=2, then 3, 4, and so on, until the destination host replies. Figure 18-13 shows the packet from the second set with TTL=2. In this case, one router (R1) actually forwards the packet, while another router (R2) happens to decrement the TTL to 0, causing a TTL Exceeded message to be sent back to host A.

The figure shows these four steps:

1. The traceroute command sends a packet from the second set with TTL=2.

2. Router R1 processes the packet and decrements TTL to 1. R1 forwards the packet.

3. Router R2 processes the packet and decrements TTL to 0. R2 discards the packet.

4. R2 notifies the sending host of the discarded packet by sending a TTL Exceeded ICMP message. The source IP address of that message is 172.16.4.2.

Finally, the choice of source IP address to use on the time-exceeded message returned by routers has a big impact on the output of the traceroute command. Most routers use simpler logic that also makes command output like traceroute more consistent and meaningful. That logic: choose the TTL Exceeded message’s source IP address based on the source interface of the original message that was discarded due to TTL. In the example in Figure 18-13, the original message at Step 2 arrived on R2’s G0/0/1 interface, so at Step 3, R2 uses G0/0/1’s IP address as the source IP address of the TTL Exceeded message, and as the interface out which to send the message.

Standard and Extended traceroute

The standard and extended options for the traceroute command give you many of the same options as the ping command. For instance, Example 18-5 lists the output of a standard traceroute command on Router R1. Like the standard ping command, a standard traceroute command chooses an IP address based on the outgoing interface for the packet sent by the command. So, in this example, the packets sent by R1 come from source IP address 172.16.4.1, R1’s G0/0/0 IP address.

R1# traceroute 172.16.2.101
Type escape sequence to abort.
Tracing the route to 172.16.2.101
VRF info: (vrf in name/id, vrf out name/id)
   1 172.16.4.2 0 msec 0 msec 0 msec
   2 172.16.2.101 0 msec 0 msec *

The extended traceroute command, as shown in Example 18-6, follows the same basic command structure as the extended ping command. The user can type all the parameters on one command line, but it is much easier to just type traceroute, press Enter, and let IOS prompt for all the parameters, including the source IP address of the packets (172.16.1.1 in this example).

R1# traceroute
Protocol [ip]:
Target IP address: 172.16.2.101
Source address: 172.16.1.1
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to 172.16.2.101
VRF info: (vrf in name/id, vrf out name/id)
   1 172.16.4.2 0 msec 0 msec 0 msec
   2 172.16.2.101 0 msec 0 msec *

Both the ping and traceroute commands exist on most operating systems, including Cisco IOS. However, some operating systems use a slightly different syntax for traceroute. For example, most Windows operating systems support tracert and pathping, and not traceroute. Linux and OS X support the traceroute command.