Using a Troubleshooting Scenario
The problem in this fictional scenario is that end systems A and B cannot ping each other. (Assume that the Frame Relay network is fully functional.) End system A is a Sun Classic running Solaris 2.5, and end system B is a PC running Windows 95, as shown in Figure 6-1.
Figure 6-1. A and B cannot ping each other
Checking the Available Routes
In the following output of the show ip route 168.71.8.2 from RouterA, you can see that RouterA has a route to the subnet that PC-B is on:
RouterA#show ip route 168.71.8.2
Routing entry for 168.71.8.0 255.255.255.0
Known via "rip", distance 120, metric 1
Redistributing via rip
Last update from 168.71.6.3 on Serial1, 00:00:13 ago
Routing Descriptor Blocks:
* 168.71.6.3, from 168.71.6.3, 00:00:13 ago, via Serial1
Route metric is 1, traffic share count is 1
RouterB#
In the following output of the show ip route 168.72.5.2 from RouterB, you can see that RouterB has a route to this subnet:
RouterB#show ip route 168.72.5.2
Routing entry for 168.71.8.0 255.255.255.0
Known via "rip", distance 120, metric 1
Redistributing via rip
Last update from 168.71.6.1 on Serial0, 00:00:23 ago
Routing Descriptor Blocks:
* 168.71.6.1, from 168.71.6.1, 00:00:13 ago, via Serial0
Route metric is 1, traffic share count is 1
RouterB#
The previous test has validated that the two routers have knowledge of the subnets that Sun-A and PC-B are on.
Tracing the Route
Another useful tool is the TraceRoute utility. It is used to trace a route (path) between a device and a host on a remote network or subnetwork address. Most systems with a TCP/IP protocol stack have a version of this utility. In Windows 95, it is called tracert.exe. It is usually found in the Windows directory.
The following output of the tracert 168.71.8.2 command from SUN-A shows that the trace dies after reaching RouterB. Note that SUN-A resolved the Domain Name System (DNS) entry for PC-B before attempting the trace:
SUN-A> traceroute 168.71.8.2
traceroute to pc-b.cisco.com (168.71.8.2), 30 hops max, 40 byte packets
1 routerb (168.71.6.3) 3 ms 3 ms 3 ms
2 * * *
3 * * *
4 * * *
5 * * *
SUN-A>
The following output of the tracert 168.72.5.2 command from PC-B shows that the trace never really starts. It was necessary to kill the tracert session by pressing CTRL-C (^C).This typically means that the PC cannot reach its first gateway:
c:windows > tracert 168.72.5.2
c:windows >
You can test whether the PC is running tracert correctly by tracing the route to itself:
c:windows > tracert 168.71.8.2
Tracing route to 168.71.8.2 over a maximum of 30 hops:
1 2 ms 3 ms 2 ms 168.71.6.3
Trace complete.
c:windows >
If, for any reason, the PC fails to run tracert correctly, it appears to freeze after you enter the tracert command.
c:windows > tracert 168.71.8.2
If this happens, you need to terminate the operation by pressing CTRL-C (^C).You can then use the winipcfg.exe utility to verify that you have an IP address. At a DOS command prompt, type winipcfg, which will bring up the Wi ndows IP configuration tracking utility. It can tell you everything there is to know about the IP configuration of the PC Figures 6-2 and 6-3.
c:windows > winipcfg
Figure 6-2. The Simple View of Winipcfg
Figure 6-3. The detailed View of Winipcfg
If your IP address appears as 0.0.0.0, your system doesn't have an IP address configured. Contact your system administrator or refer to the Microsoft documentation on configuring the TCP/IP protocol. If an IP address appears but is not the address you expected, try running tracert on this address.It should work.
In this case, PC-B appears to be running tracert.exe correctly. Note that the address was not resolved to a DNS entry, possibly because PC-B could not reach its DNS server.
The problem now appears to be that PC-B cannot reach its local gateway, which prevents PC-B from reaching any hosts on different subnets or networks.
HINT
Cisco routers also have a TraceRoute utility that can be run from a command prompt. The syntax is traceroute xxx.xxx.xxx.xxx, where the xs represent a host address, a major network, or a subnet. See the Cisco IOS documentation for more information on Cisco's implementation of TraceRoute.
Using Extended Pings to Track Connectivity
Another method for quickly determining whether an end system—such as PC-B—has connectivity to its gateway is to use the extended ping function available on Cisco routers (see Figure 6-4).
Extended ping allows you to select a source address for the pings from any valid IP address in the router. Normally, pings from a router use the address from the interface attached to the subnet that the pings will exit over as the source address. Although a normal ping from a router to an end system proves that IP connectivity exists, it only proves it for the subnet that the end system and the router are connected to.
A more useful test is to determine whether the end system knows how to get packets off of its local network or subnetwork.
Figure 6-4. Using an extended ping from RouterB
The following output from RouterB pinging PC-B shows that the pings have failed. Note that the source IP address— 168.71.6.3—is not on the same subnet as PC-B, proving that PC-B is having a problem reaching its gateway.
RouterB#ping
Protocol [ip]:
Target IP address: 168.71.8.2
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address: 168.71.6.3
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 168.71.8.2, timeout is 2 seconds:
…
Success rate is 0 percent (0/5)
RouterB#
Note
See the Cisco IOS documentation for more information on Cisco's implementation of ping.
Other Possible Problems
It is too early to assume that the problem lies with PC-B's gateway address. Although it might make sense at this point to skip to checking the gateway address of PC-B, doing so would eliminate the opportunity to show some other useful troubleshooting techniques.
An ARP Problem
It is not unheard of to encounter a problem with the ARP process. An ARP entry can be corrupted. Another example is that the gateway interface MAC address changes when someone installs a new interface. The end system's ARP entry for the old IP/MAC address combination might not have timed out yet, which would cause the end system to send packets to the wrong layer two (MAC) address.
Remember that layer two addresses represent point-to-point connectivity on the physical LAN, to which the end system and the gateway are attached. IP addresses, in contrast, represent end-to-end connectivity.
Note
Cisco routers do not have the problem of changing the MAC addresses for interfaces when the physical interfaces themselves are changed because Cisco routers download the MAC address for interfaces from a table of addresses held in memory on the system board that holds the CPU. In a 7500-based system, the MAC addresses are held on the RSP.
In the following output from the show arp command on RouterB, you can see the IP/MAC addresses in use. On Cisco routers, the show interface command shows the MAC addresses assigned to the interfaces. On a Windows 95 system, the winipcfg.exe utility shows the MAC address in use by the PC.
RouterB#show arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 168.71.8.1 - 0000.0c0a.50ca ARPA Ethernet0
Internet 168.72.8.2 1 0060.9733.e9f5 ARPA Ethernet0
In this scenario, the IP/MAC addresses appear correct. You can compare them to the addresses on PC-B. The arp -a command is a fairly universal method for displaying an ARP table on a device running TCP/IP:
C:WINDOWS>arp -a
Interface: 168.71.8.2
Internet Address Physical Address Type
168.71.8.200-60-97-33-e9-f5 dynamic
168.71.8.100-00-0c-0a-50-ca dynamic
C:WINDOWS>
The following output of arp -a from SUN-A is provided for reference only. You already know that SUN-A can reach its local gateway.
SUN-A > arp -a
Net to Media Table
Device IP Address Mask Flags Phys Addr
------ -------------------- --------------- ----- ---------------
le0 168.72.5.1 255.255.255.255 00:00:0c:32:93:95
le0 168.72.5.2 255.255.255.255 SP 00:80:5f:78:79:71
SUN-A>
Because the addresses match, it is probably safe to assume that they are correct. Therefore, the problem lies elsewhere.
Validating End System Routing Tables
The problem now appears to be with PC-B's gateway entry. This section explains how to verify this hypothesis and how to make a temporary change. You previously verified that SUN-A's use of its gateway is correct. The methods for analyzing its routing table are presented in this section for reference purposes only.
Although neither end system is running a dynamic routing protocol, they both still have routing tables. The information in these tables is derived from configuration files when the systems start.
Note
It is possible to manipulate these routing tables temporarily by using the procedures provided in this section. Making the changes permanent requires that the configuration files themselves be modified, which is beyond the scope of this book. Consult the system administrator or the system reference guide for the system in question.
By displaying the routing table on PC-B, you can find the problem. The default route of 0.0.0.0 0.0.0.0 is pointing to the wrong gateway address: 168.71.8.10. The correct gateway is 168.71.8.1:
C:WINDOWS>netstat -rn
Route Table
Active Routes:
Network Address Netmask Gateway Address Interface Metric
0.0.0.0 0.0.0.0 168.71.8.10 168.71.8.2 1
168.71.8.0 255.255.255.0 168.71.8.1 168.71.8.2 1
168.71.8.2 255.255.255.255 127.0.0.1 127.0.0.1 1
168.71.0.0 255.255.0.0 168.71.8.1 168.71.8.2 1
168.71.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 224.0.0.0 168.71.8.2 168.71.8.2 1
255.255.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
Active Connections
Proto Local Address Foreign Address State
C:WINDOWS>
The following is how you flush existing gateways and add a new gateway dynamically in Windows 95 using a DOS command prompt:
C:WINDOWS>route -f add 0.0.0.0 mask 0.0.0.0 168.71.8.1
The following routing table from PC-B shows the correct gateway address in place:
C:WINDOWS>netstat -rn
Route Table
Active Routes:
Network Address Netmask Gateway Address Interface Metric
0.0.0.0 0.0.0.0 168.71.8.1 168.71.8.2 1
168.71.8.0 255.255.255.0 168.71.8.1 168.71.8.2 1
168.71.8.2 255.255.255.255 127.0.0.1 127.0.0.1 1
168.71.0.0 255.255.0.0 168.71.8.1 168.71.8.2 1
168.71.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 224.0.0.0 168.71.8.2 168.71.8.2 1
255.255.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
Active Connections
Proto Local Address Foreign Address State
C:WINDOWS>
To make the same kind of a change on a Solaris system (and most other UNIX systems), use the following syntax (as ROOT):
SUN-A# route add default 168.71.5.1 1
add net default: gateway 168.71.5.1
HINT
The second line shown in the previous UNIX output (add net default: gateway 168.71.5.1) is SUN-A echoing the command back to the terminal prompt. The word default and the address 0.0.0.0 can be used interchangeably.
Again, you use the netstat -rn command to display the routing table—the default equals 0.0.0.0:
SUN-A# netstat -rn
Routing Table:
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
127.0.0.1 127.0.0.1 UH 0 80 lo0
224.0.0.0 168.71.5.2 U 3 0 le0
default 168.71.5.1 UG 0 3622
SUN-A>
The following output of the route command from PC-B shows the various options this command can accept. The syntax on UNIX systems and NT systems is very similar. You can typically find out the available options by entering the route command without any options after it. This is how the information that follows was created.
Remember that these commands are only temporary. Rebooting the PC will restore the defaults.
C:WINDOWSDesktop>route
Manipulates network routing tables.
ROUTE [-f] [command [destination] [MASK netmask] [gateway]]
-f Clears the routing tables of all gateway entries. If this is
used in conjunction with one of the commands, the tables are
cleared prior to running the command.
command Specifies one of four commands
PRINT Prints a route
ADD Adds a route
DELETE Deletes a route
CHANGE Modifies an existing route
destination Specifies the host to send command.
MASK If the MASK keyword is present, the next parameter is
interpreted as the netmask parameter.
netmask If provided, specifies a subnet mask value to be associated
with this route entry. If not specified, if defaults to
255.255.255.255.
gateway Specifies gateway.
All symbolic names used for destination or gateway are looked up in the
network and host name database files NETWORKS and HOSTS, respectively. If
the command is print or delete, wildcards may be used for the destination and
gateway, or the gateway argument may be omitted.
C:WINDOWSDesktop>
Following is an example of adding and then deleting a route on PC-B:
C:WINDOWS>route add 171.68.97.0 mask 255.255.255.0 168.71.8.1
C:WINDOWS>netstat -rn
Route Table
Active Routes:
Network Address Netmask Gateway Address Interface Metric
0.0.0.0 0.0.0.0 168.71.8.1 168.71.8.2 1
168.71.8.0 255.255.255.0 168.71.8.1 168.71.8.2 1
168.71.8.2 255.255.255.255 127.0.0.1 127.0.0.1 1
168.71.0.0 255.255.0.0 168.71.8.1 168.71.8.2 1
168.71.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
171.68.97.0 255.255.255.0 168.71.8.1 168.71.8.2 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 224.0.0.0 168.71.8.2 168.71.8.2 1
255.255.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
Active Connections
Proto Local Address Foreign Address State
C:WINDOWS>
C:WINDOWS>route delete 171.68.97.0 mask 255.255.255.0 168.71.8.1
C:WINDOWS>netstat -rn
Route Table
Active Routes:
Network Address Netmask Gateway Address Interface Metric
0.0.0.0 0.0.0.0 168.71.8.1 168.71.8.2 1
168.71.8.0 255.255.255.0 168.71.8.1 168.71.8.2 1
168.71.8.2 255.255.255.255 127.0.0.1 127.0.0.1 1
168.71.0.0 255.255.0.0 168.71.8.1 168.71.8.2 1
168.71.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
224.0.0.0 224.0.0.0 168.71.8.2 168.71.8.2 1
255.255.255.255 255.255.255.255 168.71.8.2 168.71.8.2 1
Active Connections
Proto Local Address Foreign Address State
C:WINDOWS>
It is safe to practice with these commands because they are reset when the PC is rebooted or power cycled.
This section has focused on some basic troubleshooting techniques that are useful when the problem appears to be an end system configuration issue.