Trouble Ticket 2 Documentation Lab
Baselining and documentation are very important prerequisites to supporting and continuing to support networks. Add any additional notes to your drawings and tables and take time to redraw them if necessary. Often colored pens/pencils enable you to highlight different encapsulations, protocols, addresses, settings, and such.
Logging and time stamps are extremely helpful for documentation purposes. Tools such as Network Time Protocol (NTP), syslog, TFTP, FTP, and Simple Network Management Protocol (SNMP) are quite valuable when it comes to supporting internetworks. Put some of these tools to practical use as you perform the following tasks:
Set up the duck router as the master timekeeper and have the other devices get their time from duck.
Configure the devices so that they display log and debug output with a time and date stamp in milliseconds.
Set up a syslog server such as the free 3CDaemon on hosta and configure the other devices to send their informational logs to the syslog server.
Increase the size of your logging buffers.
Perform a show tech-support and other baseline commands on your routers.
Save all configurations. Send them to a TFTP/FTP server of your choice. This could be part of the 3CDaemon that is running on hosta. Alternatively, you can set up one of your routers as a TFTP server.
Make sure you know where to locate specific data on Cisco's website to properly support all of your devices. Go back and review Chapter 2, “What's in Your Tool Bag?” for more specifics. For now, research the differences between a manual memory dump and an exception dump. Manually generate a memory dump from the duck router to the existing TFTP server.
Don't forget about your hosts and servers. Make sure you know Layer 2 and Layer 3 addresses, gateways, routes, frame types, protocols, and so on. Refer back to the previous chapters to assist you with this. Add this information to your diagram or create a separate table.
(Optional) Use a protocol analyzer such as Sniffer for your baseline. Use the Sniffer as well as show and debug commands where you can to analyze all background traffic on your network. You need to know what normal is for your environment. Refer back to Chapter 2 for a refresher on protocol analysis with Sniffer Pro.
Look at the “Trouble Ticket 2 Documentation Lab Solution.” Can you spot the issues? Better yet, can you fix the issues? Make a list of each issue that you fix to compare it to the issue list and ending configuration in the sample solutions. Fix everything you identify, and use ping and trace to test all connectivity. For example, my 804 (pingme router) is a good ping target on the backbone, but use what you have. If you are really struggling, this Trouble Ticket includes sample output in a file named tt2 testing, and so does the next. Use this as a last resort. Now is your chance to apply methods and tools on your own.
Trouble Ticket 2 Documentation Lab Solution
Look at Figure 10-3. Because the drawing is getting a little crowded, you may want to take the information gathered and compile it into a table such as in Figure 10-4. This method to collect data is what I used in Chapter 5, “Shooting Trouble with Ethernet.” Refer back to it for the templates to use for your devices and make any necessary changes for this scenario.
Examine the examples and the supporting files for this chapter's Trouble Ticket 1 and Trouble Ticket 2. Review the ending configurations in the preceding chapter and the following examples to get more familiar with the environment you will be supporting in this chapter. NTP, time stamps, logging, and TFTP configuration start in Example 10-9 (and go through Example 10-17). Open the supporting file for more detailed data, including the configuration and a show tech-support output for the devices (tt2 testing). The TFTP server ending configurations for this Trouble Ticket are available in case you have the need to use TFTP to copy them to your devices.
Example 10-9. Document the duck Router
duck#clock set 9:59:00 July 30 2002 duck#configure terminal duck(config)#service timestamps debug datetime msec duck(config)#service timestamps log datetime msec duck(config)#logging 172.16.1.42 duck(config)#logging trap ? <0-7> Logging severity level alerts Immediate action needed (severity=1) critical Critical conditions (severity=2) debugging Debugging messages (severity=7) emergencies System is unusable (severity=0) errors Error conditions (severity=3) informational Informational messages (severity=6) notifications Normal but significant conditions (severity=5) warnings Warning conditions (severity=4) <cr> duck(config)#logging trap informational duck(config)#logging buffer 500000 duck(config)#ntp ? access-group Control NTP access authenticate Authenticate time sources authentication-key Authentication key for trusted time sources broadcastdelay Estimated round-trip delay clock-period Length of hardware clock tick master Act as NTP master clock max-associations Set maximum number of associations peer Configure NTP peer server Configure NTP server source Configure interface for source address trusted-key Key numbers for trusted time sources duck(config)#ntp master duck(config)#end duck#copy running-config startup-config |
Set up your syslog server before you configure the other routers. Figure 10-5 displays the 3CDaemon free syslog server running on hosta.
Example 10-10. Document the heron Router
heron(config)#service timestamps debug datetime msec heron(config)#service timestamps log datetime msec heron(config)#ntp server 172.16.1.1 heron(config)#ntp source e0 heron(config)#end heron#show ntp status Clock is synchronized, stratum 9, reference is 172.16.1.1 nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**19 reference time is C0F0E5E3.EBC9E623 (10:16:03.921 UTC Tue Jul 30 2002) clock offset is -0.1268 msec, root delay is 6.52 msec root dispersion is 0.21 msec, peer dispersion is 0.05 msec heron#configure terminal heron(config)#logging 172.16.1.42 heron(config)#logging trap informational heron(config)#logging buffer 500000 heron(config)#end heron#copy running-config startup-config |
Example 10-11. Document the goose Router
goose(config)#logging 172.16.1.42 goose(config)#logging trap informational goose(config)#logging buffer 500000 goose(config)#service timestamps debug datetime msec goose(config)#service timestamps log datetime msec goose(config)#ntp server 172.16.1.1 goose(config)#ntp source s0/0 goose(config)#end goose#show clock 10:19:32.447 UTC Tue Jul 30 2002 goose#copy running-config startup-config |
Example 10-12. Document the osprey Router
osprey(config)#logging 172.16.1.42 osprey(config)#logging trap info osprey(config)#logging buffer 500000 osprey(config)#service timestamps debug datetime msec osprey(config)#service timestamps log datetime msec osprey(config)#ntp server 172.16.1.1 osprey(config)#ntp source s0/0 osprey(config)#end osprey#copy running-config startup-config |
Example 10-13. Document the crab Router
crab(config)#logging 172.16.1.42 crab(config)#logging trap informational crab(config)#logging buffer 500000 crab(config)#service timestamps debug datetime msec crab(config)#service timestamps log datetime msec crab(config)#ntp server 172.16.1.1 crab(config)#ntp source s0 crab(config)#end crab#copy running-config startup-config |
Example 10-14. Document the swan Router
swan(config)#logging 172.16.1.42 swan(config)#logging trap informational swan(config)#logging buffer 500000 swan(config)#service timestamps debug datetime msec swan(config)#service timestamps log datetime msec swan(config)#ntp server 172.16.1.1 swan(config)#ntp source s1 swan(config)#end swan#copy running-config |
Example 10-15. Document the ferry Router (Frame Relay Switch)
ferry(config)#logging 172.16.1.42 Cannot open logging port to 172.16.1.42 ferry(config)#logging trap informational ferry(config)#logging buffer 500000 ferry(config)#service timestamps debug datetime msec ferry(config)#service timestamps log datetime msec ferry(config)#ntp server 172.16.1.1 ferry(config)#ntp source s0 ferry(config)#end ferry#copy running-config startup-config |
Example 10-16. Document the chesapeakebay Switch (2900 CatOS)
chesapeakebay> (enable)set logging server 172.16.1.42 172.16.1.42 added to System logging server table. chesapeakebay> (enable)set logging timestamp enable System logging messages timestamp will be enabled. chesapeakebay> (enable)set logging buffer 500000 Usage: set logging buffer <buffer_size> (buffer_size = 1..500) chesapeakebay> (enable)set logging buffer 500 System logging buffer size set to <500> chesapeakebay> (enable)set ntp server 172.16.1.1 NTP server 172.16.1.1 added. chesapeakebay> (enable)show ntp Current time: Tue Jul 30 2002, 10:48:28 Timezone: '', offset from UTC is 0 hours Summertime: '', disabled Last NTP update: Broadcast client mode: disabled Broadcast delay: 3000 microseconds Client mode: disabled NTP-Server ---------------------------------------- 172.16.1.1 |
Example 10-17. Document the kentnarrows Switch (3512XL IOS)
kentnarrows(config)#logging 172.16.1.42 kentnarrows(config)#logging trap informational kentnarrows(config)#logging buffer 500000 kentnarrows(config)#service timestamps debug datetime msec kentnarrows(config)#service timestamps log datetime msec kentnarrows(config)#ntp server 172.16.1.1 kentnarrows(config)#ntp source vlan1 kentnarrows(config)#end kentnarrows#show ntp status Clock is synchronized, stratum 9, reference is 172.16.1.1 nominal freq is 381.4697 Hz, actual freq is 381.4697 Hz, precision is 2**17 reference time is C0F0E9A4.9AD26EC9 (10:32:04.604 UTC Tue Jul 30 2002) clock offset is 4.1159 msec, root delay is 15.59 msec root dispersion is 7.81 msec, peer dispersion is 3.68 msec kentnarrows#copy running-config startup-config !!! the output of the 1900 is not shown for this trouble ticket |
Now that you have some common documentation tools configured, such as NTP, time stamps, logging, and syslog, you should continue to use these tools to assist you with supporting the network. Verify that NTP is working on duck in Example 10-18 and then be sure to save all your configurations.
Example 10-18. Viewing NTP on duck
duck#show ntp associations address ref clock st when poll reach delay offset disp *~127.127.7.1 127.127.7.1 7 15 64 377 0.0 0.00 0.0 * master (synced), # master (unsynced), + selected, - candidate, ~ configured duck#show ntp status Clock is synchronized, stratum 8, reference is 127.127.7.1 nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**19 reference time is C0F0EC36.D3096102 (10:43:02.824 UTC Tue Jul 30 2002) clock offset is 0.0000 msec, root delay is 0.00 msec root dispersion is 0.02 msec, peer dispersion is 0.02 msec |
Example 10-19 shows the copying of the configurations to the TFTP server that I have running on hosta. I did not bother with the 1900 or 804 because they would be just as easy to reconfigure as to download the configuration.
Example 10-19. Copying the Configurations to the TFTP Server
duck#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [tt2duck-confg]? !! 1936 bytes copied in 8.744 secs (242 bytes/sec) heron#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [tt2heron-confg]? !! 1963 bytes copied in 7.676 secs (280 bytes/sec) goose#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [goose-confg]? tt2goose-confg !! 2880 bytes copied in 1.664 secs (2880 bytes/sec) osprey#copy running-config tftp Remote host []? 172.16.1.42 Name of configuration file to write [osprey-confg]? tt2osprey-confg Write file tt2osprey-confg on host 172.16.1.42? [confirm] Building configuration... Writing tt2osprey-confg !! [OK] crab#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [crab-confg]? tt2crab-confg !! 2695 bytes copied in 10.64 secs (269 bytes/sec) swan#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [swan-confg]? tt2swan-confg !! 2333 bytes copied in 12.356 secs (194 bytes/sec) ferry#copy running-config tftp Address or name of remote host []? 172.16.1.42 Destination filename [ferry-confg]? tt2ferry-confg %Error opening tftp://172.16.1.42/tt2ferry-confg (Socket error) ferry#!!!ip is not running on the frame switch ferry#!!!use your terminal program to get a copy of that config chesapeakebay>enable Enter password: chesapeakebay> (enable)copy ? Usage: copy tftp flash copy flash tftp chesapeakebay> (enable)write ? Usage: write network write terminal write <host> <file> chesapeakebay> (enable)write network IP address or name of remote host? 172.16.1.42 Name of configuration file? tt2chesapeakebay-confg Upload configuration to tt2chesapeakebay-confg on 172.16.1.42 (y/n) [n]? y ... Finished network upload. (7861 bytes) chesapeakebay> (enable) kentnarrows#copy running-config tftp Source filename [running-config]? Destination IP address or hostname []? 172.16.1.42 Destination filename [running-config]? tt2kentnarrows-confg Building configuration... !! 1952 bytes copied in 0.322 secs |
My supporting TFTP files are included as well. Reference their names in Example 10-19.
Another task in this Trouble Ticket is the memory dump. Example 10-20 displays the output of the memory dump to the TFTP server, which takes some time if you plan on letting it finish.
Example 10-20. Memory Dump to TFTP
duck#ping 172.16.1.42 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.1.42, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 16/16/20 ms duck#write core Remote host [0.0.0.0]? 172.16.1.42 Base name of core files to write [duck-core]? writing uncompressed tftp://172.16.1.42/duck-core !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ... writing uncompressed tftp://172.16.1.42/duck-coreiomem |
The core dump created two files on the TFTP server for the duck router:
duck-core
duck-coreiomem
NOTE
Cisco is moving away from relying on writing an exception dump to an external TFTP/FTP server. Whenever there is sufficient space in boot flash or on a Flash PC card, a crashinfo file is usually written. This includes the output of show stack and show tech-support, as well as the most recent exec mode and config mode commands.
You should also create a table to document your hosts as in Table 10-1. If you are the server/workstation group rather than the network group, your table will probably look a lot more detailed than what this provides. The point here is that troubleshooting is end to end. Normally this means from host to host with any switches and routers in between. The main tools on Windows hosts include ipconfig/winipcfg and the network property sheet. Ensure you can ping, trace, and telnet to every device from each host before you proceed to the other Trouble Tickets.
Documentation and baselining are extremely important but are very time-consuming. In a practical environment, it many times gets put off until later, and later never comes. However, supporting the LAN and WAN is much easier when you have done your homework up front.
NOTE
You can never have too much documentation. Now that you have it, consider experimenting a bit. If you really want to verify your configurations, erase them all on your devices. Be brave and reload them, too. Then set up your devices for basic IP connectivity to the TFTP server and download your previously saved configurations.
Now that you are quite familiar with the scenario through discovering or configuring it, supporting it, and documenting it, be prepared for some more challenging Trouble Tickets. The name of the Trouble Ticket obviously gives you a hint of some (not all) of the issues. You need to “spot the issues” on your own instead of following my lead page by page, step by step. Issue lists are provided after each Trouble Ticket so that you can make sure you at least found the issues I intended.