Switches, like routers, contain a CPU. Unlike routers, however, a switch performs most of the packet forwarding without impacting the CPU. Switching decisions are performed on the switch ASICs and, depending on the switch type, the bridging table may be stored on an ASIC as well.

Some of the operations performed by the switch's CPU include spanning tree, Telnet services, Cisco Discovery Protocol (CDP), security (such as Terminal Access Controller Access Control System [TACACS]), remote monitoring (RMON), VLAN Trunk Protocol (VTP), port aggregation, dynamic VLANs, and SNMP processing.

As such, measuring the CPU of a switch is of little importance when determining the switch's packet-forwarding performance of the switch.

Bridges/switches keep track of the ports from which they have received certain Media Access Control (MAC) addresses to isolate unicast conversations to the ports involved. The alternative is to forward all traffic, unicast and broadcast, to all switched ports, which would be excessive. This feature is traditionally done using a forwarding database (sometimes called a bridging table) and content-addressable memory (CAM) tables.

The forwarding database is usually a single centralized table or database on a bridge that contains all the MAC addresses (and their ports) the bridge has learned. It is used to forward packets to the correct ports.

The content of the CAM can be accessed using the contents rather than an address. A CAM typically exists at each physical port. In non-promiscuous mode (meaning not bridging), the CAM contains the burned-in MAC address of the network interface card (NIC), and is programmed with other addresses (like multicasts/broadcast) to which it needs to listen. Any packet that fails the CAM lookup is dropped by the NIC.

Bridges operate in promiscuous mode, which means the CAM on each port is cleared, so all MAC addresses are accepted. As the bridging table is built, the CAM is populated with destination MAC addresses of stations generating traffic on that port. In promiscuous mode, packets received on a port that have a destination MAC address that matches an entry in the CAM for that port are ignored. If MAC addresses are stored in the CAM, the MAC address is used to look up the entry, rather than a sequential search, because sequential searches are much slower. This makes the CAM quicker than the forwarding database. Many NICs have a CAM, but the Catalyst 5000 does not have CAM on Ethernet ports.

On the Catalyst 5000, the term "CAM" is used for historical reasons; no actual CAM is present. The Catalyst 5000 uses a central Enhanced Address Recognition Logic (EARL) for speed and additional functionality by combining the VLAN ID with the MAC address in the learning and lookup functions performed by the Catalyst 5000 hardware. The EARL combines the functions of the forwarding database (central complete list of MACs, VLANs, and ports) and the CAM (fast lookup using the contents of memory rather than a search). The MAC addresses are stored on an SRAM chip on the supervisor engine and do not impact memory or processor usage on the network-management processor (NMP). There is enough SRAM on the chip to store 16,000 MAC addresses.

Generally, the path of a packet through a switch is as follows:

All of these operations are performed without impacting the switch's CPU at all.

When bridging, Catalyst switches perform both transparent and translational bridging. Transparent bridging is the process of forwarding traffic from one port (such as Ethernet) to another common media port (such as another Ethernet port). The packet will not be modified in any form as it is taken from the receiving port and passed on to the destination port.

Translational bridging goes one step further by bridging traffic between two different media, such as from Ethernet to Fiber Distributed Data Interface (FDDI). Translational bridging requires a switch to take a received packet and "translate" it to a different media. This entails, at a minimum, the replacement of one topology's part of a packet with another. Translational switching requires extra processing time because a packet must be received, analyzed, and translated before sending the packet on to its final destination.

A VLAN is an administratively defined broadcast domain that can span multiple switches. Only end stations within the VLAN receive packets that are unicast, broadcast, or multicast (flooded). A VLAN enhances performance by limiting traffic; it allows the transmission of traffic among stations that belong to it and blocks traffic from other stations in other VLANs. VLANs can provide security barriers (firewalls) between end stations on different VLANs within the same switch.

In addition, some of the VLAN components supported on the Catalyst series include the following:

When creating fault-tolerant networks, a loop-free path must exist between all nodes in the network. A spanning-tree algorithm is used to calculate the best loop-free path through a Catalyst-switched network. Spanning-tree packets are sent and received by switches in the network at regular intervals. The packets are not forwarded by the switches participating in the spanning tree, but are instead processed to determine the spanning-tree itself. The IEEE 802.1D bridge protocol, called STP, performs this function for Catalyst switches.

The Catalyst series switches can use STP on all VLANs. The STP detects and breaks loops by placing some connections in a stand-by mode, which are activated in the event of a failure. A separate STP runs within each configured VLAN, ensuring valid Layer 2 topologies throughout the network.

The supported STP states are as follows:

The state for each port is initially set by the configuration and later modified by the STP process. After the port state is set, the 802.1D bridge specification (RFC 1493) determines whether the port forwards or blocks packets.

The SPAN feature enables you to monitor traffic on any port for analysis by a network-analyzer device or RMON probe. Enhanced SPAN ( E-SPAN) enables you to monitor traffic from multiple ports with the same VLAN.

SPAN redirects traffic from an Ethernet, Fast Ethernet, FDDI port, or VLAN to an Ethernet or Fast Ethernet monitor port for analysis and troubleshooting. You can monitor a single port or VLAN using a dedicated analyzer such as a Network General Sniffer, or an RMON probe such as a Cisco SwitchProbe.

CDP is media and protocol independent and runs on all Cisco-manufactured equipment, including routers, bridges, access and communication servers, and switches. With CDP, network-management applications can retrieve the device type and SNMP agent address of neighboring devices. This allows applications to send SNMP queries to neighboring devices.

CDP meets a need created by the existence of lower-level, virtually transparent protocols. CDP allows network-management applications to discover Cisco devices that are neighbors of already known devices, in particular neighbors running lower-layer, transparent protocols. CDP runs on all media that support the Subnetwork Access Protocol (SNAP), including LAN and Frame Relay. CDP runs over the data link layer only, not the network layer. With CDP, two systems that support different network layer protocols can learn about each other.

Cached CDP information is available to network-management applications. Cisco devices never forward a CDP packet. When new information is received, old information is discarded. CiscoWorks for Switched Internetworks (CWSI) uses this information during network discovery.

CDP is also very useful in troubleshooting. In a network of routers only, the Address Resolution Protocol (ARP) tables, routing tables, and other information are used to discover the topology, or to confirm the connectivity of the topology that is already known. In a network of bridges, these tables aren't used; instead, a forwarding database (which is not useful to discover bridges; it's for end stations) and spanning-tree information (which may be disabled, and gives information upstream toward the root only) are used. CDP will discover all Cisco devices that are neighbors and provide information such as host name, Cisco IOS version, and management IP address.

See the CCO Web site for information on how to configure CDP in a Catalyst 5000.

RFC 1757 defines a portion of the Management Information Base (MIB) for monitoring data link layer information of remote segments, in particular, traffic characteristics and error rates.

Remote network-monitoring devices, often called monitors or probes, are instruments that exist for the purpose of managing a network. Often these remote probes are standalone devices and devote significant internal resources for the sole purpose of managing a network. An organization may employ many of these devices, one per network segment, to manage its Internet. In addition, these devices may be used for a network-management service provider to access a client network, often geographically remote.

The objects defined in the RFC are intended as an interface between an RMON agent and an RMON management application and are not intended for direct manipulation by users.

Although some users may tolerate the direct display of some of these objects, few will tolerate the complexity of manually manipulating objects to accomplish row creation. These functions should be handled by the management application. Cisco provides applications to perform these functions.

RMON, as described in RFC 1757, is implemented on Cisco Catalyst products and routers as follows:

Telnet (also known as CLI) allows direct login into a switch or router to have access to configuration and monitoring commands.

As defined in RFC 1157, the Simple Network Management Protocol (SNMP) is based on the concept of a SNMP Manager communicating with one or more SNMP agents, using the SNMP. SNMP Get, Get-Next, and Set operations are performed by the SNMP Manager to the agent to either retrieve or set management variables supported by the SNMP agent. Management information is usually equally available from SNMP and through Telnet. SNMP agents can notify SNMP managers by issuing SNMP traps. Traps can contain any number of management information to better qualify the trap.

Based on the SNMP technology, the remote network monitoring capabilities are supported by a dedicated RMON (remote monitoring) data-collection and monitoring engine residing in a device. Communicating—for example, setting collection rules and receiving notifications from the engine—is performed through SNMP. RMON addresses ISO Layers 1 through 3, while RMON2 addresses ISO Layers 4 and above.

The syslog protocol is used by Cisco devices to issue unsolicited notifications to a management station. Although similar in nature to SNMP traps, the syslog protocol is only used for event notification. The CISCO-SYSLOG-MIB is being implemented on Cisco devices as an alternative to issuing syslog messages, to allow any SNMP manager to receive all events through SNMP.

All Cisco devices generate SNMP traps to notify NMS applications of activity and error conditions. In addition, IOS-based devices generate syslog messages.

  snmp-server enable traps syslog

  logging history level

Event processing ischaracterized by the following four activities.

This section introduces a model to manage a high volume of traps and syslog messages and correlate them to your network; so now the information becomes practical rather than overwhelming.

Theoretical Event Model

The following proposed event model is a conceptual model for correlating Cisco events. It assumes syslog messages as well as SNMP traps generated either directly by the device or as a result of SNMP thresholding through the RMON alarm and event groups. All these event types serve as input into the event model.

Events are first filtered to eliminate as many events deemed of no interest as possible, as driven from a knowledge database. This operation minimizes the processing required in the subsequent phases of the event model.

Events of interest are then normalized to a common form to ease the processing in the event-correlation engine. Normalization may also involve modifying the severity of an event to more accurately reflect the priority of an event at a specific site.

The event-correlation engine is composed of several correlation conditions that may eliminate events, pass them straight through at all times, only pass them straight through if one or more other conditionsoccur within a specific time window, modify events, or create new events. Each condition is specific to a device, a type of device, or to all devices depending on the scope of the correlation rules. Correlation rules can make use of external information such as physical or logical topology to better isolate a faulty device. Correlation rules can also issue additional requests to one or more devices if they need to obtain additional information at any point in a correlation rule.

Figure 14-1 illustrates the conceptual model.

Although Figure 14-1 shows CLI commands to communicate with a device, this option is usually not needed because syslog messages and SNMP MIBs provide all information needed to manage these devices.

Figure 14-1. Conceptual Cisco Event Model

The simplestway to monitor device availability is to check responses to a ping (ICMP) or SNMP get request. Standard NMS products monitor all the objects they manage through these simple mechanisms. Although a ping response is not a guarantee that the switch is functioning properly, the lack of such a response is a definite sign of a problem. Using an NMS with availability monitoring linked to the color of a device on a topology map is the most common example of fault monitoring.

Configure routers and switches to send their console messages to a syslog server, typically being the NMS. A Catalyst 5000 series switch can buffer the last n syslog console messages it generates by saving them in a 1-KB buffer. Although these are useful for quick reference from the command-line interface (CLI), it is recommended to log these messages to a syslog server for persistence, later reference, and correlation.

Log messages are system messages that would normally be sent out on the console port; for example: "11/4/1996,13:52:54:SYS-5: Module 3 failed configuration". Cisco routers and switches generate many different kinds of system messages.

Switch syslog messages are formatted with two parameters: the facility and severity. You must choose one of eight facilities (local0–local7), and then choose the facility based on the facilities your syslog server is already logging. It is best to choose a facility that will be dedicated to your network devices. Cisco routers choose local7 by default, which is a good choice unless there is a conflict.

You can configure the severity level of each type of message you want the switch to send. You should log a severity level of "warning" and below of each type of system message so that you do not clog up the logs with superfluous messages that do not add useful information.

Although syslog information can be used for fault management, it is not a real-time fault alert mechanism like an SNMP trap (unless you do additional work to make it behave so). Syslog information is useful for event correlation, however. Parsing the syslog files for messages correlating to a certain event can provide useful insight to the cause of the problem. You can configure the syslog server to log messages of different severities to different log files or messages of all severities to one large file. You can then use tools such as Perl to search the syslog log files for the desired information. Tools such as Cisco Resource Manager (CRM) can gather syslog information and create reports based on severity and dates. Such tools can also summarize the syslog messages to make interpreting the messages easier.

SNMP traps provide a mechanism for a device to unilaterally notify a network operator of a certain condition in real time. SNMP traps are unsolicited notifications, independent of any SNMP polling activities. The SNMP traps generated by Cisco switches and routers provide useful information on potentially harmful environmental conditions, processor status, and port status. Every device will also generate SNMP traps based on the features it supports. For example, a Cisco switch will generate SNMP spanning-tree topology changes SNMP traps when a spanning-tree configuration changes because of a trunk or switch being added or removed from the network.

Initially, it is recommended to enable the module, chassis, bridge, authentication, and port linkup/down traps. The port-level traps require enabling the port trap on the "critical" ports with this command:

  set port trap module|port enable|disable

Many NMS products have trap receivers that log traps and can be configured to react to traps in fairly sophisticated ways. You can be informed of the trap through a pop-up window, an audible alarm, or an electronic page.

To generate useful alarms you have to define sensible thresholds. Unfortunately, this is not always easy to do. In general, you should set thresholds on the data you have baselined. Set one threshold that will alert you when conditions are problematic and the opposite threshold to alert you when conditions have returned to normal.

Here is a list of guidelines for identifying appropriate thresholds:

Root bridges should be statically defined in the core distribution switches when using spanning tree. Spanning tree should be activated on all trunk ports. Optionally, you may want to activate spanning tree on end-user switch ports as well, in case a loop is mistakenly created in the network. Such a small precaution can prevent your network from grinding to a halt when a loop is created.

You can obtain the same data from a switch in a variety of ways. The method you use depends on many factors:

For very small networks, the simple CLI may suffice. For larger, more complex networks, an NMS with SNMP pollers, SNMP trap processors, and a relational database may be necessary. For most situations, it will end up being a combination of both along with some Perl and Expect scripts to fill in the gaps. The following subsections cover what data to collect, how to get that data, and then how to turn that data into useful information. The following sections describe the data you can obtain from CLI commands and specific SNMP MIB objects.

This section also covers basic fault management through SNMP traps and syslog messages.

The aim of monitor polling is to detect when network changes have occurred and to generate an immediate alarm. Monitor polling is to detect "hard errors" such as a device not responding or an interface status change. Any alarm generated by monitor polling should be acted on immediately. If your network management system is monitored on a 24×7 basis, a monitor polling alarm should create a visible and audible alarm on the system, so that the operator can take immediate action. If you do not have your network management system monitored on a 24×7 basis, it is recommended that any monitor polling alarm generate an alarm to the appropriate person, possibly via pager or e-mail.

Monitor polling applications are included with most commercial SNMP platforms and SNMP-based management applications. The purpose of monitor polling applications is to analyze the data points and generate an alarm when necessary.

The aim of threshold polling is to determine when escalating error conditions are occurring and take actions before performance is severely impacted. A wide variety of problems surface originally as increased error conditions. These conditions may eventually become a "hard error," or may present themselves as phantom problems that come and go. Threshold polling is a tool to detect these problems.

To implement threshold polling, the first decision is to determine which MIB variables to poll for (MIB variables for switches follow in this section). Users who are starting out with threshold polling may start with the suggested variables that follow, and then add any other applicable MIB variables that are appropriate for their network.

After the threshold MIB variables are decided upon, you must establish baseline values for these variables. This can be done by starting a poll process on the MIB variables for a period of time (1 week, for example). You must then review the data collected. Baselining should be performed during peak traffic patterns to be most representative of unusual conditions. The more static your network is (yearly adds, moves, and changes), the less baselining you need to perform, while the more dynamic your network is (monthly adds, moves, and changes), the more baselining you need to perform. With this data in hand, determine what values would be out of specification. A rule of thumb is to set thresholds 10%–20% higher than the maximum value seen.

The threshold values for any particular MIB variable may be applied uniformly across all switches, or they could be customized for groups of switches that have similar characteristics (for example, core, distribution).

Another decision to be made is what kind of notification is appropriate for threshold violations. Threshold violations are not indications of hard errors, so immediate notification is usually not in order. Logging all threshold values and reviewing on a daily basis is usually the most appropriate means of notification. It is very important that repeated threshold violations be investigated. It can then be determined if a problem has occurred that can be corrected, or if the violations are a result of the threshold values being too low for the circumstances.

The aim of performance polling is to gather data that can be analyzed over time to determine trends and to aid in capacity planning. As with threshold polling, the determination of what MIB variables to poll for is the first consideration. Later in this section, some suggestions are made on which MIB variables would be most useful for performance polling of switches.

For performance polling, individual data points (raw data) are stored intermittently on the polling machine. Depending on what polling mechanism you are using, the data could be in either a raw format or in a relational database.

To best keep the data manageable, the raw data should periodically be aggregated, and stored in another database or file to be kept for future reporting. The raw data can be kept for a period of time for backup purposes, but eventually this data should be purged and only the aggregate data is kept. The last step in the process is to produce reports from the aggregate data that will be periodically reviewed for trending or capacity planning purposes.

To best describe this process, here's an example of a company's performance polling system:

For more detailed information on performance monitoring, see the IBM reference book (IBM Red Books) titled Monitoring Performance In Router Networks, Document GG24-4157-00.

As mentioned previously, most Catalyst switches support "mini- RMON," which consists of four basic RMON-1 groups: statistics, history, alarms, and events. For baselining purposes, the etherStats group provides a useful range of Layer 2 traffic statistics. You can use the objects in Table 14-3 to get statistics on unicast, multicast, and broadcast traffic as well as a variety of Layer 2 errors. The RMON agent on the switch can be configured to store these samples in the history group. This mechanism enables you to reduce the amount of polling without reducing the sample rate. Using RMON histories can give you more accurate baselines without substantial polling overhead. The more histories you collect, however, the more switch resources you use.

The most powerful part of RMON-1 is the thresholding mechanism provided by the alarm and event groups. RMON thresholding enables you to configure the switch to send an SNMP trap informing you of an anomalous condition. Now that you have identified all your critical ports and used SNMP polling (and maybe RMON histories) to create baselines showing normal traffic activity for those ports, you are ready to set RMON thresholds. Set the thresholds to generate an alarm when there is a large variance from the baseline for that particular port. Then, set the thresholds to notify you when the traffic returns to baseline levels for that port.

The setting of these thresholds is best done using an RMON management package. Properly creating the rows in the alarm and event tables is tedious and complex. Commercial RMON NMS packages such as TrafficDirector incorporate graphical user interfaces (GUIs), which make the setting of RMON thresholds much easier.

Although switches provide only four basic groups of RMON-1, it is important to not forget the rest of RMON-1 and RMON-2. You can get Layer 3 and higher information on your switches using the SPAN port feature and an external RMON probe such as a Cisco SwitchProbe. A SwitchProbe supports RMON-2 and can take a feed from a SPAN port to monitor full RMON data on any given port or a whole VLAN on a particular switch. You can use a SPAN port and a SwitchProbe to capture a packet stream for a particular port (using the packet capture group of RMON-1) and upload the packets for decoding to an RMON management package. You can also use the SPAN port and an external SwitchProbe to give network and application layer statistics on a particular port or VLAN. The SPAN port is SNMP-controllable via the SPAN group in the CISCO-STACK-MIB, so this process is easy to automate. TrafficDirector makes use of these features with its "roving agent" feature.

There are caveats to spanning a whole VLAN. Even if you use a 100-Mbps probe, the entire packet stream from one VLAN or even one 100-Mbps full-duplex port may exceed the bandwidth of the SPAN port. Use care to see that you are not overloading the SPAN port. If the SPAN port is running at full bandwidth continuously, chances are you are losing data.

It is important to remember that the primary function of a switch is to switch frames—not to act as a large multiport RMON probe. Therefore, as you are setting up histories and thresholds on multiple ports for every condition imaginable, keep in mind that you are stealing resources the switch could otherwise use for its primary purpose—forwarding network traffic. Also remember the critical port rule: Only poll and set thresholds on the ports which you identified as critical.

RMON memory usage is constant across all switch platforms relating to statistics, histories, alarms, and events. RMON uses what is called a "bucket" to store histories and statistics on the RMON agent (which is the switch in this case). The bucket size is defined on the RMON probe (SwitchProbe) or RMON application (TrafficDirector), and then sent to the switch to be set.

Configuring syslog on the Catalyst 5000 series switches consumes 1 KB of memory (supported in software version 2.2 and higher) to save the last syslog messages generated.

Some standard MIBs assume that a particular SNMP entity contains only one instance of the MIB. Therefore, the standard MIB does not have any index that would enable users to directly access a particular instance of the MIB. In these cases, we provide community string indexing to access each instance of the standard MIB. The syntax is community string@instance number.

The Catalyst switch includes one instance of the standard BRIDGE-MIB for each VLAN in the switch, for example. If the read-only community string is "public" and the read-write community string is "private," you could use public@25 to read the BRIDGE-MIB for VLAN 25 and use private@33 to read and write the BRIDGE-MIB for VLAN 33. Only using the community string public or private will result in always accessing the BRIDGE-MIB for VLAN 1 (default behavior).

Traps sent from a MIB indexed by a community string also indicate the instance of the MIB to which it corresponds by using community string indexing. An STP newRoot trap from the BRIDGE-MIB for VLAN 25, for example, would have a community string ofpublic@25 in the trap community field, assuming the read-only community string is public.

Also note that community string indexing does not affect access to MIBs that have only one instance. Therefore, public@25 can be used to access RFC1213-MIB at the same time as the BRIDGE-MIB for VLAN 25 is being accessed.

Another example for the Catalyst switch is the SNMP-REPEATER-MIB. To access this MIB for a particular repeater in the Catalyst switch, use community string@module number/port number. If the read-only community string is public, for example, you can use public@3/1 to read the SNMP-REPEATER-MIB for the repeater attached to port 1 on module 3.

To determine the available VLANs on a given Catalyst switch, you must query the vlanTable MIB group from the CISCO-STACK-MIB using the VLAN ID as index.

Starting with Cat5xxx Release 4.1(1), the newRoot and topologyChange traps defined in the BRIDGE-MIB have the vlanIndex appended in their varBind list. This eliminates the need for SNMP platforms and applications to decode the community string within the trap. Here are two examples of the new traps supported:

  Received SNMPv1 Trap:
  Community: public@2
  Enterprise: dot1dBridge
  Agent-addr: 172.10.17.31
  Enterprise Specific trap.
  Enterprise Specific trap: 2    (This is a topologyChange trap)
  Time Ticks: 27654316
  vtpVlanIndex.1.2 = 2           (This is the VLAN number)
  ifName.97 = 4/2                (This is the interface index)

  Received SNMPv1 Trap:
  Community: public@3
  Enterprise: dot1dBridge
  Agent-addr: 172.10.17.31
  Enterprise Specific trap.
  Enterprise Specific trap: 1    (This is a newRoot trap)
  Time Ticks: 27651818
  vtpVlanIndex.1.3 = 3           (This is the VLAN number)

For a summary of community string indexing, see the CCO Web site.

Onefrustrating issue is trying to decipher which port created a certain trap. Knowing that a trap came from ifIndex 100 does not help pinpoint the source of the trap, but knowing that the trap came from port 1/1 is very useful. We use the IF-MIB to map ifIndex to a port number. If you poll the ifXTable for ifName, you can get a correspondence between ifIndex and the actual port. If you received a trap for ifIndex 17, for example, you could poll ifName to find the port, as shown here:

  $ snmpwalk robotron ifName

  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.1 : DISPLAY STRING-   (ascii):  sc0
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.2 : DISPLAY STRING-   (ascii):  sl0
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.3 : DISPLAY STRING-   (ascii):  1/1
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.4 : DISPLAY STRING-   (ascii):  1/2
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.5 : DISPLAY STRING-   (ascii):  2/1
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.6 : DISPLAY STRING-   (ascii):  2/2
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.7 : DISPLAY STRING-   (ascii):  2/3
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.8 : DISPLAY STRING-   (ascii):  2/4
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.9 : DISPLAY STRING-   (ascii):  2/5
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.10 : DISPLAY STRING-   (ascii):  2/6
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.11 : DISPLAY STRING-   (ascii):  2/7
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.12 : DISPLAY STRING-   (ascii):  2/8
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.13 : DISPLAY STRING-   (ascii):  2/9
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.14 : DISPLAY STRING-   (ascii):  2/10
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.15 : DISPLAY STRING-   (ascii):  2/11
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.16 : DISPLAY STRING-   (ascii):  2/12
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.17 : DISPLAY STRING-   (ascii):  2/13
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.18 : DISPLAY STRING-   (ascii):  2/14
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.19 : DISPLAY STRING-   (ascii):  2/15
  ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName.20 : DISPLAY STRING-   (ascii):  2/16

The output of this indicates that port 2/13 generated the trap.

Use the set snmp commands to set all SNMP parameters: community strings, RMON, and other SNMP traps. Table 14-4 explains these commands.

Details of all these set commands for Release 4.2 can be found on the Cisco Web site.

The common tools for fault management include logging SNMP traps (which include the RMON threshold traps) and syslog messages, and then processing the data and reacting appropriately. Most NMS packages provide a trap daemon that receives and logs SNMP traps, and then provides some mechanism for reacting to that trap. Some typical mechanisms used are pop-up messages and audible alarms on the NMS station or the execution of a paging script. More complex reactions may be to execute an Expect script that collects time-critical data to determine the state of the switch immediately following the occurrence of a fault, or perhaps trigger a packet capture from the appropriate port using an external RMON probe. Several such scenarios are described later in this chapter.

Regarding syslog messages, most (if not all) UNIX systems provide a syslog daemon. Commercial and public domain syslog daemons are also available for Win95 and WinNT systems. These daemons usually log all switch messages into a particular file. Then, a Perl script or some other reporting tool can be used to parse the log file into usable information. Cisco Resource Manager has a Syslog Analyzer, for example, which creates reports from syslog messages based on message severity levels or based on the devices generating the messages.

Most commercial NMS products understand standard basic SNMP traps (such as linkUp and linkDown traps) and RMON thresholds. But for Cisco-specific traps, you must configure the NMS to format the traps so that they are readable and recognizable. The CCO Web site contains information on formatting traps for HP OpenView Network Node Manager and Tivoli Net View in the trapd.conf file.

Cisco-specific MIBs can be found in Cisco MIBs, such as those listed in Table 14-5.

The following variables are useful in determining the backplane utilization of the switch. The same counters are used when displaying the red traffic level indicators on the front of the supervisor card. However, these variables should not be confused with CPU or memory utilization.

The following information is available at the CCO Web site:

Use the commands described in this section as an additional method of gathering switch resource data.

  switch 5000 (enable) show biga
  BIGA Registers:
      cstat:       00  upad :     FFFF   pctrl :     0000   nist :     0000
      sist :     0018  hica :     0000   hicb  :     0000   hicc :       00
      dctrl:     F5FF  dstat:     0000   dctrl2:       80   npim :     00F8
      thead: 101F196C  ttail: 101F196C   ttmph : 101F196C   tptr : 104347E2
      tdsc : 00000500  tlen :     0000   tqsel :       05
      rhead: 101F1220  rtail: 101F1204   rtmph : 101F123C   rptr : 10586C80
      rdsc : 804D0000  rplen: 101F1234   rtlen : 00000000   rlen :     1600
      fltr :     00FF  fc   :       00   Rev   :       04   CFG  : 02020202
  BIGA Driver:
      Initializd:     TRUE  SpurusIntr: 00000000  NPIMShadow:     00F8

  BIGA Receive:
      RxDone    :    FALSE
      First RBD : 101EF894  Last  RBD : 101F1478
      SoftRHead : 101F1210  SoftRTail : 101F11F4
      FramesRcvd: 04572393  BytesRcvd : 589914384
      QueuedRBDs: 00000256  RsrcErrors: 00000000

  BIGA Transmit:
      First TBD : 101F1494  Last  TBD : 101F1B78
      SoftTHead : 101F19D4  SoftTTail : 101F19D4
      Free TBDs : 00000064  No TBDs   : 00000000
      AcknowErrs: 00000000  HardErrors: 00000000
      QueuedPkts: 00000000  XmittedPkt: 11353833
      XmittedByt: 909016542 Panic     : 00000000
      Frag<=4Byt: 00000306

  switch 5000 (enable) show inband
  Inband Driver:
  DriverPtr:  A0559F20    Initializd:     TRUE  SpurusIntr: 00000000
      RxDone:        FALSE  TxDMAWorking:  FALSE  RxRecovPtr: 00000000(-1)
      FPGACntl:      004F  Characteristics:0000  LastISRCause:     04

      Transmit:
       First TBD : A055E7A4(0  )  Last  TBD : A055F784(0  )
       TxHead    : A055F5A4(112)  TxTail    : A055F5A4(112)
       AvailTBDs : 00000128       QueuedPkts: 00000000
       XmittedPkt: 07626990       XmittedByt: 581073462
       PanicEnd  : 00000000       PanicNullP: 00000000
       BufLenErrs: 00000000       Len0Errs  : 00000000
       Frag<=4Byt: 00000162       SpursTxInt: 00000000
       No TBDs   : 00000000       NullMbuf  : 00000000

      Receive:
       First RBD : A0559FA4(0  )  Last  RBD : A055E780(511)
       RxHead    : A055AE44(104)  RxTail    : A055AE20(103)
       AvailRBD  : 00000512       RsrcErrors: 00000824
       PanicNullP: 00000000       PanicFakeI: 00000000
       FramesRcvd: 18507368       BytesRcvd : 1676456769
       RuntsRcvd : 00000000       HugeRcvd  : 00000000

  GT64010 IntMask: F00F0000  IntCause: 0330E083
  GT64010 TX DMA (CH 1):
      Count:       0000  Src  :     0134B062   Dst   :     4ff10056   NRP  :     0
  0000000
      Cntl :       15C0
  GT64010 RX DMA (CH 2):
      Count:       0680  Src  :     4FF20000   Dst   :     01c3e580   NRP  :     0
  0558590
      Cntl :       55C0

  PSI (PCI SAGE/PHOENIX Interface) FPGA:
      Control : 004F  TxCount : 0056
      RxDMACmd: 35C0  RxBufSiz: 0680  MaxPkt  : 0680
      IntCause: 0002  IntMask : 0003

  switch 5000 (enable) show mbuf
  MBSTATS:
          mbufs                   10224   clusters        3932
          free mbufs              9946    clfree          3675
          lowest free mbufs       9935    lowest clfree   3665

  MALLOC STATS :
  Block Size       Free Blocks
    16             1
    48             2
    112            1
    144            1
    208            1
    240            1
    400            1
    496            4
 Largest block available : 7510096
 Total Memory available  : 7546400
 Total Memory used       : 563952

  switch 5000 (enable) ps -c
  CPU usage information:
  Name             CPU-Usage     Invokations
  ---------------  ------------- -------------
  Kernel           93%           1
  SynDiags         0%            1
  SynConfig        0%            1
  Earl             1%            1
  THREAD           0%            1
  Console          0%            1
  telnetd          0%            1
  cdpd             0%            1
  cdpdtimer        0%            1
  SptTimer         0%            1
  SptBpduRx        0%            1
  VtpTimer         0%            1
  VtpRx            0%            1
  DISL_Rx          0%            1
  DISL_Timer       0%            1
  sptHelper        0%            1
  ..etc
  ..etc
  System Idle - Current: 41% High: 51% Low: 8% Average: 47%

  switch 5000 show log
  Network Management Processor (ACTIVE NMP) Log:
    Reset count:   100
    Re-boot History:   Mar 19 1998 16:06:10 3, Mar 11 1998 12:03:03 3
                       Mar 10 1998 04:34:52 3, Mar 08 1998 08:38:30 3
                       Mar 08 1998 08:09:51 3, Mar 08 1998 06:28:31 3
                       Mar 08 1998 05:33:23 3, Mar 02 1998 09:02:13 3
                       Feb 20 1998 05:02:35 3, Feb 20 1998 04:55:45 3
    Bootrom Checksum Failures:      0   UART Failures:                   0
    Flash Checksum Failures:        0   Flash Program Failures:          0
    Power Supply 1 Failures:       39   Power Supply 2 Failures:         1
    DRAM Failures:                  0

    Exceptions:                     9
      Last Exception occurred on Feb 18 1998 17:14:18 ...
      Software version = 2.3(1)
      Error Msg:
      PID = 0 Co_\__Ô_?
      PC: 1015A3AC, Status: 2009, Vector: 007C
      sp+00: 20091015 A3AC007C 00000000 00000001
      sp+10: 00400000 AAAA0000 107F0008 1025EEC0
      sp+20: 107FFFAC 1017563E 00000000 107FFFE8
      sp+30: 10175EA0 00000000 000006C7 00000000
      sp+40: 00000000 00000000 00000000 00000000 
      sp+50: 00000000 00000000 50000200 00000007
      sp+60: 68000000 00000000 00000000 00000000
      sp+70: 00000000 00000000 00000000 00000000
      sp+80: 00000000 00000000 00000000 00000000
      sp+90: 00000000 00000000 00000000 00000000
      sp+A0: 00000000 00000000 00000000 00000000
      sp+B0: 00000000 00000000 00000000 00000000
      sp+C0: 00000000 00000000 00000000 00000000
      sp+D0: 00000000 00000000 00000000 00000000
      sp+E0: 00000000 00000000 00000000 00000000
      sp+F0: 00000000 4937F8E7 00000000 00000000
      D0: 00000003, D1: 00000010, D2: 00000000, D3: 00000001
      D4: 0040B5C1, D5: AAAAF0E7, D6: 00000003, D7: 10800000
      A0: 68000000, A1: 00000079, A2: 50000200, A3: 50000200 
      A4: 103FFFFC, A5: 64000000, A6: 107FFFA0, sp: 107FFF80

  NVRAM log:

  01. 12/29/97,07:05:17:convert_post_SAC_CiscoMIB:Nvram block 0  unconvertable: 2(1)
  02. 12/29/97,07:05:17:convert_post_SAC_CiscoMIB:Nvram block 1  unconvertable: 1(0)
  03. 12/29/97,07:05:17:convert_post_SAC_CiscoMIB:Nvram block 5  unconvertable: 1(0)
  04. 12/29/97,07:05:17: check_block_and_log:Block 59 has been  deallocated: (0x500
  191D8)
  05. 12/29/97,07:05:17: convert_post_SAC_CiscoMIB:Nvram block 61  unconvertable:
   1(0)

  Module 2 Log:
    Reset Count:   2
    Reset History: Thu Mar 19 1998, 16:09:04
                   Wed Mar 11 1998, 12:05:57

    FCP Flash Checksum Failures:     0  DMP Flash Checksum Failures:     0
    FCP Flash Program Failures:      0  DMP Flash Program Failures:      0
    FCP DRAM Failures:               0  DMP DRAM Failures:               0
    FCP SRAM Failures:               0  DMP SRAM Failures:               0
    FCP Exceptions:                  0  DMP Exceptions:                  0
    Path Test Failures:              0

  Module 3 Log:
    Reset Count:   2
    Reset History: Thu Mar 19 1998, 16:08:18
                   Wed Mar 11 1998, 12:05:11


  Module 4 Log:
    Reset Count:   1
    Reset History: Mon Mar 23 1998, 10:50:06


  Module 5 Log:
    Reset Count:   2
    Reset History: Thu Mar 19 1998, 16:08:18
                   Wed Mar 11 1998, 12:05:11

The SNMP MIB objects in this list should be polled if the chassisAlarmOn trap is received.

The following information is available at the CCO Web site:

What is a minor and major chassis alarm? When the system LED status turns to red, a chassisMajorAlarm is generated. When the system LED status turns orange, a chassisMinorAlarm is generated. The trap generated will be a chassisAlarmOn trap. Included with the traps are variables that indicate whether the trap is from a chassisTempAlarm, a chassisMinorAlarm, or a chassisMajorAlarm. Decoding the trap indicates what kind of alarm generated the trap.

The following conditions cause a major alarm:

The following conditions cause a minor alarm:

With either a minor or major alarm, the system status LED on the front panel turns red. This information applies to the Catalyst 5000 series switches. Other products that use the CISCO-STACK-MIB have different definitions of major and minor alarms.

This section contains samples of the output from the CLI commands that provide chassis and environmental status.

  switch 5000 (enable) show system
  PS1-Status PS2-Status Fan-Status Temp-Alarm Sys-Status Uptime d,h:m:s Logout
  ---------- ---------- ---------- ---------- ---------- -------------- ---------
  ok         none       ok         off        ok         14,21:20:38     20 min
  PS1-Type   PS2-Type   Modem   Baud Traffic Peak Peak-Time
  ---------- ---------- ------- ----- ------- ---- -------------------- -----
  WS-C5008A  none       disable 9600  0%      0%   Wed Oct 22 1997,  14:17:56
  System Name              System Location          System Contact
  ------------------------ ------------------------ ------------------- -----
  switch 5000

  switch 5000 show test 1
  Environmental Status (. = Pass, F = Fail, U = Unknown)
  PS (3.3V):   .   PS (12V): .   PS (24V):   .   PS1: .     PS2: .
  Temperature: .   Fan:      .

  Module 1 : 2-port 100BaseFX MM Supervisor
  Network Management Processor (NMP) Status: (. = Pass, F = Fail, U = Unknown)
  ROM:  .   Flash-EEPROM: .   Ser-EEPROM: .   NVRAM: .   MCP Comm: .

  EARL Status :
          NewLearnTest:         .
          IndexLearnTest:       .
          DontForwardTest:      .
          MonitorTest           .
          DontLearn:            .
          FlushPacket:          .
          ConditionalLearn:     .
          EarlLearnDiscard:     .
          EarlTrapTest:         .


  LCP Diag Status for Module 1  (. = Pass, F = Fail, N = N/A)
  CPU         : .    Sprom    : .    Bootcsum : .    Archsum  : N
  RAM         : .    LTL      : .    CBL      : .    DPRAM    : .    SAMBA : N
  Saints      : .    Pkt Bufs : .    Repeater : N    FLASH    : N

  MII Status:
  Ports 1  2
  -----------
        N  N

  SAINT/SAGE Status :
  Ports 1  2  3
  --------------
        .  .  .

  Packet Buffer Status :
  Ports 1  2  3
  - -------------
        .  .  .

  Loopback Status [Reported by Module 1] :
  Ports  1  2  3
  --------------
        .  .  .

The SNMP MIBs in this list should be used for polling if the moduleUp or moduleDown trap is received:

The following information is available at the CCO Web site:

This section contains samples of the output from CLI commands that provide module status.

  switch 5000 (enable) show module
  Mod Module-Name         Ports Module-Type           Model     Serial- Num Status
  --- ------------------- ----- --------------------- --------- ------- --- -------
  1                       2     100BaseFX MM Supervis WS-X5006   003292389   ok
  3                       12    10BaseFL Ethernet     WS-X5011   003140385   ok
  4                       12    10BaseFL Ethernet     WS-X5011   003418318   ok
  5                       12    100BaseTX Ethernet    WS-X5113   002203857   ok

  Mod MAC-Address(es)                           Hw     Fw     Sw
  --- ----------------------------------------  ------ ------ --------- -------
  1   00-60-47-96-f2-00 thru 00-60-47-96-f5-ff  1.4    2.1    2.1(9)
  3   00-60-3e-d1-86-e4 thru 00-60-3e-d1-86-ef  1.3    1.2    2.1(9)
  4   00-60-3e-c9-90-54 thru 00-60-3e-c9-90-5f  1.1    1.2    2.1(9)
          5   00-40-0b-d5-0e-10 thru 00-40-0b-d5-0e-1b  1.4    1.2    2.1(9)

  switch 5000 (enable) show test 4

  Module 4 : 12-port 10/100BaseTX Ethernet

  LCP Diag Status for Module 4  (. = Pass, F = Fail, N = N/A)
   CPU         : .    Sprom    : .    Bootcsum : .    Archsum  : N
   RAM         : .    LTL      : .    CBL      : .    DPRAM    : N    SAMBA : .
   Saints      : .    Pkt Bufs : .    Repeater : N    FLASH    : N

   SAINT/SAGE Status :
    Ports 1  2  3  4  5  6  7  8  9  10 11 12
    -----------------------------------------
          .  .  .  .  .  .  .  .  .  .  .  .

   Packet Buffer Status :
    Ports 1  2  3  4  5  6  7  8  9  10 11 12
    -----------------------------------------
          .  .  .  .  .  .  .  .  .  .  .  .

   Loopback Status [Reported by Module 1] :
    Ports 1  2  3  4  5  6  7  8  9  10 11 12
    -----------------------------------------
          .  .  .  .  .  .  .  .  .  .  .  .

   Channel Status :
    Ports 1  2  3  4  5  6  7  8  9  10 11 12
    -----------------------------------------
          .  .  .  .  .  .  .  .  .  .  .  .

The SNMP MIBs in this list should be used for polling if the newRoot or topology Change trap is received.

The following information is available at the CCO Web site:

The bridge forwarding database or "CAM table" used by the EARL is available via this MIB object:

The following contains sample output from the CLI command that provides information about the CAM table.

  switch 5000> show cam count dynamic
  Total Matching CAM Entries = 200

The following information is available at the CCO Web site:

The following sample output is from the CLI command that provides error counts by port.

  switch 5000 (enable) show port counters
  Port  Align-Err  FCS-Err    Xmit-Err   Rcv-Err    UnderSize
  ----- ---------- ---------- ---------- ---------- ---------
   1/1           0          0          0          0         0
   1/2           0          0          0          0         0
   2/7           0          0          0          0         0
   2/8         428        159          0          0       718
   2/9           0          0          0          0         0

  Port  Single-Col Multi-Coll Late-Coll  Excess-Col Carri-Sen Runts      Giants
  ----- ---------- ---------- ---------- ---------- --------- ---------  ---------
   1/1           0          0          0          0         0         0          0
   1/2           0          0          0          0         0         0          0
   2/1        4855        479          0          0         0         0          0
   2/2         775         94          0          0         0         0          0
   2/3          65          6          0          0         0         0          0
   2/4          69          9          0          0         0         0          0
   2/5         354         36          0          0         0         0          0
   2/6           0          0          0          0         0         0          0
   2/7         104         14          0          0         0         0          0
   2/8           0          0          0          0         0       621          -

The following are the SNMP MIBs; the information can be found on the CCO Web site:

The following sample output is from the CLI command that provides MAC-level statistics.

  switch 5000 (enable) show mac

  MAC      Rcv-Frms   Xmit-Frms  Rcv-Multi  Xmit-Multi Rcv-Broad  Xmit-Broad
  -------- ---------- ---------- ---------- ---------- ---------- ----------
   1/1              0          0          0          0          0           0
   1/2              0          0          0          0          0           0
   2/1        7564831    3159859      93795    2982015    7375010       47356
   2/2         839319   11317193      97413    2991019       1560     7417591
   2/3          40744   10516768       5631    3096045          6     7419192
   2/4          40989   10523185       6222    3102759          0     7419195
   2/5          87858   10577397       6179    3114275        920     7418557
   2/6              0          0          0          0          0           0
   2/7          53448   10557561       5632    3124050          0     7419181
   2/8        1646786   31124783    1645159   31124783          0           0
   2/9          45666   10610469       4980    3150899        543     7418652
   2/10         44872   10582388       5723    3158925        118     7417517
   2/11      24269744    8525203   24269742    1104787          0     7420184

Alignment errors are a count of the number of frames received that don't end with an even number of octets and have a bad CRC.

When packets less than 64 bytes in length not ending on a whole byte boundary (for example, a fragments left from a collision) are received, the Catalyst will increment the runts counter and the align-err counter.

An alignment error is an indication of a cable problem or a faulty transmitter on the network equipment connected at the other end. This count should be zero or very low. When the cable is first connected, some may occur. Also, if there is a hub connected, collisions between other devices on the hub may cause this.

FCS error count is the number of frames that were transmitted and received with a bad checksum (CRC value) in the Ethernet frame. These frames are dropped and not propagated on to other ports. A small number of these errors is acceptable, but it could also be an indication of bad cables, NICs, and so on.

Runt frames are too small for an Ethernet segment.

The behavior of runts is as follows:

A small number of these errors is acceptable, but it could also be an indication of bad cables, NICs, and so on. In a shared Ethernet environment, runt frames are almost always caused by collisions. If runt frames occur when collisions are not high or in a switched Ethernet environment, they are the result of underruns or bad software on a network interface card. Attach a protocol analyzer to try to identify the faulty network interface card by determining the source address of the runt frames.

Table 14-8 describes which counters are incremented under certain error situations.

The error types are defined as follows:

This section lists the MIB objects that should be monitored and what information each will provide.

The ability to detect problems quickly in any network is critical. Network operations personnel can rely on a graphical network map to display the operational states of critical network elements such as routers and switches. Most commercial network management can perform discovery of network devices. Each network device is represented by a graphical element on the management platform's console. Different colors on the graphical elements represent the current operational status of network devices. These network-management platforms can also receive and display events generated from network devices.

Network devices can be configured to send notifications to network-management platforms. Upon receiving the notifications, the graphical element representing the network device changes to a different color, depending on the severity of the notification received.

There are a few ways to detect faults on a network consisting of Cisco routers and switches. The most common ones are via syslog messages, SNMP/traps, and RMON. Cisco devices are capable of sending syslog messages to a syslog server. Syslog messages are system messages from routers/switches describing different conditions on a device. SNMP traps forwarded by devices are useful to notify faulty conditions, such as if an interface goes up/ down.

Not all syslog and trap messages indicate faulty conditions on a device. Some messages are informational messages and do not require any action from the user. The amount of syslog and trap messages sent by a network device can be limited by specific commands on the configuration file.

  %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1, changed state to down

  gateway> show interface ethernet 0

  Ethernet0 is up, line protocol is up  (OperStatus)
  Hardware is Lance, address is 0000.0c38.1669 (bia 0000.0c38.1669)
  Internet address is 172.16.97.1/24
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255
  Encapsulation ARPA, loopback not set, keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never

RMON

The preceding section showed how SNMP traps can provide alerts to a management station. This reactive approach is useful for informing network operators after a problem has occurred. A more proactive approach, however, would be to inform the operators before a potential problem hits the device. For example, performance will become an issue when CPU utilization on a router hits a high value. The regular approach of polling the router using SNMP to find out its utilization could miss the event depending on the polling interval. By utilizing RMON support on Cisco devices, they can be configured to monitor CPU utilization and only send an alert when a threshold is reached (that is, CPU utilization hits 90).

Figure 14-2 shows how RMON can be used to monitor CPU utilization. The device will sample the value of CPU utilization at predefined intervals. If the value hits Threshold 1, either an event can be generated to inform the user or an entry is logged in the router's retrievable RMON table. Threshold 2 is set to reactivate the monitoring when CPU utilization hits that value. RMON eliminates the regular polling from a management console, and reduces SNMP network traffic.

Figure 14-2. Setting Thresholds for CPU Utilization Using RMON

Other statistics that can be monitored using RMON include input/output drops, buffer failures, internal temperatures, number of Frame Relay BECN/FECN packets, and so on.

Events observed at the management console need to be diagnosed to determine the severity of a problem and the necessary action to correct the problem. With syslog messages, the severity and description of the problems can be determined quickly because of the message verbosity. Corrective actions can be taken after reviewing the nature of the problem.

Obtaining a baseline of network performance involves taking samples of network statistics over an extended period of time. The baseline data can be collected using standalone probes attached to a LAN segment or WAN link. The data is used to determine a normal traffic pattern in the network. Additional measurements on network performance can be compared against the baseline to determine whether they are within the normal pattern.

SLA involves the task of defining specific performance characteristics expected from the network. The agreement defines certain performance metrics used to measure the actual service level obtained from the network against the stated level of service in the SLA. It is a very common agreement between the provider of a service and a recipient of the service. The metrics for network performance can include response time, availability, and so on.

The performance of a network is directly linked to the operational state of devices within the network. Hardware and software components of a network device also affect its performance. Failed hardware components can cause a complete outage in the network. It is critical to monitor the operating environments of network devices such as voltage, temperature, airflow, and ensure they are operating within specifications. Software components such as buffers, memory, and so on can have a significant impact on the protocols running on the device.

A useful performance indicator on a router is its CPU utilization. By measuring CPU utilization over time, a trend can be established to determine traffic patterns. Routers running constantly at high-utilization levels can affect the overall performance of forwarding and processing packets. CLI commands exist on the router to display the CPU utilization and information on running processes. Information returned by the command on CPU load can be accessed using objects defined in the OLD-CISCO-CPU-MIB file. The following displays CPU utilization using the proper CLI command:

  Router# show processes
  CPU utilization for five seconds: 1%/0%; one minute: 1%; five minutes: 1%

   PID QTy       PC Runtime (ms)    Invoked   uSecs    Stacks TTY Process
     1 Mwe 6039CCC8      2203448    9944378     221 7392/9000   0 IP-EIGRP Router
     2 Lst 60133594       329612      34288    9613 5760/6000   0 Check heaps
     3 Cwe 6011D820            0          1       0 5648/6000   0 Pool Manager
     4 Mst 6015FAA8            0          2       0 5608/6000   0 Timers

CPU utilization can be read using the MIB objects in Table 14-23.

The amount of main memory left on a router's processor has a significant impact on performance. Buffers are allocated from memory into different memory pools that are used by a protocol. IPX SAP packets, for example, use middle buffers in sending out packets. The following CLI commands are commonly used to monitor the memory and buffer statistics on a router:

The values collected from CLI commands are accessible via SNMP. Cisco provides the following MIB files for obtaining the equivalent output from CLI commands: CISCO-MEMORY-POOL-MIB, OLD-CISCO-INTERFACES-MIB, and OLD-CISCO-MEMORY-MIB.

The show memory command displays memory allocation:

  Router# show memory

                 Head   Total(b)    Used(b)    Free(b)     Lowest(b)    Largest(b)
  Processor  60DB19C0   119858752   1948928    117909824   117765180   117903232
       Fast  60D919C0   131072      69560      61512       61512       61468

Memory allocation can be read using the MIB objects in table Table 14-24.

The show buffers command displays buffer allocation:

  Router# show buffers
  Buffer elements:
       499 in free list (500 max allowed)
       124485689 hits, 0 misses, 0 created

  Public buffer pools:
  Small buffers, 104 bytes (total 120, permanent 120):
       112 in free list (20 min, 250 max allowed)
       35868550 hits, 0 misses, 0 trims, 0 created
       0 failures (0 no memory)
  Middle buffers, 600 bytes (total 90, permanent 90):
       88 in free list (10 min, 200 max allowed)
       37894226 hits, 0 misses, 0 trims, 0 created
       0 failures (0 no memory)
  Big buffers, 1524 bytes (total 90, permanent 90):
       90 in free list (5 min, 300 max allowed)
       1161634 hits, 0 misses, 0 trims, 0 created
       0 failures (0 no memory)
  Large buffers, 5024 bytes (total 10, permanent 10):
       10 in free list (0 min, 30 max allowed)
       0 hits, 0 misses, 0 trims, 0 created
       0 failures (0 no memory)
  Huge buffers, 18024 bytes (total 0, permanent 0):
        0 in free list (0 min, 13 max allowed)
        0 hits, 0 misses, 0 trims, 0 created
        0 failures (0 no memory)

Buffer allocation can be read using the MIB objects in Table 14-25.

The show interface command displays interface statistics:

  Router# show interface
  Ethernet0/0 is up, line protocol is up
    Hardware is cxBus Ethernet, address is 0010.f65f.7000 (bia 0010.f65f.7000)
    Internet address is 172.16.97.1/24
    MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255
    Encapsulation ARPA, loopback not set, keepalive set (10 sec)
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:01, output 00:00:01, output hang never
    Last clearing of "show interface" counters never
    Queueing strategy: fifo
    Output queue 0/40, 0 drops; input queue 0/75, 0 drops
    5 minute input rate 0 bits/sec, 0 packets/sec
    5 minute output rate 0 bits/sec, 0 packets/sec
       12072853 packets input, 1379751443 bytes, 0 no buffer
       Received 1824605 broadcasts, 0 runts, 0 giants
       0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
       0 input packets with dribble condition detected
       11283674 packets output, 1218604416 bytes, 0 underruns
       0 output errors, 24888 collisions, 1 interface resets
       0 babbles, 0 late collision, 0 deferred
       0 lost carrier, 0 no carrier
       0 output buffer failures, 0 output buffers swapped out

Interface information can be read using the MIB objects in Table 14-26.

Another way to look at MIB objects is to sort them by types of interfaces, as shown in Table 14-27.

The final tasks involved in performance management are setting thresholds, exception reporting, analysis, and tuning.

A few key syslog messages need to be filtered to reduce the number of notifications to network operators. These filters need to suppress identical messages repeated within n seconds.

Selected syslog messages apply to Cisco 7500 routers and are listed in Table 14-28.

The SNMP trap in Table 14-29 should be filtered similarly.

Platform: . 7500 router.

Objective: . Detect when a device reports it is shutting down (as opposed to not reaching it, which is covered in the Router/Switch Down scenario).

Symptoms: . Cisco IOS logs syslog message before (SYS-5-RELOAD) and after (SYS-5-RESTART) a device restarts.

Correlation Logic: . When the SYS-5-RELOAD message is received, the event-correlation engine must wait for a SYS-5- RESTART to arrive within n minutes and cancel the original message. If no SYS-5-RESTART message is received within n minutes, the operator should be notified of a critical alert. If the router reports a SYS-5- RESTART message within n minutes, the condition will only be logged as informational, along with the value of the whyReload MIB object.

If the router is detected as down, this information will be fed as input into the correlation rule for the router up/down problem, so that the background reachability check will not report it as down again.

Probable Cause: . Software error or operator intervention.

Actions/Resolution: . Notify operator if router does not issue a SYS-5-RESTART message within n minutes (n = time for router to reload + 1 minute).

Platform: . Catalyst 5000.

Objective: . Detect when a device reports it is being reset from the console (as opposed to not reaching it, which is covered in the Router/Switch Down scenario).

Symptoms: . Cisco IOS logs syslog message before the switch is reset from the console (SYS-5:System reset). An SNMP cold start trap and/or the IOS message (SNMP-5:Cold Start Trap) after the device restarts.

Correlation Logic: . When the SYS-5:System reset message is received, the event-correlation engine must wait for an IOS message SNMP-5:Cold Start Trap or a cold start trap to arrive within n minutes and cancel the original message. If no SNMP-5:Cold Start Trap message is received within n minutes, the operator should be notified of a critical alert. If the switch reports a SNMP-5:Cold Start Trap IOS message or cold start trap within n minutes, the condition will only be logged as informational.

If the switch is detected as down, this information will be fed as input into the correlation rule for the router/switch up/down problem, so that the background reachability check will not report it as down again.

Probable Cause: . Software error or operator intervention.

Actions/Resolution: . Notify operator if router does not issue a SNMP-5:Cold Start Trap or cold start trap message within n minutes (n = time for switch to reload + 1 minute).

Platforms: . 7500 router and other routers with BRI interfaces.

Objective: . Knowing when a link is up or down is perceived by customers as being the most important requirement. Added correlation will allow detecting normal link up/down transitions from dialup lines to more serious transitions for other links.

An interface being down can be a critical alert on a router or switch. Access routers with ISDN and ASCII links have their interfaces going up and down many times a day, however, as part of their normal activity, as dialup calls are initiated and torn down. A correlation rule should be applied to sort out the relevant link-down conditions from normal operation.

Symptoms: . LINK_3_UPDOWN syslog messages are logged.

Correlation Logic: . Two levels of correlation were identified:

For simple correlation, if the string BRI is contained in the message, the message should be discarded.

For advanced correlation, if the message does not contain the string BRI, the physical topology database should be queried to determine the type of link, namely whether it is a link between two routers, a router and a switch, or router and an end-user port.

The syslog message should be further processed only if it is a link between two Cisco devices.

Probable Cause: . Link was either disconnected or broken.

Actions/Resolution: . Notify the network operator, except if the interface is that of an end user or an ASCII or BRI interface.

Platform: . Catalyst 5000 only.

Objective: . STP reconfiguration can have disastrous effects, such as some parts of the network not being accessible any longer. Upon detection of an STP reconfiguration, the event-correlation engine should ensure that the network is still functional.

Symptoms: . If you received any of the following syslog messages:

Or any of the following RFC 1493 (Bridge MIB) SNMP traps:

These traps contain the community string in the format community@vlanId, where vlanId is the VLAN identifier in which the spanning tree changed. For Catalyst 5000 switch Release 4.1 and higher, these traps will contain the vtpVlanIndex and ifName varBinds to provide additional information to the application processing these traps.

Correlation Logic: . Three types of correlation are possible, depending on the extent the ECS can query the network:

The simple STP correlation refers to ensuring that every device reporting an STP change through the means referred to earlier is monitored to ensure that it is restored to a valid state. This will be accomplished as follows, based on the Catalyst 5000 MIB.

First of all, the CISCO-STACK-MIB object sysEnableBridgeTraps should be verified to be set to enabled(1), or an alert should be raised.

The community field in the topologyChange or newRoot trap must be read and used to further interrogate the switch the trap came from. This enables you to query the specific VLAN the topology change occurred on.

First, you need to determine whether any ports were a trunk before the STP change. To do so, the list of all trunks per VLAN must be maintained and verified against the list after the STP change. A trunk port is defined as having its vlanPortIslOperStatus MIB object in the vlanPortTable of the CISCO-STACK MIB with a value of trunking(1).

Second, you need to determine which trunk ports in this VLAN had a state change by scanning the vlanPortTable for those entries with vlanPortVlan set to the VLAN ID specified in the trap.

For each of the selected ports, identify the vlanPortIslOperStatus MIB object with a value of trunking(1). For each trunk, you need to monitor its status as identified in RFC 1493. This is accomplished as follows.

Read the vlanPortIslAdminStatus object in the vlanPortTable and ensure it is not set to off(2). Otherwise, generate an alert that the port was disabled.

For each trunk port in the selected VLAN, read the corresponding vlanPortModule and vlanPort from the vlanPortTable. Use these two values of vlanPortModule and vlanPort to index into the CISCO-STACK-MIB portTable to retrieve the corresponding portIfIndex value. Use the portIfIndex to read the MIB-II ifOperStatus to determine whether the interface is still up. If not, generate an alert.

Read the portCrossIndex value in the portTable in the CISCO-STACK-MIB for the selected entry and use this value to index into the RFC 1493 dot1dStpPortTable to verify that the dot1dStpPortEnable is set to enabled(1).

Then, use the same portCrossIndex value in the portTable in the CISCO-STACK-MIB to locate the corresponding entry in the RFC 1493 dot1dStpPortTable to verify that dot1dStpPortState is either in blocking(2) or forwarding(5) mode, 2 minutes after the trap is received. Otherwise, send an alert.

Verify also that the ports that were known to be trunks before the trap occurred are still trunks (that is, that all trunks known to be valid trunks are still functioning as trunks). Otherwise, an alert should be generated. Similarly, the situation when a port was known not to be a trunk, but is found to be a trunk while processing the trap, should be reported.

Alerts should also include e-mail summarizing the discrepancies found as a result of this correlation. Whether a separate e-mail is generated for each port discrepancy or all port discrepancies associated with an STP change are accumulated in a single e-mail is implementation dependent.

The next correlation type is a medium correlation. An STP reconfiguration is always limited to a subnet. Assuming the event-correlation engine knows the devices in the subnet prior to the reconfiguration, a simple correlation could ensure that all these devices are still up and running (by performing a reachability test) and that these are still the same devices (by comparing the previously known and the current RFC1213-MIB sysDescr string for each device). A logical topology discovery will also be initiated to detect whether any new device was added to the network. An alternative is to rely on switches newly added to the network to generate either trap, so that the ECS can detect them.

Although not 100% perfect, this correlation rule can assist a network operator in narrowing down the scope of the error more quickly.

The final correlation type is an advanced correlation. One way to realize why an STP configuration occurred is to know ahead of time the list of devices involved in each spanning tree in the network. An STP topology can be collected by querying the dot1Stp MIB group from the RFC 1493 MIB. CWSI 1.2 and greater can also be used to display a graphical map of the STP tree over the physical topology discovered using CDP. Note that Cisco supports one spanning tree per VLAN. The VLAN ID is reported either in the syslog message or in the community string contained in the SNMP trap. If no VLAN is configured in the network, it is assumed that there is only one VLAN with a VLAN ID of 1. Therefore, the term VLAN in this section is used to represent the subnet if no VLANs are configured.

When any spanning tree in a VLAN reconfigures, the event-correlation engine can then identify which VLAN reconfigured and determine what changed in the spanning tree.

The event correlation listens for RFC 1493 (Bridge MIB) newRoot or topologyChange SNMP traps to detect when an STP reconfiguration occurs. These traps are always sent by a device whose port changed state as defined in the RFC.

If a topologyChange trap is received, the event-correlation engine can easily compile the list of devices involved in the STP reconfiguration.

If the ECS cannot adequately extract the community string from the trap and does not process syslog messages, the event-correlation rule needs to relate the trap to a VLAN by detecting which VLAN had its topology change counter increased by one. For that purpose, the CISCO-STACK vlanTable MIB group can be queried to list the VLANs configured in the switch. Using the SNMP community string to address individual VLANs (using community@vlanId), the NMS could alternately poll for the RFC 1493 dot1dStpTopChanges MIB object, which gives the number of topology changes since the device rebooted or initialized. This method would enable you to isolate which VLAN had a topology change so that further verification could be performed in that VLAN only.

Because STP reconfiguration is automatic, the event-correlation logic needs to determine the following:

The event-correlation engine needs to determine whether a node was detected as going down prior to an STP topology change being detected. The event-correlation rule will calculate the difference between the known STP topology prior to the STP reconfiguration and the new STP configuration with the following objectives:

In any of these three situations, the event-correlation engine needs to assess which ports involved in a spanning tree changed from a forwarding state to a blocking state, or from a learning to forwarding state. It also needs to correlate whether any of these ports had their RFC1213-MIB operStatus MIB object toggle from up to down, in an attempt to identify a port failure or disconnection.

The operator will be notified of the actual device and port that caused the topology change. It may be that the event correlation identifies more than one possible failure cause or device, in which case it should report all possible causes or devices to the operator.

The notion of correctness can be very complex, depending on the network characteristics.

The simplest means to determine correctness of an STP configuration is to ensure that every device in the VLAN spanning tree is reachable. Therefore, the correlation rule will trigger a reachability test to all devices known to have been members in the spanning tree prior to the reconfiguration. The correlation rule will also trigger a discovery of the new STP tree to detect whether a new device was added to the spanning tree.

The correlation engine of the device(s) involved in the reconfiguration will notify the network operator and will specify the most likely cause of the STP reconfiguration, namely:

A more difficult test is to detect which interfaces went down or up, and which would therefore have caused the STP reconfiguration, while not reporting interfaces which were known to be down before the STP topology change occurred. This correlation rule can be performed through the same mechanisms described earlier. STP discovery must include the ports involved in an STP tree with their operStatus and adminStatus as defined in the ifTable of RFC1213 MIB and compare the values of these MIB objects prior to and after an STP reconfiguration. Another way to detect whether STP reconfiguration occurred because of a down interface is to correlate it to a linkDown SNMP trap or syslog message received within seconds of a topologyChange or newRoot SNMP trap or topology change syslog message within a VLAN.

An alternative method is to monitor the dot1dStpPortForwardTransitions MIB object to detect when this object increments by one.

Probable Cause: . A device involved in the spanning tree was removed from service, failed, or was added to the network, or a link was added or removed between two switches.

Actions/Resolution: . Notify operator as specified earlier. No automatic action is expected.

Platforms: . Catalyst 5000 and 7500 router.

Objective: . Detect when a switch or router is down, without reporting the unreachable switches and routers behind it.

Based on knowledge of the topology layout, a correlation engine must identify the most likely failure from a group of unreachable nodes. Assuming Cisco manages Cisco equipment only, end-user devices'status—such as PCs and servers—will not be monitored.

Symptoms: . Several devices are not reachable at the same time.

Correlation Logic: . A logical (Layer 3) topology database can serve to detect a faulty router from a group of unreachable routers. However, a physical (for example, Layer 2) topology database is needed to detect a faulty switch from a group of unreachable switches.

Every device reported as not reachable within n minutes will be queried for RFC1213-MIB sysObjectID MIB object and use Cisco's mapping table.

Then, the event-correlation engine would either perform a simple or advanced correlation. The recommendation is n = 5 minutes.

Simple correlation involves using the existing NMS topology database.

This will work best for determining a faulty router from other routers, but will not differentiate between switches (because most NMS do not support L2 topology).

Advanced correlation allows determining a faulty switch from multiple switches in a subnet by making use of a physical-connectivity database describing how switches are interconnected. The ECS would need to first use L3 topology to identify which subnet is most likely to have a failure. Then an L2 topology database can be used to identify which switch may have failed.

Probable Cause: . A single router or switch is down.

Actions/Resolution: . Notify operator with a single notification identifying the faulty device.

Platform: . 7500 router.

Objective: . Detect when excessive traffic causes excessive CPU load.

Network managers have only a limited understanding of the traffic flowing through their Cisco switches and routers. Detecting high traffic conditions becomes increasingly important. CPU overload conditions as reported by SYS-3-CPUHOG syslog messages will be monitored and an alert will be generated when excessive conditions are detected— conditions not related to transient traffic flows as reported by NetFlow, which may indicate a temporary overload condition.

Symptoms: . The SYS-3-CPUHOG syslog message is logged, indicating a CPU overload condition.

Correlation Logic: . Two levels of correlation were identified.

The first type of correlation is simple correlation. The event-correlation engine will 0trigger traffic statistics collection every 20 seconds for n minutes for the ifInOctets and ifOutOctets from the RFC1213 MIB ifTable. If any interface traffic utilization exceeds 40% of the link capacity (ifSpeed) for more than 50% of the collection period, the operator will be notified with a reference to the interface(s) having excessive traffic.

If the interface with excessive traffic is the same interface used for ping and SNMP reachability, it is likely that the condition will result in the device not being reachable temporarily, which would report the device as down. The device low-performance condition will be fed as input into the reachability correlation rule so that the operator is not notified twice for the same problem, or that other devices are assumed down/unreachable because this interface is overloaded (see the Router Down problem).

The low-performance condition will be deemed a minor alarm for n minutes and not reported to the network operator if it goes away within n minutes. The recommendation is n = 15 minutes.

If no performance condition exists, the lack of reachability will be treated as a critical alarm because the device is probably totally down.

The second type of correlation is advanced correlation. NetFlow can be used to perform more advanced correlation and to avoid reporting transient overloads by verifying which protocols are responsible for the high-performance impact. This is explained as follows.

The IOS command show ip cache flow will report a table of IP addresses that are communicating on which port and which device interface. The table columns are labeled SrcIPaddress, DstIPaddress, SrcP, DstP, SrcIf, and DstIf. From this information, you can determine whether CPU overload is due to excessive traffic on a given interface, and whether the traffic is transient (FTP on port 21, for example). If the traffic is identified as transient, the event-correlation engine will ignore the CPUHOP syslog message for 5 minutes for that specific session. If the CPUHOP syslog message is repeated because of the same NetFlow session, over a period of 30 minutes, the operator will be notified as a warning that an unusual NetFlow session is taking place. The IOS command export ip flow can also be used for this purpose.

If the CPUHOP syslog message occurs repeatedly for different sessions (such as 10 times per hour), the operator will be notified as an indication that the device is consistently being overloaded and should be looked at further.

Probable Cause: . The cause may be due to a transient session (such as FTP) or the router is underpowered for the traffic going through it.

Actions/Resolution: . Notify operator if performance problem persists as described earlier.

Platform: . Catalyst 5000 with redundant supervisor module.

Objective: . Detect when temperature rises and may cause a device shutdown.

Symptoms: . The Catalyst 5000 switch with a redundant supervisor module reports a syslog message: SYS-0: "Temp high Failure" or an SNMP trap: CISCO-STACK MIB chassisAlarmOn with the varBindList containing chassisTempAlarm = on(2), chassisMinorAlarm = on(2) or off(1), and chassisMajorAlarm = on(2), whenever the temperature exceeds 50° Celsius.

Correlation Logic: . If, within 5 minutes, you receive another syslog message: SYS-0: "Temp Critical Recovered" or SYS-2: "Temp high Okay" or the SNMP trap: CISCO-STACK MIB chassisAlarmOff, the alert should be cleared from the event-correlation engine.

If none of these messages or an SNMP trap is received, the alert should be fed as input into the Device Up/Down correlation rule reporting that the device is down.

Probable Cause: . The air-conditioning in the room failed or the fan failed.

Actions/Resolution: . The original error should be reported immediately as a paging notification to the network operator. If the problem clears itself, the operator should be notified again.

Platform: . 7000 router.

Objective: . Detect when temperature rises and may cause a device shutdown.

Symptoms: . Any of the following syslog messages are received:

The -TEMP messages will generate a major alarm; the SHUTDOWN messages will generate a critical alarm.

More granularity may be supported if the ECS can decode the actual temperature from the syslog message and apply its own thresholds and alarm severities.

The alarm would eventually have to be cleared manually by the operator.

Correlation Logic: . Pass through.

Probable Cause: . The air-conditioning in the room failed.

Actions/Resolution: . Notify operator.