Chapter 10

Troubleshooting Nexus Route-Maps

This chapter covers the following topics:

• Conditional matching with ACLs, Prefix Lists, and Regular Expression

• Route-Maps

• Troubleshooting RPM

• Redistribution

• Policy-Based Routing

Nexus Operating System (NX-OS) route-maps provide the capability to filter routes and modify route attributes and routing behavior. These technologies use conditional match criteria to allow actions to occur based upon route characteristics.

Before route-maps are explained, the concepts involved with conditional matching using access control lists (ACL), prefix lists, and conditional matching of BGP communities must be explained.

ACLs in NX-OS are generic expressions for filtering traffic based on Layer 2, Layer 3, or Layer 4 information. ACLs are composed of access control entries (ACE), which are entries in the ACL that identify the action to be taken (permit or deny) and the relevant packet classification. Packet classification starts at the top (lowest sequence) and proceeds down (higher sequence) until a matching pattern is identified. After a match is found, the appropriate action (permit or deny) is taken and processing stops. At the end of every ACL is an implicit deny ACE, which denies all packets that did not match earlier in the ACL.

ACLs are classified into two categories:

• Standard ACLs: Define the packets based solely on the source network.

• Extended ACLs: Define the packet based upon source, destination, protocol, port or combination of other packet attributes. Standard ACLs use the numbered entry 1–99, 1300–1999, or a named ACL. Extended ACLs use the numbered entry 100–199, 2000–2699, or a named ACL. Named ACLs provide relevance to the functionality of the ACL, are used with standard or extended ACLs, and are generally preferred.

The behavior for selecting a network prefix with an extended ACL varies depending on whether the protocol is an IGP such as Enhanced Interior Gateway Protocol (EIGRP), Open Shortest Path First (OSPF), Intermediate System-to-Intermediate System (IS-IS) or Border Gateway Protocol (BGP).

Using ACLMGR, the following functionalities are available with ACL:

• Object-Groups (Matching IP Addresses, TCP or UDP ports)

• Time ranges

• IPv6 wildcard matching

• Packet length based matching

• Stateful restarts

• Per-entry statistics

Along with applying ACLs on the interface or using ACLs along with route-maps, which is then used by routing protocols for route filtering purposes, ACLs have the following applications:

• Control Plane Policing (CoPP)

• Dynamic Host Configuration Protocol (DHCP)

• Policy-Based Routing (PBR)

• Web Cache Communication Protocol (WCCP)

NX-OS supports the following ACL formats:

• IPv4/IPv6 ACLs

• Media access control (MAC) ACL

• Address resolution protocol (ARP) ACL

• Virtual LAN (VLAN) access-map

In NX-OS, when an ACL is applied to a target, a policy is created. NX-OS supports the following types of ACL policies:

• Router ACL (RACL)

• Port ACL (PACL)

• VLAN ACL (VACL)

• Virtual terminal line (VTY) ACL

Example 10-1 illustrates the various ACL configurations supported on NX-OS. In the following example, the command statistics per-entry is configured to enable the statistics for the ACEs configured under the ACLs. If the command statistics per-entry is not configured, the command show ip access-list does not display any statistics for the packets hitting a particular ACE.

Example 10-1 ACL Formats

Click here to view code image

Example 10-2 illustrates the difference between the output of the show ip access-list command when the statistics per-entry command is configured, compared to when it is not configured. In the following example, the ACL configuration that has the statistics per-entry command configured displays the statistics for the confirmed hits.

Example 10-2 ACL Statistics

Click here to view code image

When an ACL is attached to an interface or any other component, the ACL gets programmed in the ternary content addressable memory (TCAM). The TCAM programming for the access-list is verified using the command show system internal access-list interface interface-id input statistics [module slot]. This command displays under which bank the ACL is programmed and what kind of policy is created when the ACL is attached to an attach point. Along with this, the command displays the statistics of each ACE entry in the ACL.

If the statistics per-entry command is not configured, the counters in the TCAM increment only for the entry for all traffic; that is, permit ip 0.0.0.0/0 0.0.0.0/0. Example 10-3 demonstrates the ACL entry on TCAM and the TCAM statistics when statistics per-entry command is not configured.

Example 10-3 Verifying Access-List Counters in TCAM

Click here to view code image

As stated before, the ACLMGR takes care of creating the policies when an ACL is attached to an attach point. The policies created by the ACLMGR are verified using the command show system internal aclmgr access-lists policies interface interface-id. This command displays the policy type and interface index, which points to the interface where the ACL is attached, as shown in Example 10-4.

Example 10-4 Verifying Access-List Counters in Hardware

Click here to view code image

NX-OS has a packet processing filter (PPF) API, which is used to filter the security rules received and processed by the ACLMGR to the relevant clients. The clients can be an interface, a port-channel, a VLAN manager, VSH, and so on. It is important to remember that the ACLMGR stores all the data in the form of a PPF database, where each element is a node. Based on the node ID received from the previous command, more details about the policy can be verified by performing a lookup in the PPF database on that node.

Example 10-5 illustrates the use of the command show system internal aclmgr ppf node node-id to perform a lookup on the PPF database of the ACLMGR for the policy node created when the policy is attached to an attach point. This command is useful when troubleshooting ACL/filtering-related issues, such as ACL not filtering the traffic properly or not matching the ACL entry at all on NX-OS platform.

Example 10-5 Verifying the PPF Database

Click here to view code image

Figure 10-1 BGP Extended ACL Matches

Table 10-2 demonstrates the concept of the wildcard for the network and subnet mask.

In Figure 10-2, the prefix match specification has the high-order bit pattern of 192.168.0.0 and a high-order bit count of 16. The high-order bit pattern has been converted to binary to demonstrate where the high-order bit count lays. Because no additional matching length parameters are included, the high-order bit count is an exact match.

Figure 10-2 Basic Prefix Match Pattern

The prefix match specification logic might look identical to the functionality of an access-list. The true power and flexibility comes by using matching length parameters to identify multiple networks with specific prefix lengths with one statement. The matching length parameter options are as follows:

• le (less than or equal to <=)

• ge (greater than or equal to >=), or both.

Figure 10-3 demonstrates the prefix match specification with a high-order bit pattern of 10.168.0.0, high-order bit count of 13, and the matching length of the prefix must be greater than or equal to 24.

Figure 10-3 Prefix Match Pattern with Matching Length Parameters

The 10.168.0.0/13 prefix does not qualify because the prefix length is less than the minimum of 24 bits, whereas the 10.168.0.0/24 prefix does meet the matching length parameter. The 10.173.1.0/28 prefix qualifies because the first 13 bits match the high-order bit pattern, and the prefix length is within the matching length parameter. The 10.104.0.0/24 prefix does not qualify because the high-order bit-pattern does not match within the high-order bit count.

Figure 10-4 demonstrates a prefix match specification with a high-order bit pattern of 10.0.0.0, high-order bit count of 8, and the matching length must be between 22 and 26.

Figure 10-4 Prefix Match with Ineligible Matched Prefixes

The 10.0.0.0/8 prefix does not match because the prefix length is too short. The 10.0.0.0/24 qualifies because the bit pattern matches and the prefix length is between 22 and 26. The 10.0.0.0/30 prefix does not match because the bit pattern is too long. Any prefix that starts with 10 in the first octet and has a prefix length between 22 and 26 will match the prefix match specification.

NX-OS prefix lists are configured with the global configuration command ip prefix-list prefix-list-name [seq sequence-number] {permit | deny} high-order-bit-pattern/high-order-bit-count [{eq match-length-value | le le-value | ge ge-value [le le-value]}].

If a sequence is not provided, the sequence number auto-increments by 5 based off the highest sequence number. The first entry is 5. Sequencing allows the deletion of a specific entry. Because prefix lists cannot be resequenced, it is advisable to leave enough space for insertion of sequence numbers at a later time.

Example 10-6 provides a sample prefix list named RFC 1918 for all the networks in the RFC 1918 address range. The prefix list only allows /32 prefixes to exist in the 192.168.0.0 network range and not exist in any other network range in the prefix list.

Example 10-6 Sample Prefix List

Click here to view code image

Notice that sequence 5 permits all /32 prefixes in the 192.168.0.0/13 bit pattern, then sequence 10 denies all /32 prefixes in any bit pattern, and then sequence 15, 20, 25 permit routes in the appropriate network ranges. The sequence order is important for the first two entries to ensure that only /32 prefixes exist in the 192.168.0.0 in the prefix list.

The command show ip prefix-list prefix-list-name high-order-bit-pattern/high-order-bit-count first-match provides the capability for a specific network prefix to be checked against a prefix-list to identify the matching sequence, if any.

Example 10-7 displays the command being executed against three network prefix patterns based upon the RFC1918 prefix-list created earlier. The first command uses a high-order bit count of 32, which matches against sequence 5, whereas the second command uses a high-order bit count of 16, which matches against sequence 25. The last command matches against sequence 10, which has a deny action.

Example 10-7 Identification of Matching Sequence for a Specific Prefix Pattern

Click here to view code image

Route-maps are composed of four components:

• Sequence Number: Dictates the processing order of the route-map.

• Conditional Matching Criteria: Identifies prefix characteristics (network, BGP path attribute, next-hop, and so on) for a specific sequence.

• Processing Action: Permits or denies the prefix.

• Optional Action: Allows for manipulations dependent upon how the route-map is referenced on the router. Actions include modification, addition, or removal of route characteristics.

Route-maps use the following command syntax route-map route-map-name [permit | deny] [sequence-number]. The following rules apply to route-map statements:

• If a processing action is not provided, the default value of permit is used.

• If a sequence number is not provided, the sequence number defaults to 10.

• If a conditional matching statement is not included, an implied all prefixes is associated to the statement.

• Processing within a route-map stops after all optional actions have processed (if configured) after matching a conditional matching criteria.

• If a route is not conditionally matched, there is an implicit deny for that route.

Example 10-8 provides a sample route-map to demonstrate the four components of a route-map shown earlier. The conditional matching criteria is based upon network ranges specified in an ACL. Comments have been added to explain the behavior of the route-map in each sequence.

Example 10-8 Sample Route-Map

Click here to view code image

Example 10-9 Sample Route-Map

Click here to view code image

As you can see, a number of conditional matching options are available. Some of the options, like vlan and mac-list, are applicable only for policy-based routing. Table 10-3 provides the command syntax for the most common methods for matching prefixes and describes their usage.

In Example 10-10, sequence 10 requires that a prefix pass ACL-ONE or ACL-TWO. Notice that sequence 20 does not have a match statement, so all prefixes that are not passed in sequence 10 qualify and are denied.

Example 10-10 Multiple Match Variables Example Route-Map

Click here to view code image

If multiple match options are configured for a specific route-map sequence, both match options must be met for the prefix to qualify for that sequence. The Boolean logic uses an and operator for this configuration.

In Example 10-11, sequence 10 requires that the prefix match ACL ACL-ONE and that the metric be a value between 500 and 600. If the prefix does not qualify for both match options, the prefix does not qualify for sequence 10 and is denied because another sequence does not exist with a permit action.

Example 10-11 Multiple Match Options Example Route-Map

Click here to view code image

Example 10-12 Complex Matching Route-Maps

Click here to view code image

Reading configurations like this must follow the sequence order first, conditional matching criteria second, and only after a match occurs should the processing action and optional action be used. Matching a deny statement in the conditional match criteria excludes the route from that sequence in the route-map.

The prefix 172.16.1.0/24 is denied by ACL-ONE, so that infers that there is not a match in sequence 10 and 20; therefore, the processing action (permit or deny) is not needed. Sequence 30 does not contain a match clause, so any remaining routes are permitted, The prefix 172.16.1.0/24 passes on sequence 30 with the metric set to 20. The prefix 172.16.2.0/24 matches ACL-ONE and passes in sequence 10.

Verifying that all components of a route-map are defined is a vital component of troubleshooting.

• Operates as a stateful, restartable, and multithreaded process

• Maintains state in a database that exists in private memory

• Supports IPv4 and IPv6 address families

• Provides API interaction for other processes like BGP or OSPF

• Interacts with the unicast RIB (URIB), ARP table

NX-OS’s RPM provides additional methods to verify route-map interaction on a Nexus switch. Figure 10-5 provides a sample topology where NX-2 is redistributing OSPF learned routes into EIGRP.

Figure 10-5 Sample Topology Displaying Redistribution

Example 10-13 displays the relevant EIGRP configuration that redistributes OSPF and directly connected routes into EIGRP. The route-map is selective with the routes that are being redistributed into EIGRP.

Example 10-13 NX-2 Redistribution Configuration

Click here to view code image

When routes are not installed in EIGRP as anticipated, the first step is to check to make sure that any relevant policies were bound to the destination routing protocol. The command show system internal rpm event-history rsw displays low-level events that are handled by RPM.

Example 10-14 displays the command. Notice that two different route-maps were applied to EIGRP: one was for OSPF and the other for directly connected interfaces as the source redistribution protocols.

Example 10-14 Viewing RPM Event-History

Click here to view code image

It might be simpler to get a count of the number of RPM processes that have attached to a protocol using the command show system internal rpm clients, as demonstrated in Example 10-15. If the Bind-count did not match the anticipated number of route-maps used by that protocol, viewing the RPM event-history will indicate the error.

Example 10-15 Viewing the Number of RPM Clients per Protocol

Click here to view code image

In addition to viewing the accuracy of a prefix-list as shown in Example 10-2, the capability to view the internal programming for a prefix-list is beneficial. The command show system internal rpm ip-prefix-list displays all the prefix-lists configured on the Nexus switch for the route-map from Example 10-13. Notice that the clients show you the route-map referencing the prefix list. In addition, each prefix-list entry displays the number of sequences and version history for that prefix-list. Example 10-16 displays the use of the show system internal rpm ip-prefix-list command for NX-2.

Example 10-16 Viewing Prefix-Lists from RPM Perspective

Click here to view code image

The last method is to view relevant changes from a debug perspective. Traditionally, the route-map option needs to be enabled from the destination protocol. Example 10-17 displays the debugs for the route-maps associated to the redistribution into EIGRP.

Example 10-17 Viewing Debug Information for Redistribution

Click here to view code image

Policy-based routing (PBR) allows for conditional forwarding of packets based on the following packet characteristics:

• Routing by protocol type (Internet Control Message Protocol [ICMP], Transmission Control Protocol [TCP], User Datagram Protocol [UDP] and so on)

• Routing by source IP address, destination IP address, or both

• Manually assigning different network paths to the same destination based upon tolerance for latency, link-speed, or utilization for specific transient traffic

Packets are examined for PBR processing as they are received on the router interface. PBR verifies the existence of the next-hop IP address and then forwards packets using the specified next-hop address. Additional next-hop addresses are configured so that in the event that the first next-hop address is not in the RIB, the secondary next-hop addresses are used. If none of the specified next-hop addresses exist in the routing table, the packets are not conditionally forwarded.

NX-OS PBR configuration use a route-map with match and set statements that are then attached to the inbound interface. The following steps are used:

Step 1. Enable the PBR feature. The PBR feature is enabled with the global configuration command feature pbr.

Step 2. Define a route-map. The route-map is configured with the global configuration command route-map route-map-name [permit | deny] [sequence- number].

Step 3. Identify the conditional match criteria. The conditional match criteria is based upon packet length with the command match length minimum-length maximum-length, or by using the packet ip address fields with an ACL using the command match ip address {access-list-number | acl-name}.

Step 4. Specify the next-hop. The route-map configuration command set ip [default] next-hop ip-address [... ip-address] is used to specify one or more next-hops for packets that match the criteria. The optional default keyword changes the behavior so that the next-hop address specified by the route-map is used only if the destination address does not exist in the RIB. If a viable route exists in the RIB, that is the next-hop address that is used for forwarding the packet.

Step 5. Apply the route-map to the inbound interface. The route-map is applied with the interface parameter command ip policy route-map route-map-name.

Step 6. Enable PBR statistics (optional). Statistics of PBR forwarding are enabled with the command route-map route-map-name pbr-statistics.

Figure 10-6 displays a topology to demonstrate how PBR operates. The default path between NX-1 and NX-6 is NX-1 → NX-2 → NX-3 → NX-5 → NX-6 because the link cost of 10.23.1.0/24 is lower than 10.24.1.0/24 link. However, specific traffic sourced from NX-1’s Loopback 0 (192.168.1.1) to NX-6’s Loopback 0 (192.168.6.6) must not forward through NX-3. These packets must forward through NX-4 even though it has a higher path cost.

Figure 10-6 Policy-Based Routing Topology

Example 10-18 displays NX-2’s PBR configuration.

Example 10-18 NX-2’s PBR Configuration

Click here to view code image

Example 10-19 displays a traceroute from NX-1, which displays traffic flowing through NX-3. A source interface was not specificed, so traffic is sourced from the 10.12.1.1 IP address. This is confirmed based upon the routing table on NX-2 for the 192.168.6.6 network prefix.

Example 10-19 Normal Traffic Flow to NX-6’s Loopback 0 Interface

Click here to view code image

Example 10-20 displays a traceroute from NX-1’s Loopback 0 interface to be redirected on NX-2 toward NX-4. The traceroute displays that PBR is working as intended.

Example 10-20 Verification of PBR-Based Traffic

Click here to view code image

PBR statistics were enabled on NX-2 that allows for network engineers to see how much traffic was forwarded by PBR. Example 10-21 displays output for PBR statistics before and after traffic was conditionally forwarded.

Example 10-21 Sample Output of PBR Statistics

Click here to view code image

Cisco. Cisco NX-OS Software Configuration Guides, www.cisco.com.