Chapter 11

Troubleshooting BGP

This chapter covers the following topics:

• BGP Fundamentals

• Troubleshooting BGP peering issues

• BGP Route Updates and Route Propagation

• BGP Route Filtering

From the perspective of BGP, an autonomous system (AS) is a collection of routers under a single organization’s control. Organizations requiring connectivity to the Internet must obtain an autonomous system number (ASN). ASNs were originally 2 bytes (16-bit) providing 65,535 ASNs. Due to exhaustion, RFC 4893 expands the ASN field to accommodate 4 bytes (32-bit). This allows for 4,294,967,295 unique ASNs, providing quite a leap from the original 65,535 ASNs. The Internet Assigned Numbers Authority (IANA) is responsible for assigning all public ASNs to ensure that they are globally unique.

Two blocks of private ASNs are available for any organization to use as long as they are never exchanged publicly on the Internet. ASNs 64,512 to 65,535 are private ASNs within the 16-bit ASN range, and 4,200,000,000 to 4,294,967,294 are private ASNs within the extended 32-bit range.

Network engineers and vendors continue to add functionality and feature enhancements to BGP. BGP now provides a scalable control plane for signaling for overlay technologies like Multiprotocol Label Switching (MPLS) Virtual Private Networks (VPN), IPsec Security Associations, and Virtual Extensible Lan (VXLAN). These overlays provide Layer 3 connectivity via MPLS L3VPNs, or Layer 2 connectivity via Ethernet VPNs (eVPN).

Every address-family maintains a separate database and configuration for each protocol (address-family + subaddress-family) in BGP. This allows for a routing policy in one address-family to be different from a routing policy in a different address-family, even though the router uses the same BGP session to the other router. BGP includes an AFI and SAFI with every route advertisement to differentiate between the AFI and SAFI databases. Table 11-1 provides a small list of common AFI and SAFIs used with BGP.

• Well-known mandatory

• Well-known discretionary

• Optional transitive

• Optional nontransitive

Per RFC 4271, well-known attributes must be recognized by all BGP implementations. Well-known mandatory attributes must be included with every prefix advertisement, whereas well-known discretionary attributes may or may not be included with the prefix advertisement.

Optional attributes do not have to be recognized by all BGP implementations. Optional attributes can be set so that they are transitive and stay with the route advertisement from AS to AS. Other PAs are nontransitive and cannot be shared from AS to AS. In BGP, the Network Layer Reachability Information (NLRI) is the routing update that consists of the network prefix, prefix-length, and any BGP PAs for that specific route.

The BGP attribute AS_PATH is a well-known mandatory attribute and includes a complete listing of all the ASNs that the prefix advertisement has traversed from its source AS. The AS_PATH is used as a loop-prevention mechanism in the BGP protocol. If a BGP router receives a prefix advertisement with its AS listed in the AS_PATH, it discards the prefix because the router thinks the advertisement forms a loop.

• Internal BGP (iBGP): Sessions established with an iBGP router that are in the same AS or participate in the same BGP confederation. iBGP sessions are considered more secure, and some of BGP’s security measures are lowered in comparison to EBGP sessions. iBGP prefixes are assigned an administrative distance (AD) of 200 upon installing into the router’s Routing Information Base (RIB).

• External BPG (EBGP): Sessions established with a BGP router that are in a different AS. EBGP prefixes are assigned an AD of 20 upon installing into the router’s RIB.

BGP uses TCP port 179 to communicate with other routers. Transmission Control Protocol (TCP) allows for handling of fragmentation, sequencing, and reliability (acknowledgement and retransmission) of communication (control plane) packets. Although BGP can form neighbor adjacencies that are directly connected, it can also form adjacencies that are multiple hops away. Multihop sessions require that the router use an underlying route installed in the RIB (static or from any routing protocol) to establish the TCP session with the remote endpoint.

BGP can be thought of as a control plane routing protocol or as an application, because it allows for the exchanging of routes with peers multiple hops away. BGP routers do not have to be in the data plane (path) to exchange prefixes, but all routers in the data path need to know all the routes that will be forwarded through them.

Router-IDs typically represent an IPv4 address that resides on the router, such as a loopback address. Any IPv4 address can be used, including IP addresses not configured on the router. NX-OS uses the command router-id router-id under the BGP router configuration to statically assign the BGP RID. Upon changing the router-id, all BGP sessions reset and need to reestablish.

The Hold Time attribute sets the Hold Timer in seconds for each BGP neighbor. Upon receipt of an UPDATE or KEEPALIVE, the Hold Timer resets to the initial value. If the Hold Timer reaches zero, the BGP session is torn down, routes from that neighbor are removed, and an appropriate update route withdraw message is sent to other BGP neighbors for the impacted prefixes. The Hold Time is a heartbeat mechanism for BGP neighbors to ensure that the neighbor is healthy and alive.

When establishing a BGP session, the routers use the smaller Hold Time value contained in the two router’s OPEN messages. The Hold Time value must be set to at least 3 seconds, or zero. For Cisco routers the default hold timer is 180 seconds.

• Idle

• Connect

• Active

• OpenSent

• OpenConfirm

• Established

Figure 11-1 displays the BGP FSM and the states in order of establishing a BGP session.

Figure 11-1 BGP Finite State Machine

If an error causes BGP to go back to the Idle state for a second time, the ConnectRetryTimer is set to 60 seconds and must decrement to zero before the connection is initiated again. Further failures to leave the Idle state result in the ConnectRetryTimer doubling in length from the previous time.

If the ConnectRetry timer depletes before this stage is complete, a new TCP connection is attempted, the ConnectRetry timer is reset, and the state is moved to Active. If any other input is received, the state is changed to Idle.

During this stage, the neighbor with the higher IP address manages the connection. The router initiating the request uses a dynamic source port, but the destination port is always 179.

• BGP versions must match.

• The source IP Address of the OPEN message must match the IP address that is configured for the neighbor.

• The AS number in the OPEN message must match what is configured for the neighbor.

• BGP Identifiers (RID) must be unique. If a RID does not exist, this condition is not met.

• Security Parameters (Password, Time to Live [TTL], and so on)

If the Open messages do not have any errors, the Hold Time is negotiated (using the lower value), and a KEEPALIVE message is sent (assuming the value is not set to zero). The connection state is then moved to OpenConfirm. If an error is found in the OPEN message, a Notification message is sent, and the state is moved back to Idle.

If TCP receives a disconnect message, BGP closes the connection, resets the ConnectRetryTimer, and sets the state to Active. Any other input in this process results in the state moving to Idle.

Step 1. Create the BGP routing process. Initialize the BGP process with the global configuration command router bgp as-number.

Step 2. Assign a BGP router-id. Assign a unique BGP router-id under the BGP router process. The router-id can be an IP address assigned to a physical interface or a Loopback interface.

Step 3. Initialize the address-family. Initialize the address-family with the BGP router configuration command address-family afi safi so it can be associated to a BGP neighbor.

Step 4. Identify the BGP neighbor’s IP address and autonomous system number. Identify the BGP neighbor’s IP address and autonomous system number with the BGP router configuration command neighbor ip-address remote-as as-number.

Step 5. Activate the address-family for the BGP neighbor. Activate the address-family for the BGP neighbor with the BGP neighbor configuration command address-family afi safi.

Examine the topology shown in Figure 11-2. This topology is used as reference for the next section as well. In this topology, Nexus devices NX-1, NX-2, and NX-4 are part of AS 65000, whereas router NX-6 belongs to AS 65001.

Figure 11-2 Reference Topology

Example 1-4 displays the BGP configuration for router NX-4 demonstrating both IBGP and EBGP peering. For this example, NX-4 is trying to establish an IBGP peering with NX-1 and an EBGP peering with NX-6. While configuring a BGP peering, it is important to ensure the following information is correct:

• Local and remote ASN

• Source peering IP

• Remote peering IP

• Authentication passwords (optional)

• EBGP-multihop (EBGP only)

In Example 11-1, NX-4 is forming an IBGP peering with NX-1 and an EBGP peering with NX-6 router. The NX-4 device is also advertising its loopback address under the IPv4 address family using the network command.

Example 11-1 NX-OS BGP Configuration

Click here to view code image

After the configuration is performed on NX-4, peering should be established between NX-1 and NX-4 as well as between NX-4 and NX-6. The BGP peerings are verified using the command show bgp afi safi summary, where afi and safi are used for different address families. In this case,  IPv4 unicast address family is used. Examine the verification of BGP peering between the NX-1, NX-4, and NX-6, as shown in Example 11-2. Notice that both the IBGP and EBGP peering is established on the NX-4 switch, and a prefix is being learned from each neighbor.

Example 11-2 NX-OS BGP Peering Verification

Click here to view code image

After the BGP peering is established, the BGP prefixes are verified using the command show bgp afi safi. This command lists all the BGP prefixes in the respective address families. Example 11-3 displays the output of the BGP prefixes on NX-4. In the output, the BGP table holds locally advertised prefixes with the next-hop value of 0.0.0.0, the next-hop IP address, and a flag to indicate whether the prefix was learned from an IBGP (i) or EBGP (e) peer.

Example 11-3 NX-OS BGP Table Output

Click here to view code image

On NX-OS, the BGP process is instantiated the moment the router bgp asn command is configured. The details of the BGP process and the summarized configuration are viewed using the command show bgp process. This command displays the BGP process ID, state, number of configured and active BGP peers, BGP attributes, VRF information, redistribution and relevant route-maps used with various redistribute statements, and so on. If there is a problem with the BGP process, this command can be viewed to verify the state of BGP along with the memory information of the BGP process. Example 11-4 displays the output of the command show bgp process, highlighting some of the important fields in the output in Example 11-3.

Example 11-4 NX-OS BGP Process

Click here to view code image

• BGP peering down

• Flapping BGP peer

BGP peering issues are one of the most common issues that are experienced by network operators in the production environment. Though one of the common issues, the impact of down peer or a flapping BGP peer can be from very minimal (if there is redundancy in the network) to huge (where the peering to the Internet provider is completely down). This section focuses on troubleshooting both issues.

• During establishment of BGP sessions because of misconfiguration

• Triggered by network migration or event, software or hardware upgrades

• Failure to maintain BGP keepalives due to transmission problems

A down BGP peer state is in either an Idle or Active state. From the peer state standpoint, these states would mean the following possible problems:

• Idle State

  • No connected route to peer

• Active State

  • No route to peer address (IP connectivity not present)

  • Configuration error, such as update-source missing or wrongly configured

• Idle/Active State

  • TCP establishes but BGP negotiation fails; for example, misconfigured AS

  • Router did not agree on the peering parameters

The following subsections list the various steps involved in troubleshooting BGP peering down issues.

• Local AS number

• Remote AS number

• Verifying the network topology and other documentations

It is important to understand the traffic flow of BGP packets between peers. The source IP address of the BGP packets still reflects the IP address of the outbound interface. When a BGP packet is received, the router correlates the source IP address of the packet to the BGP neighbor table. If the BGP packet source does not match an entry in the neighbor table, the packet cannot be associated to a neighbor and is discarded.

In most of the deployments, the iBGP peering is established over loopback interface, and if the update-source interface is not specified, the session does not come up. The explicit sourcing of BGP packets from an interface is verified by ensuring that the update-source interface-id command under the neighbor ip-address configuration section is correctly configured for the peer.

If there are multiple hops between the EBGP peers, then proper hop count is required. Ensure the ebgp-multihop [hop-count] is configured with the correct hop count. If the hop-count is not specified, the default value is set to 255. Note that the default TTL value for IBGP sessions is 255 whereas the default value of EBGP session is 1. If an EBGP peering is established between two directly connected devices but over the loopback address, users can also use the disable-connected-check command instead of using the ebgp-multihop 2 command. This command disables the connection verification mechanism, which by default, prevents the session from getting established when the EBGP peer is not in the directly connected segment.

Another configuration that is important, although optional, for successful establishment of a BGP session is peer authentication. Misconfiguration or typo errors in authentication passwords will cause the BGP session to fail.

Example 11-5 Ping with Source Interface as Loopback

Click here to view code image

Using the preceding ping methods, reachability is verified for both the IBGP and EBGP peers. But if there is a problem with the reachability, use the following procedure to isolate the problem or direction of the problem.

Identify the direction of packet loss. The show ip traffic command on NX-OS is used to identify the packet loss or direction of the packet loss. If there is a complete or random packet loss of the ping (ICMP) packets from source to destination, use this method. The command output has the section of ICMP Software Processed Traffic Statistics, which consists of two subsections: Transmission and Reception. Both the sections consist of statistics for echo request and echo reply packets. To perform this test, first ensure that the sent and receive counters are stable (not incrementing) on both the source and the destination devices. Then initiate the ping test toward the destination by specifying the source interface or IP address. After the ping is completed, verify the show ip traffic command to validate the increase in counters on both sides to understand the direction of the packet loss. Example 11-6 demonstrates the method for isolating the direction of packet loss. In this example, the ping is initiated from NX-1 to NX-4 loopback. The first output displays that the echo request packets received at 10 and the echo reply sent are 10 as well. After the ping test from NX-1 to NX-4 loopback, the counters increase to 15 for both echo request and echo reply.

Example 11-6 Ping Test and show ip traffic Command Output

Click here to view code image

Similarly, the outputs are verified on NX-1 as well for echo reply received counters. In the previous example, the ping test is successful, and thus both the echo request received and echo reply sent counters incremented, but in situations when the ping test is failing, it is worth checking these counters closely and with multiple iterations of test. If the ping to the destination device is failing but still both the counters increment on the destination device, the problem could be with the return path, and the users may have to check the path for the return traffic.

ACLs prove to be really useful when troubleshooting packet loss or reachability issues. Configuring an ACL matching the source and the destination IP helps to confirm whether the packet has actually reached the destination router. The only caution that needs to be taken is that while configuring ACL, permit ip any any should be configured at the end, or else it could cause the other packets to get dropped and thus cause a service impact.

Example 11-7 shows how the ACL configuration should look if BGP is passing through that link. The example shows the configuration for both IPv4 as well as ipv6 access-list in case of IPv6 BGP sessions. For applying IPv4 ACL on interface, ip access-group access-list-name {in|out} command is used on all platforms. For IPv6 ACL, ipv6 traffic-filter access-list-name {in|out} interface command is used on NX-OS.

Example 11-7 ACL for Permitting BGP Traffic

Click here to view code image

Other than having ACLs configured on the edge devices, lot of deployments have firewalls to protect the network from unwanted and malicious traffic. It is a better option to have a firewall installed than to have a huge ACL configured on the routers and switches. Firewalls can be configured in two modes:

• Routed mode

• Transparent mode

In routed mode, the firewall has routing capabilities and is considered to be a routed hop in the network. In transparent mode, the firewall is not considered as a router hop to the connected device but merely acts like a bump in the wire. Thus, if an EBGP session is being established across a transparent firewall, ebgp-multihop might not be required, and even if it is required to configure ebgp-multihop due to multiple devices in the path, the firewall is not counted as another routed hop.

Firewalls implement various security levels for the interfaces. For example, the ASA Inside interface is assigned a security level of 100 and the Outside interface is assigned security level 0. An ACL needs to be configured to permit the relevant traffic from the least secure interface going toward the higher security interface. This rule applies for both routed as well as transparent mode firewalls, and ACL is required in both cases.

Bridge groups are configured in transparent mode firewall for each network to help minimize the overhead on security contexts. The interfaces are made part of a bridge group and a Bridge Virtual Interface (BVI) interface is configured with a management IP address.

Example 11-8 displays an ASA ACL configuration that allows ICMP as well as BGP packets to traverse across the firewall and shows how to assign the ACL to the interface. Any traffic that is not part of the ACL is dropped.

Example 11-8 Configuration on Transparent Firewall

Click here to view code image

In the access-list named Out, though, both the statements permitting the BGP packets are not required, but it is good practice to have both.

Another problem users might run into with a firewall in middle is with a couple of features on an ASA firewall:

• Sequence number randomization

• Enabling TCP Option 19 for MD5 authentication

ASA firewalls by default perform sequence number randomization and thus can cause BGP sessions to flap. Also, if the BGP peering is secured using MD5 authentication, enable TCP option 19 on the firewall’s policy.

Example 11-9 TCP Socket Connections

Click here to view code image

If BGP peering is not getting established, it may be possible that there is a stale entry in the TCP table. The stale entry may show the TCP session to be in established state and thus prevent the router from initiating another TCP connection, thus preventing the router from establishing a BGP peering.

A good troubleshooting technique for down BGP peers is using Telnet on TCP port 179 toward the destination peer IP and using local peering IP as the source. This technique helps ensure that the TCP is not getting blocked or dropped between the two BGP peering devices. This test is useful for verifying any TCP issues on the destination router and also helps verify any ACL that could possibly block the BGP packets.

Example 11-10 shows the use of Telnet on port 179 from NX-1 (192.168.1.1) to NX-4 (192.168.4.4) to verify BGP session. When this test is performed, the BGP TCP session gets established but is closed/disconnected immediately.

Example 11-10 Using Telnet to Port 179

Click here to view code image

If the telnet is not sourced from the interface or IP that the remote device is configured to form a BGP neighborship with, the Telnet request is refused. This is another way to confirm that the peering device configuration is as per the documentation or not.

When troubleshooting TCP connection issues, it is also important to check the event-history logs for a netstack process as well. Netstack is an implementation of a Layer-2 to Layer-4 stack on NX-OS. It is one of the critical components involved in the control plane on NX-OS. If there is a problem with establishing a TCP session on a Nexus device, it could be a problem with the netstack process. The show sockets internal event-history events command helps understand what TCP state transitions happened for the BGP peer IP.

Example 11-11 demonstrates the use of the show sockets internal event-history events command to see the TCP session getting closed for BGP peer IP 192.168.2.2, but it does not show any request coming in.

Example 11-11 show sockets internal event-history events Command

Click here to view code image

• Unsupported version number

• Wrong Peer AS

• Bad or wrong BGP router ID

• Unsupported optional parameters

Out of the reasons  listed, wrong peer AS or bad BGP Identifier are the most common OPEN message errors and are usually caused due to documentation or human error. The notification messages are also self-explanatory for the two errors and clearly indicate the wrong value and the expected value in the notification message, as shown in Example 11-12. In this example, the router is expecting the peer AS to be in AS 65001 but it's receiving the AS 65002.

Example 11-12 BGP Wrong Peer AS Notification Message

Click here to view code image

During the initial BGP negotiation between the BGP speakers, certain capabilities are exchanged. If any of the BGP speakers are receiving a capability that they do not support, BGP detects an OPEN message error for unsupported capability (or unsupported optional parameter). For instance, one of the BGP speakers is having the capability of enhanced route refresh, but the BGP speaker on the receiving end is running an old software that does not have the capability, then it detects this as an OPEN message error. The following optional capabilities are negotiated between the BGP speakers:

• Route Refresh capability

• 4-byte AS capability

• Multiprotocol capability

• Single/Multisession capability

To overcome the challenges of unsupported capability, use the command dont- capability-negotiate under the BGP neighbor configuration mode. This command disables the capability negotiations between the BGP peers and allows the BGP peer to come up.

When a BGP peer is down, and all the other troubleshooting steps are not helping figure out where the problem is, enable debugs enabled to see if the router is generating and sending the necessary BGP packets, and if it's receiving the relevant packets or not. However, debug is not required on NX-OS because the traces in BGP have sufficient information to debug the problem. There are several debugs that are available for BGP. Depending on the state in which BGP is stuck, certain debug commands are helpful.

For a BGP peering down situation, one of the key debugs used is for BGP keepalives. The BGP keepalive debug is enabled using the command debug bgp keepalives. In the debug output, the two important factors to consider for ensuring a successful BGP peering are as follows:

• If the BGP keepalive is being generated at regular intervals

• If the BGP keepalive is being received at regular intervals

If the BGP keepalive is being generated at regular intervals but the BGP peering still remains down, it may be possible that the BGP keepalive couldn’t make it to the other end, or it reached the peering router but was not processed or dropped. In such cases, BGP keepalive debugs are useful. Enable the debug command debug bgp keepalives to verify whether the BGP keepalives are being sent and received. Example 11-13 illustrates the use of BGP keepalive debug. The first output helps the user verify that the BGP keepalive is being generated every 60 seconds. The second output shows the keepalive being received from the remote peer 192.168.1.1.

Example 11-13 BGP Keepalive Debugs

Click here to view code image

Figure 11-3 BGP Notification Header

In addition to the fixed-size BGP message header, a notification contains the following, as shown in Figure 11-4.

Figure 11-4 Notification Section Information in BGP Header

The Error code and Error-Subcode values are defined in RFC 4271. Table 11-3 shows all the Error codes, Error-Subcode and their interpretation.

Whenever a notification is generated, the error code and the subcode are always printed in the message. These notification messages are really helpful when troubleshooting down peering issues or flapping peer issues.

The methodology for troubleshooting IPv6 BGP peers is same as that of IPv4 BGP peers. Here are a few steps you can use to troubleshoot down peering issues for IPv6 BGP neighbors:

Step 1. Verify the configuration for correct peering IPv6 addresses, AS numbers, update-source interface, authentication passwords, EBGP multihop configuration.

Step 2. Verify reachability using the ping ipv6 ipv6-neighbor-address [source interface-id | ipv6-address].

Step 3. Verify the TCP connections using the command show socket connection tcp on NX-OS. In case of IPv6, check for TCP connections for source and destination IPv6 addresses and one of the ports as port 179.

Step 4. Verify any IPv6 ACL’s in path. Like IPv4, the IPv6 ACLs in the path should permit for TCP connections on port 179 and ICMPv6 packets that can help in verifying reachability.

Step 5. Debugs. On NX-OS switches, use the debug bgp ipv6 unicast neighbors ipv6-neighbor-address debug command to capture IPv6 BGP packets. Before enabling the debugs, enable the debug logfile for BGP debug. For filtering the debugs for a particular IPv6 neighbor, use the IPv6 ACL to filtering the debug output for that particular neighbor.

• Idle/Active: Discussed in previous section

• Idle/Established: Bad update, TCP problem (MSS size in multihop deployment)

Flapping BGP peers could be due to one of several reasons:

• Bad BGP update

• Hold Timer expired

• MTU mismatch

• High CPU

• Improper control-plane policing

• Bad link carrying the update or bad hardware

• Problem with BGP update packaging

• Malicious update generated or the UPDATE packet modified by an attacker (hacker)

Whenever a BGP update is corrupted, a BGP notification is generated with the error code of 3, as shown in Table 11-3. When an error is noticed in the BGP update, BGP generates a hex-dump of the bad update message, which can be further decoded to understand which section of the update was corrupted. Along with the hex-dump, BGP also generates a log message that explains what kind of update error has occurred, as shown in Example 11-14.

Example 11-14 Corrupt BGP Update Message

Click here to view code image

Use the command debug bgp packets to view the BGP messages in hexdump, which can be further decoded. If too many BGP updates and messages are being exchanged on the NX-OS devices, a better option is to perform an Ethanalyzer or SPAN to capture a malformed BGP update packet to further analyze it.

• Interface/platform drops

• MTS queue stuck

• Control-plane policy drops

• BGP Keepalive generation

• MTU issues

One reason may be Interface/platform drops. Various Interface issues like a physical layer issue or drops on the interface can lead to the BGP session getting flapped due to Hold Timer expiry. If the interface is carrying excessive traffic or even the line card itself is overloaded or busy, the packets may get dropped on the interface level or on the line card ASIC. If the BGP keepalive or update packets are dropped in such instances, BGP may notify the peer of Hold Timer expiry.

Another possibility is that the MTS queue is stuck. Sometimes, BGP Keepalives have arrived at the TCP receiving queue but are not being processed and moved to the BGP InQ. This is noticed when the BGP InQ queues are empty and a BGP neighbor goes down due to Hold Timer expiry. The most common reason for such a scenario on Nexus switches is because the MTS queue is stuck on either the BGP or TCP process. MTS is the main component that takes care of carrying information from one component to another component within NX-OS. In such scenarios, it may be possible that multiple BGP peers may get impacted on the system. To recover, a supervisor switchover or a reload may be required.

In addition, CoPP policy drops can also be a cause. The CoPP policy is designed to prevent the CPU from excessive and unwanted traffic. But a poorly designed CoPP policy causes control-plane protocol flaps. If the CoPP policy has not been accommodated to take care of all the BGP control-plane packets and the number of BGP peers on the router, there might be instances where those packets get dropped. In such situations, users might experience random BGP flaps due to CoPP policy dropping certain packets.

• At what time of the day is the BGP flap happening?

• How frequently is the flap happening?

• How is the traffic load on the interface/system when the flap occurs?

• Is the CPU high during the time of the flap? If yes, is it due to traffic or a particular process?

These questions help lay out a pattern for the BGP flaps, and relevant troubleshooting can be performed around the same time. To further troubleshoot the problem, understand that the BGP flap is due to two reasons:

• Either keepalives getting generated at regular intervals but not leaving the router or not making it to the other end.

• Keepalives are not getting generated at regular intervals.

If the keepalives are getting generated at regular intervals but not leaving the router, then notice that the OutQ for the BGP peer keeps piling up. The OutQ keeps incrementing due to keepalive generation, but the MsgSent does not increase, which may be an indication that the messages are stuck in the OutQ. Example 11-15 illustrates such a scenario where the BGP keepalives are generated at regular intervals but do not leave the router, leading to a BGP flap due to hold timer expiry. Notice that in this example, the OutQ value increases from 10 to 12, but the MsgSent counter is stagnant at 3938. In this scenario, the peering may flap every BGP hold timer.

Example 11-15 BGP Message Sent and OutQ

Click here to view code image

But if the device experiences random BGP flaps and at irregular intervals, it is possible that the BGP keepalives are getting generated at regular intervals, although the flaps may still happen frequently. For instance, a BGP peering flaps between 4 to 10 minutes. These issues are hard to troubleshoot and may require a different technique than just running show commands. The reason is that it is not easy to isolate which device is not generating the keepalive in a timely manner, or if the keepalive is generated in a timely manner but there is a delay that occurs when the keepalive makes it to the remote peer. To troubleshoot, follow the two-step process between the two ends of the BGP connection.

Step 1. Enable BGP keepalive debug on both routers along with the debug logfile.

Step 2. Enable Ethanalyzer on both routers.

The purpose of enabling Ethanalyzer or any other packet capture tool (based on the underlying platform) is that it is possible that the BGP keepalives reach the other end in a timely manner, but those keepalives may be delayed before reaching BGP process itself. Based on the outputs of the BGP keepalive debug and the Ethanalyzer from the far end device, the timelines could be matched to conclude where exactly the delay might be happening that is causing the BGP to flap. It may be the BGP process that is delaying the keepalive generation, or it may be the other components that interact with BGP to delay the keepalive processing.

• Improper planning and network design

• Device not supporting Jumbo MTU or certain MTU values

• Change due to application requirement

• Change due to end customer requirement

BGP sends updates based on the Maximum Segment Size (MSS) value calculated by TCP. If Path-MTU-Discovery (PMTUD) is not enabled, the BGP MSS value defaults to 536 bytes as defined in RFC 879. The problem with that is, if a huge number of updates are getting exchanged between the two routers at the MSS value of 536 bytes, convergence issues will be noticed and thus an inefficient use of the network. The reason is that the interface with an MTU size of 1500 is capable of sending nearly three times the MSS value and can be much higher if the interface supports jumbo MTU, but it has to break down the updates in chunks of 536 bytes.

Defined in RFC 1191, PMTUD is introduced to reduce the chances of IP packets getting fragmented along the path and thus helping with faster convergence. Using PMTUD, the source identifies the lowest MTU along the path to the destination and thus decides what packet size should be sent.

How does PMTUD work? When the source generates a packet, it sets the MTU size equal to the outgoing interface with a DF (Do-Not-Fragment) bit set. For any intermediate device that receives the packet and has an MTU value of its egress interface lower than the packet it received, the device drops the packet and sends an ICMP error message with Type 3 (Destination Unreachable) and Code 4 (Fragmentation needed and DF bit set) along with the MTU information of the outgoing interface in the Next-Hop MTU field back toward the source. When the source receives the ICMP unreachable error message, it modifies the MTU size of the outgoing packet to the value specified in the Next-Hop MTU field above. This process continues until the packet successfully reaches the final destination.

BGP also supports PMTUD. PMTUD allows a BGP router to discover the best MTU size along the path to a neighbor to ensure efficient usage of exchanging packets. With Path MTU discovery enabled, the initial TCP negotiation between two neighbors has MSS value equal to (IP MTU − 20 byte IP Header − 20 byte TCP Header) and DF bit set. Thus, if the IP MTU value is 1500 (equal to the interface MTU) then the MSS value is 1460. If the device in the path has a lower MTU or even if the destination router has a lower MTU—for example, 1400, then the MSS value is negotiated based on 1400−40 bytes = 1360 bytes. To derive MSS calculation, use the following formulas:

• MSS without MPLS = MTU − IP Header (20 bytes) − TCP Header (20 bytes)

• MSS over MPLS = MTU − IP Header − TCP Header − n*4 bytes (where n is the number of labels in the label stack)

• MSS across GRE Tunnel = MTU − IP Header (Inner) − TCP Header − [IP Header (Outer) + GRE Header (4 bytes)]

Now the question is why the MTU mismatch causes BGP sessions to flap? When the BGP connection is established, the MSS value is negotiated over the TCP session. When the BGP update is generated, BGP updates are packaged in the BGP update message, which can hold prefixes and header information to the maximum capacity of the MSS bytes. These BGP update messages are then sent to the remote peer with the do-not-fragment (df-bit) set. If a device in path or even the destination is not able to accept the packets with a higher MTU, it sends an ICMP error message back to BGP speaker. The destination router either waits for the BGP Keepalive or BGP Update packet to update its hold down timer. After 180 seconds, the destination router sends a Notification back to Source with a Hold Time expired error message.

Example 11-16 illustrates a BGP peer flapping problem when there is a MTU mismatch in the path. Consider the same set of devices NX-1, NX-2, NX-4, and NX-6 from the topology shown in Figure 11-2. In this topology, assume the devices have ICMP unreachable disabled on its interfaces. The NX-6 device is advertising 10,000 prefixes to NX-4, which is being further advertised toward NX-1. The interface MTU on NX-1 and NX-4 is set to 9100, whereas the MTU on the interface on NX-2 facing NX-1 is still set to the default; that is, 1500. Because the path MTU discovery (PMTUD) is enabled, the MSS is negotiated to value 9060. The ICMP unreachable message is denied because the lower MTU setting on the NX-2 interface is not received by NX-1.

Example 11-16 BGP Flaps due to MSS Issue

Click here to view code image

The BGP flap does not occur when a small amount of prefixes are exchanged between the peers because the BGP packet size is under 1460 bytes. One symptom of BGP flaps due to MSS/MTU issues is a repetitive BGP flap that occurs because the Hold Timer expires.

The following are the few possible causes of BGP session flapping due to MTU mismatch:

• The interface MTU on both the peering routers do not match.

• The Layer 2 path between the two peering routers does not have consistent MTU settings.

• PMTUD didn’t calculate correct MSS for the TCP BGP session.

• BGP PMTUD could be failing due to blocked ICMP messages by a router or a firewall in path.

To verify there are MTU mismatch issues in the path, perform an extended ping test by setting the size of the packet as the outgoing interface MTU value along with DF bit set. Also, ensure that ICMP messages are not being blocked in the path to have PMTUD function properly. Ensure that the MTU values are consistent throughout the network with a proper review of the configuration.

Perform a ping test to remote peer with the packet size as the MTU of the interface and do not fragment (df-bit) set as shown in Example 11-17.

Example 11-17 PING with DF-Bit Set

Click here to view code image

Figure 11-5 BGP Route Processing

Let’s now understand the various fundamentals of route advertisement in the sections that follow. For this section, examine the topology shown in Figure 11-6.

Figure 11-6 BGP Route Propagation Topology

• Network statement: Using network ip-address/length command.

• Redistribution: Redistribute directly connected links, static routes, and IGP, such as Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Enhanced Interior Gateway Routing Protocol (EIGRP), Intermediate System to Intermediate System (IS-IS), and Locator ID Separation Protocol (LISP). A route-map is required when a prefix is being redistributed from another routing protocol including directly connected links.

• Aggregate Route: Summarizing a route, though the component route must exist in the BGP table.

• Default Route: Using the default-information originate command.

Example 11-18 Prefix Advertisement Using network Command

Click here to view code image

Example 11-19 BGP and IGP Redistribution

Click here to view code image

There are a few caveats when performing redistribution for OSPF and IS-IS as listed:

• OSPF: When redistributing OSPF into BGP, the default behavior includes only routes that are internal to OSPF. The redistribution of external OSPF routes requires a conditional match on route-type under route-map.

• IS-IS: IS-IS does not include directly connected subnets for any destination routing protocol. This behavior is overcome by redistributing the connected networks into BGP.

Example 11-20 displays the various match route-type options available under the route-map. The route-type options are available for both OSPF and IS-IS route types.

Example 11-20 match route-map Command Options

Click here to view code image

Example 11-21 demonstrates the use of the summary-only attribute with the aggregate-address command. Notice that NX-2 has 3 prefixes but only a single aggregate prefix gets advertised to NX-5. Notice that on NX-2, when the summary-only command is configured, the more specific routes are suppressed.

Example 11-21 Route Aggregation

Click here to view code image

BGP uses three tables for maintaining the network prefix and path attributes (PA)s for a route. The following BGP tables are briefly explained:

• Adj-RIB-in: Contains the NLRIs in original form before inbound route policies are processed. The table is purged after all route-policies are processed to save memory.

• Loc-RIB: Contains all the NLRIs that originated locally or were received from other BGP peers. After NLRIs pass the validity and next-hop reachability check, the BGP best path algorithm selects the best NLRI for a specific prefix. The Loc-RIB table is the table used for presenting routes to the ip routing table.

• Adj-RIB-out: Contains the NLRIs after outbound route-policies have processed. A BGP route may contain multiple paths to the same destination network. Every path’s attributes impact the desirability of the route when a router selects the best path. A BGP router advertises only the best path to the neighboring routers.

Inside the BGP Loc-RIB table, all the routes and their path attributes are maintained with the best path calculated. The best path is then installed in the RIB of the router. In the event the best path is no longer available, the router can use the existing paths to quickly identify a new best path. BGP recalculates the best path for a prefix upon four possible events:

• BGP next-hop reachability change

• Failure of an interface connected to an EBGP peer

• Redistribution change

• Reception of new paths for a route

The BGP best path selection algorithm influences how traffic enters or leaves an autonomous system (AS). BGP does not use metrics to identify the best path in a network. BGP uses path attributes to identify its best path. But even before BGP influences the best path selection using PAs, the router looks for the longest prefix match for the routes present in the RIB and prefers that route to be installed in the forwarding information base (FIB).

BGP path attributes are modified upon receipt or advertisement to influence routing in the local AS or neighboring AS. A basic rule for traffic engineering with BGP is that modifications in outbound routing policies influence inbound traffic, and modifications to inbound routing policies influence outbound traffic.

BGP installs the first received path as the best path automatically. When additional paths are received, the newer paths are compared against the current best path. If there is a tie, processing continues onto the next step, until a best path winner is identified.

The following list provides the attributes that the BGP best path algorithm uses for the best route selection process. These attributes are processed in the order listed in Table 11-5.

The best path algorithm is used to manipulate network traffic patterns for a specific route by modifying various path attributes on BGP routers. Changing of BGP PA influences traffic flow into, out of, and around an autonomous system (AS). The BGP routing policy varies from organization to organization based upon the manipulation of the BGP PAs. Because some PAs are transitive and carry from one AS to another AS, those changes could impact downstream routing for other SPs, too. Other PAs are nontransitive and influence only the routing policy within the organization. Network prefixes are conditionally matched on a variety of factors, such as AS-Path length, specific ASN, BGP communities, or other attributes.

Examining the topology shown in Figure 11-6, NX-5 and NX-6 advertise their loopback toward AS 65000. When NX-1 receives the loopbacks, it receives it via NX-2 and NX-3 but only one of them is chosen as the best. The command show bgp afi safi ip-address/length displays both the received paths but also displays one of the paths that was not chosen as the best path, as shown in Example 11-22. In this example, initially the path for 192.168.5.5/32 is chosen via NX-2 due to the lowest RID, but when an inbound policy on NX-3 is defined to set a higher local preference, the path via NX-3 is chosen as the best.

Example 11-22 BGP Best Path Selection

Click here to view code image

BGP supports three types of equal cost multipath (ECMP): EBGP multipath, IBGP multipath, or eiBGP multipath. In all three types of BGP multipath, the following BGP path attributes (PA) must match for multipath to be eligible:

• Weight

• Local Preference

• AS-Path length and content (confederations can contain a different AS_CONFED_SEQ path)

• Origin

• MED

• Advertisement method must match (IBGP or EBGP); if the prefix is learned via an IBGP advertisement, the IGP cost must match to be considered equal

Examine the topology shown in Figure 11-6. In this topology, NX-1 learns same prefixes from both NX-2 and NX-3. Because there is an IBGP peering between NX-1, NX-2, and NX-3, the paths learned via NX-1 are internal. To have multiple BGP paths installed in the RIB and BRIB, multipath IBGP is configured on NX-1. Example 11-23 demonstrates the IBGP multipath functionality as explained.

Example 11-23 IBGP Multipath

Click here to view code image

The BGP event-history logs are used to verify the second-best path being added to the Unicast Routing Information Base (URIB). Use the command show bgp event-history detail to view the details for both the best path and the second-best path of a prefix being added to URIB, as shown in Example 11-24. In Example 11-24, first the best path is selected, which is via 192.168.2.2, and then another path is added to the URIB, which is learned via nexthop 192.168.3.3.

Example 11-24 Event-History Logs for BGP Multipath

Click here to view code image

Example 11-25 Debugs for BGP Update and Route Installation in BRIB

Click here to view code image

On NX-OS, debugs are not necessarily required to understand the update generation process. Use the command show bgp event-history detail to view the detailed event logs. The detail option is not available by default and thus is required to be configured under the router bgp configuration using the command event-history detail [size large | medium | small]. Example 11-26 displays the detailed output of the BGP event-history logs showing the same update process. In this example, the update is being generated for NX-3. If the event-history logs are rolled over and the issue still keeps occurring again and again, in such situations debugs can be enabled, as demonstrated in Example 11-25.

Example 11-26 Event-History Logs for BGP Update Generation

Click here to view code image

• Establishing sessions with a number of peers.

• Locally generate all the BGP paths (either via network statement, redistribution of static/connected/IGP routes), and/or from other component for other address-family (for example, Multicast VPN (MVPN) from multicast, L2VPN from l2vpn manager, and so on).

• Send and receive multiple BGP tables; that is, different BGP address-families to/from each peer.

• Upon receiving all the paths from peers, perform best path calculation to find the best path and/or multipath, additional-path, backup path.

• Installing the best path into multiple routing tables like default or VRF routing table.

• Import and export mechanism.

• For other address-family like L2VPN or multicast, pass the path calculation result to different lower layer components.

BGP uses a lot of CPU cycles when processing BGP updates and requires memory for maintaining BGP peers and routes in BGP tables. Based on the role of the BGP router in the network, appropriate hardware should be chosen. The more memory a router has, the more routes it can support, much like how a router with a faster CPU supports larger number of peers.

There are various steps that should be followed to verify whether the BGP has converged and the routes are installed in the BRIB.

If there is a traffic loss, before BGP has completed its convergence for a given address-family, verify the routing information in the URIB and the forwarding information in the FIB. Example 11-27 demonstrates a BGP route getting refreshed. The command show bgp event-history [event | detail] is used to validate that the prefix is installed in BRIB table and that the command show routing event-history [add-route | modify-route | delete-route] used to check the route has been installed in the URIB. In the URIB, verify the timestamp of when the route was downloaded to the URIB. If the prefix was recently downloaded to the URIB, there might have been an event that caused the route to get refreshed. Also, the difference in the time between when the prefix was installing in BRIB and when it was further downloaded to URIB will help understand the convergence time.

Example 11-27 BRIB and URIB Route Installation

Click here to view code image

BGP convergence for relevant address-family is checked using the command show bgp convergence detail vrf all. Example 11-28 shows the output of the show bgp convergence details vrf all command. This command shows when the best-path selection process was started and the time to complete it. Not only that, the command also displays the time taken to converge the prefix to URIB, which can be used to understand how the device is performing from BGP and URIB convergence perspective.

Example 11-28 show bgp convergence detail Command Output

Click here to view code image

If the best-path runs before EOR is received, or if a peer fails to send EOR marker, it can lead to traffic loss. In such situations, enable debug for BGP updates with relevant debug-filters for VRF, address-family, and peer, as shown in Example 11-29.

Example 11-29 Debug Commands with Filter

Click here to view code image

From the debug output, check the event log to look at the timestamp to see when the most recent EOR was sent to the peer. This also shows how many routes were advertised to the peer before the sending of the EOR. A premature EOR sent to the peer can also lead to traffic loss if the peer flushes stale routes early.

If the route in URIB has not been downloaded, it needs to be further investigated because it may not be a problem with BGP. The following commands can be run to check the activity in URIB that could explain the loss:

• show routing internal event-history ufdm

• show routing internal event-history ufdm-summary

• show routing internal event-history recursive

BGP is a heavy protocol because it uses the most CPU and memory resources on a router. Many factors explain why it keeps utilizing more and more resources. The three major factors for BGP memory consumption are as follows:

• Prefixes

• Paths

• Attributes

BGP can hold many prefixes, and each prefix consumes some amount of memory. But when the same prefix is learned via multiple paths, that information is also maintained in the BGP table. Each path adds to more memory. Because BGP was designed to give control to each AS to manage the flow of traffic through various attributes, each prefix can have various attributes per path. This is put down as a mathematical function, where N represents the number of prefixes, M represents the number of paths for a given prefix, and L represents the attributes attached to given prefix:

• Prefixes: (O(N))

• Paths: (O(M × N))

• Attributes: (O(L × M × N))

• Aggregation

• Filtering

• Partial routing table instead of full routing table

With the use of aggregation, multiple specific routes can be aggregated into one route. But aggregation is challenging when tried on a fully deployed running network. After the network is up and running, the complete IP addressing scheme has to be looked at to perform aggregation. Aggregation is a good option for green field deployments. The green field deployments give more control on the IP addressing scheme, which makes it easier to apply aggregation.

Filtering provides control over the number of prefixes maintained in the BGP table or advertised to BGP peers. BGP provides filtering based on prefix, BGP attributes, and communities. One important point to remember is that complex route filtering, or route filtering applied for a large number of prefixes, helps reduce the memory, but the router takes a hit on the CPU.

Many deployments do not require all the BGP speakers to maintain a full BGP routing table. Especially in an enterprise and data center deployments, there is no real need to having the full Internet routing table. The BGP speakers can maintain even a partial routing table containing the most relevant and required prefixes or just a default route toward the Internet gateway. Such designs greatly reduce the resources being used throughout the network and increase scalability.

• Reduce the number of peerings.

• Use RRs instead of IBGP full mesh.

Multiple BGP paths are a direct effect of the multiple BGP peerings. Especially in an IBGP full-mesh environment, the number of BGP sessions increases exponentially and thus the number of paths. A lot of customers increase the number of IBGP neighbors to have more redundant paths, but two paths are sufficient to maintain redundancy. Increasing the number of peerings can cause scaling issues both from the perspective of the number of sessions and from the perspective of BGP memory utilization.

It is a well-known fact that IBGP needs to be in full mesh. Figure 11-7 illustrates an IBGP full-mesh topology. In an IBGP full-mesh deployment of n nodes, there are a total of n*(n−1)/2 IBGP sessions and (n−1) sessions per BGP speaker.

Figure 11-7 IBGP Full Mesh

This not only affects the scalability of an individual node or router but the whole network. To increase the scalability of IBGP network, two design approaches can be used:

• Confederations

• Route Reflectors

• Reduce the number of attributes.

• Filter standard or extended communities.

• Limit local communities.

On NX-OS, use the command show bgp private attr detail to view the various attributes attached to the BGP prefixes. Example 11-30 displays the various global BGP attributes on NX-1. These attributes were learned across various prefixes, including the community attached to the prefix learned from NX-4.

Example 11-30 BGP Attributes Detail

Click here to view code image

There is no method to get rid of the default BGP attributes, but the use of other attributes can be controlled. Using attributes that make things more complex is of no use. For example, using MED and various MED-related commands, such as the command bgp always-compare-med or bgp deterministic-med, can have an adverse impact on the network and can lead to route instability or routing loop conditions. At the same time, the user-assigned attributes will consume more BGP memory, which can easily be avoided.

A peer-policy defines the address-family dependent policy aspects for a peer, including inbound and outbound policy, filter-list and prefix-lists, soft-reconfiguration, and so on. A peer-session template defines session attributes, such as transport details and session timers. Both the peer-policy and peer-session templates are inheritable; that is, a peer-policy or peer-session can inherit attributes from another peer-policy or peer-session, respectively. A peer template pulls the peer-session and peer-policy sections together to allow cookie-cutter neighbor definitions. Example 11-31 illustrates the configuration of BGP templates on NX-1.

Example 11-31 BGP Template Configuration

Click here to view code image

• Hard Reset: Dropping and reestablishing BGP session. Performed by the command clear bgp afi safi [* | ip-address].

• Soft Reset: A soft reset uses filtered prefixes stored in the memory to reconfigure and activate BGP routing tables without tearing down the BGP session. Performed using the command clear bgp afi safi [* | ip address] soft [in | out].

Hard reset of a BGP session is disruptive to an operational network. If a BGP session is reset repeatedly over a short period of time due to multiple changes in BGP policy, it can result in other routers in the network dampening prefixes, causing destinations to be unreachable and traffic to be black-holed.

Soft reconfiguration is a traditional way to allow route-policy to be applied on the inbound BGP route update. BGP soft reconfiguration is enabled using the command soft-reconfiguration inbound under the neighbor configuration mode. When configured, the BGP stores an unmodified copy of all routes received from that peer at all times, even when the routing policies did not change frequently. Enabling soft reconfiguration means that the router also stores prefixes/attributes received prior to any policy application. This causes an extra overhead on memory and CPU on the router.

To manually perform a soft reset, use the command clear bgp ipv4 unicast [* | ip-address] soft [in | out]. The soft-reconfiguration feature is useful when the operator wants to know which prefixes have been sent to a router prior to the application of any inbound policy.

To overcome the challenges of soft-reconfiguration inbound configuration, BGP route refresh capability was introduced and is defined in RFC 2918. The BGP route refresh capability has a capability code of 2 and the capability length of 0. Using the route refresh capability, the router sends out a route refresh request to peer to get the full table from the peer again. The good part of route refresh capability is there is no preconfiguration needed to enable this capability. The ROUTE-REFRESH message is a new BGP message type described in Figure 11-8.

Figure 11-8 BGP Route Refresh Message

The AFI and SAFI in the ROUTE-REFRESH message points to the address-family where the configured peer is negotiating the route refresh capability. The Reserved bits are unused and are set to 0 by the sender and ignored by the receiver.

A BGP speaker sends a ROUTE-REFRESH message only if it has negotiated the route refresh capability with its peer. This implies that all the participating routes should support the route refresh capability. The router sends a route refresh request (REFRESH_REQ) to the peer. After the speaker receives a route refresh request, the BGP speaker readvertises to the peer the Adj-RIB-Out of the AFI and SAFI carried in the message, to its peer. If the BGP speaker has an outbound route filtering policy, the updates are filtered accordingly. The route refresh requesting peer receives the filtered routes.

The clear ip bgp ip-address in or clear bgp afi safi ip-address in command tells the peer to resend the full BGP announcement by sending a route-refresh request. Whereas the clear bgp afi safi ip-address out command resends the full BGP announcement to the peer, it does not initiates a route refresh request. The route refresh capability is verified using the show bgp afi safi neighbor ip-address command. Example 11-32 displays the route refresh capability negotiated between the two BGP peers.

Example 11-32 BGP Route Refresh Capability

Click here to view code image

The BGP refresh request (REFRESH_REQ) is sent in one of the following cases:

• clear bgp afi safi [* | ip-address] in command is issued.

• clear bgp afi safi [* | ip-address] soft in command is issued.

• Adding or changing inbound filtering on the BGP neighbor via route-map.

• When configuring allowas-in for the BGP neighbor.

• Configuring soft-reconfiguration inbound for the BGP neighbor.

• Adding a route-target import to a VRF in MPLS VPN (for AFI/SAFI value 1/128 or 2/128).

RFC 1966 introduces the concept that an iBGP peering can be configured so that it reflects routes to another iBGP peer. The router reflecting routes is known as a route reflector (RR), and the router receiving reflected routes is a route reflector client. The RR design turns an IBGP mesh into a hub-and-spoke design where the RR is the hub router. The RR clients are either regular IBGP peers—that is, they are not directly connected to each other—or the other design could have RR clients that are interconnected. Three basic rules involve route reflectors and route reflection:

• Rule #1: If an RR receives an NLRI from a non-RR client, the RR advertises the NLRI to a RR client. It will not advertise the NLRI to a non-RR client.

• Rule #2: If an RR receives an NLRI from an RR client, it advertises the NLRI to RR client(s) and non-RR client(s). Even the RR client that sent the advertisement receives a copy of the route, but it discards the NLRI because it sees itself as the route originator.

• Rule #3: If an RR receives a route from an EBGP peer, it advertises the route to RR client(s) and non-RR client(s). Only route-reflectors are aware of this change in behavior because no additional BGP configuration is performed on route-reflector clients. BGP route reflection is specific to each address-family. The command route-reflector-client is used on NX-OS devices under the neighbor address-family configuration.

Examine the two RR design scenarios shown in Figure 11-9. The topology in (a) has R1 acting as the RR, whereas R2, R3, and R4 are the RR clients. The topology shown in (b) has a similar setup to that of (a) with a difference that the RR clients are fully meshed with each other.

Figure 11-9 Topology

The RR and the client peers form a cluster and are not required to be fully meshed. Because the topology in (b) has an RR along with fully meshed IBGP client peers, which actually defies the purpose of having RR, the BGP RR reflection behavior should be disabled. The BGP RR client-to-client reflection is disabled using the command no bgp client-to-client reflection. This command is required only on the RR and not on the RR clients. Example 11-33 displays the configuration for disabling BGP client-to-client reflection.

Example 11-33 Disabling BGP Client-to-Client Reflection

Click here to view code image

If a router receives an NLRI with its RID in the Originator attribute, the NLRI is discarded.

If a route reflector receives an NLRI with its cluster-id in the Cluster List attribute, the NLRI is discarded.

Example 11-34 provides a sample prefix output from a route that was reflected by the route reflector NX-1, as shown in Figure 11-9. Notice that the originator ID is the advertising router and that the cluster list contains the route-reflector ID. The cluster list contains the route-reflectors that the prefix traversed in the order of the last route-reflector that advertised the route.

Example 11-34 Output of an RR Reflected Prefix

Click here to view code image

If a topology contains more than one RR and both the RRs are configured with different cluster IDs, the second RR holds the path from the first RR and hence consumes more memory and CPU resources. Having either single cluster-id or multiple cluster-id has its own disadvantages.

• Different cluster-id: Additional memory and CPU overhead on RR

• Same cluster-id: Less redundant paths

If the RR clients are fully meshed within the cluster, no bgp client-to-client reflection command can be enabled on the RR.

NX-OS supports the BGP maximum-prefix feature that allows you to limit the number of prefixes on a per-peer basis. Generally, this feature is enabled for EBGP sessions, but it is also used for IBGP sessions. Although this feature helps scale and prevent the network from an excess number of routes, it is very important to understand when to use this feature. The BGP maximum-prefix feature is enabled in the following situations:

• Know how many BGP routes are anticipated from the peer.

• What actions need to be taken if the number of routes are exceeded. Should the BGP connection be reset or should a warning message be logged?

To limit the number of prefixes, use the command maximum-prefix maximum [threshold] [restart restart-interval | warning-only] for each neighbor. Table 11-6 elaborates each of the fields in the command.

An important point to remember is that when the restart option is configured with the maximum-prefix command, the only other way apart from waiting for the restart-interval timer to expire, to reestablish the BGP connection, is to perform a manual reset of the peer using the clear bgp afi safi ip-address command.

Example 11-35 illustrates the use of the maximum-prefix command. NX-2 is receiving over 10 prefixes from neighbor 10.25.1.5, but the device has set the maximum-prefix limit to 10 prefixes. In such an instance, the BGP peering is shut on the device where maximum-prefix is set, but the remote end peer remains in Idle state. While troubleshooting BGP peering issues, validate the show bgp afi safi neighbors ip-address command to verify the reason for last reset.

Example 11-35 Maximum-Prefixes

Click here to view code image

A lot of times, the as-path prepend option is used to increase the AS-PATH list to make a path with lower AS-PATH list preferred. This operation does not have much of an impact. But from the perspective of the Internet, a longer AS-PATH list cannot only cause convergence issues but can also cause security loopholes. The AS-PATH list actually signifies a router’s position on the Internet.

To limit the maximum number of AS-PATH length supported in the network, the maxas-limit command was introduced. Using the command maxas-limit 1-512 in NX-OS, any route with AS-PATH length higher than the specified number is discarded.

• Prefix-lists

• Filter-lists

• Route-maps

The BGP route-maps provide more dynamic capability as compared to prefix-lists and filter-lists, because it not only allows you to perform route filtering but also allows the network operators to define policies and set attributes that can be further used to control traffic flow within the network. All these route filtering and route policy methods are discussed in future sections.

Example 11-36 displays the BGP table of Nexus switch NX-2 in the topology shown in Figure 11-9. The NX-2 switch is  used as the base to demonstrate all the filtering techniques shown further in this chapter.

Example 11-36 BGP Table on NX-2

Click here to view code image

The prefix-list can be applied directly to a BGP peer and also as a match statement within the route-map. A prefix-list is configured using the command ip prefix-list name [seq sequence-number] [permit ip-address/length | deny ip-address/length] [le length | ge length | eq length]. Examine the same topology as shown in Figure 11-6. Example 11-37 illustrates the configuration of BGP inbound and outbound route filtering using prefix-lists on NX-2. The inbound prefix-list permits for 5 networks, whereas the outbound prefix-list permits for host network entries is /32 prefixes matching in subnet 192.168.0.0/16. When the prefix-lists are configured, use the command show bgp afi safi neighbor ip-address to ensure that the prefix-lists have been attached to the neighbor.

Example 11-37 Prefix-List-Based Route Filtering

Click here to view code image

Example 11-38 displays the output of the BGP table after the prefix-lists have been configured and attached to BGP neighbor 10.25.1.5. Notice that in this example, on the NX-2 switch, only 5 prefixes are seen from neighbor 10.25.1.5. On NX-5, all the loopback addresses of the nodes in AS 65000 are advertised apart from 192.168.44.0/24.

Example 11-38 BGP Table Output After Prefix-List Configuration

Click here to view code image

On the inbound direction on NX-2, use the command show bgp event-history detail to view the details of the prefixes being matched against the prefix-list Inbound. Based on the match, the prefixes are either permitted or denied. If no entry exists for the prefix in the prefix-list, it is dropped by BGP and will not be part of the BGP table. Example 11-39 displays the event-history detail output demonstrating how a prefix 100.1.30.0/24 is rejected or dropped by BGP prefix-list and the prefix 100.1.5.0/24 being permitted at the same time.

Example 11-39 BGP Event-History for Inbound Prefixes

Click here to view code image

For the outbound direction, the show bgp event-history detail command output displays the prefixes in the BGP table being permitted and denied based on the matching entries in the outbound prefix-list named Outbound. After the filtering is performed, the prefixes are then advertised to the BGP peer along with relevant attributes, as shown in Example 11-40.

Example 11-40 BGP Event-History for Outbound Prefixes

Click here to view code image

NX-OS also has CLI to verify policy-based statistics for prefix-lists. The statistics are verified for the policy implied in both inbound and outbound directions and shows the number of prefixes permitted and denied in either direction. Use the command show bgp afi safi policy statistics neighbor ip-address prefix-list [in | out] to view the policy statistics for prefix-list applied on a BGP neighbor. The counters of the policy statistics command increment every time a BGP neighbor flaps or a soft clear is performed on the neighbor. Example 11-41 demonstrates the use of a policy statistics command for BGP peer 10.25.1.5 in both inbound and outbound directions to understand how many prefixes are being permitted and dropped in both inbound and outbound directions. In this example, a soft clear is performed on the outbound direction, and it is seen that the counters increment for the outbound prefix-list policy statistics by 4 for permitted prefixes and 1 for a dropped prefix.

Example 11-41 BGP Policy Statistics for Prefix-List

Click here to view code image

If at any point there is a problem noticed with an inbound or outbound prefix-list on a BGP neighbor, verify the route policy manager (RPM) NX-OS component. The first step of verification is to ensure that the prefix-list is attached to the BGP process or not. This is verified using the command show system internal rpm ip-prefix-list. This command displays the name of the prefix-list and its client information. This command also displays the number of entries present in the prefix-list.

After verifying the prefix-list and its clients, use the command show system internal rpm event-history rsw to ensure the correct prefix-list has been bound to the BGP process. An incorrect binding or a missing binding event-history log can indicate that the prefix-list is not properly associated with the BGP process or the BGP neighbor.

Example 11-42 shows the output of both the preceding commands.

Example 11-42 RPM Client Info and Event-History for Prefix-Lists

Click here to view code image

Example 11-43 BGP Filter-Lists

Click here to view code image

Example 11-44 displays the prefixes in the BGP table received from peer 10.25.1.5 after being filtered by the filter-list. Notice that all the prefixes shown in the BGP table have AS 274 in their AS_PATH list.

Example 11-44 BGP Table with Filter-List Applied

Click here to view code image

The easiest way to verify which prefixes are being permitted and denied is to use the show bgp event-history detail command output, but if the event-history detail command is not enabled under the router bgp configuration, you can enable debugs to verify the updates. The debug bgp updates command can be used to verify both the inbound and the outbound updates. Example 11-45 demonstrates the use of debug bgp updates to verify which prefixes are being permitted and which are being denied. The action of permit or deny is always based on the entries present in the AS-path list.

Example 11-45 debug bgp updates Output

Click here to view code image

Similar to policy statistics for prefix-lists, the statistics are also available for filter-list entries. When executing the command show bgp afi safi policy statistics neighbor ip-address filter-list [in | out], notice the relevant AS-path access list referenced as part of the filter-list command and the number of matches per each entry. The output also displays the number of accepted and rejected prefixes by the filter-list, as displayed in Example 11-46.

Example 11-46 BGP Filter-Lists

Click here to view code image

Because the filter-list uses AS-path access-list, RPM information can be verified for as-path-access-list using the command show system internal rpm as-path-access-list as-path-acl-name. This command confirms if the AS-path access-list is associated with the BGP process. The command show system internal rpm event-history rsw is used to validate if the AS-path access-list is bound to the BGP process. Example 11-47 displays both the command outputs.

Example 11-47 BGP Filter-Lists

Click here to view code image

Example 11-48 illustrates a sample configuration of a multisequence route-map that is applied to a neighbor. Notice that in this example, the route-map sequence 10 is matching on prefix-list to match certain set of prefixes and on sequence 20 matches AS-Path access-list. Note that there is no sequence 30. Absence of any other entry in the route-map acts as an implicit deny statement and denies all prefixes.

Example 11-48 BGP Route-Map Configuration

Click here to view code image

Example 11-49 shows the BGP table after inbound route-map filtering. Notice that the prefixes 100.1.1.0/24 to 100.1.5.0/24 are set with the local preference of 200, whereas the prefixes that match AS 274 in the AS-path list are set with the local preference of 300. Because there is no route-map entry matching sequence 30, all the other prefixes are denied by the inbound route-map filtering.

Example 11-49 BGP Table Output with Route-Map Filtering

Click here to view code image

The show bgp event-history detail command can be used again to verify which prefixes are being permitted or denied based on the route-map policy. Based on the underlying match statements, relevant set actions are taken (if any). Example 11-50 displays the event-history detail output demonstrating prefixes being permitted and denied by route-map.

Example 11-50 BGP Event-History

Click here to view code image

You can also validate the policy statistics for the route-map similar to prefix-list and filter-list. The command show bgp ipv4 unicast policy statistics neighbor ip-address route-map [in | out] displays the matching prefix-list or AS-path access-list or any other attributes under each route-map sequence and its matching statistics, as shown in Example 11-51.

Example 11-51 BGP Policy Statistics for Route-Map

Click here to view code image

Within route-maps, various conditional matching features are used, such as prefix-lists, regular expressions (regex), AS-Path access-list, BGP communities, and community-lists. When multiple filtering mechanisms are configured under the same neighbor, the following order of preference is used for both inbound and outbound filtering:

• Inbound Filtering

  • Route-map

  • Filter-list

  • Prefix-list, distribute-list

• Outbound Filtering

  • Filter-list

  • Route-map

  • Advertise-map (conditional advertisement)

  • Prefix-list, distribute-list

Network prefixes are conditionally matched by a variety of routing protocol attributes, but the following sections explain the most common techniques for conditionally matching a prefix.

To parse through the large amount of available ASNs (4,294,967,295), regular expressions (regex) are used. Regular expressions are based upon query modifiers to select the appropriate content. The BGP table is parsed with regex using the command show bgp afi safi regexp “regex-pattern” on Nexus switches.

Table 11-7 provides a brief list and description of the common regex query modifiers.

The following section provides a variety of common tasks to help demonstrate each of the regex modifiers. Example 11-52 provides a reference BGP table for displaying scenarios of each regex query modifier for querying the prefixes learned via Figure 11-10.

Figure 11-10 BGP Regex Reference Topology

Example 11-52 BGP Table for Regex Queries

Click here to view code image

Scenario: Only display ASs that passed through AS 100. The first assumption is that the syntax show bgp ipv4 unicast regex “100” as shown in Example 11-53 is ideal. The regex query includes the following unwanted ASNs: 1100, 2100, 21003, and 10010.

Example 11-53 BGP Regex Query for AS 100

Click here to view code image

Example 11-54 uses the underscore (_) to imply a space left of the 100 to remove the unwanted ASNs. The regex query includes the following unwanted ASNs: 10010.

Example 11-54 BGP Regex Query for AS _100

Click here to view code image

Example 11-55 provides the final query by using the underscore (_) before and after the ASN (100) to finalize the query for the route that passes through AS 100.

Example 11-55 BGP Regex Query for AS _100_

Click here to view code image

Scenario: Only display routes that were advertised from AS 300. At first glance, the command show bgp ipv4 unicast regex “_300_” might be acceptable for use, but in Example 11-56 the route 192.168.129.0/24 is also included.

Example 11-56 BGP Regex Query for AS 300

Click here to view code image

Because AS 300 is directly connected, it is more efficient to ensure that AS 300 was the first AS listed. Example 11-57 shows the caret (^) in the regex pattern.

Example 11-57 BGP Regex Query with Caret

Click here to view code image

Scenario: Only display routes that originated in AS 40. In Example 11-58 the regex pattern “_40_” was used. Unfortunately, this also includes routes that originated in AS 50.

Example 11-58 BGP Regex Query with AS 40

Click here to view code image

Example 11-59 provides the solution using the dollar sign ($) for the regex the pattern “_40$”.

Example 11-59 BGP Regex Query with Dollar Sign

Click here to view code image

Scenario: Only display routes with an AS that contains 11 or 14 in it. The regex filter “1[14]” can be used as shown in Example 11-60.

Example 11-60 BGP Regex Query with Brackets

Click here to view code image

Scenario: Only display routes with the last two digits of the AS of 40, 50, 60, 70, or 80. Example 11-61 uses the regex query “[5-8]0_”. See the output in Example 11-60.

Example 11-61 BGP Regex Query with Hyphen

Click here to view code image

Scenario: Only display routes where the second AS from AS 100 or AS 300 does not start with 3, 4, 5, 6, 7, or 8. The first component of the regex query is to restrict the AS to the AS 100 or 300 with the regex query “^[13]00_”, and the second component is to filter out AS starting with 3-8 with the regex filter “_[^3-8]”. The complete regex query is “^[13]00_[^3-8]” as shown in Example 11-62.

Example 11-62 BGP Regex Query with Caret in Brackets

Click here to view code image

Scenario: Only display routes where the AS_PATH ends with AS 40 or 45 in it. The regex filter “_4(5|0)$” is shown in Example 11-63.

Example 11-63 BGP Regex Query with Parentheses

Click here to view code image

Scenario: Only display routes with an originating AS of 1–99. In Example 11-64, the regex query “_..$” requires a space, and then any character after that (including other spaces).

Example 11-64 BGP Regex Query with Period

Click here to view code image

Scenario: Only display routes where they contain at least one or more ‘11’ in the AS path. The regex pattern is “(11)+” as shown in Example 11-65.

Example 11-65 BGP Regex Query with Plus Sign

Click here to view code image

Scenario: Only display routes from the neighboring AS or its directly connected AS (that is, restrict to two ASs away). This query is more complicated and requires you to define an initial query for identifying the AS, which is “[0-9]+”. The second component includes the space and an optional second AS. The “?” limits the AS match to one or two ASs as shown in Example 11-66.

Example 11-66 BGP Regex Query with Dollar Sign

Click here to view code image

Scenario: Display all routes from any AS. This may seem like a useless task, but may be a valid requirement when using AS-Path access lists, which are explained later in this chapter. Example 11-67 shows the regex query.

Example 11-67 BGP Regex Query with Asterisk

Click here to view code image

Example 11-68 provides two sample AS-Path access lists. AS-Path access-list 1 matches against any local IBGP prefix, or any prefix that passes through AS 300 where as AS-Path access-list 2 provides a more complicated AS-Path access list that matches the 16-bit private ASN range (64,512 – 65,536).

Example 11-68 AS-Path Access List Configuration

Click here to view code image

• No-Advertise: The No_Advertise community (0xFFFFFF02 or 4,294,967,042) specifies that routes with this community should not be advertised to any BGP peer. The No-Advertise BGP community can be advertised from an upstream BGP peer or locally with an inbound BGP policy. In either method, the No-Advertise community is set in the BGP Loc-RIB table that affects outbound route advertisement.

• No-Export: The No_Export community (0xFFFFFF01 or 4,294,967,041) specifies that when a route is received with this community, the route is not advertised to any EBGP peer. If the router receiving the No-Export route is a confederation member, the route is advertised to other sub ASs in the confederation.

• Local-AS: The No_Export_SubConfed community (0xFFFFFF03 or 4,294,967,043) known as the Local-AS community specifies that a route with this community is not advertised outside of the local AS. If the router receiving a route with the Local-AS community is a confederation member, the route is advertised only within the sub-AS (Member-AS) and is not advertised between Member-ASs.

• Internet: Advertise this route to the Internet community and all the routers that belong to it.

The private community value is of the format (as-number:16-bit-number). Conditionally matching BGP communities allows for selection of routes based upon the BGP communities within the route’s path attributes so that selective processing occurs in a route-map.

NX-OS devices do not advertise BGP communities to peers by default. Communities are enabled on a neighbor-by-neighbor basis with the BGP address-family configuration command send-community [standard | extended | both] under the neighbor’s address family configuration. Standard communities are sent by default, unless the optional extended or both keywords are used.

Conditionally matching on NX-OS devices requires the creation of a community list. A community list shares a similar structure to an ACL, is standard or expanded, and is referenced via number or name. Standard community lists match either well-known communities or a private community number (as-number:16-bit-number), whereas Expanded community lists use regex patterns.

Examining the same topology as shown in Figure 11-6. In this topology, NX-5 assigns a community value of 65001:274 for the prefixes that have AS 274 in their AS_Path list. Example 11-69 illustrates the configuration on NX-5 to a community value attached to prefixes.

Example 11-69 Advertising Community Value

Click here to view code image

On NX-2, if an operator wants to set a BGP attribute based on the matching community value, community-list is used in the matching statement under route-map. Example 11-70 illustrates the configuration for using BGP community values for influencing route policy.

Example 11-70 Influencing Route Policy Using BGP Community

Click here to view code image

• show tech bgp

• show tech netstack

If there is some issue seen with BGP route policies, collect the following logs along with show tech bgp:

• show tech rpm

In case the routes are not being installed in the routing table but are present in the BGP table, you can also collect the following show tech output:

• Show tech routing ipv4 unicast [brief]

Collect and share these logs  with Cisco TAC for a root-cause analysis of the problem.

Halabi, Sam. Internet Routing Architectures. Indianapolis: Cisco Press, 2000.

Zhang, Randy, and Micah Bartell. BGP Design and Implementation. Indianapolis: Cisco Press 2003.

White, Russ, Alvaro Retana, and Don Slice. Optimal Routing Design. Indianapolis: Cisco Press, 2005.

Jain, Vinit, and Brad Edgeworth. Troubleshooting BGP. Indianapolis: Cisco Press, 2016.

Edgeworth, Brad, Aaron Foss, and Ramiro Garza Rios. IP Routing on Cisco IOS, IOS XE and IOS XR. Indianapolis: Cisco Press, 2014.

Cisco. Cisco NX-OS Software Configuration Guides, www.cisco.com.