4.2.1 Kernel Software

Currently, there is a large range of developed and refined/enhanced real-time kernel software, both licensed and open-source, to address the real-time software design constraints seen in switch/routers. An equipment vendor may choose to develop a common switch/router interface model that permits different kernels to be used to create specific platforms as required. In many cases, a vendor would use a common portable kernel that permits quick retargeting of the total switch/router software for short time-to-market development needs as well as reuse of already developed software.

4.2.2 Software Implementation

The following approaches are typically used today in the development of routing, bridging, and network management protocol software for switch/routers:

• Designer develops/implements software independently and directly from standards-based or proprietary specifications.
• Designer imports and refines/enhances software from other well-known implementations.
• Designer uses software from open-source implementations, sometimes with refinements/enhancements.
• Designer licenses software from a software supplier, for example, a software development company, government-funded university projects, and so on.

In the early years of networking, until the mid-1990s, designers developed in-house high-performance implementations of the main routing and bridging protocols (IP, IPX, Integrated IS–IS, Ethernet, Token Ring, FDDI, etc.). Designers also used specific software kernels and provided extensions for any special and proprietary networking features required. In addition, designers enhanced/extended the real-time software kernels available at that time with software interfaces commonly used in public domain kernel software (e.g., the Berkeley Software Development (BSD) UNIX socket model and system services). The addition of these software interfaces facilitated and accelerated the addition of new software from external sources.

In today's practices, a vendor would use a common switch/router and router software across its many internetworking platforms. A majority of the vendor's routing and bridging software is independent of the underlying hardware and is developed to support the evolving networking standards and also to allow portability. When designing each platform, the vendor would customize the performance-intensive and hardware-specific software code to optimize the design and maximize performance for each instance of a switch/router or router architecture.

4.2.3 Switch/Router Software Design Issues

The switch/router designer will typically have to tackle a number of technical problems related to software design when building the switch/router. This section describes the most challenging issues and how they were commonly addressed in the design process. The following are the most significant issues:

• Stability of the Layer 3 (routing) and Layer 2 (bridging) protocols and algorithms
• Requirements for network management
• Switch/router performance
• Interactions between Layer 3 (routing) and Layer 2 (bridging) functions

In addition, the total amount of memory available (system-wide) is a major constraint on the design of the switch/router. The key issues to watch out for are memory size and its usage. Typically, the memory is largely consumed by software code and by the routing and forwarding tables. These tables can be very large and are used by the switch/router to maintain the best routes to network destinations.

In the case of switch/routers that also support virtual circuits or connections (e.g., ATM, MPLS, and connection-oriented Ethernet), significant amounts of memory can be consumed by the per connection state and counter-information the device maintains. Examples of connection-oriented Ethernet technologies include IEEE 802.1ah (Provider Backbone Bridges (PBB)), IEEE 802.1Qay-2009 (Provider Backbone Bridge Traffic Engineering (PBB-TE)), and MPLS-TP (formerly T-MPLS).

Routers and switch/routers do communicate with their peers in the network to determine the best routes to a network destination. To accomplish this, all such routing devices must each be able to support the route database required to maintain a view of the network. A switch/router or router design may include an automatic shutdown mechanism that takes effect should the device run out of memory in which the routing information is stored. This mechanism, when used, can help prevent routing loops in the network.

Another design issue is the mechanisms needed to handle internal control plane and data plane congestion. To implement congestion control for traffic generated by protocols such as TCP, a router determines whether or not a packet experiences congestion by calculating the average queue length at an interface over a given time period and comparing this with predefined thresholds.

Packets may be dropped or marked for dropping later when these thresholds are breached. The calculation of the average queue values and the execution of the dropping/marking of packets must be performed in an efficient manner in real time given the potential traffic load that can converge at the device. For these reasons, and also to support real-time control, designers develop and implement algorithms specific to the particular queue structures and hardware architecture in the switch/router or router.

4.3.1 Requirements on Processing Power

A majority of routing protocols (link-state protocols in particular) requires that routing protocol updates be received and then propagated within a specified time frame of their arrival. It is recognized that the calculations required for maintaining the routing tables are CPU-intensive, but this processing time is proportional to the number of links reported in link-state packets (LSPs), in OSPF routing, for example.

The routing update process time requirement/constraints (i.e., the time within which a routing update has to propagate at a node) mean that the CPU time must be fairly allocated between the routing decision process (i.e., time required for routing database calculations) and the routing update process itself. If the switch/router or router is required to wait until the routing decision process is completed before the routing update process is handled, then the delays on forwarding LSPs would be too large. To address these concerns, designers consider three possible solutions:

1. Process Prioritization: In this approach, the routing update process is given a strict priority over the routing decision process to allow the routing database to be updated as required. Here, the designer will have to find a solution as to how to synchronize access to the shared link-state database (LSDB) and at the same time allow the routing decision process to complete, if a misbehaving peer router generates LSPs at an excessive rate.
2. Time-Slicing or Time-Sharing the CPU: In this approach, both the routing update and routing decision processes are allowed to run simultaneously, thus sharing the control CPU. This approach also requires synchronizing access to the LSDB.
3. Voluntary Preemption: In this approach, the routing decision process periodically checks if the routing update process requires execution and, if so, allows it. The checks can be configured to happen at time intervals frequent enough to satisfy the delay constraints/requirements and at times suitable/appropriate for the routing decision process to avoid the need for synchronizing access to the LSDB.

Designers typically choose the third approach to avoid synchronizing access to the LSDB for the following two reasons:

• In complex systems, synchronization issues can often create system problems that are serious and difficult to debug. An architecture that avoids these issues entirely can often lead to increased software simplicity and reliability.
• Adding synchronization mechanisms for parallel tasks can decrease total system performance, for example, because it causes excessive rescheduling of operations.

As a result, an architecture that uses voluntary preemption of processes allows a simpler yet very efficient solution that still meets the system requirements.

4.3.2 Queuing Requirements

By using appropriate queuing mechanisms, the control CPU can ensure that during high network traffic loads, routing control information will not be discarded. In a centralized CPU routing/forwarding architecture, separating the end-user data (to be directly forwarded) from routing control messages is a more logical solution to preserving the processing routing control information. However, this works well only if the CPU can process all the routing control messages without losing messages. Low-end switch/routers or routers typically do not have a control CPU that is fast enough to guarantee such routing information processing performance.

Even if a router can guarantee the timing requirements on the routing update and decision processes even under worst-case traffic loads (for OSPF routing, for example) and if that load is combined with a flood of Hello, Database Description (DBD), Link-State Request (LSR) messages, Link-State Update (LSU), and Link-State Acknowledgment (LSAck), some of these messages have to be queued for later processing or simply discarded. However, even in a large network, the worst-case scenario would be if all these nodes were to send Hello and other control messages at the same time, which is very unlikely.

With careful software design, the switch/routers or routers can meet the network stability requirements and still not lose connectivity to their peer routing devices on the network. In architectures with limited control CPU power, the current practice is to design and implement a traffic management policy that differentiates between routing protocol message types (e.g., OSPF messages: Hello, DBD, LSR, LSU, LSAck) to meet their respective processing requirements for network stability.

The traffic management parameters that control the minimum and maximum numbers of packets allowable for each (differentiated) routing control message type have to be carefully calculated based on each message's architect behavior and the network· configuration in which the router is deployed. For example, a router designed to support a given maximum number of adjacent routers in a network will influence the selection of the traffic management policy for managing the OSPF Hello message queues and packet buffers. Such traffic management mechanisms (for the control messages) should be implemented to guarantee that the minimum levels of message exchanges (per message type) are met to ensure overall system forwarding performance, network convergence, and stability.

The traffic management policy for the routing control messages can be designed to also use buffer memory pools that support priority queuing. The priority queues allow various traffic management policies to be implemented. Inbound control traffic can be placed in the priority queues according to routing message type. The queues are then serviced using a number of priority scheduling algorithms (weighted round-robin (WRR), weighted fair queuing (WFQ), etc.).

These algorithms can be configured with different weights assigned to each priority queue to ensure that all message types are adequately served, although at different rates. The actual weights can be made configurable to take into account the performance characteristics of the router and expected network configuration it would be placed in.

Other routing control traffic management policies may include the following:

• Separate Buffer Pools Per Routing Control Message Type: An architecture may have a separate buffer pool for each of the different control message types. The disadvantage of this approach is that, in small network configurations or ones that do not carry heavy routing control traffic, the pool of buffers may be constantly underutilized and also denying the actual data plane packet forwarding process from utilizing these buffers when it needs them.
• Strict Priority Scheduling of Routing Control Message Queues: Configuring strict priority scheduling for processing the different routing control message types is undesirable, because the flooding of one routing control message type could starve one or more other message types of CPU processing opportunity for a long time. In such a case, the system works better when each message type is given some processing time than to giving one type absolute priority.

4.3.3 Requirements on Memory Allocation

Network devices, in general, must have adequate buffering for storing and processing packets, but switch/routers and routers, in particular, must have sufficient buffering to handle routing control messages. Today's routing devices strive to guarantee this buffering requirement for control messages. The line cards must never run out of buffers; otherwise, routing control messages required by the route processor may be discarded depriving the routing protocols of important network information. The route processor must also have sufficient buffering for the control messages it receives so that they are properly processed.

4.4.1 Installation and Loading of Software

With the current proliferation of Internet services, switch/router or routers are able to update their software (software updates or new software) over the network. Also switch/router or routers are able to store their software in nonvolatile memory and so do not need to be reloaded on each boot. However, some older systems load the software each time they are booted.

The use of nonvolatile memory (e.g., flash memory) allows for fast and reliable software loading combined with backup load operation when software updates are required. The load can be from a network server using, for example, the trivial file transfer protocol (TFTP). This provides an easy way to update software when required.

4.4.2 Router/Software Configuration

The general view about routers is that they are notoriously very difficult to configure. Whether this view is justified or not depends on which network manager you talk to and which brand of router is up for discussion. Router vendors have gone to great lengths to dispel this as a myth. In this regard, vendors have been developing tools (using the latest software technologies available) to assist network managers with configuration.

Each switch/router or router comes with configuration tools: CLI based, Secure Shell (SSH), menu-driven programs, scripts, web-based tools, and so on. For example, a menu-driven program could lead the network manager through a series of forms to be filled to define the information required to configure the router (or switch/router) or to modify an existing configuration.

The tools often come with online help information to provide extra information needed for configuration and sometimes tutorials. The help information may also provide information about how steps may be retraced. Consequently, the network manager has no need to learn very lengthy, complex, and often very technical information about configuring the device.

During the design and development of the configuration tools, router vendors normally use formal human factors testing to ensure that these tools meet the expectations of the device user (network manager). The vendor uses the router's customer field testing and trials to provide additional feedback on the configuration tool's effectiveness and ease of use. Even throughout the life of the product, the vendor continues to use human interface testing to refine the quality and performance of the tools.

A network manager can use an SSH client at a remote location to make a secure, encrypted connection to a SSH server in a switch/router or router. SSH supports strong encryption for authentication, and the user authentication can be via username and password, TACACS+ and RADIUS (see TACACS+ and RADIUS below). The authentication and encryption in SSH allows an SSH client to carry out a secure communication over a network that is even insecure.

4.4.3 Network Monitoring

It is now common practice for network devices including switch/routers to include capabilities for the network to detect faults, isolate them, log, and notify the network manager about the fault conditions. The network manager or any automated mechanism can then take the necessary remedial measures to correct the faults encountered in the network to keep it running effectively. Other capabilities provided in the devices allow them to monitor and measure a number of network parameters and metrics so that the performance of the network can be maintained at the desired operational level.

Switch/router or router incorporate capabilities to allow for a wide range of performance and system metrics/parameters at the device, interface, and protocol levels to be collected at regular intervals using, for example, the Simple Network Management Protocol (SNMP). With this, a Network Management System (NMS), supporting a polling engine, allows data collection from the network (switch/routers, routers, etc.). Most current NMSs have extensive network capabilities that include, at a minimum, collecting, storing, and presenting polled data.

Currently, there exist many commercial and public domain tools that are capable of collecting, storing, and presenting data from switch/router, routers, and other network equipment. Network managers can use graphical interfaces that present network maps and show the operational states of the network devices such as switch/routers and routers. Current network devices including switch/routers and routers support web-based interfaces that enable the device's performance data to be accessible from remote locations in the network. In addition to SNMP, many switch/routers and routers support other features such NetFlow, sFlow, and RMON (Remote Network MONitoring).

RMON probes and NetFlow in the switch/routers and routers provide the ability to collect traffic profiles from them. User traffic has grown significantly in networks and continues to place greater demands on network resources. Consequently, the need to collect information on user traffic and network load has become even more important. Network managers, who in the past had a limited view or knowledge of the types of traffic running in the network, can use NetFlow, for example, to profile the network traffic. User and application traffic profiling gives the network manager even a better view of the traffic in the network.

The RMON can be deployed in a network at various locations where agents (implemented in stand-alone RMON probes or embedded in network devices) pass on information to a central management station through SNMP. The latest RMON version allows remote monitoring of a network device from the Media Access Control (MAC) sublayer up to the network and even application layers. The statistics, alarms, history, and other data provided by RMON can be used by a network manager to proactively monitor and take steps to improve network availability at various protocol layers in the network.

NetFlow allows detailed traffic flows statistics to be collected from network devices, which can then be used for billing, network and device troubleshooting, and network capacity planning. NetFlow can be implemented on individual ports, links, and interfaces on a network device, allowing information on traffic passing through these points to be collected. The NetFlow data/information gathered on a network device is exported to an entity referred to as a collector. Some of the functions performed by the collector include reducing the volume of data it receives (which involves filtering and aggregation functions), hierarchical data storage, and file system management.

Some routers provide features that can be utilized for measuring response times between routing devices. One router with such a feature is capable of measuring the response time to other routing devices. SNMP traps can be configured on the source router to alert an NMS if the response times cross some predefined thresholds.

4.4.4 System and Access Control

Switch/routers and routers, as standard practice, do have mechanisms to control administrative access to them and network resources. These access control mechanisms, for example, can monitor users logins to the device and refuse access to unauthorized users. Some of the methods of controlling access to switch/routers and routers include entering local user IDs and passwords on the device, configuring Access Control Lists (ACLs), and using protocols such as TACACS+ (Terminal Access Controller Access Control System plus) or RADIUS (Remote Authentication Dial-In User Service).

Both TACACS+ and RADIUS, although different, are protocols that provide secure remote access to networks and network services based on an Authentication, Authorization, and Accounting (AAA) architecture. AAA, generally, is an architectural framework for managing access to network services and resources, enforcing access and usage policies, auditing service and resource usage, and generating information needed for billing purposes.

The process of authentication provides mechanisms for identifying a user and determining whether that user is allowed access to the network. This often involves, in the simplest case, the user supplying a username and password that are checked against a database of usernames and passwords. The switch/router or router supports mechanism for identifying a user (network manager or administrator) prior to being allowed access to the device.

The authorization process provides mechanisms to determine what type of resources, services, or activities are permitted for an authenticated user. Authorization only occurs after authentication. Authorization can be used do define the specific services, resources, or activities allowable for the (different) users. Also, in the context of configuring a switch/router or router, authorization can be used to limit the commands that are permissible to the different authenticated users. Authorization can also include a one-time authorization and can be for each service that is requested by the user.

The accounting process provides mechanisms for tracking what a user carried out and when it was done (on authorized or unauthorized resources, services, or activities). Accounting can be used for billing for service or resource usage and for performing an audit trail. Accounting can be used to gather information on user identities, start and stop times, and the commands executed on the switch/router or router. Accounting does not need to happen after authentication and authorization, it can be carried out independent of them.

Both TACACS+ and RADIUS are client/server protocols. TACACS+ runs on TCP, while RADIUS runs on either UDP or TCP (in IETF RFC 6613, RADIUS over TCP). TACACS+ provides separate AAA services, while RADIUS combines both authentication and authorization services. TACACS+ is mostly used by the network administrator for secure access to network devices such as switches, switch/routers, and routers, while RADIUS is commonly used to authenticate and login remote users to a network.

SNMP can also be used to carry out configuration changes on switch/routers and routers similar to the capabilities offered by a CLI for system configuration. However, the right security measures have to be implemented on the switch/router or router to prevent unauthorized access and change via SNMP. SNMPv3 (SNMP version 3) supports secure exchanges of management data (by authenticating and encrypting data) between a management station and network devices.

SNMPv3's encryption and authentication features ensure that data transported to a management station are secured than in SNMPv1 and SNMPv2. When SNMP is used, the standard practice is to provide the necessary security features needed to ensure that only authorized network management stations (NMPs) are allowed to perform configuration changes on switch/router or router.

An SNMP privilege level can be used to specify/limit the types of operations that a management station can perform on the switch/router or router. A Read-Only (RO) privilege level allows a management station (NMP) to only query the device's data. This does not permit the management station to issue configuration commands such as rebooting the device and shutting down network interfaces. The Read-Write (RW) privilege level is the only mode that allows such operations to be performed. SNMP ACLs can be used with SNMP privilege levels to specify which specific management stations are allowed to request management information from the switch/routers or routers.

4.4.5 Problem Solving and Debugging

Problem solving and debugging tasks in a network constitute the biggest, most time-consuming, and expensive aspects of a network manager's responsibilities. Fortunately, there exist today many networking techniques and tools for performing extensive and very complex tasks required for testing and debugging switch/routers, router, and other network devices.

Building on initial experience gained during the design, testing, and debugging of a switch/router (as a matter of fact, other network devices), vendor normally produces problem-solving guides for the device. These user and problem-solving guides typically provide a step-by-step description of how various problems associated with the deployment of the device can be isolated and solved.

Typically, the vendor conducts human factor testing on these guides to determine how effective they are and also investigate different forms of making this information available. The guide is normally made available to the user in the form of portable flash memory drives, hard copy (in some cases), and also online on the vendor's Web site.

Another tool for problem solving is incorporating software and hardware features into the networking device that allows it to record or copy packets that are sent to and received on its interfaces (some sort of port-mirroring and network analyzer capabilities). The copied packets can then be analyzed and displayed automatically by software routines on an external management station.

The mirroring capability functions like having a built-in analyzer on the network interface. This feature can serve as an effective diagnostic tool used by the design engineers when debugging system hardware and software development problems, and by field engineers when investigating a problem on the customer site. Switch/router and router vendors also include diagnostic facilities such as loopback testing over the device's interfaces.

Network problem solving and debugging can also be done using SNMP, RMON, and other similar protocols. A network management platform (NMP) receives events (or notifications) from network elements (switches, switch/routers, routers, severs, etc.) in the network and processes them for the purpose of carrying out fault, configuration, performance, security, and accounting management. A standard NMP supports the following functions: network discovery, mapping the topology of network elements, event handler, collection and plotting of performance data, and browser for data management.

An NMP can perform the discovery of the network devices and then represent/display each network device by a graphical element on the NMP's console. The graphical elements on the console typically use dissimilar colors to indicate the current working state of a switch/router, router, and other devices in the network. To carry out the needed functions, the network devices can be configured to send notifications (SNMP traps) to the NMP. When the NMP receives the SNMP traps (also stored in a log file), it changes the graphical element representing the particular network device to a different color based on the severity of the SNMP trap received.

Important tools for network troubleshooting that can be used by the network manager are Trivial File Transfer Protocol (TFTP) and system log (syslog) servers. The key use of the TFTP server is for storing system configuration information (files) and software code/images for switches, switch/routers, routers, and other network devices. Switch/routers and routers are capable of sending system log messages to a syslog server located somewhere in the network. The syslog messages can be used by the network manager to perform troubleshooting tasks when network problems occur. The syslog messages can also be used by network field personnel to carry out root cause analysis.

The primary goal of fault management is to detect faults encountered in a network, possibly (automatically) isolate them, and then notify other entities who will be responsible for correcting them. To do this, switches, switch/routers, routers, and other devices are now capable of alerting management stations (NMPs) when a fault occurs on the network. Sending SNMP trap messages, using SNMP polling, logging messages in a syslog server, and employing RMON thresholds all form critical components of an effective fault management system. An NMP can then alert the network manager when a network fault is reported and remedial/corrective actions are required.

SNMP traps describing the environmental conditions of a switch/router or router can be sent to the NMP when an environmental threshold is exceeded:

• A voltage notification can be sent if the voltage measured at a given point in a network device is outside the normal range (at warning, critical, or shutdown stage).
• A shutdown notification can be sent if a system parameter at a given test point in a network device is reaching a critical state and the system is about to initiate a shutdown.
• A redundant supply notification can be sent if a redundant power supply in the network device fails.
• A fan notification can be sent if any one of the cooling fans in the network device's cooling fan array fails.
• A temperature notification can be sent if the temperature at a given point in the network device is outside the normal range (at warning, critical, or shutdown stage).

A switch/router or router can implement RMON alarms and events capabilities to allow it to monitor itself for the rising and falling of a system parameter/metric against thresholds (input/output packet drops, CPU utilization, buffer failures, etc.). With this, the switch/router or router will take samples of the parameter/metric at predefined time intervals and compare it against the configured thresholds. The device will send an SNMP trap to a management station if the measured value falls outside the configured thresholds. The network manager can use RMON alarms and events provided by the network devices for proactive management of the network before potential problems occur.

4.5.1 Performance Metrics

A number of common metrics have been developed for benchmarking the performance of switch//routers and routers. The following three major metrics are commonly used:

• Throughput: This refers to the maximum forwarding rate (e.g., packets per second) at which no offered packets are dropped by the device.
• Packet Loss Rate: This is the fraction of packets out of the total offered traffic load that the network device should have forwarded while operating under a constant traffic load, but were not forwarded due to lack of resources.
• Latency (or Transit Delay): This refers to the time interval from when the last bit of the packet reaches the input port to when the first bit of the packet is seen on the output port.

4.5.2 Packet Throughput/Forwarding Rate

Since the packet forwarding rate is the most important performance metric for a switch, switch/router, or router, designers carefully optimized both the hardware and software of the device to allow packet forwarding to occur at as much higher speed as possible. In architectures with centralized forwarding, the designers try to optimize the design such that all the forwarding on the central CPU is done with every available software and hardware assistance possible at minimal cost.

In architectures with distributed forwarding, the design is optimized such that forwarding and filtering operations are handled by line cards. Both designs would typically employ various forms of hardware assist for the performance-critical destination address lookups in the forwarding engine. This allows the design to support the necessary requirements for very high-speed packet forwarding.

On each line card in a distributed forwarding architecture, a streamlined software kernel may be developed for the local processor along with all its required software. The line card processor software kernel and modules have to be carefully designed to have the minimum number of instructions and the lowest number of execution cycles necessary to perform the high-speed filtering and forwarding operations.

On the control or route processor of the switch/router or router (which is typically centralized), the software kernel also has to be designed to be fully capable of forwarding packets on its own. However, the route processor is mainly required to provide the software processing for the nonperformance-intensive operations of the system (i.e., the processing of routing protocol updates used to build the routing database and network management messages). The partitioning of processing of received packets (between the line cards and the route processor) in the distributed architecture allow such systems (and the networks they reside in) to remain highly stable when traffic overloads occur.

In some architectures with centralized forwarding, the software-based forwarding operation has no hardware assist at all. Instead, the software-based lookup algorithms have to be optimized and implemented to meet the performance-intensive requirement of the (software) forwarding engine. In these systems, the software is highly optimized and tuned for the processor supporting the forwarding engine to allow higher performance forwarding. The amount of software code is also kept to a minimum to minimize the additional maintenance overhead associated with highly optimized and tuned software.

4.5.3 Packet Transit Latency

The packet latency (or transit delay) in switch/routers and routers also depends heavily on the hardware and software processing and storage characteristics of the device. One design factor that affects the packet latency is whether packet processing (and address lookup) starts prior to the packet being completely received or not. Another factor (mainly in Layer 2 forwarding) is whether the packet received can be sent out to the output port for transmission prior to its complete reception.

In Layer 2 forwarding (mainly), when packet reception, forwarding, and transmission occur in parallel, the architecture is referred to as cut-through. With cut-through switching, the switch delays the arriving packet only long enough to read the relevant Layer 2 packet header and make a forwarding decision (destination MAC address and, if necessary, VLAN tag and IEEE 802.1p priority field). The cut-through switch does not have to wait for the full packet serialization by the sending device before it begins packet processing.

The internal switch latency during cut-through switch is reduced because packet processing is limited to the Layer 2 header only rather than to the entire packet. The fundamental drawback of cut-through switching is that corrupted packets cannot be identified and discarded because the packets are forwarded before the FCS field is received and thus no CRC calculation can be performed. In spite of this shortcoming, cut-through switching, which was the predominant mode of switching in the 1990s, has enjoyed resurgence in recent years because of the reduction in latency it provides.

Cut-through switching is applicable for networks that do not require speed changes and are limited enough in diameter/extent to have very low packet error rates. Low error rates mean that only a negligible amount of bandwidth will be wasted on bad packets. Typically, networks that have optical fiber connectivity have lower packet corruption than those based on copper. Fortunately, many of today's networks tend to be fiber based, which have greater immunity to electrometric interference and radio-frequency interference (EM/RFI), crosstalk, and so on. Fiber-based network have relatively low attenuation and support greater distances and bandwidths.

The store-and-forward switch has to wait for the full packet to be received (i.e., serialized by the sending device) before it begins packet processing and forwarding. The switch latency for a packet is the delay between the last bit into the switch and the first bit out (LIFO) of the switch. After the output port has been determined, the switch has to reserialize the packet to send it on its way to the receiving device.

In some routing (Layer 3) device designs, the packet reception and forwarding can be done in parallel prior to a packet being completely received. However, such designs do not start transmitting the packet at the output port until the packet is completely received because Layer 3 forwarding and address lookups are relatively indeterministic compared to Layer 2 forwarding.

In routing (Layer 3) devices, the forwarding model is store-and-forward. The factors that affect the packet latency in such devices are as follows:

• Packet Reception: The Layer 2 and 3 headers of the packet must be completely received before processing can start.
• Forwarding Table Lookup Operations: This includes Layer 2 packet header verification, parsing and analyzing the Layer 3 header, performing the required Layer 3 destination address lookup, performing any required Layer 2 and 3 packet field rewrites, and queuing the packet for transmission on the destination port/interface.
• Queuing Due to Congestion: If the destination interface is busy when it receives a packet, it will have to be queued until the interface is ready for transmission. The internal packet latency for a device must reflect the potential latency delays due to congestion at various points in the device including the output interface. Congestion avoidance algorithms in protocols such as TCP are designed to minimize this congestion delay when overload conditions occur.
• Packet Transmission Delay: This delay refers to the time taken to clock/place the bits of the packet out of the interface onto the transmission medium but may also include medium access delays (particularly in media that require media contention). In such media, delays occur due to another node already using the common shared transmission medium.

Now let us look at a number of methods used by switch/routers and routers to minimize the packet transit delay. Most single processor systems minimize the packet reception and transmission delays by allowing the line interface hardware to perform the receive and transmit functions while using their direct memory access (DMA) features. In these systems, the same data path (fast-path) optimizations used to improve the packet forwarding rate are used at the same time to minimize the forwarding delay.

In architectures with distributed forwarding, the line cards generally have no DMA, and the onboard processor receives each packet byte-by-byte and parses the Layer 2 and 3 headers as soon as they are received. The data link (Layer 2 header) fields are immediately examined and decoded even before the network (Layer 3) address bytes have been received. The destination network address lookup is initiated as soon as the address fields have been received (i.e., before the packet payload or data have been received). The destination address lookup is then performed by the local forwarding engine without direct involvement from other modules in the system.

In some designs, the line cards receive packets one chunk (segment) at a time, that is, a complete packet is received in smaller segments. Typically, these line cards use, internally, small fixed-size buffers (smaller than the largest maximum transmission unit (MTU) of the system) that are linked together, as required, to store a complete packet. The line card's forwarding engine performs the forwarding table lookup as soon as the header information is available (i.e., when the first segment of the packet is received).

The advantage of the above line card architecture is that, for a large packet, the forwarding engine is able to complete a forwarding decision well before the last byte of the packet has been received. However, the disadvantage is that, until the last byte of the packet has been received by the system, it is impossible to tell whether the CRC of the packet is correct or the packet has not been corrupted.

For these reasons, the receive line card does not pass the packet to the destination line card until the CRC check has been successfully completed. This system design is still store-and-forward (not cut-through) even though the packet forwarding decision is initiated immediately after the Layer 3 header is received.

Before a packet is transmitted to the destination interface, certain modifications must be made in the packet. In IP, the time-to-live (TTL) field is updated and, in some cases, other fields like those for QoS indication may be modified, after which the IP checksum is recalculated. Layer 2 packets may need header field modification for QoS indication or conversion between untagged and tagged Ethernet frames. To speed up packet forwarding and reduce packet latency, some line cards have hardware assistance for modifying the packet header information including updating checksums and CRCs.

The above design options all attempt to reduce the packet transit delay as much as possible using different mechanisms. These mechanisms try to reduce the transit delay by controlling the time it takes to receive a packet to the time it departs the system. The designs incorporate modules that blend some aspects of store-and-forward processing with cut-through design.

Some of these architecture process packets in distributed processors (in the line cards) but transmit packets only after complete reception and validation of the packets (store-and-forward processing). This design limits the packet forwarding rate of the system, because packets that are found to be in error are discarded, but these can only be discovered if the system receives complete packets. The full cut-through designs in comparison have relatively higher packet forwarding rates, but they would propagate corrupted/error or invalid packets that eventually get discarded by the end user.

4.6.1 Switch/Router Design with Protocol Split

In this switch/router design approach (Figure 4.1), traffic from nonroutable protocols are Layer 2 forwarded (bridged), and those from routable protocols are routed, for example, IP, Internetwork Packet Exchange (IPX), Systems Network Architecture (SNA), and AppleTalk. The Layer 2 and Layer 3 functions are completely separate, the only common feature is that they share the same link interfaces. Every packet received by the switch/router in this design is sent either to the Layer 3 function (if routable) or to the Layer 2 functions.

Unlike routable protocols such as IP, traffic from protocols such as DEC's Local Area Transport (LAT), NetBIOS Extended User Interface (NetBEUI) cannot be routed over a network. The nonroutable protocols do not have network (Layer 3) addresses but only have device (Layer 2) addresses. The protocols do not support any network addressing scheme that can be used to forward packets from one network to another network.

A nonroutable protocol is mainly limited and used within a LAN where all the devices belong to the same network. If the devices are located in different networks (LANs), the nonroutable protocol they use does not allow them to exchange data. They can only do that when the data they generate are encapsulated in a routable protocol.

Unless a nonroutable protocol is involved, this switch/router design approach is not usable and has no significant benefits.

4.6.2 Integrated Switch/Router Design with Shared External Interface

In this design approach (Figure 4.2), the Layer 3 function is interfaced to and layered on top of the Layer 2 function. In this case, the Layer 3 function uses its own internal Layer 2 address(es) and logical interface to interface to the Layer 2 function, which in turn connects to the external networks. Incoming packets are passed to the bridging process first, but if they are addressed to the Layer 3 function (router), they are then forwarded to the routing process. In this design, the routing function uses a logical interface within the switch/router over which it receives routable packets.

In this design, the Layer 3 and 2 functions can be logical entities within the same physical platform or they can be physically separate collocated entities. The latter architecture is similar to a Layer 2 switch connected to a “one-armed router” that provides Layer 3 functionality for inter-VLAN communications.

A one-armed router, also called a “router on a stick,” is a Layer 3 forwarding device that has a one logical or physical connection to a network that has more than one VLANs (subnets) requiring routing between them. A one-armed router is often used to forward inter-VLAN traffic and connects to these VLANs via a Layer 2 switch.

4.6.3 Integrated Switch/Router with Multiple External Interfaces

This switch/router design (Figure 4.3) is similar to the design with shared external interface shown in Figure 4.2, except that the Layer 3 function uses all the interfaces also available to the Layer 2 function and logically connects to the same extended network through all these interfaces. This design constitutes the standard design found in a majority of today's switch/routers. Some older designs adopt the approach presented in Figure 4.2.

4.7.1 Watchdog Polling of the Line Cards

This feature is needed to enable the system guard against its software being stuck in an infinite loop and thereby preventing it from responding to system control and management instructions and messages. The control processor most likely will be protected by a hardware watchdog timer, but the line cards, on the other hand, may not support such a timer feature. To enable the line card software guard against such problems, the control processor software can be designed to poll each line card periodically, say, every 500 ms. With this, if the control processor receives no response, the line card is reset.

4.7.2 Statistics Counters in the Line Cards

The line cards in the distributed forwarding architecture are designed to handle packet forwarding, which in many cases creates the additional requirement of maintaining counters (e.g., number of packets in error, number of data bytes received). A designer may choose to implement these using 32 bit counters. However, to avoid implementing 64 bit counters in the line card (which may increase the line card memory costs and require 64 bit arithmetic for the counters), the designer may implement the full (bigger) counters in the centralized control processor and then allow it to poll the line cards periodically, but frequently enough to ensure that the line card counters do not overflow and wrap around. Each line card counter may be sized (i.e., number of bits) to support, for example, 400 ms polling frequency by the control processor.

4.7.3 Control of the Line Cards

The line cards can be designed such that when a Layer 2 (data link) protocol or a Layer 3 (routing) protocol is started or stopped on any one of its network interfaces, the centralized control processor would receive the necessary management commands/notifications and also allow it to issue appropriate control commands and messages to the line card.