First, TCP is a transport protocol that provides connection-oriented service, which means a connection must be established between two nodes that wish to exchange application data. This connection, or socket, identifies ports, or logical identifiers, that correlate directly to an application process running on each of the two nodes. In this way, a client will have a port assigned to the application that wishes to exchange data, and the server that the client is communicating with will also have a port. Once the connection is established, application processes on the two connected nodes are then able to exchange application data across the socket.

The process of establishing the connection involves the exchange of control data through the use of TCP connection establishment messages, including the following:

TCP connections are bidirectional in that the two nodes are able to exchange information in either direction. That is, the originating peer should be able to send data to the receiving peer, and the receiving peer should be able to send data to the originating peer. Given that information needs to be exchanged in both directions, a SYN must be sent from the originating peer to the receiving peer, and an ACK must be sent from the receiving peer to the originating peer. To facilitate transmission of data in the reverse direction, the receiving peer will also send a SYN to the originating peer, causing the originating peer to respond with an ACK.

The SYN and ACK messages that are exchanged between peers are flags within a TCP datagram, and TCP allows for multiple flags to be set concurrently. As such, the process of exchanging SYN and ACK messages to establish a connection is simplified, because the receiving peer can effectively establish a connection in the reverse path while acknowledging the connection request in the forward path.

Given that the SYN and ACK flags can be set together within a TCP datagram, you will see a message called a SYN/ACK, which is the receiving node acknowledging the connection request and also attempting connection in the reverse path. This is used in favor of exchanging separate SYN and ACK control datagrams in the reverse direction.

Figure 2-8 illustrates the process of establishing a TCP connection.

Figure 2-8. TCP Connection Establishment

Now that a connection has been established, application processes can begin exchanging data. Data exchanged between application processes on two communicating nodes will use this connection as a means of guaranteeing delivery of data between the two nodes, as illustrated in Figure 2-9.

Figure 2-9. Applications and TCP Connections

Second, TCP provides guaranteed delivery. Providing guarantees for delivery involves not only acknowledgment upon successful receipt of data, but also verification that the data has not changed in flight (through checksum verification), reordering of data if delivered out of order (through sequence number analysis), and also flow control (windowing) to ensure that data is being transmitted at a rate that the recipient (and the network) can handle.

Each node must allocate a certain amount of memory to a transmit buffer and a receive buffer for any sockets that are connected. These buffers are used to queue data for transmission from a local application process, and queue data received from the network until the local application process is able to read it from the buffer. Figure 2-10 shows how TCP acts as an intermediary between applications and the network and manages movement of data among data buffers while managing the connection between application processes.

Figure 2-10. TCP Transmit and Receive Buffers

Each exchange of data between two connected peers involves a 32-bit sequence number (coordinating position within the data stream and identifying which data has been previously sent) and, if the ACK flag is set, an acknowledgment number (defining which segments have been received and which sequence is expected next). An ACK message acknowledges that all segments up to the specified acknowledgment number have been received.

For each segment that is transmitted by an originating node, a duplicate copy of the segment is placed in a retransmission queue and a retransmission timer is started. If the retransmission timer set for that segment expires before an acknowledgment number that includes the segment is received from the recipient, the segment is retransmitted from the retransmission queue. If an acknowledgment number is received that includes the segment before the retransmission timer expires, the segment has been delivered successfully and is removed from the retransmission queue. If the timer set for the segment continues to expire, an error condition is generated resulting in termination of the connection. Acknowledgment is generated when the data has passed checksum verification and is stored in the TCP receive buffer, waiting to be pulled out by the application process.

Each exchange of data between two connected peers involves a 16-bit checksum number. The recipient uses the checksum to verify that the data received is identical to the data sent by the originating node. If the checksum verification fails, the segment is discarded and not acknowledged, thereby causing the originating node to retransmit the segment. A checksum is not fail-safe, however, but does serve to provide a substantial level of probability that the data being received is in fact identical to the data transmitted by the sender.

The TCP checksum is not the only means of validating data correctness. The IPv4 header provides a header checksum, and Ethernet provides a CRC. As mentioned previously, these do not provide a fail-safe guarantee that the data received is correct but do minimize the probability that data received is incorrect substantially.

As each segment is received, the sequence number is examined and the data is placed in the correct order relative to the other segments. In this way, data can be received out of order, and TCP will reorder the data to ensure that it is consistent with the series in which the originating node transmitted the data. Most connection-oriented transmission protocols support some form of sequencing to compensate for situations where data has been received out of order. Many connectionless transmission protocols, however, do not, and thus might not even have the knowledge necessary to understand the sequencing of the data being sent or received.

Flow control is the process of ensuring that data is sent at a rate that the recipient and the network are able to sustain. From the perspective of an end node participating in a conversation, flow control consists of what is known as the sliding window. The sliding window is a block of data referenced by pointers that indicates what data has been sent and is unacknowledged (which also resides in the retransmission buffer in case of failure), and what data has not yet been sent but resides in the socket buffer and the recipient is able to receive based on available capacity within the window.

As a recipient receives data from the network, the data is placed in the socket buffer and TCP sends acknowledgments for that data, thereby advancing the window (the sliding window). When the window slides, pointers are shifted to represent a higher range of bytes, indicating that data represented by lower pointer values has already been successfully sent and written to the recipient socket buffer. Assuming more data can be transmitted (recipient is able to receive), the data will be transmitted.

The sliding window controls the value of the window size, or win, parameter within TCP, which dictates how much data the recipient is able to receive at any point in time. As data is acknowledged, the win value can be increased, thereby notifying the sender that more data can be sent. Unlike the acknowledgment number, which is incremented when data has been successfully received by a receiving node and written to the recipient socket buffer, the win value is not incremented until the receiving node’s application process actually extracts data from the socket buffer, thereby relieving capacity within the socket buffer.

The effect of a zero win value is to stop the transmitter from sending data, a situation encountered commonly when a recipient’s socket buffer is completely full. As a result, the receiver is unable to accept any more data until the receiver application extracts data from the socket buffer successfully, thereby freeing up capacity to receive more data. The network aspects of flow control are examined in the later section, “Throughput.” Figure 2-11 shows an example of how TCP windows are managed.

Figure 2-11. TCP Sequence, Acknowledge, and Window Values

The connection-oriented, guaranteed-delivery nature of TCP causes TCP to consume significant resources and add latency that can impair application performance. Each connection established requires some amount of CPU and memory resources on the communicating nodes. Connection establishment requires the exchange of control data across the network, which incurs transmission and network delays. Managing flow control, acknowledging data, and retransmitting lost or failed data incur transmission and network delays.

How do other transport protocols fare compared to TCP? Other transport protocols, such as UDP, are connectionless and provide no means for guaranteed delivery. These transport protocols rely on other mechanisms to provide guaranteed delivery, reordering, and retransmission. In many cases, these mechanisms are built directly into the application that is using the transport protocol.

While transport protocols such as UDP do not directly have the overhead found in TCP, these overhead-inducing components are often encountered in other areas, such as the application layer. Furthermore, with a transport protocol that does not have its own network-sensitive mechanism for flow control, such transport protocols operate under the management of an application and will consume as much network capacity as possible to satisfy the application’s needs with no consideration given to the network itself.

Although the transport layer can certainly add incremental amounts of latency due to connection management, retransmission, guaranteed delivery, and flow control, clearly the largest impact from the perspective of latency comes from physics. Put simply, every network connection between two nodes spans some distance, and sending data across any distance takes some amount of time.

In a complex internetwork, a network connection might be between two nodes but span many intermediary devices over tens, hundreds, thousands, or tens of thousands of miles. This delay, called propagation delay, is defined as the amount of time taken by a signal when traversing a medium. This propagation delay is directly related to two components:

The maximum propagation velocity found in networks today is approximately two-thirds the speed of light (the speed of light is 3 × 10⁸ m/sec, therefore the maximum propagation velocity found in networks today is approximately 2 × 10⁸ m/sec). The propagation delay, then, is the physical separation (in meters) divided by the propagation delay of the link (2 × 10⁸ m/sec).

For a connection that spans 1 m, information can be transferred in one direction in approximately 5 nanoseconds (ns). This connection presents a roundtrip latency of approximately 10 ns.

For a connection that spans 1000 miles (1 mile = 1600 m), or 1.6 million meters, information can be transferred in one direction in approximately 8 ms. This connection presents a roundtrip latency of approximately 16 ms.

For a satellite connection that spans 50,000 miles (1 mile = 1600 m), or 80 million meters, information can be transferred in one direction in approximately 400 ms. This connection presents a roundtrip latency of approximately 800 ms.

These latency numbers are relative only to the amount of time spent on the network transmission medium and do not account for other very important factors such as serialization delays, processing delays, forwarding delays, or other delays.

Serialization delays are the amount of time it takes for a device to extract data from one queue and packetize that data onto the next network for transmission. Serialization delay is directly related to the network medium, interface speed, and size of the frame being serviced. For higher-speed networks, serialization delay might be negligible, but in lower-speed networks, serialization delay can be significant. Serialization delay can be calculated as the size of the data being transferred (in bits) divided by the speed of the link (in bits). For instance, to place 100 bytes of data (roughly 800 bits) onto a 128-kbps link, serialization delay would be 6.25 ms. To place 100 bytes of data onto a 1-Gbps link, serialization delay would be 100 ns.

Processing delays are the amount of time spent within a network node such as a router, switch, firewall, or accelerator, determining how to handle a piece of data based on preconfigured or dynamic policy. For instance, on an uncongested router, a piece of data can be compared against a basic access list in under 1 millisecond (ms). On a congested router, however, the same operation might take 5 to 10 ms.

The forwarding architecture of the network node impacts processing delay as well. For instance, store-and-forward devices will wait until the entire packet is received before making a forwarding decision. Devices that perform cut-through forwarding will make a decision after the header is received.

Forwarding delays are the amount of time spent determining where to forward a piece of data. With modern routers and switches that leverage dedicated hardware for forwarding, a piece of data may move through the device in under 1 ms. Under load, this may increase to 3 to 5 ms.

The sum of all of these components (physics, transport protocols, application protocols, congestion) adds up to the amount of perceived network latency. For a direct node-to-node connection over a switch, network latency is generally so small that it does not need to be accounted for. If that same switch is under heavy load, network latency may be significantly higher, but likely not enough to be noticeable. For a node-to-node connection over a complex, hierarchal network, latency can become quite significant, as illustrated in Figure 2-12.

Figure 2-12. Network Latency Between Two Nodes

This latency accounts for the no-load network components in the path between two nodes that wish to exchange application data. In scenarios where intermediary devices are encountering severe levels of congestion, perceived latency can be dramatically higher.

With this in mind, you need to ask, “Why is latency the silent killer of application performance?” The reason is quite simple. Each message that must be exchanged between two nodes that have an established connection must traverse this heavily latent network. In the preceding example where the perceived roundtrip network latency was 48 ms, we must now account for the number of messages that are exchanged across this network to handle transport protocol management and application message transfer.

In terms of transport protocol management, TCP requires connection management due to its nature as a connection-oriented, guaranteed-delivery transport protocol. Establishing a connection requires 1.5 round trips across the network. Segment exchanges require round trips across the network. Retransmissions require round trips across the network. In short, every TCP segment sent across the network will take an amount of time equivalent to the current network delay to propagate.

To make matters worse, many applications use protocols that not only dictate how data should be exchanged, but also include semantics to control access, manage state, or structure the exchange of data between two nodes. Such application protocols are considered “chatty” or said to “play network ping-pong” or “ricochet” because of how many messages must actually be exchanged before a hint of usable application data is exchanged between the two nodes.

Take HTTP, for example, as illustrated in Figure 2-13. In a very basic web page access scenario, a TCP connection must first be established. Once established, the user’s browser then requests the container page for the website being viewed using a GET request. When the container page is fully transferred, the browser builds the structure of the page in the display window and proceeds to fetch each of the embedded objects listed in the container page. Oftentimes, this is done over one or two concurrent TCP connections using serialized requests for objects. That is, when one object is requested, the next object cannot be requested until the previous object has been completely received. If the browser is using a pair of TCP connections for object fetching, two objects can be transferred to the browser at any given time, one per connection. For a website with a large number of objects, this means that multiple GET requests and object transfers must traverse the network.

Figure 2-13. Application Protocol Latency

In this example, assuming no network congestion, packet loss, or retransmission, and that all objects are transferable within the confines of a single packet, a 100-object website using two connections for object fetching over a 100-ms network (200-ms roundtrip latency) would require the following:

The rendering of this website would take approximately 11 seconds to complete due to the latency of the application protocol being used. This does not account for network congestion, server response time, client response time, or many other very important factors. In some cases, based on application design or server configuration, these objects may already be cached on the client but require validation by the application protocol. This leads to additional inefficiency as objects that are stored locally are verified as valid and usable.

HTTP is not the only protocol that behaves this way. Many other application protocols are actually far worse. Other “ping-pong” application protocols that require extensive messaging before useful application data is ever transmitted include the following:

More details on application protocol inefficiencies will be discussed later in this chapter in the “Application Protocols” section.

CIFS sessions between a client and file server can be used for many functions, including remote drive mapping, simple file copy operations, or interactive file access. CIFS operations perform very well in a LAN environment. However, given that CIFS was designed to emulate the control aspects of a shared local file system via a network interface, it requires a high degree of control and status message exchanges prior to and during the exchange of useful data. This overhead, which is really due to how robust the protocol is, can be a barrier to application performance as latency increases.

As an example, accessing a 1.5-MB file over the network that resides on a Windows file server requires well over 1000 messages to be exchanged between the client and the server. This is directly related to the fact that CIFS requires the user to be authenticated and authorized, the file needs to be found within the directory structure, information about the file needs to be queried, the user permissions need to be verified, file open requests must be managed, lock requests must be handled, and data must be read from various points within the file (most likely noncontiguous sections).

CIFS is a protocol that has a high degree of application layer chatter; that is, a large number of messages must be exchanged to accomplish an arbitrary task. With protocols such as CIFS that are chatty, as latency increases, performance decreases. The combination of chatty applications and high-latency networks becomes a significant obstacle to performance in WAN environments because a large number of messages that are sent in a serial fashion (one at a time) must each traverse a high-latency network.

Although TCP/IP is a reliable transport mechanism for CIFS to ride on top of, latency that is introduced in the network will have a greater impact on CIFS performance, because the CIFS protocol is providing, through the network as opposed to being wholly contained within a computer, all of the semantics that a local file system would provide.

CIFS does support some built-in optimization techniques through message batching and opportunistic locks, which includes local caching, read-ahead, and write-behind. These optimizations are meant to grant a client the authority necessary to manage his own state based on the information provided by the server. In essence, the server will notify the user that they are the only user working with a particular object. In this way, the CIFS redirector, which is responsible for transmission of CIFS messages over the network, can respond to its own requests to provide the client with higher levels of performance while offloading the server. Although these techniques do provide value in that fewer messages must be handled by the origin server, the value in a WAN environment is nullified due to the extreme amounts of latency that exist in the network.

For Internet traffic that uses HTTP as a transfer mechanism, several factors contribute to latency and the perception of latency, including the following:

Intranet web traffic is generally much more dynamic than traditional client/server traffic, and Internet traffic today is much more dynamic than it was 10 years ago. With the introduction of Java and other frameworks such as Microsoft’s .NET, the client browser is empowered to provide a greater degree of flexibility and much more robust user experience.

Along with the capabilities provided by such frameworks, the browser is also required to process and render a larger amount of information, which may also increase the burden on the network. Although Internet browsers have matured to include functions such as local caches, which help mitigate the transmission of locally stored objects that have been previously accessed, the browser still cannot address many of the factors that contribute to the perception of latency by the user.

For objects embedded within a web page, the browser may be required to establish additional TCP connections to transfer them. Each TCP connection is subject to the roundtrip latency that natively exists between the client and server in the network, and also causes resource utilization increases as memory is allocated to the connection. Negatively adding to an already challenged protocol, the average Internet web page is populated with a significant number of small objects, increasing the number of TCP connections required when accessing the web page to be browsed and the number of roundtrip exchanges required to fetch these embedded objects.

In a network environment with several users located in a remote branch sharing a common WAN link, browser caching aids in improving the overall performance. Browser caching does not, however, address the limitations created by the fact that every user must request his own copy of content at least once because he is not sharing a common repository. Compounding the number of users across a limited-bandwidth WAN link adds latency to their browsers’ ability to receive requested HTTP data. Application response times increase, and network performance declines predictably.

The most common browser, Microsoft Internet Explorer, opens two TCP connections by default. Though HTTP 1.1 supports pipelining to allow multiple simultaneous requests to be in process, it is not widely implemented. As such, HTTP requests must be serially processed over the two available TCP connections.

A typical web page is composed of a dynamic object or container page and many static objects of varying size. The dynamic object is not cached, whereas the static objects may be locally cached by the client browser cache or perhaps an intermediate public cache. In reality, even the static objects are often marked noncacheable or immediately expired because the application owner either is trying to account for every delivery or has at some point experienced problems with stale objects in web caches.

The application owner’s knee-jerk reaction was to immediately expire all objects allowing the client to cache an object but then force the client to issue an If-Modified-Since (IMS) to revalidate the objects’ freshness. In this case, a web page with 100 objects that may be cached in the client browser cache would still have to IMS 100 times over two connections, resulting in 50 round trips over the WAN. For a 100-msec RTT network, the client would have to wait 5 seconds just to revalidate while no real data is traversing the network.

By nature, HTTP does not abide by any given bandwidth limitation rules. HTTP accepts however much or little bandwidth is made available via the transport layer (TCP) and is very accepting of slow links. Consider that a web browser will operate the same if connecting to the Internet via a 2400-bps modem or a 100-Mbps Ethernet card, with the only difference being the time required to achieve the same results.

Some HTTP implementations are forgiving of network disruption as well, picking up where the previous transmission was disrupted. The lines that separate an application from the traditional browser have been blurring over the past several years, moving the browser from an informational application to the role of critical business application. Even with the evolution in role for the web browser, HTTP still abides by its unbiased acceptance of the network’s availability.

With the transition of many core business-related applications from client/server to HTTP based, some applications use client-based software components while still using HTTP as the transport mechanism. With the vast success of Java, application infrastructure can be centralized, yet, segments of the application (such as applets) can be distributed and executed on the remote user workstation via his web browser. This benefits the application and server in two ways:

Following this model a step further, it is common for only the database entries to traverse the network and for any new Java applets to be distributed to the client as needed. The client continues to access the application via the local Java applets that are active in their browser. The applet remains available until the browser is closed or a replacement applet is served from the application server. This process works very well when the client has the Java applet loaded into the browser, but getting the initial Java applet(s) to the client consumes any bandwidth made available to HTTP during the transfer.

Many web-accessible database applications are available today and are commonly tested in a data center prior to their deployment within the corporate network. As previously shown in Chapter 1, “Strategic Information Technology Initiatives,” the testing of new applications commonly takes place within a lab and not in a network environment that mirrors a full production remote branch. If a branch location has 20 users, each requiring access to the same application, that application must traverse the shared WAN link 20 times.

Testing of applications that use an actual production branch may not test the application and WAN to their fullest capacity, that is, all users simultaneously within the branch. Some applications may load in seconds when tested in a lab with high-speed LAN links connecting the client, but when tested over a WAN link with 20 simultaneous users sharing the common WAN link, it could take 30 minutes or more for the same application to become available to all users in that branch.

Due to the forgiving nature of HTTP, the loading of the application becomes slow to the user but does not time out or fail the process of loading. Branch PCs commonly have slow access in the morning, because applications must be loaded on workstations, and other business-critical functions, such as e-mail and user authentication, traverse the network at the same time. HTTP does not care about what else is happening on the network (TCP does), as long as the requested content can eventually transfer to the client’s browser.

Many network administrators consider the FTP a “necessary evil.” Legacy applications and logging hosts commonly use FTP for simple, authenticated file transfers. FTP is viewed as a simple solution but with a potential for a major impact. Everything from the NIC of the server to the WAN link that the traffic will traverse is impacted by the manner in which FTP transfers content. FTP can disrupt the ability to pass other traffic at the same time an FTP transfer is taking place. By nature, FTP consumes as much bandwidth as possible during its transfer of content, based on what TCP is allowed to consume. FTP tends to use large data buffers, meaning it will try to leverage all of the buffer capacity that TCP can allocate to it.

Precautions exist within many operating systems and third-party applications that allow the administrator to define an upper limit to any given FTP transfer, preventing congestion situations. FTP is not fault tolerant and, by nature, is very unforgiving to disruptions. In many cases, a network disruption requires that the file be retransmitted from the beginning. Some application vendors have written client and server programs that leverage FTP as a control and data transfer protocol that allow for continuation of a failed transfer, but these mechanisms are not built into FTP as a protocol itself.

NFS, which was originally introduced by Sun Microsystems, provides a common protocol for file access between hosts. NFS provides read and write capabilities similar to CIFS, but NFS is stateless in nature whereas CIFS is extremely stateful. This means that the server requires use of idempotent operations, that is, each operation contains enough data necessary to allow the server to fulfill the request. With CIFS, the server knows the state of the client and the file and can respond to simple operations given the known state.

Since the acceptance of NFS by the Internet Engineering Task Force (IETF), vendors such as Microsoft, Novell, Apple, and Red Hat have adopted the NFS protocol and implemented a version into their server and client platforms.

Although the standards are clearly defined to implement NFS, there are currently three common versions of NFS that all have different abilities and create and react differently to latency challenges introduced on the network. NFS version 2 (NFSv2) leverages UDP as a transport layer protocol, which makes it unreliable in many network environments. Although some implementations of NFSv2 now support TCP, NFSv3 was more easily adopted due to its native support of TCP for transport.

NFS is similar to CIFS in that a physical location within a file system is made accessible via the network, and clients are able to access this shared file system as if it were local storage. NFS introduces some challenges that are similar to those of CIFS, including remote server processing delay when data is being written to the remote host, and sensitivity to network latency when data is being read from or written to a remote host.

For file transfers, NFS generally uses 8-KB blocks for both read and write actions. NFS impacts more than just the network; NFS also impacts the CPU of the server, disk throughput, and RAM. There is a direct parallel between the volume of data being transferred via NFS and the amount of server resources consumed by NFS.

Fortunately, vendors such as Sun have implemented bandwidth management functionality, allowing for NFS traffic flows to be throttled for both incoming and outgoing traffic independently of each other. If bandwidth management is not applied to NFS traffic, as with other protocols, the client application will consume as much bandwidth as allowed by the transport protocol. As network speeds become faster, file sizes will trend toward becoming larger as well. NFS has been used by applications as a transport for moving database and system backup files throughout the network, requiring that traffic management be planned in advance of implementing the NFS protocol.

The ability to control NFS traffic on the WAN will aid administrators in preventing WAN bottleneck slowdowns. Much like CIFS, if 50 users in a remote office each need a copy of the same file from the same mount point, the file must be transferred across the network at least once for each requesting user, thereby wasting precious WAN bandwidth.

MAPI provides a common message exchange format that can be leveraged in client/server and peer-to-peer applications. MAPI is commonly used as a standardized means of exchanging data over another protocol such as a Remote Procedure Call (RPC). For instance, Microsoft Exchange uses MAPI for the structuring of messages that are exchanged using remote-procedure calls (RPC). Although not all vendors’ implementation of MAPI may be the same, not to mention the use of the protocol that is carrying data (such as RPC), several applications use MAPI to dictate the way messages are exchanged.

Throughput of many client/server applications that leverage MAPI, such as Microsoft Outlook and Exchange, which use MAPI within RPC, decreases by as much as 90 percent as latency increases from 30 ms to 100 ms. As e-mail continues to become an increasingly important enterprise application, poor performance of e-mail services within the network is generally brought to the attention of network administrators faster than other poorly performing business-critical applications. The majority of this decline in application performance is directly related to bandwidth, latency, throughput, and the inefficiencies of the protocol being used to carry MAPI message exchanges.

Microsoft’s use of MAPI and RPC differs between Exchange Server 2000 and Exchange Server 2003. Traffic patterns between the two different server versions differ from the perspective of network traffic due to the introduction of cached mode in Exchange Server 2003, when used in conjunction with the matching Microsoft Outlook 2003 product. When Exchange 2003 and Outlook 2003 are combined, cached mode allows for the majority of user operations to be performed locally against a “cached copy” of the contents of the user’s storage repository on the server. In essence, cached mode allows a user to keep a local copy of his storage repository on the local PC. This local copy is used to allow operations that do not impact data integrity or correctness to be handled locally without requiring an exchange of data with the Exchange server. For instance, opening an e-mail that resides in the cached copy on the local PC does not require that the e-mail be transferred again from the Exchange server. Only in cases where updates need to be employed, where state changes, or when synchronization is necessary do the server and client actually have to communicate with one another.

E-mail vendors such as Lotus, with its Domino offering, support compression and other localization functions as well. Lotus recognizes that allowing less traffic to traverse the WAN between the client and server will improve application performance for its users.

Although compression reduces the overall volume of data that traverses the network, e-mail protocols are still very sensitive to latency and packet loss. When operating on networks that are subject to a predictable amount of traffic loss, e-mail may not function at all, making services appear as if they are offline. Most e-mail applications have a default interval of 60 seconds for synchronization between the client and server. With such offline synchronization, traffic is greatly reduced once the e-mail client has sent and received all of its messages and has synchronized the local cached copy. However, if the amount of data that needs to be synchronized to the server is large and transfer of this data extends beyond the synchronization window, it could be quite difficult for a client to catch up. Similar to file-sharing protocols, if 50 users in a remote office each received a copy of the same e-mail with the same 5-MB attachment, that 5-MB attachment would be transferred across the network once per recipient, consuming large amounts of network capacity.

Understanding network infrastructure and network management is the first step in venturing to assess and manage application performance over the network. If you do not know how your network is designed and implemented, you will not be in a position to effectively monitor and manage your network or the applications that use it. Take the time to understand your network topology, and be sure to become familiar with each component, including the following:

Similarly, become familiar with each of the non-network components that have an impact on application performance, including these:

From the network and non-network information, an administrator can easily identify how his network is configured, as well as where to take action if an outage is observed.

The next step in understanding the network’s impact on application performance is to know the limits of your network. Every router, switch, network card, and cable has a capacity limit. If the limiting factor is not the speed of the interface, it could be the CPU or memory utilization of each device that the traffic must traverse.

A router or switch that is running at capacity may drop an excessive number of packets (that is, become a point of congestion). If aggregation and oversubscription are not planned appropriately, packet loss and queuing delays could have a significantly adverse effect on the performance of applications traversing that device.

Network switches are commonly designed with inherent oversubscription. For instance, a switch that supports 24 ports of 1000-Mbps traffic per port may not have an adequate backplane capacity to support 24 ports under full load. Understanding the limits of a router, switch, firewall, or other network element will help identify potential bottlenecks in the network. Adequately sizing network infrastructure components will help prevent network stability issues and better guarantee higher levels of application performance.

Several monitoring utilities exist today that allow you to monitor all aspects of a router or switch. Monitoring utilities range from simple shareware tools to monitoring and reporting systems that cost millions of dollars. The following sections describe SNMP-compliant managers and syslog.

Routers and switches that support network management and monitoring via the Simple Network Management Protocol (SNMP) should be configured to report to an SNMP-compliant manager, because this will aid in providing visibility into the overall health of each of the devices. SNMP monitoring will inform an administrator of a device that exceeds a configured threshold, generates specific warnings, or triggers alerts. If a defined threshold such as CPU utilization is exceeded, the stability of the network may be at risk due to a device that cannot support or scale beyond its current workload.

SNMP supports several functions that help secure and stabilize the monitoring information that is provided by the network routers and switches. Functions include the ability to encrypt and authenticate the requests that the router or switch must respond to.

Although SNMPv1 is the most commonly implemented version, versions 2c and 3 provide additional levels of protection that help deter network threats from snooping around the network. Table 2-2 provides a simplistic overview of SNMP version differences.

In addition to SNMP, syslog can be used to monitor system-related events, which can prove useful in many network environments. Syslog tracks just about every event that occurs on the network, depending on the level of visibility desired by the network administrator. If an application is suspect in the stability or performance of a given network, implementing syslog may help in tracking down specific suspect events. Also, depending on what information you are tracking on the network, there may be legal implications that protect the employee’s rights. Syslog will help in understanding network traffic patterns, and in a way that helps determine if there are application or network failures being observed on the network.

Many network vendors today have results of performance validation tests and may make these available to customers. This information may not be publicly available due to the sensitive nature of the data. Most vendors make this information available upon request, and understanding how each device in your network behaves will help you to know what the observed limits are and how to deploy accordingly. For example, deploying a router that cannot meet the packets per second requirement or that has insufficient resources may not scale to meet the needs of the location it connects to.

One of the most important factors in network stability is the redundancy plan. Redundancy planning and implementation can be leveraged at any place within the network, ranging from the data center application servers to the remote branch WAN connectivity or even the LAN switch. With the intelligence that exists in today’s infrastructure components, redundancy has become easier to implement. Although items such as redundant WAN connections and the associated routers can be costly, the technology exists today to allow for near-immediate failover with granular control over which applications are allowed to take advantage of the alternate WAN connectivity.

Take the time to consider the options that exist when considering redundancy in the network. Proper planning is the key to providing network access that is scalable, reliable, and stable for the remote user. In most cases, the cost of downtime is far greater than the cost of redundancy.

The client and server also are large factors in network stability and application performance, starting with the connection point to the network, the NIC. For all devices, if the client or server network connection is not stable (because of improper device drivers, faulty network cards, bad cables, or a poorly implemented or misconfigured TCP/IP stack, for example), then the network will appear to lack stability and application performance will suffer.

The device driver selected by the operating system may not always be the most appropriate, accurate, or current. Ensuring that servers, client workstations, and their respective NICs are operating in the most efficient and stable manner will impact the stability of the LAN and WAN directly. The stability of the network offers no value if the server or client devices cannot reliably transmit data through it. There is a possibility that the operating system’s default driver selection will lead to packet loss or a network interface exhibiting intermittent functionality. The host operating system may not be equipped to communicate the lack of functionality to the administrator.

TCP Chimney Offload is Microsoft’s software implementation that directly supports TCP Offload Engine (TOE) NICs. The subject of TOE is discussed later in this chapter, in the section “Network Interface Cards.” TCP Chimney Offload, when combined with a Microsoft-approved TOE, allows for a stateful offload of traffic from the operating system to the TOE NIC. This mitigates the need for the server CPU to spend cycles managing TCP connections. Instead, it can focus on application-centric efforts such as serving web pages, files, or database content.

Receive-side Scaling addresses the common challenge of load allocation to a single processor within a multiprocessor server. A single network processor in a server handles all network traffic. For servers that have high utilization levels and multiple processors, one processor will commonly show a high level of utilization, while other processors will not receive network traffic. When RSS is implemented on the server with supported NICs, all inbound network traffic is load-balanced across all recognized processors, allowing for additional network traffic to be accepted by the server.

NetDMA is the third offering of the Windows Server 2003 Scalable Networking Pack. NetDMA requires supporting hardware, which optimizes data flow through a server. NetDMA reduces the dependency on the server’s processor to manage moving data between the network adapter and application buffers within the server’s RAM. NetDMA, through supporting hardware that optimizes data flow through a server, manages the memory data transfer processes, enabling the server’s processor to be allocated to other application-related functions.

Layer 2 and Layer 3 caches reside in the processor chip. These caches accelerate the processor’s ability to receive data. Processors today support caches that range in size from 1 MB to 24 MB or larger. Typically, the larger the processor’s cache, the faster the processor will function in a server that runs under high utilization.

Frequently accessed data, which would typically be held in the server’s RAM memory, may be partially or entirely stored in the processor’s cache memory to reduce the latency of data lookups. The processor will always check its local cache before checking with the server’s installed RAM.

The front side bus is a significant factor in the server’s ability to move data about the server. The numerical rating of a server’s front side bus designates at what speed the processor can transfer data to other devices, including RAM and I/O cards (such as those attached via PCI or other interconnect). The speed rating of a server’s front side bus will directly correlate into how quickly the server can pass data between its components; this distribution of data has a direct impact on the server’s ability to process and serve data to clients, thereby impacting application performance overall.

A dual-core processor is simply a single processor chip that contains two processors that can work in parallel. A common benefit of a dual-core processor is that the individual processors will each have their own primary cache, or Level 1 cache. The traffic of the individual processors will pass through a common Level 2 cache, or secondary cache, allowing the individual processor instances to share a common cache. The dual-core, shared-cache architecture provides a faster processor response time for both processors within the same chip.

Because no direct dual-core support is built into operating systems, the support must be built into the applications that use the server’s processors. To produce an application that will optimally perform on a dual-core server, the application vendor must be aware of the processors installed within the server to support the available resources offered by the dual-core processor. Figure 2-17 illustrates the shared L2 cache architecture of the dual-core processor.

Figure 2-17. Dual-Core Processor

Hyper-threading and Super-threading are offerings that are specific to processors manufactured by Intel Corporation. Hyper-threading allows a single processor to appear as if two separate processors exist. Super-threading allows a thread, or thread of execution, to be shared with a processor that has available thread resources.

For servers, hyper-threading is best taken advantage of by applications that are aware of and support hyper-threading, although the operating system may or may not need to be aware of the processor’s ability. Hyper-threading support must exist in the BIOS and chipset of the server to take advantage of a hyper-thread-capable processor. Intel has combined the dual-core and hyper-threading capabilities in some of its more powerful processor offerings.

Some processors support a 32-bit architecture, whereas others support a 64-bit architecture. As operating systems evolve toward a greater array of 64-bit support, applications must also support the ability to communicate with a 64-bit processor. Applications that are written specifically to support the 32-bit server architecture may operate in a diminished capacity when run on a 64-bit architecture, because the server must emulate the 32-bit architecture.

In the case of database applications, overall performance improves if the database application was compiled specifically to support a 64-bit architecture. Database application performance improves due to the ability to better support multitasking and the ability to transfer larger blocks of data among components within the server.

Proper selection of the server’s operating system will depend on which processor has been installed in the server. 128-bit processors will be the next numerical evolution of the server processor.

Serial attached SCSI (SAS) is the next generation of SCSI interconnect where communication with the drives occurs in serial, not parallel, between the controller and attached drives. With serial operation, clock rates can be increased due to minimized cross-talk, which also enables greater distances to be achieved.

SAS disk drives and controllers are the result of 20 years of learning from traditional SCSI storage devices. SAS controllers allow drives to be attached in either RAID or JBOD configurations, just as traditional SCSI allowed. RAID configurations of SAS-capable disks can produce throughput ratings of up to 3 GB per second, aiding in affordable and optimal application access.

Fiber Channel (FC) attached storage is commonly connected to the server via Lucent connectors (LCs) embedded within a small form-factor pluggable (SFP) optical adapter, such as a PCI host bus adapter (HBA). HBAs have throughput ratings of up to 4 Gbps and have become reasonably priced to implement in the data center.

FC HBAs are manufactured to support either 32-bit architectures, 64-bit architectures, or both, when interfacing with the PCI bus. This allows application performance to take advantage of a combination of an operating system and processor that may already be 64-bit enabled. If a server is already operating in 64-bit mode, it is advised to implement FC with a 64-bit-capable adapter as well.

FC HBAs are commonly supported by operating systems such as Microsoft Windows Server 2003, Red Hat Linux, and Sun Solaris. Furthermore, FC HBAs generally support direct connection to a disk subsystem (DAS) or to a Fibre Channel fabric (SAN) to access a provisioned portion of a shared array. Interestingly enough, SCSI command sets are most commonly used on top of FC interconnect.

Figure 2-18 examines how current-generation storage interconnect compares to legacy storage interconnect. This table clearly articulates the performance differences that can be found with newer-generation technology.

Figure 2-18. Comparing Storage Interconnect Throughput

Along with examining aspects such as CPU subsystem, memory subsystem, and storage interconnect, the file system configuration should also be examined for its impact on application performance.

Solaris uses blocks in a consecutive manner for a logical method of disk space allocation. Applications such as Solaris Volume Manager can be used to configure and optimize a server’s file system. There are two common types of file system: a raw I/O file system and the UNIX File System (UFS). Solaris 8 and later releases allow for database applications to be run directly from the UFS partition, with very good performance.

Solaris block sizes are 4 KB and 8 KB, allowing for data to be written to blocks in smaller incremental sizes. For 4-KB block sizes, data may be written in 512-B, 1-KB, 2-KB, and 4-KB fragment sizes. For 8-KB block sizes, data may be written in 1-KB, 2-KB, 4-KB, and 8-KB fragment sizes. To check the current block size configuration of a server, execute the command given in Example 2-1 as the super user or administrator of the server.

# df –g
/             (/dev/dsk/c0t0d0s0 ):     8192 block size     1024 fragment size
/proc         (/proc             ):      512 block size      512 fragment size
/etc/mnttab   (mnttab            ):      512 block size      512 fragment size
/dev/fd       (fd                ):     1024 block size     1024 fragment size
/tmp          (swap              ):     8192 block size     8192 fragment size

Database application vendors offer guidelines for configuring an application server’s file systems to operate optimally with their application. If the disk block sizes are 4 KB, and the fragment sizes are configured to utilize 4 KB each, then buffering and memory utilization will be improved, because the amount of data that needs to be passed to disk is identical to the size of the block being written. In this way, data does not need to be manipulated or otherwise shaped to match the block size of the file system. The same rule applies to 8-KB blocks and 8-KB fragment sizes, allowing for fewer read operations prior to writing data to a specific block on the disk.

Using the forcedirectio option allows for data writing that spans several blocks to occur as a single I/O event. This allows database applications to bundle together events scheduled to be written to disk.

Microsoft Windows Server 2003 defaults to 4-KB allocation units within the NTFS file system. Several functions of the NTFS are dependent on the 4-KB allocation unit, such as disk compression. Any other size will impact the ability to enable compression, which may not improve overall server performance on an application server.

Disk partitions of 2 GB to 2 TB utilize the 4-KB application unit by default. To change the default allocation unit size of Windows Server 2003, the Disk Management snap-in is required to change a disk’s partition properties. Once an application is installed on a server and the partition housing the database is created, you should not change the allocation unit size. Any time an administrator attempts to change the allocation unit size, the disk partition must be reformatted. Reformatting will eliminate all data within the altered partition. Windows Server 2003 supports allocation unit sizes up to 64 KB in size.

For HP-UX, there are three common file systems, VxFS (Veritas File System), UFS (UNIX File System), and HFS (Hierarchical File System). All three of these file systems have different characteristics for their block size allocation parameters. For example, VxFS, which has a default block allocation unit size of 1 KB, supports a maximum of 8 KB per allocation unit. HFS defaults to 8 KB per block, allowing for several different options at the time the disk partition is created, ranging from 4 KB to 64 KB in size.

It is common for database applications running under an HP-UX operating system to leverage 8-KB block sizes for their file systems. 8 KB is a recommended block size for applications such as Oracle 11i and SAP. 8-KB block sizes are ideal for large transactions on the server, whereas smaller block sizes are most appropriate for database applications that call for a significant amount of random access. In the case of Oracle and other database applications, the application itself must be configured to match the disk block size for optimal and efficient application performance.

The 64-KB block sizes are viewed by many as the optimal disk allocation size for both HFS and JFS, with HFS supporting 8-KB fragment sizes, and JFS having no fragment support. These sizes may not be optimal for the application’s database on the server, requiring the application vendor or author to be consulted to validate which file system format and block size should be implemented at the time of the server installation.

Red Hat Linux supports three commonly deployed operating systems:

Each file system has characteristics that are critical to database application performance. Areas such as block size, maximum file size supported, journaling, and maximum supported partition size all become factors when selecting the proper Linux file system.

The second extended file system (ext2) is considered the benchmark of Linux file systems. Although ext2 is supported by many database applications, ext2 is not considered to be suitable by some vendors due to a lack of journaling. Journaling is a process in which a segment of the actual disk is used to store a temporary copy of the data prior to being written to the actual data partition specific to the disk. In the event of a disk failure, any disk access intentions are read from the journaling section of the disk, allowing the system to recover after power is restored to the server.

ext2 supports a maximum partition size of 4 TB, allowing for a maximum file size within the 4-TB partition of 2 TB. The ext2 file system also supports block sizes ranging from 1 KB to 4 KB. The ext2 file system is considered to be more efficient in terms of disk utilization than ext3, nearly eliminating the need for the use of any defragmentation tools.

The third extended file system (ext3) shares many commonalities with the ext2 file system. The primary differences can be classified as the support of journaling, as well as a method to change the file system and partition sizes of existing partitions. Journaling has been added to ext3, using a section of disk that is separate from the general data storage section of the disk.

ext3 supports three different levels of journaling:

Each level of journaling provides the application server increased storage reliability. Although file systems that support journaling may be a requirement, it does have a minor impact on the performance of the storage, requiring some data to be written to the disk twice. ext3 file systems do not require utilities to defragment storage to optimize disk block usage.

ReiserFS is the latest file system supported by Red Hat Linux. ReiserFS is a journaling file system, much like ext3 but supports additional features. The ReiserFS file system supports a maximum partition size of 16 TB and a maximum file size of 8 TB per file. For block sizes, the ReiserFS file system supports only a 4-KB block size, which is optimal for applications that leverage files of 4 KB or smaller. Some database applications do not support the ReiserFS file system, which may be a concern when large file support must exceed the 2-TB limit of the ext2 and ext3 file systems.

The differences in file system limitations across several operating system vendors are illustrated in Table 2-5.

The Reiser4 file system, which has not yet been merged into the Linux operating system, has added some unique enhancements to the Linux file system. Disk block sharing allows a block that may not have been completely filled to share space with other files. Block sharing will allow for significantly better disk space utilization. Reiser4 also introduces a new concept of journaling, termed the wandering log. Wandering logs change the method of the initial write of file data during the journaling process, allowing data to be written anywhere in the data portion of the disk. If data is written throughout the disk, data defragmentation may be required to optimize the Reiser4 file system. The Reiser4 file system is too new to be supported by major application vendors and will require support of the Linux kernel prior to becoming supported by application vendors.

Some database applications require an alternative to traditional disk partitions that use a formatted file system. These applications use raw I/O disks or raw partitions. Eliminating the traditional barriers of file systems, raw I/O is written to a specific disk, array, or host. When using raw I/O, the application does not allow for the same type of disk administration as a traditional operating system managed partition, and is commonly managed by the database application. Raw I/O partitions are commonly faster than operating system formatted partitions, because there is less operating system overhead managing the raw partitions or disks. Raw I/O partitions are typically specified as a requirement by the application, and not the operating system. Management of the data that resides in the raw I/O partition is commonly a function of the database application.