Disk and Storage Basics

Many criteria should be considered when choosing a data storage solution for the database server. Performance is only one of many factors that needs to be considered, but it is certainly one of the most important factors because of its direct influence on the response times and availability of the overall SAP system.

Database Server Perspective

The database server manages the I/O requests from the SAP application to the data. A database server is an intelligent I/O engine that serves up the data in a structured way. The database server contains processors, memory, and I/O controllers or interfaces to the storage systems. The processors and memory of the database server play a significant role in determining the overall response time of the SAP system.

Typically the case with production SAP systems, the size of all the database files combined is significantly larger than the size of the physical memory (RAM) available, whether in the server or in the storage system's cache. Otherwise, the database files could be completely stored in faster RAM, eliminating the need for many disk-tuning considerations. Because most servers are not yet available with hundreds of gigabytes of physical memory, the data transfer to or from the storage system plays a crucial role in the overall performance of the database server.

Storage System Perspective

The performance of the storage system is sensitive to the pattern of I/O requests coming from the business applications and the database. The database decides if the request for data from the application can be satisfied from the contents of its memory (a data buffer cache hit), or if it needs to retrieve the data from the disk storage system. Determining how many processors and how much memory to use in the database server is covered in more detail in Chapter 3 . The performance section of this chapter focuses more narrowly on the storage systems themselves, usually external to the server(s) they are attached to, in relation to the business applications.

Figure 4-1 shows the typical database server and storage system configuration, including: database memory, file system, host bus adapters in the server and disk I/O channel or interface, as well as the storage system's controller, cache, and disk mechanisms.

Figure 4-1. Disk Storage System Components on a Database Server


From a purely theoretical performance perspective, the database server's memory and the storage system's cache should be sized large enough to fit the entire contents of the database. Because this isn't realistic for production SAP databases, at least the smaller working set of data must be stored in these available caches.

The most important areas of sizing affecting a storage system are (1) its cache size and (2) the individual number of disks. The following section focuses on the disk and storage system basics, including how individual disk mechanisms perform in the SAP application environment.

Disk I/O Patterns: A Business Application Perspective

Computing systems running business applications have two generally recognized kinds of disk I/O operations: (1) file-server-oriented, a mix of random and larger sequential I/O requests; and (2) transaction-oriented, that is, small, random I/O requests. An I/O request is either a read from or a write to the disk(s) from the database host computer. The I/O requests can come in different sizes, depending on the application software making the request and on the storage system configuration. In addition, the read accesses versus write-I/O request pattern will also be different depending on the business application.

Standard File Sharing and Sequential I/O

A typical file server storing a workgroup's office data files tends to make a mix of random and sequential I/O requests to or from the storage system. Random disk access comes into play when searching for files. When files are written to or read from the file server's disks, the I/Os tend to be large and sequential in nature. Typical data files include word processing documents, presentations and graphic images, spreadsheet data files, and so on. Most of these documents are several kilobytes (KB) in size and can commonly reach several megabytes (MB). Because disks tend to perform best with larger, sequential I/O requests, few physical disks are needed to provide an adequate level of file server performance.

In the SAP system landscape, the application servers for SAP R/3, SAP BW, SAP APO, et cetera, are the ones that have a disk I/O access profile most like a standard file server. In addition, the required Windows NT primary domain controller (PDC) server also matches this disk I/O profile, as well as the web servers needed for mySAP.com configurations. The SAP data- base server cannot be compared with standard file servers, however, except for the database log volumes.

OLTP Applications

Online Transaction Processing (OLTP) applications, such as SAP R/3, generate transaction oriented disk I/O patterns with small, random I/O. The random I/O request size is dependent on the database used. It is common for a database used with SAP R/3 to store data pages on the disk(s) in 8KB sizes, although the actual physical I/O made to disks with random I/O may be smaller. A production SAP R/3 system makes millions of small reads and writes to random locations on the database server's storage system—normally during the daily online hours.

In addition to using smaller, random I/O requests, an SAP system also tends to make more read requests than write requests. A ratio of 80% reads from and 20% write I/Os to the database server is common for SAP R/3 systems. However, every SAP R/3 implementation is unique, depending on the business modules and customization used, so the example 80/20% read/write ratio does not always apply. In most OLTP environments, excluding heavy batch data upload activity, the percentage of write-I/Os to the database is usually less than 20%.

In a few cases, but not often, the storage system access read/write ratio of an SAP R/3 system has been measured as “bad,” as 60/40%. Having a higher write ratio is considered bad because of the impact it has on performance with certain data protection levels—RAID 5, for example. More detail on data protection levels and I/O performance for certain read/write disk access ratios is provided later in this chapter.

NOTE

SAP R/3 (OLTP) System Disk I/O Activity Assumption

In the SAP R/3 system landscape, the production database server has a typical OLTP disk I/O access profile of an 80/20% Read/Write (4R:1W) ratio, considered as default in the rest of this chapter, unless otherwise stated. The OLTP I/O profile does not necessarily apply to the database host of the SAP BW system or other mySAP.com application components, however.


OLAP Applications

Online Analytical Processing (OLAP), decision support, or data warehousing systems, such as SAP's Business Information Warehouse (BW), are used to run reports and to view all kinds of business data in ways normally not possible with or contained in the ERP system.

Although running reports is possible on a production SAP R/3 system, doing so usually has a significant performance impact on the transaction processing system. This is because a lot of data needs to be read from the database to create the various views and table constructs needed for report generation. Many disks will be tied up searching for this huge amount of data, essentially locking out others from using a part of the database. Running one large report during the business day along with OLTP activity can bring the entire SAP R/3 system to a crawl, causing response times to be unacceptable, sometimes even preventing other users from logging on. To address this kind of performance problem, SAP BW is available to offload reporting from the SAP R/3 systems.

The SAP BW system, for example, has two distinct operational modes. First, there's the nightly process to update the database by (a) copying in new data and (b) creating the Infocubes (specialized views of the data). Data is copied or transferred into SAP BW from other sources (SAP R/3 or other). This is a purely 100% sequential write activity with large disk I/O requests—essentially a large file copy. The copied data is then inserted into the SAP BW database as table and row inserts with certain requirements to update the predefined views of the data warehouse (Infocubes). Unless the data is sorted ahead of time, this update phase contains many random disk I/O writes to the database. This first operational mode usually happens on a nightly or weekend basis, depending on the business needs and the volume of data to be analyzed, and may have a narrow timeframe in which to complete. However, it is rarely performed during online business hours. The performance bottleneck in this process, besides the server's CPU and memory, is the ability to quickly update the database—a possible storage system performance issue, with special focus on the transaction logs and updating of indexes.

The second operational mode of SAP BW is the online query phase. Here, users run queries or reports against the appropriately structured data warehouse. Running reports and viewing data is normally a very read-intensive activity on the disks. However, significant amounts of writes also happen in order to run complex reports of the data that sort, merge, or join existing tables into new, temporary tables or views. In SAP BW terms, many ad hoc queries against Infocubes generate high amounts of write-I/O activity in the temporary DB space, which can result in many physical I/Os if the server memory is not large enough.

Disk Drive Mechanisms

The disk drive mechanisms, or spindles, are one of the most critical elements to choose for a disk-storage solution. Disk drives can vary in size, performance, and cost, so finding the ideal combination is not always straightforward. Disk drives come in different form factors or different sizes (half-height and low profile), and with various connector types, including hot swap connectors (combined power, data, and address signals) for online replacement. Among the various disk drive characteristics, the performance and capacity aspects are the focus for this chapter.

Drive Types and Capacity versus Performance

Disk drives are usually specified by their capacity (how many gigabytes of data they hold) and by their performance (rotational speed and seek times). There are always trade-offs between capacity, speed, and price. Higher capacity disk drives may not have the same rotational speed as smaller disks but are good for sequential I/O because of their high density. Smaller but faster disks are more cost effective for random I/O. Disks that provide both high performance and high capacity are usually the most expensive.

An SAP system needs both capacity and performance. Ultimately, the combination of storage system with the number and type of disks chosen must match the transaction load generated by the SAP application. Raw disk speed is typically less important than matching the total number of disks to the I/O requirements coming from the application. The sizing aspects of this chapter will address sizing the storage system to the requirements of SAP database servers.

Performance Characteristics (IOPS)

The number of I/O operations a disk drive can perform per second (IOPS) is a critical measure of performance for OLTP, random disk I/O request types. Disk drive mechanisms consist of mechanical and electrical components. The disk controller interfaces with the peripheral bus via a UDMA, SCSI, or FC connector. It also contains a local disk cache, usually 512KB to 2MB in size. Data transfers from the disk cache to the host bus in the server are considered burst transfers because they happen at the full speed of the disk I/O channel. For an SAP system, data is almost never located exclusively in the disk mechanism's cache, so the sustained transfer rate from the disk magnetic platter or media is more relevant.

Two main disk characteristics have contributed to the constant improvements of sustained disk IOPS performance: rotational speed and track-bit densities. Because the density of information stored on a magnetic disk platter increases, the capacities increase. In addition, seek time delay is reduced because the head reading the data does not have to travel as far compared with older generation disks. Newer generation drives of the same speed also have improvements in the seek algorithms, which contribute to higher OLTP performance gains. The newer the drive, the faster is its OLTP and sequential performance.

When measuring a disk's IOPS rate, the size of the data to be transferred must be specified. The typical random disk I/O transfer size of 8KB is assumed in this chapter. Of course, the larger the disk I/O size to be transferred, the longer it takes, and the fewer IOPS a disk can handle (transferring a 128KB block size takes longer than an 8KB block). However, relative to the overhead of finding the data block on the disk, transferring a larger I/O block from disk is more efficient. Table 4-1 lists the important performance characteristics of the common disk mechanisms, including rotational speed, random IOPS, and sequential I/O transfers.

Table 4-1. Disk Drive Performance Estimates
Disk Drive Rotational Speeds Estimated SAP R/3 OLTP OPS per Disk [a] Sequential /O Read/Writes (MB/s) [b]
5400 rpm 45 6/3
7200 rpm (1st Generation) 60 7/4
7200 rpm (2nd Generation or newer) 70 9/5
10K rpm (1st Gen: 4–9GB) 90 12/6
10K rpm (2nd Gen: 9–18GB) 110 15/9
12K rpm 125 18/12

[a] Estimated I/Os per second for SAP R/3 OLTP environments with random I/O requests of 8KB in size, 80% reads and 20% writes, with a response time of 25 ms or less. Actual IOPS per disk achieved in production environments may be different due to varying I/O profiles and disks used.

[b] The peak sequential MB/s throughput estimates per disk are relevant only for specific disk volumes in an SAP system (ex: DB log volumes). Depending on the disk drive, peak rates may be higher.

The disk device performance numbers presented in Table 4-1 are approximations only for purposes of discussion in this book. They are based on typical disks available during 1999 and are not intended to represent any one specific disk manufacturer's device specifications. Newer generation and bigger disks will generally have better performance than shown in the table. These estimates relate specifically to the physical disk drive's IOPS and sustained or sequential transfer rates from the media. They do not consider transfers from the disk's cache to the server host, which is simply a measure of disk I/O channel or interface speed.

From Table 4-1 , one can see steady improvements in disk technology; improvements that are welcome for database servers. The number of I/Os per second increases with each new generation of disks. An interesting trend, however, is that the overall capacities of the disks have been increasing faster than improvements are being made in disk performance. For SAP database servers, using many disks is usually good, whereas using many fast disks is best. Faster disks employed correctly (along with disk cache) can help maintain reasonable response times.

TIP

The Fast versus Fat Choice

Smaller, faster disks are often in the same price range as larger capacity, but slower disks. For best storage system performance, using the fastest available disks is the ideal choice. Often, this means deploying more of the smaller disks in order to make up the capacity needed for the database. This may mean a higher price tag for the system, but it may be a small price to pay when compared to slow response times or an unavailable system due to performance problems.

An alternative is to deploy both types of disks where they are most needed. However, mixing different types of disks may have configuration limits and has an impact on the management of failures or replacement parts, including hot spare devices.


Disk I/O Channels

For standard, high-volume servers, including Unix mid-range and Intel-based servers, several disk channel I/O technologies can be used. These choices include UDMA, SCSI, or Fibre Channel I/O technologies. This section covers the basic capabilities, terminology, and some issues associated with using these standard technologies with SAP systems.

UDMA

The Ultra Direct Memory Access technology (also called Ultra ATA) is primarily designed for smaller systems with a single drive configuration. Most Intel-based servers have UDMA controllers built in on the motherboard supporting two devices per channel. Using DMA to transfer data frees up the CPU to do other tasks. The actual performance of a single UDMA disk compared with a single SCSI disk is typically equivalent or better in a normal office application environment. UDMA is the evolution of the EIDE, or Enhanced Integrated Drive Electronics standard.

UDMA is definitely the lowest cost disk solution on the market, mostly due to the tremendous PC production volume benefits. The problem is that it doesn't scale well. This I/O technology is optimized for sequential I/O to one or a few number of disks (single-threaded), but it cannot handle the multiple, simultaneous I/O request patterns common with SAP database servers. It is simply not practical to use UDMA beyond a few disks per system with a large number of users.

Welcome to the World of SCSI

The Small Computer System Interface (SCSI) has become the industry standard I/O technology for server system disk I/O. Almost all server systems shipped today either have SCSI disk support built in or have an option to easily add it. SCSI provides higher channel speed and more performance in multi-user environments because it can do scatter/gather DMA (devices are busy seeking/gathering while another is transferring data). It's also more flexible, more reliable, and accommodates a larger number of peripherals. SCSI contains both a hardware and a software protocol specification. Some of the SCSI hardware specifications are listed in Table 4-2 .

Table 4-2. Common SCSI Technology Comparisons
SCSI Terms Parallel Interface [a] Bus Speed MB/s (max) Bus Width (bits) Bus Freq. MHz Max. Bus Lengths [b] (meters) Max. Devices [c]
SE HVD LVD
SCSI-1 5 8 5 6 25  8
Fast SCSI-2 10 8 10 3 25  8
Fast Wide SCSI 20 16 10 3 25  16
Ultra SCSI (Wide) 40 16 20 1.5 25  8, 16
Ultra2 SCSI (Wide) 80 16 40  25 12 16
Ultra3 SCSI 160 16 80   12 16

[a] Only the variations common in servers are shown.

[b] SE = Single-Ended, HVD = High Voltage Differential, LVD = Low Voltage Differential. Maximum bus lengths officially supported may differ, depending on the configuration.

[c] Includes the SCSI host bus controller as one device.

The SCSI terminology refers only to the disk channel technology and not to the logical disk volume structure commonly referred to as JBOD or RAID (explained later in this chapter). Even disk array systems use SCSI disks and channel I/O technology for the basic data transfer and hardware connection.

Either dealing with SCSI hardware configurations can be fun and relatively easy, or it can turn into a troubleshooting nightmare, depending on whether the electrical configuration rules were followed. The basic SCSI hardware specifications for the copper-based cables and parallel interface include many elements. There are several types of connectors that go along with the SCSI channel I/O specifications. In addition, SCSI offers two types of bus terminators for signal dampening: passive and active. The maximum bus lengths are determined by the electrical signaling and termination types, as well as by the frequency of the SCSI bus. The higher the frequency, the shorter the SCSI bus length. SCSI IDs, whether on an 8-bit or 16-bit bus, are one more factor to deal with when configuring the storage system (see Figure 4-2 ).

Figure 4-2. SCSI Bus Configuration


The SCSI electrical signaling type called single-ended (SE) was the lowest cost version to implement and was thus deployed in most Intel-based servers by default. High-voltage differential SCSI (HVD) supports more devices over longer cable distances but was more expensive to implement. Thus differential SCSI was common only among the higher-end server systems. This created the problem of not being able to use SE-based SCSI devices on a high-voltage differential SCSI bus. Ultra2 SCSI, and beyond, offers a cost-effective, low-voltage differential (LVD) SCSI specification with a 12-meter cable length, which is generally available for all types of servers and is compatible with SE devices.

TIP

SCSI Bus Configuration and Performance

There's a lot to think about when using SCSI with all the combinations of cables, connectors, terminators, and device types. When it is configured correctly, it works very reliably. However, SCSI bus termination, ID settings, and cabling issues are some of the most common root causes of storage system problems. In addition, performance problems can easily occur when mixing slower devices, such as CD-ROM or tapes, along with newer high-speed disks. Segmenting disks from other peripherals onto a separate SCSI bus is highly recommended for optimum performance.

Note that the SCSI bus speeds in Table 4-2 are theoretical. The actual sustained data transfer rates are usually about 25% less due to overhead of the SCSI protocol. Therefore, an 80MB/s LVD SCSI bus realistically only sustains about 60MB/s of data transfer, and so on. This may be important when determining how many disks to attach on one SCSI bus and when estimating the overall I/O capability of a server.


Fibre Channel

Fibre Channel (FC) is a new communications protocol designed to overcome the limitations of existing architectures. From an SAP system perspective, Fibre Channel technology enables the database server to be located far away from the backup devices, enables high data transfer rates between the database server and the storage system, and is the foundation for a more comprehensive, reliable storage area network (SAN).

The original purpose of the Fibre Channel specification was to support any number of existing protocols over a variety of physical media and existing cable options. Table 4-3 shows some of the cables and lengths supported at the FC specification's data transfer rates.

Table 4-3. Fibre Channel Cables and Distances Supported
Fibre Channel—Transfer Rate Fiber Optic Single-Mode @9 μ m Fiber Optic Multi-Mode @50 μ m Co-axial (copper) STP (copper)
1063 Mbits/s ~100Mbytes/s 10 km 500 m 50 m 28 m
4250 Mbits/s ~400Mbytes/s 02 km 175 m   
The existing 62.5 μm fiber optic cables for FDDI are also supported, but at shorter distances.

The standard Fibre Channel implementation uses the SCSI-3 I/O protocol stack for disks, tape libraries, and other data storage devices, although it can simultaneously transport other protocols. Fibre Channel, however, didn't become the high-speed networking standard. Instead, the networking industry leveraged the Fibre Channel technology for the Gigabit Ethernet standard.

Fibre Channel can address from 2 to 15.5 million devices using 24-bit address identifiers in a fabric topology. Devices may have more than one FC port. A simple disk array device, for example, may have two FC ports for redundancy purposes. An enterprise storage system can have 32 FC ports for scalability, availability, and connectivity. A server typically has one or more Fibre Channel I/O host bus adapters (two minimum for redundancy), usually a 32-bit or 64-bit PCI card, which contain a GBIC (Giga-Bit-Interface-Card). There's a specific GBIC plug-in module for each type of cable and connector style (short-wave fiber optic, coaxial, STP, etc.).

There are a few common topologies used with Fibre Channel, and they can be mixed. A common configuration is a two-node configuration, as shown in Figure 4-3 . The two-node con figuration can be either a small loop or a point-to-point (direct) configuration. It's the lowest cost and most secure FC connection between a server and one storage device. This has the highest maximum bandwidth possible because there is no arbitration. This is a common FC configuration for mission-critical server and storage systems.

Figure 4-3. Two-Node Fibre Channel Configuration


The arbitrated loop (FC_AL), as shown with a hub in Figure 4-4 , is a shared, 100 MB/s (200 MB/s full duplex) architecture. Because the loop is a shared transport, devices must arbitrate for access to the loop before sending data, analogous to Token Ring. A typical way to connect more than two FC devices to the same loop segment is via an FC hub. Arbitrated loops allow up to 126 devices, although the majority of them are deployed with from 4 to 30 devices. One port in the arbitrated loop may be attached to a Fibre Channel switch. With two or more target devices being transferred to or from at once, the performance impact becomes increasingly noticeable.

Figure 4-4. FC Arbitrated Loop (FC_AL) Topology


Although it is possible to connect more than two devices in a loop without using a hub, it's not recommended for availability reasons. The SPOF would be any device, which may be difficult to troubleshoot. In addition, an FC hub can isolate ports that have problems with cables or FC host bus adapters in the server or storage devices. This allows the remaining FC devices in the loop to remain accessible.

Typical hubs have 8–16 ports and are rack mountable. A device-to-hub connection uses short-wavelength laser technology, limiting the distance to 500 meters. These hubs typically have a special port called the long-wave port. This is used to interconnect to long-wave capable hubs to span distances beyond 500 meters, up to 10 km. The 10-km distance between hubs typically uses the more expensive, single-mode, 9 μm fiber optic cable.

Storage Area Networks (SAN)

The third Fibre Channel topology is the fabric, using Fibre Channel switches. The fabric topology is the foundation for Storage Area Networks (SANs). FC switches provide enhanced services to facilitate registration of devices and the discovery of targets (disks) by initiators (servers). FC switches may be connected together to form extended storage networks for large enterprises (the SAN worldwide-name addressing supports up to 15.5 million devices versus the 126 loop-IDs). Hubs may also be attached to switches, as seen in Figure 4-5 , but would be logically configured with arbitrated loop limits.

Figure 4-5. FC Fabric Topology—Server and Disk Nodes


FC switches are similar to the Ethernet-based switches. They contain a high-speed back plane that allows multiple, dedicated point-to-point connections to occur without a loss in performance. In addition, the FC fabric can be made redundant with a mesh design (described in more detail in the networking chapters). While FC switches remain significantly more expensive, it is recommended to deploy them when there is more than one simultaneous FC target to send data to or from. If there's one storage system with a single FC connection (the target), but two or more servers are attached to it via FC, then an FC hub will suffice. This is because a standard storage system can only transfer data between one server at a time. However, as prices per port on FC switches drop, they will be the dominant FC interconnectivity device for SAN implementations.

Performance at Large Distances

An additional benefit of FC switches over hubs is the throughput at longer distances. FC arbitrated loops, using long-wave hubs, will experience exponential performance degradation as the 10-km limit is reached (due to the round-trip delays of the loop). In the worst case, this could mean that an FC_AL configuration might only be able to reach 50MB/s instead of the advertised 100MB/s. FC switches, on the other hand, experience only a linear degradation of 10% at the full 10-km distance (i.e., 90MB/s). This may be important for configurations that span two data centers, especially when using software mirroring.

SAN Security

In a SAN environment, all server hosts can potentially be connected to all disk resources. Some level of security is needed to prevent unauthorized access to the SAP database server's disk volumes. SAN security products operate either at the host-level, the SAN fabric-level, or the storage-level. Table 4-4 shows the current support of typical SAN product types for security of operating systems, SAN FC switches, or storage products.

Table 4-4. Support of SAN Security Levels
Security Product Product Types Supported Intruder Node Protection Granularity of Security
OS Switch Storage
Host-based ++ +++ +++ (−) LUN level
Fabric Zoning +++ + +++ (+) FC Port
Storage-based +++ +++ + (+) LUN level
++ Limited availability or support across the industry
+++ Good availability or support available across the industry
+ Proprietary solution or limited to supporting only a particular vendor's solution
(−) Cannot prevent an unconfigured or intruding host/OS from accessing SAN storage
(+) Can prevent access against intruding host/OS

SAN security for mission-critical systems is best granted per LUN (disk logical unit number or volume), not at the port level of an FC switch. Which security product should be chosen depends on the number and variety of SAN components in these levels. The host-based LUN security SAN manager software works best if the host OS configuration is controllable (best with homogeneous OS), and works independently of the storage device type.

Storage level security is storage device type dependent and is typically available only on enterprise storage systems or newer mid-range storage designed specifically for the SAN environment. The LUN level of security is provided by software specific to the storage system and is best used to protect against other server hosts; it cannot be intruded upon by other operating systems. It is a good security method to use with heterogeneous server configurations if the same type of storage is used within the organization.

The SAN Promise

The promise of SAN environments includes universal data access among heterogeneous systems—a data storage network accessible by various types of server/OS combinations, including mixtures of Unix servers, Windows NT/2000, and others. SAN security solutions will also be more tightly integrated across the levels in the future. In addition, managing shared storage growth should be possible—disk and tape systems could be shared among different servers and repaired (hot plug), added, or increased (scaled up) without downtime. The performance will alsfo increase—FC is planned to grow to 400Mbytes/s.

TIP

SAN Fabric Compatibility

Not all FC devices (servers or PCs), FC hubs, and FC switches support all types of fabric interconnectivity. If a SAN is chosen for the SAP system, use only FC switches that support multiple access types for maximum flexibility. When using both disk and tape on the same SAN, choose only solutions that have been integrated, tested, and certified to work together. However, a SAN is not required for an SAP system. A simple FC direct connection could suffice in most cases.


The Fibre Channel standard is significantly larger and more complex than SCSI, thus some compatibility issues existed. At first, SAN solutions were limited to only manufacturer-supported configurations, with some limits regarding tape and storage systems—a separate SAN network topology is often needed. However, as OS support and the Fibre Channel standard mature, many of the compatibility issues will be resolved.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.135.197.201