System-Level Stress Testing and Pre-Tuning

Although one of the goals of SAP stress testing is to guide and direct tuning of your soon-to-be-production system, it makes sense to engage in a certain level of pre-tuning as well. This involves testing and tweaking the individual solution stack layers and components to operate well in a standalone manner. When things run smoothly in this respect, you can then more intelligently optimize the entire technology stack to perform as a cohesive solution of integrated components and technologies.

Testing the individual components or layers in your particular SAP Solution Stack is also often referred to as system-level testing or systems testing, and will save you countless hours and much energy down the road. Key reasons for system-level testing include

• To establish a baseline for individual component-level system performance today, to be measured again as your change control processes upgrade these components or introduce new components or solution stack changes.

• To simply understand where potential performance bottlenecks might lie in the future. This is also called characterization testing or capacity testing.

• To therefore understand where future budget dollars can be best spent—where you get the most bang for the buck when you need your first performance boost. For example, if your network infrastructure is capable of supporting 200% growth in the number of online SAP users, but your disk subsystem can only handle another 30% growth in online users before performance becomes unacceptable, this information will help drive intelligent upgrades.

As you have been accustomed to by now, I will look at system-level stress testing by leveraging the SAP Solution Stack model. Specifically, the next few sections will drill down into server hardware and OS, disk subsystems and databases, network infrastructure, and mySAP.com component testing.

Server Hardware and OS Testing Tools and Best Practices

Before you commence a full-blown stress test, I believe that it is important to verify that the server is optimized for its role in your SAP solution. For Web and other Application Servers, CPU and memory performance are key. In a few cases, I have helped my clients benchmark how well different server and memory hardware configurations perform.

However, server performance is greatly affected by memory configuration at an operating system level, too. Windows-based systems must be set to maximize data throughput for network applications, for example, as displayed in Figure 16.2. And in all cases, memory swapping or paging needs to be minimized. With this in mind, I have found a number of utilities that support server hardware and OS-level testing:

• SmithMicro’s CheckIt utilities, inexpensive and effective, offer an easy way to quickly benchmark different processors as well as different servers. CheckIt is granular in its reporting, identifying metrics as diverse as system, video, network, and even disk performance.

• Various hardware vendors offer their own proprietary server load tools, usually available via download or on supporting software CDs. HP, for example, uses MeatGrinder and Thrasher for internal testing. And HP has made other tools available in the past, like Performance Stress Test Utility, which exercises the memory, disk, and network resources of your system. Another, System Stress Test, exercises memory, caching, and paging capabilities of Windows-based platforms.

• I recommend checking with each of your SAP Solution Stack partners to obtain testing tools specific to their platforms or technology products.

• Microsoft’s WAST—Web Application Stress Tool—is ideal for baselining your future IIS-based servers, like SAP ITS WGate server, for example.

Ziff-Davis Media provides a number of load testing tools, too, that they use to benchmark different systems. Both NetBench and WebBench are excellent products, relatively easy to set up and configure, and not exactly pricey. Download these load test utilities yourself from www.eTestingLabs.com.

Disk Subsystem and Database Testing

Quite a few tools are available that focus on proving the performance of disk subsystems. Each disk subsystem vendor, in fact, makes various tools available through its respective service and support organizations. Plus, other parties make tools available (that usually help promote their own goals), such as the following:

• Microsoft publishes a simple command-line utility called SQLIO that can be used to quickly provide MB/second and I/Os per second throughput data. This tool and sample scripts can be found on the Planning CD.

• Microsoft also makes its SQL Profiler testing tool available, used to record and play back SQL Server transactions.

• Intel used to publish Iometer controller and Dynamo workload generator utilities; today, these utilities are now in the open-source community. Refer to http://sourceforge.net/projects/iometer/ for further information. Iometer is excellent for testing network throughput as well as disk subsystem and disk controller performance.

Why would Intel (at one time) and Microsoft go to the trouble of creating utilities that test disk subsystem performance? Simple—these tools help prove that their respective products perform well in the enterprise.

But other tools are useful from a pre-tuning perspective, too. One of the simplest is Microsoft’s chkdsk utility, which can be executed from a command prompt for any locally attached drive letter. For years now, Microsoft has recommended that data and log drives for SAP database servers be configured for large block sizes. When SQL Server 2000 was introduced, Microsoft’s ERP support team went so far as to say they recommended 64KB block sizes, in fact. It’s simple to set this up when initially formatting a new data or log drive, using either the Windows Disk Administrator or the format command-line utility. But what becomes more difficult to check after the fact is whether a system has truly been formatted for 64KB. Getting back to chkdsk, I have found that running this utility is the simplest way to verify that a disk subsystem has indeed been set up correctly from an OS disk partition perspective. As illustrated in Figure 16.3, simply ensure that the number of bytes in each allocation unit equals 65,536 (which is 64KB), and that the file system is NTFS.

I recommend that you execute this utility without any switches, by the way, to ensure that you do not inadvertently write to the partition; without switches like /F, this utility is simply a display-only device in that it executes in read-only mode. Also, keep in mind that you must execute this command directly from each drive letter upon which a database data file or log resides. Thus, you need true access to these drives; a network share will not allow you to execute chkdsk.

Network Infrastructure Testing in the Real World

Testing the network infrastructure underpinning your mySAP solution can be quite important, especially if you are testing unproven technology or new user interfaces. I think back to the EnjoySAP initiative SAP AG embarked upon a few years ago, and remember the “unknowns” everyone encountered that could only be solved through testing or lessons learned. To level-set those readers unfamiliar with this time, SAP was well known for its thin GUI since practically its inception; the SAPGUI was tremendously efficient in the days of R/3 3x through version 4.5x. One of the many goals of my second SAP load test ever was to prove just how efficient the GUI was, in fact. My client and I arranged for the installation of a T1 network connection between their client LAN and the SAP server LAN, and we ran a number of stress test runs over this 1.544mbps link.

The results were impressive. A total of 600 SAPGUI front-end clients utilized that link concurrently, running typical SAP R/3 transactions related to sales order entries, materials management, production planning, HR, and finance/controlling. And the link showed less than a 20% average utilization, which we figured later represented a range of 40 bytes and 1,500 bytes per transaction. These were real numbers, mind you, and proved not only the efficiency of the SAPGUI but also the scalability of network links typical at the time. If 600 users could run so well over a relatively slow T1 connection, think of how fast they could operate over a 10baseT or 100Mbit network—they would figuratively scream. And at the time, with the throughput benefits of Gigabit Ethernet around the corner, it seemed as though testing the network tying SAP and other systems and their front-end clients together would simply be a waste of time in the future.

But things changed, as they tend to do. Remarkably, it was an end user who responded to a question asked by Hasso Plattner, a co-CEO and co-chairman of SAP AG, that sparked one of the biggest changes to SAP up until that time. The question? “What do you think of the SAPGUI?” From what I understand, the end user was not aware of Hasso’s role at SAP AG, and quickly gave Hasso a rundown of the SAPGUI’s perceived shortcomings—it was a bit difficult to learn, unintuitive, and just plain ugly. And just like that, a new initiative was born.

From the onset, EnjoySAP was all about improving the end user’s experience, and by all accounts I believe that SAP AG was successful in most every aspect. The only downside was that the GUI gained a bit of weight, so to speak. Not that it couldn’t be trimmed down on the fly—the new EnjoySAP SAPGUI boasted controls from nearly the beginning that allowed its innovative features to be throttled back, thus returning it to fighting condition when it came to running over slower network links. But many of my colleagues and I learned of these changes the hard way—from our customers, through testing.

In response to this customer-led education, a few of us at the competency center got together with some networking gurus, and we set out to see exactly how well the EnjoySAP GUI would perform. We built out a test lab with 10, 100, and 1000mbit network infrastructure, brought in a pile of 32-bit and (at the time) newer 64-bit NICs, and went to work building and testing SAP systems. The results, I believe, stunned all of us on two fronts. First, the SAPGUI testing showed that the public network traffic from a host of typical R/3 functional areas had grown to between 4,000 and 15,000 bytes (with everything enabled), a huge difference from the so-called classic GUI. Second, compared to 100mbit, the new Gigabit Ethernet gear was making little to no performance difference. In fact, in quite a few cases the NIC drivers were so poorly written that a server’s CPU utilization actually increased a couple of percentage points, while the number of network packets and the average transaction response time remained nearly identical across the board. Even in the best of cases, we could not manage to get more than 15 percentage points of extra bandwidth out of these first Gigabit NICs and infrastructure. And these best-case examples were not even representative of typical SAP implementations.

Fortunately, things changed again and subsequent versions of SAP’s user interface reclaimed much of what it had lost. The WebGUI interfaces were introduced in both HTML and Java formats, adding to the stable of SAP-supported user interfaces. Alongside this, Gigabit offerings matured into the products we know and use everywhere today. I hope I’m given an opportunity to do some more network testing for mySAP one day in the future. Until then, I feel pretty comfortable with things as they stand.

mySAP.com Component Layer Testing

When you think of optimizing specific SAP Solution Stack layers and components, you might not automatically think of the different SAP products themselves. But optimizing the mySAP components unique to your environment before you travel the road of SAP stress testing is huge! Consider the following areas:

• The sizing and utilization of your Program, Screen, and other buffers, various caches, Nametab buffers, SAP memory configuration/allocation, and so on dramatically impacts performance. Much of this can be reviewed by running transaction ST02.

• Verify that your “Top 50” online and batch transactions have been optimized in terms of ABAP and Java coding. You also want to pay particular attention to the Top 50 with regard to database request time, CPU time, dialog steps processed, and so on—this and much more is available through ST03.

• Review and validate that your Logon Load Balancing not only works, but that it effectively segregates different functional areas between different application servers. Don’t forget to factor in the throughput differences inherent to different application server platforms; older processor technologies and older servers in general cannot host the same number of users as newer platforms, and more obvious differences also exist between completely different hardware vendors/platforms.

• Using SM50 or reviewing your SAP Profiles, analyze the distribution and number of the different work processes across each of your application servers and the Central Instance. Pay particular attention to dialog, update, and background work processes.

Much of this can be addressed through the judicious review and application of SAP Notes. I recommend that an SAP Basis person experienced with performance tuning and SAP CCMS spend a day or two with your systems, to both review and then optimize them. You might also consider bringing in a utility to do this for you. For example, Precise’s Optistore for HP will improve your SAP solution’s off-the-shelf performance, touting itself as a key way to your SLAs while reducing overall costs. More information and a number of free white papers can be obtained at: http://www.precisesoft.com/.