13.1. Overview—SAP Technology Stack Performance Data
First of all, there's really no such thing as a typical stress test for SAP—as you have read already, objectives differ, test tools vary, and success criteria range widely. Nonetheless, I've found over the years that the kind of data collected at different technology-stack layers tend to stay pretty consistent, even though the stack itself varies from customer to customer. The following list reflects the key data points or performance metrics germane to a performance-oriented test. Note the importance I place on collecting data at two different layers in the SAP Technology Stack—in some cases, this approach is only used to validate another layer's performance figures. In other cases, though, such an approach allows me to capture what I believe is the maximum performance a particular layer or component is capable of, and then compare this to what is observed at other layers to better understand bottlenecks or additional factors that limit performance elsewhere:
Server hardware. Data relevant to CPU performance are the biggest key, and therefore it's important to validate from a hardware perspective how rapidly the CPU subsystem can process data when no bottlenecks elsewhere slow things down. I most often correlate these high-water data with application-layer response-time statistics observed via SAP CCMS (e.g., ST03 and STAD). Just as important, though, I'll use the CPU data points in comparative analyses to better understand the potential processing throughput deltas between different processor speeds, cache sizes, or 32-bit versus 64-bit architectures.
OS statistics. The inherent overhead an OS incurs can be difficult to quantify and is really only practical (from an apples-to-apples perspective) on systems that support more than one OS. Thus, running a delta test between two identically configured HP Proliant servers, one loaded with Windows Server 2003 and the other with a particular version of SuSe Linux, can make sense. More often, this type of testing involves running CPU-or disk-based tests. Another key OS factor to watch is pagefile or swap file utilization. Your goal is to minimize the amount of paging, doing what you can to utilize physical RAM rather than the logical RAM that slow disk-based pagefiles or swap files represent. To that end, it's especially important to avoid “paging in” for Windows-based systems and any kind of paging for Unix systems.
Disk subsystem. Understanding maximum MB per second and I/Os per second throughput are essential keys to achieving good disk subsystem performance. But these two metrics need to be captured in terms of sequential and direct/random reads and writes. And, to be real-world, a load should represent a realistic customer-specific mix of all four combinations of reads and writes, sequential and direct. This is all put into context when disk queues are taken into account as well. Finally, it also may make sense to try and understand which component within a particular disk subsystem is limiting performance—caching algorithms, controller load-balancing algorithms, the optimal number of disks per disk group or LUN, the storage infrastructure or “fabric” interconnecting disk storage systems with servers and tape solutions, and so on. All of these contribute to a disk subsystem's overall performance in both positive and negative ways.
Database statistics. The maximization of cache hit rates for data and various buffers, the success of your database table buffering plan, the reduction of expensive SQL, the use of effective indexing, and the proper physical layout of the database in terms of data and logs all play key roles in maximizing database performance. So, too, does the selection of the optimal OS-level blocksizes, along with database-specific and hardware/controller-specific blocksizes.
Interfaces and middleware. Network throughput and latencies, and the time spent waiting on a middleware component to do its thing and pass control back to SAP are key areas to monitor, measure, and tune. Thus, RFCs, ALE connections, and SAP CPIC times need to be considered, along with characterizing the performance of the network interconnecting everything. Once core latencies are understood, it makes sense to look at each middleware solution as its own technology stack, and accordingly optimize it layer by layer.
SAP application layer. Characterizing online user and batch job response times, broken down by response-time specifics like those available in STAD and ST03N, is very important. So, too, is understanding workload metrics—throughput numbers, like the number of sales orders processed in an hour, give meaning to the previously described response-time numbers.
Multiple systems and components. Like middleware components, when it comes to tracking a cross-application business process, it's important to understand the load borne by each discrete technology stack “island” along with the performance of the network infrastructure or system component (e.g., the SAP XI) effectively bridging these islands together. The islands themselves may include SAP ITS components, any number of mySAP Business Suite components, non-SAP/legacy enterprise applications, SAP enhancements or bolt-ons, firewalls, routers, and so on.
Front-end client. At the end of the day, the speed an end user ascribes to a particular system is subjective. The response time observed after pressing the Enter key, for example, includes the time it takes the end user's request to traverse everything already described in this list, and then make the roundtrip back to the user—a lot of movement for a seemingly simple user request. The pieces often missing in this equation include lag time associated with the front-end network, however, along with roll-time and a factor known simply as “GUI build time,” or the time it takes to create and update the SAPGUI or other user interface back at the user's computer.
Many more areas of configuration may be important to a particular test run. Note, for example, that I don't include characterizing RAM performance as a key metric at the hardware layer. Although RAM is obviously critical, and more RAM is always better than less, the fact that it's so fast generally removes the need for deep analysis in stress tests. On the other hand, in the past I've benchmarked different systems using different RAM technologies or system busses/architectures based on specific customer requests or the need to simply characterize new customer platforms. How to best divvy up RAM into the various buffers and pools used by SAP and its underlying database is another common test scenario, too. In the end, what to test depends on your situation.