12.3. Ready . . . Set . . .
We're getting close—just a few more steps to complete before packages may be released and workloads may finally get underway. As I review the steps remaining in the execution checklist to be sure nothing has been missed, I need to take the time to note any problems thus far, or identify potential problems (e.g., the fact that a client-driver's connection seems flaky or a script is having trouble reaching its peak user load). And most important, I need to stay relaxed—this is no time to rush through anything. That is, my methodical approach to stress testing only really pays off in the next few minutes, and then only if I continue along the same path and carefully work my way through starting all of the various monitoring tool sets necessary to capture performance data that can later be analyzed.
12.3.1. Gentlemen, Start Your Monitors!
Depending on the kind of test I'm executing, my project's success criteria, and the tools I've been granted, I may need to start only a single monitor (e.g., Microsoft's PerfMon or, in other cases, SAP's CCMS Monitor, RZ20) or I might need to start a slew of technology-stack-specific monitors. Of course, because I would have already been through all of the pretesting described in Chapter 11, at this point I'm keenly aware which monitors are required, and I know not only that the data I seek are indeed collected by these monitors but how to collect these data as well. So depending on my needs, I may start any or all of the following monitoring tools:
Network layer. Any number of hardware-based sniffer products (or Microsoft Network Monitor) are intrinsically valuable to testing that must characterize the load between a front-end client and the SAP Application Servers (especially if slow links are in the picture). The same holds true for application server to database server traffic, too, or the traffic between ITS/Agate servers and their SAP Application Servers.
Server hardware and OS layer. These tools are server- and OS-specific. For Windows-based solutions, I nearly always lean on PerfMon to provide a broad base of performance statistics (from network activity up through the OS to the database), configuring it to dump performance snapshots to a log file. Sometimes, I'll capture all statistics, though more often than not I'll narrow this down to specific hardware subsystems or OS constructs. For Tru64 or Digital Unix systems, collect provides just about all of the value I need in a monitoring tool while requiring only a very small fraction of a system's horsepower to execute (certainly not true for all Unix versions).
Disk subsystem. For classic Digital- and Compaq HSG80–based and similar SANs, I depend on vtdpy or its GUI-enabled counterpart DSview to provide the raw MB per second throughput numbers and I/Os per second data points valuable to many stress-test exercises. These tools also provide detailed information related to the average blocksize, load on individual physical or logical drives, and more. A connection to the particular disk subsystem in question is required to use either of these tools. With newer HP/Compaq Enterprise Virtual Arrays or HP's Virtual Array (VA) product line, however, I need only a browser pointed to the system's SAN Management Appliance.
Database layer. I typically do not need to start a database-specific monitor, because the data I need are readily available in part through the OS and in more detail via SAP CCMS (ST04 and DB02, in particular).
SAP application layer. CCMS transactions executed in real time during the test run and via scripted Basis monitoring transactions started before the run commences form the foundation of my SAP-specific monitoring. In addition to the slew of transactions covered later in this chapter, I might also start RZ20 for “daily load” stress tests (after having cleared all events so that the CCMS monitor I am executing starts “green”), because this allows me to visually monitor the real-time status of buffers, workloads, response-time thresholds, and more.
ITS layer. The ITS Administrator serves as the backbone for monitoring transaction-level activities and how they correspond to server loads.
Outside of these stack-specific tools, I might be inclined to execute one or more generic utilities or applications as well. Sometimes, the application depends on my customer's environment. For example, my customers that depend on BMC Patrol or HP OpenView may already have a set of monitors and monitoring criteria that could prove useful to me in a stress-test run. Indeed, they might even require using these tools to leverage standard or custom reports with which they are already familiar.
Other times, the decision to use a general systems-management tool may rest solely in my hands, rather than as an absolute requirement requested by my customer. For example, given my background with Compaq Computer Corporation, I often found myself running Compaq Insight Manager (CIM), not as a primary data collection tool but instead as a secondary way to gather network-, disk-, and CPU-related load and performance data. CIM acted as a “backup” to my primary data collection tools, helping me essentially to corroborate the data yielded from these other tools and approaches. And, because I've used it for years, issues with learning curves are minimal. In the same way, I've used PerfMon even when there's been no requirement to provide OS- or database-level performance statistics—the numbers still prove useful in validating throughput metrics pulled from ST03, ST04, ST06, and other SAP CCMS transactions.
As I mentioned before, another tool I use in monitoring a test run is a utility like eTimer, which is especially handy when it comes to executing a countdown and helping track when a particular milestone occurs (e.g., the login of the final SAPGUI client, signaling the start of the actual business processes). For a particular test run, I might set eTimer for a 40-minute countdown reflecting a 5-minute login period, a 15-minute ramp-up period, a 15-minute execution period, and a 5-minute cool-down period. I might use two or more timers to track multiple events, too, like the start of different functional scripts or after achieving a certain workload factor. And for test tools that support it, like AutoController, I might use both AutoController and eTimer to help me manage timing—I'll use AutoController to automatically launch a stress test at a prescribed time and use eTimer separately to track key events or milestones like those already discussed.
12.3.2. Log In to SAP
With all of my monitors up and running, and eTimer set for a countdown of perhaps 35 or 40 minutes, I'm at a point where the users may log in to SAP. I usually script this as a subroutine and use my staggered login approach to get everyone logged in as quickly as possible. However, I might choose instead to simply log in each virtual user and then have them all wait until I'm ready to commence the actual ramp-up or execution of a test. Thus, I might simply wait for the prescribed time established in each AutoController package to start the business process scripts, or I might manually release these test packages based on a set time tracked by eTimer (e.g., releasing a package exactly every 2 minutes until all packages are released). If I choose to manually release test packages, I try to choose a nice round clock time (e.g., 10:15 a.m. and 0 seconds precisely) to make calculations a bit easier going forward. Because the SAP login process itself is so resource-intensive and potentially time-consuming, it is important that I carefully note the start time of each login cycle, because I will also want to later ignore any performance statistics associated with this typically irrelevant phase of a stress-test run.
12.3.3. Go!
When the time is right and either all users automatically start logging in or executing or I manually control this by releasing the test packages myself, the stress-test run has finally commenced. At this point, it's critical that I maintain strict control over timing. Why? Because everything from this point on is tightly linked to timing. If I allow a portion of a test to go too long, or foul up the timing and release a test package too soon, the following will occur:
The load will be skewed, and therefore the test run will not faithfully reproduce the details associated with the baseline's load.
The amount of work done, or throughput, will also be skewed, because more or fewer front-end clients will be involved and therefore the potential to create more or fewer sales orders, for example, will exist.
Response times will accordingly differ from what they otherwise would have been (too great a load will naturally cause each business process to execute more slowly than it should have, for instance).
In effect, then, mucking around with the test run's timing represents a variable or change to the execution of the test run itself (in addition to the change or delta being tested in the first place), therefore breaking the cardinal rule: only one thing may change at a time. And the test run would therefore need to be tossed out and re-executed. This explains why I take extra precautions like using checklists, automating the release of test runs, or using timers—one less thing for me to worry about makes it that much more likely I won't screw things up. And maybe I'll make it home in time for dinner that night, too.