Stress Test Execution During “Test Week”

With the big week finally upon you, there is much to do, from preparation to final review of both the system to be tested and the supporting test infrastructure, to last-minute assessment of scripts, data, and more. By the time this is done, you probably have time to run a test or two before lunch. I suggest performing a reboot of all servers involved in the test—everything, including the system to be tested and the systems driving the test—prior to officially starting a “run.” In this way, all caches are flushed, everything comes up “clean,” and a very repeatable test foundation is established. Bring down the test infrastructure first, followed by the test system’s ITS or other Web servers, and then the test system’s application servers, and finally the database server. Start everything up in the reverse order—database up first, then application servers, then ITS and other Web servers, and then all of the test infrastructure client drivers, and finally the test consoles.

Next, start the various monitors you will use to collect data on the run. Start two different SAPGUI sessions, kick off PerfMon or start your UNIX utility of choice, and so on. Note the wall clock time on your test run checklist, give the run a name, and then start the actual stress test run by releasing the first AutoController package. This starts your ramp-up period. Be certain to release each package in the same sequence and using the same time intervals each time you start a run, and note any deviations. I strongly suggest recording the sequence and timing in your checklist, to make sure there is little room for error. Why? Because even a couple of minutes difference in starting hundreds of test users will result in different throughput numbers, and generally skew the test run when compared to other test runs.

Leveraging Your Testing Tools

With the first test started, you need to begin monitoring the run immediately. I recommend focusing first on the testing tool’s console (AutoController, for instance), to ensure that the scripts are indeed running and that virtual clients are indeed being generated and making a connection to your SAP system(s). While monitoring this activity, don’t forget to continue the ramp-up process, though! In my experience, it’s common to release between 5 and 10 different packages in the course of 15 to 30 minutes, while simultaneously monitoring the test run.

Monitoring the Stress Test via SAP Transaction Codes

With your SAPGUI and a number of dedicated console sessions per mySAP component, you will monitor the activity generated by each virtual user. The initial thing to look for is the system activity spikes associated with each virtual client logging in to each system. CCMS transactions ST07 and AL08 are perfect in this regard, with the latter transaction providing the added benefit of showing you how well your logon load balancing system is working (AL08 displays end users sorted by the application server each has logged into).

When the logon spikes have subsided and the test run has approached the end of the ramp-up period, another set of CCMS transactions become valuable:

• SM66, to monitor how active a particular system is across all application servers and the Central Instance. SM66 displays all active work processes—database updates, dialog, background, and so on.

• ST06, to monitor disk queues, processor/system utilization, and memory/paging statistics.

• ST04, to verify that procedure and data cache hit rates are high or otherwise acceptable.

• ST22, to monitor ABAP dumps.

• ST11, to review error logs/traces as necessary.

• SM21, to review the system log.

• ST03, though don’t worry about collecting too much data during the actual test run. Instead, I execute one or two ST03 transactions to simply ensure at a high level that actual response times are in the ball park.

While monitoring your stress test via the SAPGUI and CCMS, don’t forget to continue to look at the status of the stress test as identified by the tool’s monitor. It’s not uncommon to “lose” a couple of virtual front ends, for example. However, this needs to be noted so that its impact can be factored in after the stress test run has completed.

Operating System Utilities

Although the virtual test tool and the SAPGUI/CCMS are vital to monitoring a test run, simple utilities like Microsoft’s PerfMon and UNIX’s command-line utilities are equally important. PerfMon will gather much of the statistical source data that will later prove useful in analyzing the real performance of your test runs, for example. And this PerfMon data also helps you troubleshoot and explain anomalies that might crop up during your stress test.

The easiest way to ensure that you gather the statistics and data you might need is to collect everything—all performance counters associated with each server. But if it becomes necessary to minimize your data collection, I suggest gathering all CPU, system, disk, and network-related information. And don’t forget to be consistent across the board. The same data should be collected for and available from each test run. Not doing so risks being unable to answer questions or drill down into specific data in a specific test run. And more importantly, running different collectors actually impacts the performance of the system—collecting more statistics impacts the system more, underscoring the fact that the same process should be employed and data collected for each and every test run.

Using Test Output for Continuous Improvement

One of the neat things about a test run’s output is that it can be leveraged to tune your pre-production system “on the fly.” Further, the data can then be used for comparisons, allowing you to measure the real value gained in the tuning/changes made to the system simply by executing another test run. It’s a great process, and certainly one of the best ways to tune a system simply because the load placed on the system is so predictable and repeatable.

For this reason, collecting data expressly derived from a stress test run is a must—you cannot afford to “forget” to collect your data, or to lump the test results of many test runs into a master file or another general repository. Do your best to keep both your test runs and your test output data discrete. To this end, I suggest creating a directory structure where each test run is given a directory, and underneath each test run directory are separate folders for input, scripts used, and output data. Copy all of the relevant test-related figures into their appropriate folders. I also create a simple checklist like the one that follows to help me remember what data to collect:

• Details related to the complete “load tested” solution stack configuration. Of critical importance is to note any changes (compared to the previous run) that were made to the stack, test tool, or approach.

• Average ST03 response time statistics for the test run (with all low-level details), especially dialog and update work process response times. Include background response times if applicable.

• Dialog steps processed per application server (also available from ST03).

• The output files created from each script, which should record input data and output/results for each transaction executed during the test.

• Total number of transactions executed (sorted by whatever criteria makes sense in your particular case). This should correlate pretty well to the number of transactions processed.

• The number of virtual users that executed on average throughout the test, as well as the number of virtual users at the end of the test. Note that there is usually a drop-off of a few users during any test; however, I void a test run if I lose more than a few percentage points.

• OS-collected statistics identified previously. And don’t forget to copy the PerfMon or other utility’s login to your test run directory structure. In some cases, you might even want to add event or error logs to the output test directory.

• Add to this any data collected manually, like regular “spot checks” recording CPU utilization or average disk queue length and so on.

With all of this data collected, it is then necessary to go through the process of preparing for the next test run. Sometimes this requires restoring the “test” databases for each mySAP component. In all cases, I recommend rebooting and restarting each system as previously described to clear all cache and start the next test run with a clean slate—in this way, caches will be predictably populated during the ramp-up period of the next test.