12.6. Winding Down Test Runs
Once eTimer, another timing mechanism, or a closely watched clock indicates that the majority of my test run is nearing completion, it's time to begin thinking about winding down or “ramping down” the test run. This is necessary because most of the time I set up test runs to execute indefinitely (for flexibility in where I later mark the start and stop times for that particular run) rather than for a set period of time or a specific number of loops. Note that I set up this ability to run indefinitely at the script level (through a logic-controlled perpetual loop) rather than at the controller level, because it lets me run the same script standalone or as a scenario/member within a larger package. A number of methods are available to ramp down a test, each with its own unique benefits and pitfalls.
12.6.1. Stopping a Test Run Gracefully: External Status Files
Back in 1997, I was exposed to what I believe still represents both the most elegant and the most useful approach for gracefully stopping a perpetually executing stress-test run. In a nutshell, the approach uses an OS-based text file like STATUS.txt—called a “status” file—and compares the contents of the single line of text within this status file with a “control” word like “stop” or “start.” If “start” is found, the script's if-then logic either begins processing or continues to process; the loop of the script continues as normal. However, if the word “stop” is found, the script's if-then logic points to a subroutine within the script—or a completely different script, for that matter—that essentially closes all files, wraps things up, and logs the user off of SAP. The beauty of this approach is that it can be used with both network-based and local status files, the former of which make it easy to control the graceful shutdown of thousands of users from a single file. In addition, the content is not set in stone either—any word, any phrase will work equally well, and additional logic may be built around this content (e.g., building logic around the phrase “pause x” where “x” is a numeric value representing the number of minutes to pause).
As long as the STATUS.txt file can be opened and “read” by multiple users potentially simultaneously, this process works quite smoothly. In my experience, reading the contents is not a big deal. However, updating the contents of the file can be a challenge during a busy stress-test run simply because one of the virtual or physical clients always seems to have it opened for reading, and therefore locked. To get around this, I actually use a batch file that contains the proper word (start or stop) and simply copy a new STATUS.txt file containing the desired control word over the existing one.
Permutations of this STATUS.txt approach abound—rather than comparing the contents of a file, and therefore requiring the file to be opened and closed, you might instead simply “look” in a particular folder or directory for a file to exist. That is, your script might instead be coded such that if file c: empSTOP.txt exists, the script will begin the shutdown process. On the other hand, if the file does not exist, things would carry on as normal and the next loop in the script would be executed. And you can apply this to the status of a shared or global variable as well, as long as your test tool supports this and a method of updating the value of the variable is available during the middle of a test run.
12.6.2. Forced Aborts
When all else fails, you may need to simply abort a test run. Forcing a test run to abort should be considered only as a last resort, because it inherently impacts to some degree how well the test results are rolled up or collected—that is, the output associated with the tail end of a test run essentially becomes invalid. How is this possible? If a virtual user is only halfway through executing a scripted business process and then aborts, no performance data or workload data will be natively captured by the script (because I generally collect these data immediately after each loop in a test run), so the output logs generated from the script will be incomplete. Another shortcoming associated with aborting a test run is the fact that not all virtual and physical clients will stop executing at the same time; indeed, some may never actually stop executing at all. And in other cases, if you have hundreds or thousands of virtual users aborting simultaneously, they may or may not attempt to concurrently log their output to a shared output file, resulting in “missing” transactions that in reality were responsible for driving a portion of the load borne by the system. All of this needs to therefore be factored in when analyzing the results later, if indeed the test run was worthy of being analyzed—in most cases, it's not. Besides, because I usually abort a test run because of circumstances that would yield a poor run, the fact that the output log data are incomplete is not a problem.
An abort can be accomplished in a number of different ways or at a number of different “levels” in the SAP Technology Stack. Obviously, SAP testing tools represent the top of the heap, especially the controller application used to launch and manage test cases. Right-clicking an active AutoController package, for example, provides the option to abort it. Beyond using the test tool itself, though, test runs may be aborted at the database, OS, and even hardware layers. This becomes important if, for example, your test tool locks up or freezes, or one of the SAP components (e.g., an application server) is lost and you need to stop the current test to regroup. A number of SAP Technology Stack-specific methods are presented here:
SAP. Using the SAP MMC or other tools to stop an SAP instance is a good way of ending a stress-test run. SM04 provides this ability in a limited fashion, too, in that you can at least end particular user sessions in this manner.
Database. Bringing down the database or otherwise forcing a crash is an option, too. I don't recommend doing this unless you have to perform a restore anyway, and therefore have no risk of completely corrupting a database that must remain consistent for future test runs.
OS. Killing the processes, or stopping the services, associated with SAP or the database is another way to go, albeit a rather risky proposition unless you're already planning on restoring the database and restarting the services for the next test run.
Hardware. Bouncing or rebooting your disk subsystem along with your SAP servers and test infrastructure is a surefire method of aborting a test run. So is burning down the data center, as long as you're at it. Be sure this is what you want and that the potential ramifications are acceptable.
12.6.3. Loop Counter Testing
If you prefer to control the number of iterations a test run cycles through rather than focusing on executing a test run for a particular timeframe, a method I call loop counter testing is available. This is perhaps the simplest method of ramping down a test run, because it leverages an uncomplicated loop counter within the body of each script. Once the body of a script has executed a specific number of times—like 15—it may then be directed to write out the final results, close all files, and gracefully end the SAPGUI session. This is easy enough to do of course, because any incremental or countdown loop logic may be used. And it's great if you have an idea as to how long a script tends to run and you still wish to execute a test run for a rough period of time. In other words, if the body of your script takes about 2 minutes to completely execute and you're shooting for 30 minutes of activity, a loop counter of 15 or 16 iterations is a good way to go. Keep in mind the following points, however:
Your script will end prematurely if it runs faster than expected. This results in a “load” that was nonexistent through the final portion of your load-test period. The analysis performed afterwards would indicate a much lighter performance hit than expected, along with less work performed, and ultimately the entire test run might be of less value in an apples-to-apples comparison (if not completely worthless).
Conversely, your script might take longer to run than you expected, turning a 30-minute test into perhaps a 45- to 60-minute test. The real downside of this is that you wind up getting fewer test runs completed in a day, and you have more pre-ramp-up and post-ramp-down test data to “throw away.” If your success criteria hinge on achieving a certain transaction rate per hour, the whole test might need to be tossed in favor of executing a new one, too (presumably with updated “tighter” scripts). And for the bean counters out there, another real downside of this might be the fact that you have to keep people around later than otherwise required, adding up to potentially significant dollars.
Using a hard-coded script-based loop counter reduces flexibility. If you want the ability to stretch out a particular test run, it's simply not possible without going back and changing the loop counter within a script and then re-executing the test run. In these cases, I first recommend using a counter function within your Controller software, rather than within the script itself. If this is not possible, though, I suggest converting your hard-coded counter into one that uses a variable (which may be changed on the fly—an input-file-based text variable like the contents of COUNTER.txt makes this easy enough). It's especially easy to use a loop counter variable when your counter counts “up” rather than executing in countdown mode, in that you can change the 15 in COUNTER.txt to a 20 on the fly, and the script will be none the wiser. It's quite a bit more troublesome to change a countdown counter on the fly, though. For example, if you're at loop 6 in a test and are counting down, and wish to execute another 5 loops, not only do you need to be aware of the fact that you're in loop 6 but you need to be able to change the status of this counter to 10 within an actively executing script. On top of this, your output statistics would then reflect 2 runs of loops 6, 7, 8, 9, and 10, complicating data analysis later on.
The good news when it comes to using scripts that are bounded by a loop count is that there's realistically no chance of running out of materials or stock or running into other business process–specific supply-related constraints. That is, it's easy to plan for your specific input requirements, because you will know from the beginning exactly how many users will be executing each script and for how many iterations.