Script Development and Preparation in the Real World

The time and tasks related to developing the business process scripts to be executed during test week are discussed in this section. Before we begin, understand that the term test week is used pretty loosely—it’s not unusual for this “week” to actually run two calendar weeks. Most often, though, I have found that a calendar week—about seven days—tends to represent the average amount of time a company actually stress tests their production environment. This clearly does not include time spent in planning and preparation.

Unfortunately, we’re not talking about eight-hour days and lengthy lunches here, either. For the folks engaged in supporting test week, it seems as if no matter what level of preparation is taken up front, test week consists of something close to 12-hour days. And the days leading up to test week are usually not much better. Consider the following typical work loads I have scripted:

• In one of the first medium-sized R/3 systems I ever tested, preparation for a single test week consumed eight previous calendar weeks. During this time, I created over 50 online user transaction scripts across six different R/3 functional areas.

• Another stress test engagement required preparation to the tune of four weeks for R/3, and one additional week for BW. I coded 20 discrete online R/3 transactions, three batch processes with over a hundred variants, and five BW queries and a mix of other BW transactions in support of this test.

• Another R/3 system stress test required four weeks to script six end-to-end business processes (approximately 30 online transactions representing processes like order-to-cash and so on), and another week and a half to execute and analyze the test runs.

• Another really simple stress test consisting of only five core business transactions consumed two weeks to script and test, and another three days to execute (a short “test week”). Comparing this particular test to the previous tests illustrates the work involved up front; additional scripts become easier to develop after the core work is completed.

Why such a long time to prepare? Because there’s much to do. In terms of the big picture, it’s really not as big an investment as it might seem—a single stress test normally supports a production environment that will afterwards stay essentially the same for the next two to three years. So an investment of 2–8 weeks seems pretty reasonable in this light.

I spend much time with both the business and technical teams identifying functional process flows and work flows, and then gathering enough valid data to support a test. After installing the scripting tool on my laptop or customer-provided equipment, even more time is spent creating scripts that functionally work; like programming, script development is subject to development iterations, including fixing coding bugs, introducing standard subroutines for error handling and reporting, and so on. When I understand the business task to be scripted (such as the steps involved with creating a sales order), I must record the business process using the scripting tool. Recording involves tracking each transaction, keyboard entry, mouse-click, and so on that I execute in the SAPGUI (or WebGUI) to a text-based file called a script. This script is then edited to reflect variable input, to run in virtual user mode, and so on. Don’t worry too much about this process right now, though. I discuss the details of scripting later in this section.

During the script development process, I like to hold at least twice-a-week status reviews—these therefore consume project time as well. During each one- to two-hour meeting, I ensure that all testing assumptions still hold true, share status updates related to scripting in the various functional areas, identify issues or problem areas, and share successes. If something is changing (for example, I’m losing my test client because it’s getting refreshed, or the entire test environment is being refreshed again), we discuss what this means to the test project plan.

Fortunately, some of the tasks associated with a stress test can be performed concurrently, like working through the final test goals and success criteria, determining ways to validate success criteria, making revisions to the test plan and refining a testing methodology, working with the business units or functional teams to understand business processes or verify transaction flows, actually test-executing each transaction, and so on. This gives you an opportunity to crash the project schedule, a project management term that implies throwing more bodies at a project to achieve milestones faster; in some cases, one or two of my colleagues have assisted me with data collection and script development, allowing us to complete these tasks faster than I ever could alone. On the other hand, tasks that cannot be crashed imply things that must occur sequentially, one task after the other. A good example of this is testing a script for bugs—each script must be recorded and then set up to run in virtual mode leveraging variable input before any real bug testing can take place.

Core Script Development

In the weeks prior to test week, the script development process commences and matures or evolves as each mini-milestone is achieved. The following list assumes you are scripting in AutoTester ONE, version 2.x (other scripting languages that support virtual direct testing are similar in form and function). All of the following scripts can be found on the Planning CD:

• The correct SAP testing environment must be decided in advance, and then a SAPGUI connection needs to be made to it. To record a script using AutoTester ONE that will later be run in virtual mode (by using AutoController), for example, you need to establish a session with a particular SAP server and instance’s System ID (SID), and perhaps a logon group. I usually set up a standard SAPLOGIN.aof script for this purpose, which is then called by each functional script for the sole purpose of logging in to the system to be tested. The core functionality of this script amounts to executing a command like

  SapEstablishSession "server_name" "00" front

  where server_name is the name of the application server or central instance, 00 is the instance ID, and front tells the script to display the SAPGUI (useful for testing). For a virtual connection to an SAP logon group named FINANCE in the STG system, the SAPLOGIN.aof script might instead execute

  SapLoadEstablishSession "STG" "message_server_name" "/H/" "FINANCE" front groupconnect

  Of course, you need to ensure that the name of the logon group is correct; the SAP Basis team configures these via transaction SMLG.

• After a session is established with the SAP system, you can then log in with a scripted command like

  SapLogin "010" "gwanders" "password" "E"

  where 010 is the client, gwanders is your user ID in this particular client, and E designates English. Your password can be saved in encrypted form or as shown here, in clear-text (the password is simply “password” in this case). The great thing about establishing a session (rather than scripting a process that logs in to the SAPGUI by clicking a SAPGUI icon or in a similar fashion) and then logging in with the SapLogin command, is that many recorded commands are automatically recorded in a manner that allows for later running the script virtually. It is precisely this virtual capability that we want to leverage. Also, establishing a session has the added benefit that the SAPGUI information messages and screen names are captured by the script. These can later be used programmatically to drive if-then logic. For example, if screen “Display Material*” does not appear after 60 seconds, then execute ERROR_ROUTINE.aof, a scripted standard error routine that might write an error message to a log and then break the virtual connection to your SAP instance. Note the flexibility of some of these commands, too—an asterisk can be used to denote wildcards. In this case, any screen that is displayed within 60 seconds and starts with the title “Display Material” would not execute the error routine.

• Now we are at a point where business processes can be scripted. I assume you have already worked with your super users and functional experts and therefore understand the exact workflow that you need to duplicate via a script. It’s also helpful to understand how different business processes work, but it’s not required. Interestingly, as a Basis or technical resource, one of the best ways to gain an “Intro to SD” or “Intro to MM” education is to script in those business areas! In fact, through stress-test scripting I learned how a number of different business areas worked, enabling me to make future stress tests even more pertinent or valuable to my own clients. For example, by understanding that transaction VA01 in R/3 begins a whole slew of business processes, I was able to identify additional end-to-end work flows that needed to be scripted to represent one of my customer’s core business areas—they initially only wanted to create orders, but hadn’t thought about different sales areas, or performing repairs and credit returns, and so on.

• When a basic script has been created, it needs to be tested to ensure that it actually runs with the initial valid data provided by the functional experts tasked with assisting you. Just getting the basic functionality to “work” can be a challenge; more than once, I’ve been provided data that did not actually work in the particular system or client in which I was asked to perform the stress-testing script development.

• With the basic script validated and operational, I either collect more data myself or request more valid data from the experts. Finding this new input data was very tedious for me in the past, before I fully understood that not all data works in all business processes. For example, only specific customers may order specific items defined in a sales group, and only specific materials are actually associated with certain factories, plants, warehouses, and so on. To complicate things further, only specific storage locations within certain plants may house certain materials! It’s the rules of various business processes that make stress testing (and functional testing, for that matter) challenging. Once I began to understand that, I was able to intelligently search for valid data combinations on my own, thus freeing up my customer’s people to focus on their own tasks at hand.

• Testing my newfound data is next required. I perform my own “testing” of the validity of this new data by converting hard-coded input data (like the material number in an MM03 script that displays materials) into a variable. Thus, in an MM03 script, material 19690823 becomes variable MM03.INPUT, or something similar. I can define MM03.INPUT as a text or numeric variable, and point to various sources like an Excel spreadsheet or text file (samples of which are included on the Planning CD). I prefer Excel in that it’s easier to manipulate than a text file, but either works equally well. To test the data, I create a simple loop and inside this also include a “log” command, used to write the pass/fail nature of the data. I can then start the script and walk away; success or failure of the particular data is recorded and noted in the output log file, helping me to simply find good data combinations. More of this type of activity is discussed later in this list and in this chapter. Suffice it to say here, though, that testing data can be time-consuming in and of itself, and therefore requires as much of an automated approach as possible.

• Next, each script is adjusted or fine-tuned to better support virtual capabilities or what-if conditions. Thus, script syntax and commands are changed, necessitating more single-unit testing after these changes are made. For example, the names of SAP screens using AutoTester’s “SAPLookWindows” command are shortened and appended with an asterisk. This allows for variations in the screen name—some screens append a customer name or number to it, or a material number or some other unique value is added to it somewhere, otherwise making these not easily scripted. Further, I think about what might happen if a particular screen is not displayed by the SAPGUI when it is “supposed” to be displayed, and how to get around this—maybe a business rule only applies to certain customers, for example. So I add if-then logic to the script to address this.

• In the case of errors resulting from business rules, I also write code that directs the script to fail its current loop. In doing so, I force it to write an update to my output file for that particular transaction or business area indicating that the combination of data failed, and then I force the script to pull in a fresh combination of input data and start over. By programmatically addressing these “errors,” I avoid running my error subroutine, which ends the SAP virtual session and therefore causes me to lose one of my test users. I also avoid errors that otherwise cause the script to abort, which in turn tends to hold onto client driver resources (eventually requiring a reboot to remedy).

• Next, subroutines are added to each business transaction that cover things like explicit error conditions or failures. These kinds of conditions are displayed by the SAPGUI at the bottom of the screen, the values of which can be easily “scraped” and used in if-then statements. I pipe this data to the output file designated for the transaction or business area as well; it’s valuable to understand how many, and for what reason, business transactions fail in a particular test run, after all.

• I also build in routines for providing random, unique, or other input data not already addressed. For one of my customers, for example, I needed to provide a unique value for a configure-to-order (CTO) product. Their business rules required a unique serial number (which makes sense!), so I created a variable that concatenated the year, month, day, hour, minute, second, and the virtual client’s unique number. Just like that, I had a serial number that would never be duplicated, ever, no matter how many virtual clients ran the CTO process, or when. This and other script subroutine examples are discussed in more detail later in this section, and can be found on the Planning CD.

• I build any additional and necessary loops or do-while constructs. Sometimes, my customers request that a particular set of input files is used to feed their scripts—a loop and if-then condition must therefore be set up to determine when the end of a file has been reached. Similarly, when a virtual user establishes a session with SAP, it makes sense to execute a business process over and over again instead of logging in, executing the functional piece of the script, and then logging out. So a loop and if-then logic is built around the entire business process in this case as well.

• Now, around each core screen in a transaction or function, I build a process that collects response time statistics. This involves capturing data for whatever occurs prior to moving to the next screen in a business process. For example, if a test user presses Enter to advance from an initial VL02 “Change Outbound Delivery” screen to a “Delivery Change Overview” screen, I capture at minimum the associated SAP application server response time. Sometimes, I might also capture the “wall clock time,” or the amount of real time that has elapsed in a particular business process.

• Finally, I capture all of this response time data, error messages generated previously, and any other information captured through variables, and pipe all of this out to an output file designated for the transaction or functional area. For an example of what this entails, see the OUTPUT.aof file on the Planning CD.

When both scripting and associated single-unit testing is completed (including finding more last-minute data, which is not as unlikely as it sounds), the script is ready to be tested by multiple virtual users running concurrently. Exactly how many virtual users is up to the functional team and SAP project manager. Part of the work of stress testing involves understanding how many virtual clients will be needed to emulate the future production environment, including which functional areas and so on. When you have this data in hand, you are ready to progress to the next phase of script development—defining, installing, and configuring your load testing infrastructure.

Stress Testing Client Infrastructure

In stress testing, your goal is to run your tests in an environment that looks like production. In the best of worlds, you run your stress test on your soon-to-be actual production system. This is always preferred to any other alternatives, but is sometimes not possible. In that case, an identical staging or disaster recovery system, or something cobbled together that truly represents production, must suffice.

Just as importantly, you need to assemble the “client side” of your production system. It is this client infrastructure that I refer to when I talk about “stress test client infrastructure.” As I mentioned previously, it is rare for a customer to have enough desktop/laptop clients, network infrastructure, and time to actually assemble a mock client environment—this is precisely why virtual direct testing software products were built in the first place. But you still need some amount of in-place hardware to serve as client drivers. In other words, you will need to install one to perhaps four or five servers to provide the client load on the production system. Further, you’ll need one or more computers to control these client drivers. In the world of AutoTester products, both of these functions are performed by AutoController—an AutoController “Virtual Client” product handles the former task, while the “AutoController Console” addresses the latter. For specific and concise AutoController installation instructions, please refer to AutoTester-provided documentation and their Web site: http://www.autotester.com. And refer to the following for a high-level task list.

• Size the client infrastructure. For client drivers, I recommend running no more than 100 virtual users per processor (1.4GHz or faster). Extrapolate as necessary, based on the type and number of servers you have available. For the console, something in the neighborhood of 300–400 virtual users is supported per processor. And don’t forget to size for RAM. I suggest 512MB for 400 virtual users, which includes 128MB for your operating system and the AutoController console product itself. Much less RAM is actually required, but this gives you space if you need it, thereby ensuring minimum paging to your pagefile.

• Install the client infrastructure hardware and operating system. For an 800-user virtual test, it’s common to have two client driver servers and a single console. Often, I’ll load the console directly onto the same server running the client driver if I have the requisite CPU horsepower. Regardless, you’ll want to load Windows 2000 Advanced Server (to take advantage of all of your processing power, up to eight CPUs), and load Service Pack 2 at minimum. And stay away from /3GB and /PAE memory management, as these options are not supported by AutoController (which is actually fine, as memory beyond 2GB is not needed anyway).

• Install the SAPGUI, being careful in the installation process to include the option to also install SAP development tools, and then copy both the services and SAPMSG.ini file from one of your SAP applications servers to your client drivers and console. I’ve included samples of both these files, so that you can clearly see the kind of data that must be present.

• Install the AutoController client and console software (in any order); refer to AutoTester’s documentation for details.

• Configure each AutoController client, as documented in the “AC Virtual Client Software Notes” document found on the Planning CD. Note that it will make your life easier (and the scripts I provided on the Planning CD more useful) if you limit the number of characters in the “Remote Virtual Client Name” field to three. This is because the virtual name associated with each virtual test user generated by this particular client driver uses this as part of its name. Longer or shorter names therefore change the length of the name of each virtual test user.

At this point, you can begin building the AutoController Console packages that will execute the individual (or set of) scripts previously developed. There are a number of ways to group scripts or sets of scripts. Initially, you will want to test each individual script in a single package, however, to ensure that a single virtual user can run successfully on this new client-driver platform. This theoretically represents the final iteration of “single unit testing,” though in reality there will still be a need for it every time a change is introduced in a script.

When a single user can run in virtual mode, I like to increment the packages to run 10 virtual users; executing one package therefore starts 10 identical scripts. I verify that the input data used is indeed random, ordered, or as specified by the customer. And then I resolve any issues related to locking/contention (usually by revising a script to check for new “warning” or “error” messages generated by the virtual SAPGUI session and not seen before). If changes have been made to the scripts, I of course fall back to single-unit testing and ramp up to 10 users again, as just discussed.

After 10 virtual users run their scripts to completion successfully, I jump to the maximum required per the test plan, or to 100 virtual users, whichever is less. In an AutoController package responsible for starting many virtual users, it’s very important to ensure that all sessions are not started at the same time, but rather are staggered. In other words, ramp-up time must be taken into account—a well-thought-out stress test run should not actually commence until all virtual users have logged in to the system and are ready to “work.” Further, things like built-in delay times, waits and think times need to be carefully reviewed in lieu of the increased load 100 users place on the system, to ensure that any built-in delays still reflect the targeted think times even under load, and that everything still supports the stress test’s success criteria and goals.

Normally, 100 users is where I end my concurrent script testing—if 100 users can run the same transaction concurrently, with different data, and run to completion, in my experience quite a few more users can run concurrently, too. Plus, the number 100 is a very manageable number, as it’s large enough to speed through ramp-up, but small enough to allow me to track the status of individual virtual users as warranted. It’s therefore about the largest number of virtual users I assign to a package that I actually use for stress testing.

I perform 100-concurrent-user script-testing for each functional area, business process, or transaction that will become part of the stress test, working to ensure that the mix of online and batch transactions represents what my customer expects to see in production, therefore reflecting their stress-test goals. For example, I’ll build packages in AutoController that consist of a mix of BW, APO, and R/3 transactions, the latter of which may consist perhaps of 40 SD users, 120 financial users, 200 materials management users, and 40 production planning users if the R/3 online-user count has been identified as 400 and the functional mix has been earmarked at 10%/30%/50%/10% respectively.

Before we go on, I need to mention one final testing observation—it is very common for stress-testing client-driver utilities to appear single-threaded; indeed, some are and can therefore execute on only a single CPU. In the case of AutoController, this might appear to be the case, too. However, it’s not—I’ve run this product on 8-CPU machines and have witnessed first-hand a nice distribution of work across all of these processors. The key in this case is to open the Windows Task Manager, navigate to the Processes tab, and find the vatwrun process. Right-click this process and select the option to change processor affinity. Only the first processor is selected by default. Change this so that the vatwrun process executes on all processors (you can do this “on the fly,” in fact), and enjoy the results.

In the next few sections, I detail some of the specific scripting challenges touched upon thus far, and indicate what these scripts might look like in the real world of AutoTester and AutoController scripting.

Creating Administrative and Other Utility Scripts

Good programming or scripting often focuses on creating reusable code, like subroutines that can be accessed when needed. In the course of stress testing, I have developed a cadre of subroutines that seem to prove useful over and over again, like the following (which can be found on the Planning CD):

• Log in to the SAPGUI and establish a virtual session (SAPLOGIN.aof)

• Log out of the SAPGUI (SAPLOGOFF.aof)

• Random number generator (RANDOM_NUMBER_GENERATOR.aof)

• Unique number generator (UNIQUE_NUMBER_GENERATOR.aof)

• SAPGUI screen-scraper, useful in capturing error and warning messages or basic informational messages like “Credit limit for customer 320030613 changed” (SCREEN_SCRAPER.aof)

• A compare routine, useful for incorporating into if-then conditional statements (COMPARE_ROUTINE.aof)

• Read from an input text file (READ_FILE.aof)

• A reporting/statistics-capturing subroutine, including writing this information out to a text file (OUTPUT.aof)

Many times I actually build these subroutines directly into my script, rather than calling them—it just depends on the likelihood of having to change something, or the need for a specific subroutine, versus my willingness to manage separate subroutines. Regardless, I’m sure you will agree that these subroutines save you coding time.

Login and Establishing a Virtual User Session

There is more to running a good stress test than simply logging in a few hundred virtual users and letting them run loose. To control the ramp-up of a stress test such that all users actually have the opportunity to log in successfully, the users need to be logged in over a period of time. Why? Because logging in 500 users at once results in many of those login scripts aborting for timeout reasons.

At one time, I used to simply create small AutoController packages, and then manually release these in a controlled fashion, thereby executing the AutoTester scripts within each package in a controlled fashion as well. But I always admired the way that the SAP Benchmark Kits could log in a new benchmark user every second, and in doing so thus automatically control login “behind the scenes.” With the help of one of my very best friends and colleagues, we developed a similar programmatic approach for AutoTester ONE that I call Staggered SAPGUI Login. The core logic behind this approach is to manipulate the unique number assigned to each virtual user. Because this is held in a system variable, it is available for use even before a script is initialized or a variable file is identified. The scripted subroutine looks something like this:

• First, it’s good practice for the SAP virtual test user to check whether it is already logged in to a mySAP instance—if so, no other work is required, and the real purpose behind the script can commence (like displaying a purchase requisition, creating a sales order, and so on). Otherwise, the staggered login portion of the script must be executed. This is accomplished by the following code:

  Compare $SAP_SESSION "0"

  If this system variable is “0”, you are not logged in. At this point the rest of the subroutine in the script takes over (I use an if-then argument to control this), and therefore the lines discussed in the next few bullet points are executed. Note that if you were already logged in, execution of the script would drop out of the if-then clause and continue with the meat of the script.

• Next, I display and compare the value of the system variable $MACHINE (explained in more detail shortly) to the word “Machine.” This is because the machine variable is only set to something other than “Machine” when running a script virtually through AutoController. So if the variable is simply “Machine”, I am obviously not running a stress test—I’m probably busy coding and fixing bugs—so I hard-code my wait time to log in to the SAPGUI to a 1, for one second. The code looks like this:

  Compare $MACHINE "Machine"
  If Equal
  >Assign LOGIN.COUNTER = "1"

• The key to the whole Staggered SAPGUI Login subroutine lies in the next two lines of code.

  Otherwise
  >SubString $MACHINE 17 2 LOGIN.COUNTERTXT

  In this case, if the client is indeed a virtual client, it will have a unique number assigned to it (rather than simply the label “Machine”). This number can then be extracted from the complete machine name of the virtual user. For example, if you installed the AutoController client driver software with the remote virtual client name of “GLO,” virtual client 197 would have a full name of “GLO.VIRT.USER.0197.” Performing the SubString function “17 2” would extract the last two characters “97”. Similarly, “16 3” would extract the last three “197”—the SubString function works from left to right, counting to the character position indicated by the first number in the command, and extracting the equivalent number of characters specified by the second number in the command. Because these are alphanumeric characters, I assign the value to a text variable (LOGIN.COUNTERTXT).

• After the text variable LOGIN.COUNTERTXT is assigned, I can then manipulate the data in any way I choose, including converting it to a numeric value as shown here:

  >Assign LOGIN.COUNTER = LOGIN.COUNTERTXT
  >Message LOGIN.DELAY wait 2
  >Delay LOGIN.COUNTER

  The LOGIN.COUNTER variable is a numeric variable that can subsequently be used in comparisons, or as a starting point in a countdown, or used in a simple “delay” command as depicted in the preceding example. Note that the “message” command is optional—I like to see for myself that the subroutine actually worked while I perform my script testing. But this is commented out in the actual code finally used for scripting because it’s not necessary and in fact detrimental as it consumes video resources on the client driver machine. Think about the screen refreshes necessary for 400 virtual users running on a single client driver simultaneously, and you’ll begin to understand that any “extra” message boxes need to be avoided.

It’s important that this scripted subroutine be executed before you log in to SAP (that is, before you execute SAPLOGIN.aof). Why? Because you are executing this subroutine to control when the SAP login process should commence. So in keeping with good coding standards, it cannot exist as part of your SAP login script. Instead, it needs to be executed by the actual functional script, or an “umbrella” script used to execute multiple discrete transactions.

You might wonder why I used a system variable ($MACHINE) and not a user-defined variable. This is because most of my scripts tend to leverage different variable files, each with perhaps different variables defined. System variables, on the other hand, are consistently available even when a variable file has not been defined. And the $MACHINE variable in particular will always be unique to a particular virtual client, making it an indispensable part of the Staggered SAPGUI Login approach.

Ramping Up Users and Processes

With specific AutoController packages created for each functional area or business process to be tested, ramping up virtual users (and therefore active work processes) is pretty easy. As I said before, I define a set number of users to a package. If I need to execute 300 Sales and Distribution users on an R/3 system, I create three packages, each defined to run 100 virtual users. To control the ramp-up, at the prescribed time I simply release the first package, and monitor the SAP logins via SAPGUI transaction AL08 (as shown in Figure 16.5, which displays “Current Active Users”) or ST07 (the “Application Monitor: User Distribution”) transaction. It’s also possible through the AutoController console to monitor the activities of the individual virtual users themselves, but I tend to shy away from this functionality (I disable this feature) because it consumes a great amount of additional processing power on the AC console.

After the first 100 users are logged in, I then release the second package and follow the same monitoring process just discussed. At 200 users, I release the third and final package. When all 300 users are logged in and executing transactions, I note the specific “stop time” for ramp-up, which doubles as the “start time” for the actual stress test.

Later, when analyzing the data associated with the test run, I toss out all transactions and data associated with ramp-up. Doing so allows me to focus only on the statistics associated with the actual run. This underscores why it is so critical to track the start and stop times for each individual transaction that is executed—recording start and stop times (also called “wall clock” time) is performed programmatically in the body of the script’s loop, and written to the output file by virtue of the output subroutine called by the script.

Collecting Statistics

Statistics are critical to the stress test; failing to capture the proper source data is therefore akin to throwing away most of the budget money earmarked for stress testing. I normally collect the following data and statistics:

• In regard to housekeeping, I capture the virtual client’s machine name ($MACHINE), the test case ID associated with the script (a high-level ID, like “SAP SD” or “BW Queries”), a “Run Name” associated with the particular script (useful for versioning), and the host name of the SAP server in which the test user logged in.

• General statistics, such as the current date, the time that the script began, the time that the functional loop began executing, when the functional loop completed (which usually coincides with when the script wrote its reporting/statistics data to an output file), and when the entire script ended.

• Specific statistical data, including the name of the SAPGUI screen and how long (response time) it took to display the screen, the screen’s input data (like a customer number, material number, PO number, and so on), and the screen’s output data (like the order number created, or an information message noting the fact that a record was updated, or an update failed, and so on).

Other statistical data proves useful, too, though this data may not necessarily be collected via the scripts being executed but instead through other means. For example, I like to collect test-run data that represents a snapshot in time (like every two or three minutes) over the course of a test run. The snapshot data might include the number of concurrent processes executing, the number of users actually logged in and actively using a work process, the disk queue length associated with each of the database partitions, average CPU utilization of each application, database and central instance server, and so on. This information is captured in a variety of ways—through another SAPGUI session used for monitoring the stress test run, through Performance Monitor or other OS-based performance utilities, and even from the output generated through the testing tool itself (for example, AutoController can generate a comprehensive “results” file for your stress test run, if configured and enabled to do so).

I’ve also created specific AutoTester scripts designed to capture some of the source data just discussed. These are intended to run concurrently with my stress test scripts (another example of scriptable and repeatable background noise), and include powerful CCMS transactions like ST07, which is also shown in Figure 16.6. I am fond of ST07 because it gives me a snapshot of not only how many users are logged in, but how many are actually active simultaneously, how many are actively using a work process at that point in time, how many average user sessions are logged in, and even how many application servers are servicing various functional areas (which verifies that any logon load balancing is working as expected). Automating this transaction gives me one less thing to worry about when executing the actual stress test runs. A sample ST07 data collection script can be found on the Planning CD.

Regardless of how you collect your source data, it’s critical to understand all of the statistics you want to analyze later, as you must actively capture this data during the run. Even the simple task of starting a PerfMon log must be actively performed; I suggest creating a simple checklist of things you want to capture, so that you remember to do so during the actual test runs—without a checklist, I know personally that I would have spent more time rerunning test-runs to collect forgotten data, instead of getting to the business of analyzing that data.

Logging Out—Gracefully Ending Your Test Session

When you have completed your test run, or have decided to end or otherwise abort the run, you need to gracefully stop the test. The key here is “gracefully”! It’s pretty simple (though time-consuming) to kill all of the SAP sessions via transaction SM04. Faster still is to abort the actively running AutoController packages. But from a data-collection perspective, this is not the best answer. Why? Because correctly logging out of the virtual session is critical; an unsuccessful logout can get in the way of collecting statistics associated with the run, and skew the test results, too.

In the best of cases, simply aborting a script only causes you to lose the statistics associated with the last loop in the test run. It doesn’t sound like much, but for a test of 1,000 users, losing the results for something between 1 and 1,000 transactions will skew your results and, depending on the number of loops, actually represent a wide margin of error. Some of the 1,000 transactions may just have started, some might be halfway through a complete business process (which then might invalidate the entire business process for the test, given that it never completes successfully), and other transactions may be in the middle of writing output statistics (some of which may be actually written, while others are lost). And worst case, a whole slew of backed-up transaction output that is queued to be written might be lost. This is next to impossible to track, making the whole test run suspect. In fact, none of these cases reflect sound business process or transactional integrity, and therefore should be avoided.

Aborting the activities of a virtual user might be required in some cases, however. For example, if after executing a business process 30 or 40 times, your virtual user 197 gets hung or “stuck” in a transaction (due to poor scripting that did not take an error or other condition into account), you could very well lose all of your statistical data pertinent to that particular user—the orders created, materials moved, and so on. For one or very few users, this is easily dealt with. The output files reflecting activity for virtual user 197 can be purged or simply not considered in the final analysis, for example.

But more often than not, you will need an orderly and repeatable method of ending scripts. A few possibilities exist:

• Code each script, or set up each package of scripts, to only execute a finite number of times. This represents a method of testing in and of itself, in fact, in that the test becomes more of a race against time than an exercise in throughput.

• Code each script to check the status of a system-wide or other variable. If the contents of the variable in a special variable file are “true,” end the script. To make the contents of the variable true, and therefore end the test, I suggest copying a new variable file (with the variable already set as true) out to the file share or other location where it currently resides and is accessed by the various scripts (in effect, copying “over” the current variable file). This approach may be faster than trying to edit the contents of a variable via AutoTester’s Variable Panel (or a similar tool’s method).

• Similar to the approach described in the preceding paragraph, code each script to read the contents of a “status” file; if the contents say “stop,” then execute the SAPLOGOFF.aof script to stop the test.

I’ve used all three approaches with varying degrees of success. The last method can be troublesome, however, if a shared file is used for more than something like 100 users. This is because of the open and read activity that takes place, and resulting “lock” that one user places on the file. Subsequent users can’t get to the file, and thus sit around and wait for access. If the wait is too long, these scripts can abort. In my experience, the best way around these locking issues is to create a unique “status” file for each virtual user, like status.0197 for virtual user 197. Doing so eliminates file-contention issues, but brings up new issues including how to manage the process for quickly changing the contents of 1,000 status files. I have used simple batch files to rename the current status file (one that does not contain the word “stop”) to another name, and then renamed a file containing the word “stop” to status.0197—this works out quite nicely, avoiding every problem that I have ever come across in this regard. Other methods are certainly possible as well.

Additional Scripting Tips and Tricks

Throughout the years, I have come across or developed a few handy scripting tips and tricks that I believe will be beneficial to others. For simple “hardware delta” stress tests (where I want to compare the throughput of one SAP Solution Stack to another), one of my favorite practices is to avoid running transactions that create anything. It’s still very important to run transactions that issue writes to the database (like transactions that change sales orders, purchase requisitions, customer credit limits, and so on). But by avoiding inserts, you avoid making the database larger and therefore eliminate much of the need for restoring the database to a known state before each stress test run. This saves a lot of time on a couple of fronts—you don’t have to devise a process for warming/populating the database, nor do you have to take up time restoring the database to a known state before each test run—and makes a lot of sense for certain load-testing scenarios.

I’ve also learned to use the random number generators that ship with different load testing tools. Some of these are quite good, in that they generate a different number or “seed” as expected. Others tend to generate the same sequence of numbers, which is therefore not random in my eyes, but can be used for predictably unique sequences. I highly recommend developing your own random number generation process, too. Like the serial number example I gave earlier in this chapter, a random number can be easily created by simply concatenating a number of values that change constantly. I often use the virtual client’s unique number along with the numbers associated with the current time and date to create a truly random number.

I’ve also become a big fan of scripting Basis transactions that can help me automatically gather test statistics. Like transaction ST07 discussed previously, transaction ST03 is another excellent tool for gathering post-run response time and throughput metrics associated with a particular test run. In the past, I’ve coded ST03 in combination with SM51; using SM51, I can select a specific application server (like the top one in the list). I then follow this up with an ST03 to gather specific dialog steps processed for the run as well as average response time, wait time, load time, roll time, database request time, enqueue time, and so on. After collecting this application server-specific data, I simply run SM51 again and choose the next application server in the list. In all, this is just the kind of data that truly proves an SAP system is ready for prime time, or that one SAP system outperforms another.

In running stress tests, I noticed long ago that it took a whole lot of end users to create the load I needed to generate (as observed in SM50 or in system-wide detail in SM66). As explained earlier, these transactions allow you to display the status of dialog, update, and other work processes, as shown in Figure 16.7. The following list describes some of the ways I have artificially and consistently created a greater load on a system during stress testing:

• Use “noise” scripts. Some of my clients call this background noise; regardless, the function is the same—by executing a core set of display-only transactions behind the scenes of your real stress test, you will realize much more activity in the system and create a more-representative stress test. Noise transactions that I have scripted in the past include VA03 (display an order), MM03 (display a material), PA03 (display an employee record), ME23 (display a purchase order), and quite a few other read-only transactions.

• Similar to noise scripts, execute a predefined set of long-running queries or reports behind the scenes. A customer’s custom “Z” reports can prove beneficial in this regard, and truly help exercise an SAP system.

• End a transaction by executing the initial transaction again. For example, if you execute a VA01, you’ll go through three or four screens that need to be filled in with input data and processed. At the tail end of the transaction, an order will be created. Run the subroutine to gather your statistics followed by incrementing the loop as usual. I then suggest simply running the core VA01 transaction again, only this time refrain from going through the additional screens. Instead, execute a “/n” and repeat the loop as normal (starting again with VA01). The end result is that the load on the system is increased in a regular and repeatable manner, without introducing new or unusual transactions.

• For really basic load tests, especially in the case where no user data is yet available, it’s possible to gain a respectable amount of value from stress testing by executing a set of scripted Basis (as opposed to functional) transactions. These kinds of tests best support hardware delta testing, which I spoke of previously. Comparing one system to another can be accomplished by measuring total dialog steps processed for a finite run (of say 30 minutes), or calculating average response time, or simply monitoring the number of times a particular loop of transactions executed. Typical Basis transactions that are available even before any developers customize an SAP system include ST02, ST04, ST06, ST07, OS01, SM50, SM51, SM66, DB02, RZ10, and a host of others—the idea is to find a set of transactions that are not only relatively easy to script, but also that make database calls or CPU requests, like OS01 depicted in Figure 16.8.

  Figure 16.8. Though simple, CCMS transaction OS01 provides powerful up-to-date rolled-up user data, and in doing so creates a predictable load.

  

For scripting tools that are not SAP-aware, another useful practice is to drive the SAPGUI using “keyboard” commands as much as possible, instead of performing mouse clicks. That is, if you can navigate via the keyboard to the button you need to click, or the radio button you need to click, you tend to get a more reliable script. And because many of the screens in the SAPGUI support function keys as well as “mousing,” it’s not too difficult to find keyboard shortcuts. While scripting, right-click the background of the SAPGUI to see the available function key shortcuts for that particular screen.

Finally, yet another handy trick I picked up a few years ago involves using the number of seconds in the current time (0–59) as a way to “randomly” determine which transaction should execute. This is useful in umbrella scripts, which as I mentioned before are scripts that essentially execute other scripts based on a set of criteria. I use umbrella scripts and information provided to me by my customers to set up distributions. For example, if my customer tells me that 10% of all scripts should execute FD31, and 30% should execute VA02, I set up if-then logic in my master umbrella script like “if time = 00 through 05, then execute transaction FD31.” This represents 6 of 60 possibilities, and therefore 10%. Similarly, “if time = 06 through 23, then execute transaction VA02.” This represents 18 of 60 possibilities, of 30%. As you can see, it’s granular enough to handle different percentage loads. And if you create AutoController packages of 60 virtual clients each, you are assured of getting an excellent distribution (assuming you use the staggered login approach I explained previously).

One of the appealing things about scripting or coding is that there are usually many ways of solving a problem. I look forward to hearing about different methods that you will ultimately use to solve your own stress-testing scripting challenges. And I hope that some of the scripts discussed here and included on the Planning CD give you a framework or at least a few ideas as you seek to plan for and complete your own scripting.

Final Preparations Before “Test Week” Commences

By the time “Test Week” draws near, all of the gear required to complete the stress test should be in place. That is, the production system or system to be tested must be in-place and locked down (in terms of change control) as much as can be expected at this time. All virtual test clients—physical desktops or servers, software, and everything associated with driving virtual clients—must be in place. Any special monitors or consoles to be used for monitoring must also be set up. For example, I require at least two desktops from which to execute SAPGUI sessions, OS-monitoring tools, and so on. Finally, the AutoController (or whatever product you are using) stress-testing console needs to be installed, configured, customized in terms of packages/scripts, and tested. This will help ensure that Day 1 of Test Week is not wasted performing pre-test-week tasks. Without further adieu, on to Test Week!