Key SAP Stress-Testing Considerations

Before an SAP stress test can be conducted, much in the way of planning and preparation needs to take place. Development and use of an actual project plan is recommended to ensure that nothing falls through the cracks, as articulated next.

Creating an SAP Stress Test Project Plan

For starters, your stress test project plan will need to address the following:

• Pre-tuning your SAP Solution Stack.
  
  
• Selecting a stress-testing approach and tools, and determining the technical skillsets and other resources (and related time commitments) required to support the testing.
  
  
• Analyzing business processes to determine which mySAP components, functional areas, and so on will come into play.
  
  
• Time required to work with the business groups to understand how these business processes flow.
  
  
• When these business processes are understood, they need to be scripted.
  
  
• Developing a process for “warming up” the database, and then restoring this known state prior to each new test run. Regardless of whether you use a simple tape drive approach, or leverage the capabilities of your disk subsystem to “break off” a “snap” of your database, the process needs to be clear.
  
  
• Preparing for, executing, and monitoring each test run.
  
  
• Collecting source data, sifting through it, and performing data analysis.
  
  

Later, as you prepare for the stress test and begin to better understand the challenges associated with scripting complex business processes, you will probably make changes to the plan. It’s not unusual, for example, to initially underestimate the time needed to find enough valid test data, build the test infrastructure, and even execute the test runs. These challenges and others are covered throughout this chapter.

Analyzing Online Users and Batch Processes

You must characterize your system in terms of the business processes that run as a part of the load you want to simulate in your stress test. This is a big part of identifying the mix of business processes, and usually boils down to characterizing online users and background or batch processes. Thus, if your goal is to simulate an average day on the system, your mix of online users and batch processes will probably be skewed toward the users. It will therefore be important to characterize which functional areas are represented, and how many users will typically be “active” within each functional area—I suggest leveraging your sizing data to nail this down.

However, if you want to execute a stress test that simulates your peak load on the system, you will probably need to run more background processes representing month-end reports or quarterly financial closings and so on. The nature of the work being performed by the online users will be different, too. For example, warehouse and inventory management users will be closing out inventory reports and displaying/updating material-related information. Manufacturing and supply chain users will be updating and running new demand planning and forecasting jobs. Business management teams will be seeking to better understand forecasting models, and may be updating budgets, employee records, and so on.

My point here is that the less you understand the nature of the business transactions inherent to a particular load, the less likely your stress test will prove useful to anyone. So take care to understand which mySAP components and SAP functional areas best represent the load of your system. And identify “backup” transactions and processes as well—these may become useful if your first choice in a particular functional area proves too difficult to script or too weak in terms of acquiring useful data necessary to drive stress testing.

It’s All About the Data!

Though it’s probably not obvious at first blush, access to the right data combinations is as important a consideration in selecting a business process for scripting as is anything else. That is, an easy script is essentially worthless if only a few combinations of data are available for use in a stress test. Why? Because after the data is used it is immediately cached at a number of different levels—by the hardware cache in your disk subsystem, by the database server’s RDBMS cache, even by the application server’s cache. Thus, the next time the same transaction is executed in your stress test, this data will be immediately accessible and therefore will drive absolutely no work on your disk subsystem. If you recall from previous discussions, to truly drive your disk subsystem is a major goal of load testing—it will certainly be exercised in the real world! And because it also tends to represent the primary “bottleneck” for most enterprise applications, properly exercising your disk subsystem is paramount to a successful stress test.

When it comes to data, you can never really have too much. Valid data is crucial, of course, but quantity is equally important as just discussed. But exactly how much is enough? The right answer is simple—try to never use the same core input data twice during the execution of a test run, data like customer numbers, materials, invoices, and so on. In my experience, the following approach has proven successful:

This method can work for all transactions, obviously. But it gets a little cumbersome as the numbers get really large. For example, if my 20-member financial team became 200 people strong, I might be hard-pressed to dig up 9,000 customers that have even been set up in the system, much less able to be given credit increases. My suggestion is to work with the business, therefore, to determine what is real. Do your best to simulate the real world with the data available to you. And whenever possible, push to use business processes that enjoy a large array of data.

Updating Your Project Plan

With a basic stress test project plan beginning to take shape, and a host of underlying business processes and data better understood at this point, it’s time to take a closer look at budgets and goals, and to give some careful thought to which business processes must actually be scripted. And we must determine at what point an investment needs to be made in procuring a stress-testing utility that supports virtual users, based on which mySAP components will play a role in your stress test. Ultimately, all of this will drive intelligent project plan updates as well as underscore the type and amount of scripting required to support the test. Specifically, you must

• Evaluate the need for real or virtual users
  
  
• Analyze which stress testing tool is most appropriate for your needs and budget
  
  
• Review and consider which business processes need to be scripted
  
  

Each of these three requirements is addressed in the following sections.

Real Versus Virtual Users

Some cases in your stress test plan will probably not be worth automating or scripting. Examples might include generating a bit of background noise through launching a couple of long-running reports, or even launching an MRP or other lengthy business process. In these instances, it could save time and expense to launch each of these processes through a dedicated SAPGUI session.

However, if you need to show the impact that more than a few users have on a system, it will be neither practical nor cost-effective to do so using “real” physical desktop or laptop clients, each running multiple SAPGUI sessions. Instead, investing in an SAP testing utility that supports virtual users is the ticket. Virtual users are exactly like “real” user sessions, with one core difference—hundreds of virtual sessions may execute on a single client driver, many more than the six concurrent sessions allowed by recent SAPGUI versions. The virtual SAPGUI sessions are executed behind the scenes via direct SAP Business API (BAPI) calls. In this way, the actual graphical user interface is not required to be displayed, as scripts written to support virtual users don’t fill in the contents of a SAPGUI screen by tabbing to it and entering data; they drive business processes through referencing German field names, the APIs for drop-down boxes, and so on.

Tools that support these powerful virtual capabilities must by their very nature support excellent monitoring and reporting capabilities. These features are by no means free, however. An exercise in price versus capabilities could be warranted in some cases, in fact. But in my experience, if you need to drive more than 50 or 60 SAPGUI sessions, a simple ROI study proves you need virtual testing capabilities. Because most stress tests involve hundreds or even thousands of users, your decision should be pretty clear. This is why virtual user sessions are the subject of most of the remainder of this chapter.

SAP-Aware Versus Freeware and Inexpensive Testing Tools

The argument for testing tools that boast an SAP BAPI-certified interface, and those that do not, comes down to price. Using SAP’s BAPI makes a testing tool “SAP-aware,” in that it can communicate with an SAP system through its standard API. In this way, SAP-aware tools boast the following capabilities unavailable in other software:

• They can “screen-scrape”; thus, they pull information out of an SAP output screen like “transaction completed successfully” or “Order 0006011998 posted for customer 19940425” and use it to populate a variable.

• Similarly, they can read the contents of a field displayed by the SAPGUI even if the field is not actually “visible” on the screen.

• They can take this capability one step further, and execute transactions without even displaying the SAPGUI; the transaction actually runs, but does so “virtually” or behind the scenes without the need for a user interface. The folks at AutoTester call this VDT, or “virtual direct test” capability.

• Finally, SAP-aware tools can build on these capabilities and run many users—called virtual users—on the same physical desktops or laptops. By using a high-powered “client driver” like an 8-CPU server with 2GB of RAM, literally hundreds or even thousands of virtual users can be set in motion.

Tools like this are not cheap, however. The most popular SAP-aware test tool vendors charge between perhaps $50K and $150K for the privilege of being able to execute 500 virtual users on a single client driver. I’m most experienced with AutoTester’s products, including their excellent AutoController product for performing SAP load testing. The folks at AutoTester are some of the most flexible and responsive I’ve worked with in the IT industry, too, and their product line boasts being the first to come out with a BAPI-certified stress test product for SAP. That being said, I’ve also worked with Mercury’s WinRunner, QuickTest, and LoadRunner products, and also CompuWare’s TestPartner (which boasts the added distinction of being the first testing product to be certified for use with Web AS’s eCATT product).

Even factoring in the cost of licensing, the savings are still significant in the long run, however, in that 500 networked and managed desktops cost quite a bit more, the cost of paying real users to give up a Saturday or sacrifice normal business hours productivity costs more, and not stress testing costs even more. AutoTester allows its partners (like HP Consulting and Integration Services) to lease software licenses to their customers at a fixed price, making them even more attractive for a discrete stress test. In these cases, the customer keeps the scripts and data coming out of the testing, along with the knowledge that their system meets or exceeds expectations, but the software enabling the test is not theirs to keep. All in all, it’s an excellent way to go for the most price-conscious SAP stress-test projects.

But it is still possible to save some money if you’re willing to sacrifice the ability to easily test a lot of users running on a single server or two. Other low-end products can be used that can drive the SAPGUI simply because the SAPGUI is a Windows-based application. Similarly, tools can be used to drive the WebGUI, too. Segue’s SilkPerformer, products hailing from Empirix, and even SPECweb utilities offer management and monitoring capabilities of a test run, at prices ranging from eight hundred to a few thousand dollars. For projects truly strapped for cash, Hiddensoft’s AutoIT scripting utility can be a great way to go, too—not only is it free, but scripts are available over the Internet that have already been customized for various SAP transactions and general functions. So, if you have the client-driver hardware, the infrastructure, and the time but are lacking financial resources, this could be the answer for your particular stress test.

Developing Business Process Scripts

With all of the information gleaned thus far, it should be clear which business processes need to be run during the stress test to represent the typical load on the system. Interview the super users and functional specialists to clarify any questions you may have. Further, it should be clear as to what needs to be scripted to represent a month-end close or seasonal peak—whatever represents the system’s busiest time.

In identifying these business processes, the specific mySAP.com components upon which these core business processes depend will be clearly identified. A mySAP financials stress test might exercise R/3 FI, BW, and SRM components, for example, whereas an SAP-enabled Executive Information System might require business processes touching SAP BW, SEM, and perhaps APO and R/3 Production Planning to be scripted. Use this information to begin planning for how and where you will ultimately develop and test scripts. Will you need a dedicated test environment or technical sandbox? Or will you simply (and more likely) need access to test or development clients within each mySAP system landscape? Further, consider how you will ensure that the data in these systems or clients remains static enough to support scripting—there is nothing worse than coding a set of business process scripts one week only to come back the next and find that your data is no longer valid, or the screens in your VA01 sales order transaction have changed!

Finally, not all business processes can be scripted by all tools. Certain mySAP components defy virtual scripting, some are simply not supported by particular test tools, and others present special challenges, all of which are discussed next.