Lessons Learned in the Real World
In the last few years, I have had the honor to assist countless SAP customers with designing and executing their SAP stress tests. It’s an area of work that I really enjoy, and one that is as valuable to my clients as it is fun for me. The hours are long, but the rewards are great. So, too, have been the lessons I’ve learned—I hope the following list helps you not to repeat some of my own mistakes and oversights!
I quickly learned the value of data in one of my first stress tests—more data is always better. In this particular case, I had so little transactional and master input data that within a few minutes of a test run, all of it was accessed and just as quickly cached by the disk subsystem. With next to no load on the disk subsystem, the entire test was tremendously skewed; user response times were phenomenal, transactions flew to completion, and the whole effort was wasted until we uncovered 100x more data.
Don’t try to execute a stress test too early in your SAP project implementation. Good data can be hard to find when you begin testing too early. SAP clients change as data is brought in or created, functional areas are updated, and business processes are revamped. This fact manifested itself in scripts that no longer worked a few days before Test Week, due to data combinations being invalid and SAPGUI functional transaction screens actually changing as a result of development. This brings up another lesson learned: As much as possible, lock down your client and configuration throughout the script development and test execution phases.
Not all virtual users are created equally. Different testing tools exhibit different tendencies and strengths. Some force all users to log in to the system simultaneously, whereas others exhibit a pattern of repetitive activity. This can all be resolved programmatically, but requires time. Thus, do not wait until the last minute to actually test your scripts in a load-testing manner—follow the process I outlined earlier in this chapter, where scripts are tested for functionality, then single-user tested, 10-user tested, and 100-user tested.
Think times need to be considered and verified. If a system is designed or sized to host 1,000 concurrent medium users (therefore implying a 30-second think time), you need to be sure to include the appropriate think times in your scripting efforts, and then verify that actual “wall clock” time and your programmed think times don’t contradict each other. This is especially true as a load is put on your system. Consider the following example—after scripting a VA01 for a customer, I noted in my single-unit testing that the end-to-end process took three minutes to complete. With wall clock time of three minutes, and direction by the customer to execute 12 sales orders every hour (or one every five minutes), I figured I needed to hard-code two minutes worth of think or delay time. When I had 200 of these SD virtual users executing, however, the average wall clock time jumped to more than four minutes. I therefore had to scale back my think times in a big way; otherwise, I would have missed my target of 12 orders per hour per user by 2 orders or so. At 200 SD users, that would have equated to 400 missing orders!
The performance and monitoring data that is derived from a stress test run needs to actually be looked at and analyzed before another test run is performed. My customer and I wasted almost two days performing a series of tests only to discover that more and more of the core transactions were not completing as the user load was increased. Had I looked at the data earlier, it would have been clear that we had a problem (as I had coded an obvious “transaction unsuccessful” message in the error routine used by the scripts; the test’s output files were filled with evidence of this irritating little fact).
In one of my first stress tests, we proved that a customized R/3 Sales and Distribution implementation was poorly coded from an ABAP perspective. In another, it was uncovered that the customer’s reporting system was misconfigured (it defaulted to pulling in all data from all business units by default), bringing the entire system to its knees. Only under load were these problems obvious, though. Both problems were easily rectified (in the latter case, the users were forced to select specific parameters like profit center, cost center, and plant). But uncovering these issues was not the explicit goal of load testing in these cases, thus underscoring the unexpected extra value that can come out of a properly executed stress test.
Some of my favorite stress tests have been in support of my own long-time customers performing upgrades and updates to their in-place SAP production systems. These are usually the least time-consuming to perform, and often revolve around the introduction of new server or disk subsystem platforms, or a fundamentally new database release. I tend to see a lot of activity in this regard when new or substantially updated processors are released, or a new wave in disk subsystem technology drives massive database upgrades, or a new RDBMS is finally made public. And of course, with a new version of a core mySAP component or enabling technology introduced quarterly, I expect to enjoy many additional stress tests in the years ahead!