We have already seen in Chapter 1, Introduction to SOA Testing, how to create a performance testing script using SoapUI so let's move on further from there.
We have four load generation strategies in SoapUI:
- Simple
- Thread
- Variance
- Burst
Let's go through each of them in detail.
Simple strategy: This strategy addresses three types of performance test:
- Baseline test
- Load test
- SOAK test
The simple strategy generates a specific amount of virtual user load on the application with a ramp up time between each thread to give breathing space to the application under test.
Following screenshot shows the configuration window where the setting for the simple strategy is set.
The following window is common for each strategy type but works differently for each load generation strategy:
Let's take a detailed look at the previous screenshot:
- Threads: To specify the amount of virtual user load on the application in the preceding example we have used five threads which means five virtual users.
- Test Delay: The delays you want to keep between each thread, for example 10 threads and 10 sec delay will result in 1 sec wait after each thread.
- Random: If you want to vary the wait time, this feature is used. For example, if you put a random of .5 with a delay of 20 seconds then you will have a random delay between threads of 5 to 20 seconds.
- Limit: This is for time duration of the Test.
Variance: This technique is used usually for two types of performance testing:
- Recovery testing
- Stress testing
This strategy varies the number of threads over time. Set the Interval to the desired value and the Variance to how many the number of threads should decrease and increase.
Let's have a look at the following example.
If we start with Threads of 20, and an Interval of 60 and Variance of 0.8, the number of threads will increase from 20 to 36 within the first 15 seconds, then decrease back to 20 and continue down to four threads after 45 seconds, and finally go back up to the initial value after 60 seconds.
The Variance strategy window has the following options and looks like the following:
Following are the configuration details:
- Threads: The amount of VUser load
- Interval: The time after which the threads would start changing up
- Variance: The variance defines how much the change would be after the specified set of interval
- Thread Strategy: This strategy can be used for the following:
- Identifying the threshold point
- Analyzing a particular bottleneck
- Identifying the load after which functional bugs start to occur
Following is the screenshot which shows the configuration window of the thread strategy:
Following are the configuration values of the window:
- Threads: Specifies the number of users to run this test case
- Start Threads: The number of threads to start the test
- End Threads: The number of threads to end the test
- Limit: The time till which the test will run in seconds
- Burst Strategy: This strategy can be used for the following:
- Recovery testing
- Stress testing
- Bottleneck identification
The burst strategy simulates a scenario where you suddenly have high traffic for some duration on an application. Let's see an example:
In a holiday season, some sites like Amazon give a user more response time than usual because the number of users will be more, similarly a university website at the time of graduation will work much slower than usual as the number of users will peak when the results are announced.
These types of tests are simulated by Burst Strategy. Let's have a look at its configuration window:
- Threads: Specifies the number of threads or VUsers
- Burst Delay: Delay between each burst
- Burst Duration: The duration of the test
We have now gone through all four types of strategy for load generation using SoapUI. For more advance performance testing we can integrate SoapUI with LoadUI, we will see that in the following sections of this chapter.
Now we know how to generate load on any SOA application, let's see how to validate the response with the help of assertions.
Assertions can be added by clicking on the icon in the configuration window for each strategy:
We have the following types of assertions available in the Add Assertion panel in the SoapUI window:
- Max Errors: This assertion allows you to configure the maximum number of errors before a load test fails, following is the configuration window:
- Step Average: This sets the average response time for the whole test. If the value increases, then the value test fails. It should also be noted that this assertion provides a configuration value for setting the minimum requests after which the average is calculated. See the following configuration window:
- Step TPS: This assertion specifies the desired TPS which the test case should attain and fails if TPS is lower than expected. It should also be noted that there are a minimum number of request configuration parameters after which the TPS calculation is done. The snapshot of the configuration window shown following:
- Step Maximum: This assertion validates the maximum value for the corresponding Test Step or Test Case, going over the specified Max Time limit fails the assertion. It should be noted that there is a configuration parameter of minimum requests before which, calculation is done. Following is the snapshot of the configuration window:
- Step Status: Validates whether the underlying test case or test step is executed successfully or not. Following is the configuration window:
This ends the section for assertion we have now gone through all kinds of load testing assertions which are available in SoapUI.