JMeter is a powerful Java open source load and performance testing tool. With it, you can carry out performance tests on web application, databases, web services, and more.

Load testing (also known as stress testing) involves putting your application under continued stress over a long period of time, generally by simulating many concurrent users. This is a good way of ferreting out memory leaks or performance bottlenecks that otherwise would not appear until once the application is in production. Load testing can also be used to push an application to its limits, simulating expected peak loads, in order to study how it stands up under increasing pressure and to identify weak points in the application architecture.

Performance Testing is a little different. Performance testing involves making sure that your application performs as specified in the system requirements. For example, the requirements may include performance criteria such as the following: with 100 concurrent users, the home page must be displayed in fewer than 2 seconds on a broadband connection.

JMeter can be used very effectively to do both load and performance testing.

Before we start, a word on load testing: as with any other sort of performance testing and optimization, to load test your application well, you really need a plan. How many users is your application supposed to support? Are you writing a company application for limited internal use, or are you working on the new release of Amazon.com? What sort of response times are acceptable, and under what load?

Don’t neglect environmental issues, either. Is the test server (you’re not running your load tests on your production server, are you now?) similar in size and grunt power to the production server? What other applications will be running on the production server? How much memory will be available to your application in production?

These things should be thought through with everyone involved in the project (don’t forget the system guys!), written down for all to see, and used to build at least a basic load-test plan.

Installing JMeter is straightforward. In the tradition of many open source tools, there are no fancy installers. Just download the latest version from the JMeter web site^[14] (version 2.2 at the time of this writing), and extract the package into a directory of your choice. You will find startup scripts for Windows (jmeter.bat) and Unix (jmeter) to start JMeter in the bin subdirectory. You need to start JMeter in this directory. For example, if you had extracted the JMeter 2.3 distribution into the D: oolsjmeter directory on a Windows machine, you could run JMeter as follows:

D:>cd D:	oolsjmeterjakarta-jmeter-2.3in
D:	oolsjmeterjakarta-jmeter-2.3in>jmeter.bat

Or, on a Unix box, with JMeter installed into the /usr/local/jmeter directory, you would do the following:

$cd /usr/local/jmeter/jakarta-jmeter-2.3/bin
$jmeter

Running this command will start up the JMeter graphical console, which is were you write your test scripts (called “Test Plans” in JMeter parlance), run your tests, and view the results.

Depending on the type of tests you intend to do, you may need to provide JMeter with some additional JAR files. The simplest way to do this is to place them in the lib directory, where they will be detected automatically by the JMeter startup script. For example, if you intend to do any JDBC testing, you will need to supply the JDBC drivers, or if you are testing web services, you may need to add the mail.jar and activation.jar files.

Now, we will go though the steps involved in creating a typical test plan for a web application. The test plan will run a simple web application through its paces, simulating the expected load of 100 concurrent users. Our specifications stipulate that with 100 concurrent users, the average response time must be fewer than 2 seconds per page.

Setting Up a Thread Group

JMeter works by simulating a set of concurrent users performing various tasks on your application. In JMeter, you use a thread group to manage this set of users. As the name suggests, a thread group defines a group of threads. Each simulated user is represented by a separate thread. So the number of threads represents the number of simulated simultaneous users generated by this Thread Group. To create a new thread group, just select Edit→Add→Thread Group in the main menu, or place the cursor on the Test Plan entry and use the contextual menu (see Figure 16-1).

Figure 16-1. Creating a new thread group

The most important field in the Thread Group details screen (see Figure 16-2) is the number of users, which you define in the “Number of Threads” field.

To simulate load more realistically, the threads (think “users”) are not started all at once—there is a short delay between starting each thread. This spreads the requests over time and makes for more realistic testing. The total time spent starting the threads is called the Ramp Up Period. So, if you have a thread group with 100 threads and a ramp up period of 3,000 seconds, a new thread will be started every 30 seconds.

Scheduling Your Tests

There are several ways to define how long you want your tests to run. The most simple, and arguably the least useful, is to define the number of times that the test case should be executed, or to simply leave the tests to run forever and stop the test process manually. In practice, it is difficult to estimate how long the test plan will take to run, especially in a concurrent environment. In addition, after running for a long period, JMeter can become sluggish, making it more difficult to stop the test process manually.

The other possibility is to use the Scheduler, which allows you to schedule your load tests for some point in the future. You can specify a start and end time for your tests, useful if you want to run your tests outside working hours, for example. Alternatively, as shown in the illustration, you can specify a fixed duration (the End Time field is ignored in this case). You may also want to schedule the tests to start after a certain delay (say, in an hour’s time), by providing a value in the Startup Delay field. In this case, the Start Time will be ignored.

Figure 16-2. The Thread Group

Setting Up the HTTP Request Configuration Elements

Configuration tests scripts can be repetitive. When testing a web site, for example, all of the HTTP requests will typically be sent to the same server. In JMeter, you can use Configuration Elements to factorize a lot of this repeated data into one central location. If some pages are protected using Basic HTTP Authentication (other forms of authentication are not currently supported), user/password data can be shared. And if your web application uses cookies to manage sessions, JMeter can store these cookies when they are received and make them available to all subsequent requests.

For web application testing, the most important configuration element is “HTTP Request Defaults.” Add one to your test plan by clicking on Test Plan and selecting Add→Config Element→HTTP Request Defaults in the contextual menu. You can specify the name of your test server, the port your web application is running on, the protocol (HTTP or HTTPS), and the application context path. These values will be used by the HTTP Requests that you will set up later on.

Throughout the rest of this chapter, we will be performing some load testing on one of the open source implementations of the Java Pet Store. reference application.^[*] In Figure 16-3, we set up the default data for our test platform, which is running on a local test server called testserver, on the 8080 port. This will be used by default for all the HTTP requests in this test plan.

If you check the “Retrieve All Embedded Resources from HTML Files” option, JMeter will download the images, Javacript files, CSS stylesheets, and so on associated with an HTTP request. However, JMeter does not cache images and downloaded files as a browser does, so it is generally not a good idea to activate this option in the request defaults and only activate it in selected HTTP requests.

Figure 16-3. The HTTP Request Defaults

Another very useful configuration element is the HTTP cookie manager. The cookie manager stores cookies and makes them available for subsequent requests to the same site, as an ordinary web browser would. Because each thread represents a different user, cookies are not shared among threads.

If some or all of your site is protected by basic HTTP authentication, you can provide usernames and passwords in the HTTP authorization manager. Basic HTTP authentication is the simplest of the standard HTTP authentication methods: when a user navigates to a page protected by basic HTTP authentication, the browser will open a dialog box prompting for a username and password. JMeter lets you set up usernames and passwords for one or more URL paths in your application. Other HTTP authentication methods, such as the more secure digest authentication, are not supported. Note that JMeter does support SSL connections, which is considered to be the most reliable way to secure a web application.

Adding an HTTP Request

JMeter tests a web site by sending HTTP requests to the server, simulating the action of real users. These HTTP requests are built using samplers. Samplers are the basic building blocks of all JMeter test plans, and come in a variety of different shapes and colors. In addition to HTTP request samplers, you can also build samplers for FTP requests, web service requests, JMS messages, JDBC queries, and many others. When you are testing a web site, however, the HTTP sampler is by far the most commonly used.

The web application we are testing is an open source implementation of the JPetstore demo application from iBatis (see Figure 16-4). If you want to follow along with the examples, you can download from the IBatis web site.^[*] This is actually of little importance, as you can use JMeter to test just about any web site you like, and the techniques are similar for most sites. However, knowing what we are testing will make it easier to understand the examples.

Figure 16-4. The JPetstore application

Creating a new HTTP Request sampler is straightforward, although it can be a little laborious if there are a lot of parameters. Typically, you would start off with a request for the home page. Although you don’t need to understand all the nitty-gritty architectural details of the application you are testing, you do need to know what the request you want to send is supposed to look like. Usually, the most practical way to do this is to perform the action yourself in a web browser and look at the URL. In this case, the home page URL looks like http://testserver:8080/jpetstore/shop/index.shtml.

This is all we need to set up a JMeter HTTP sampler. Add an HTTP Request sampler to your test plan using Add →Sampler→HTTP Request (see Figure 16-5). Change the name to something meaningful, so that you will be able to identify this request easily in the test plan. You can leave the Web Server details empty; they will be retrieved from the HTTP default values we set up earlier. The only value we need to provide is the web page, independent of the application context. In this case, this would be “/index.shtml.”

As discussed above, you can also choose to retrieve embedded resources such as images, javascript files, and css stylesheets. For a real user, the home page usually has a lot of images and resources that will be fetched here and cached by the browser, so it’s often a good idea to retrieve these the first time you fetch the home page in your test plan.

Figure 16-5. Querying the home page

Next, we’ll build an HTTP request to take us into the Reptiles section of the catalog. The URL for this page looks like http://testserver:8080/jpetstore/shop/viewCategory.shtml?categoryId=REPTILES.

Now we can set up a JMeter HTTP sampler for this query (see Figure 16-6). Add an HTTP Request sampler to your test plan using Add→Sampler→HTTP Request (see Figure 16-6), and give it some appropriate name (“Reptile catalog,” for example). Then provide the Path value, which is the path to this page on the server, without any parameters. In this case, it is the following: /jpetstore/shop/viewCategory.shtml.

Next, we need to add the unique query parameter for this request. Just enter the parameter name (“categoryId”) and the corresponding value (“REPTILES”) in the parameter table. You can add as many parameters as you like.

Figure 16-6. Displaying the Reptile catalog

Using this approach, you can build up a test case query by query.

This approach works well when the queries are simple and the parameters easy to manipulate. Pages that contain large forms may require dozens of query parameters. If HTTP POSTs are being used, the parameter values will not appear in the URL, so you will need to find another way of discovering them. And this approach can be harder to use with some of the more recent frameworks such as JSF, which relies on javascripting to submit queries, and Tapestry, in which the URLs can be long, complex affairs. Another approach is to use the JMeter proxy to record a test case for you, and then tailor it to your needs afterward. We look at this approach in detail in Using the JMeter Proxy to Record a Test Case.

JMeter lets you do a lot more than simply run your HTTP requests sequentially. In JMeter, you can use the various Logic Controller components to organize your samplers in sophisticated ways, which can help you to simulate user activity in a realistic manner. Some of the more useful controllers are described here (see Figure 16-7).

The Simple Controller lets you group samplers or other Logic Controllers together. This is a useful organization tool. In Figure 16-7, the “Browse Catalog” element is a Simple Controller that regroups a logical sequence of user actions.

The Loop Controller is another popular one. As the name would suggest, it lets you loop over any elements it contains a certain number of times. You can loop for a specified number of times, or just loop forever. In Figure 16-7, we loop through this sequence of actions 50 times each time the test case is performed.

The Only Once Logic Controller can be used to define actions that should be run only once, no matter how many times the test case iterates over this element. This is useful for things like login screens, which need to be visited only once, regardless of how long the test case lasts for a particular user. In Figure 16-7, the user signs on to the site only once for each test case, despite the fact that this action is placed within a loop controller.

The Random Controller is useful when you want to simulate random or varied behavior on the part of your users. When you place a set of elements inside a Random Controller, JMeter will randomly select one of the elements each time the controller is invoked. You can use this to simulate a user browsing through different parts of a web site. In Figure 16-7, for example, the Random Controller is used to randomly pick a different details page (Iguana, Snake, Skink, or Turtle) in each iteration.

A useful alternative to the Random Controller is the Interleave Controller. As with the Random Controller, the InterLeave controller will successively choose different contained elements each time it is invoked. Unlike the Random Controller, however, it will invoke each element in the order in which they are listed.

Figure 16-7. Logic Controllers

The other vital part of any test case is the timers. Timer Controllers (see Figure 16-8) let you insert pauses at strategic places in your test case to simulate real user activity. A typical user will load a page, and then spend a second or two reading the resulting page, or at least waiting for it to be displayed. To simulate this, you typically place timers after your requests.

Timers come in different forms. Timers can be of fixed duration (the Constant Timer) or vary randomly within a specified range (the Uniform Random Timer or Gaussian Random Timer). You can also simulate high-load points using the Synchonizing Timer, which blocks threads until a certain number has been blocked and then releases them all at once. This simulates groups of users who pause and then (by intention or by bad luck) all start to work again at the same time.

Placing timers in your test case can be a bit tricky. Timers are executed before each sampler within a given scope. For example, in Figure 16-8, there is a constant timer before the signon action, which will be executed once, just before the signon. The second timer in this illustration is a Uniform Random Timer, which is placed at the start of the Simple Controller. This timer will be executed before every sampler within the Simple Controller. If several timers are present within a particular scope, the delay will be cumulated. For example, if we were to place an additional timer in the Display Details controller, there would be a supplementary delay before each details page (Iguana, Snake, Skink, and Turtle), in addition to the Uniform Random Timer at the top of this scope.

Figure 16-8. A typical Timer Controller

When you run a series of load or performance tests, you generally need to be able to record and display test results. JMeter provides several useful tools allowing you to display your results in a useful form. In JMeter terms, these components are known as listeners. You simply add one or more listeners to your test plan, configuring them as necessary.

You should be aware that listeners can consume a lot of memory. Most listeners keep a copy of every recorded sample, which, over time, will use up all of your available memory. For long-running tests, you should stick to listeners such as “Summary Report” and “Monitor Results,” which use a constant amount of memory.

When you run your load and performance tests, it is a good idea to use a monitoring tool such as jConsole (see Connecting To and Monitoring a Java Application with jConsole) to keep tabs on your server’s activity and performance. Indeed, intensive load tests are a good way to flush out memory leaks, synchronization issues, and deadlocks, as well as other hard-to-isolate problems.

Visualizing Performance with the Graph Listener

One of the most common requirements is to visualize performance results on a graph. You can set this up fairly easily with the Graph Results listener (see Figure 16-9). This graph plots server response times, and also displays average and median response times, standard deviation, and average throughput. Note that the recorded times include any delaying timers you may have added into your test plan to make it closer to the behavior of real users. This allows you to ramp up the number of users and measure throughput and response time under increasing loads in a realistic manner.

Figure 16-9. Performance graph using the JMeter Graph Listener

This graph gives a good, quick indication of how an application performs under strain. However, the JMeter graph listener is not particularly well adapted to displaying performance graphs over a prolonged period of time—after a time, the graph loops on itself and tends to become cluttered and difficult to read. A more efficient way of obtaining performance graphs over a long period is to use the Simple Data Writer listener (see Figure 16-10). This will record sampling data into a CSV or XML file. You can then load this data into your favorite spreadsheet and extract graphs to your heart’s content.

Figure 16-10. Configuring a Simple Data Writer

Getting More Details with the View Results Tree Listener

Sometimes your application may fail unexpectedly during your test runs, or your test script may return a 100 percent error rate for no apparent reason, or maybe you just want to know what a typical web request returns. A good tool to do this is the View Results Tree listener (see Figure 16-11).

This listener gives you a detailed, behind-the-scenes look at what happened when the test case was run. It records the HTTP request sent for each of your requests, the HTML data it received in response, the time it took to receive a response, as well as response codes. You can view the HTML either as text or as rendered HTML form. The HTML rendering is usually pretty sloppy, but it can give you a rough idea of what the page is trying to display. For a better rendering, you can always click on “Download embedded resources,” which will download associated stylesheets and images, although the results are still far from guaranteed.

Figure 16-11. The View Results Tree listener in action

You can also use this listener as a debugging tool. Any requests that returned an error are displayed in red, and you can visualize both the returned HTML (generally an error page of some sort) and the HTTP error code (404 for a missing page, 500 for a server error, and so on). You can also choose to record only the results that contained an error (using the “Log Errors Only” option). This is a very useful option if you need to track down hard-to-reproduce errors, as it is not very demanding in terms of memory, and it can be left on during a long-running performance test.

In addition, if you have activated the “Retrieve All Embedded Resources” option in the HTTP Request sampler, you can view the javascript files, the css stylesheets, the images, and any other resources that have been downloaded as a result of a request.

Needless to say that this listener can be very demanding on memory, so you shouldn’t leave it as part of your normal test case (unless you are using the “Log Errors Only” option). A good technique is to insert the listener into your test case, and deactivate it when you are not using it.

Getting the Executive Summary

Another useful listener is the Summary Report. This report (see Figure 16-12) provides a convenient and memory-efficient dashboard view of test results. The summary table contains a row for each of your queries, containing details such as the number of times the request was executed, the average, the minimum and maximum time taken for the requests, the throughput, the amount of data transmitted, and the percentage of pages that returned an error. This report makes an excellent executive summary for performance test results. It can also help isolate queries that take an abnormally long time to execute.

The Aggregate Report is similar, though with a more statistical orientation. In addition to the data provided by the Summary Report, this report also displays median result times and a “90% line” column. Statistically, 90 percent of requests took less time than this to complete.

Figure 16-12. The Summary Report listener

Building a test case by hand gives you a very high degree of control over the requests you build, but it can be tiresome when complex queries are involved. Complex forms, HTTP POSTs, and javascripted submits can all make it hard to work out what parameters are being sent down the wire. Fortunately, JMeter provides an alternative approach. Using JMeter’s HTTP proxy server, you can run through your test scenario using your habitual browser. JMeter will record the HTTP requests as they go to the server, and build corresponding HTTP samplers. Once you have recorded the HTTP requests you need, you can use them as building blocks to build a fully operational test case.

Using the proxy to record a test script has a number of advantages. For example, embedded resources, such as images, javascript files, and css stylesheets, are downloaded and cached by the browser. The test cases recorded in this way provide a very realistic picture of how a user’s browser will behave.

You can add an HTTP Proxy Server from the WorkBench by selecting “Add→Non-Test Elements→HTTP Proxy Server” in the contextual menu (see Figure 16-13).

Figure 16-13. Adding an HTTP proxy server

This will open the HTTP Proxy Server configuration window, which is where you set up your JMeter proxy. Make sure that the port is not already used on your local machine.

The “Target controller” field indicates the place in your test plan where the recorded elements will be stored. One useful trick is to configure an HTTP Request Defaults element with sensible default values in this element (see Setting Up the HTTP Request Configuration Elements). If any of your default values match the recorded ones (for example, server name and port), they will be left blank in the recorded elements, letting your default values do their job. This makes your test case cleaner and easier to maintain.

By default, the JMeter HTTP Proxy will try to record everything, including HTML pages, javascript files, css stylesheets, images, and so on. This results in an overwhelming number of HTTP Request elements, most of which are not of much use for your test case. Remember: if you need to, you can instruct JMeter to fetch embedded resources for particular HTTP requests. For any real application, you will need to be more selective in what you record.

There are several strategies that can help obtain only the HTTP Requests that you really want. Typically, you only need to include queries to the pages on your site. You can do this fairly easily by adding entries in the “Patterns to Include” list. These patterns take the form of regular expressions. For example, if your site uses JSP files, you could add “.*.jsp” to the list of Patterns to Include. For frameworks like Struts, JSF, or Tapestry, you might need to add patterns such as “.*.jsf” or “.*.do.”

Of course, if your site uses REST-style URLs, this approach may not work. Another possibility is to exclude the files you don’t want to record. To do this, you need to add patterns to the “Patterns to Exclude” list. For example, you may want to exclude GIF images and css stylesheets from the recording process by adding “.*.gif” and “.*.css” to the excluded patterns list.

Now, to record your tests set up your favorite browser to use the JMeter proxy port. Then just connect to your test site and run through a few test scenarios (see Figure 16-14). When you have finished, press “Stop” to stop the recording.

Figure 16-14. Recording a test scenario with JMeter

You will now have an (almost) working, albeit not very flexible, test plan. You can use it as a starting point to build your own, more sophisticated test plan. One of the first things you should do is to add timers between the recorded queries: this will simulate user behavior in a more accurate way. Then you can add more sophisticated structures such as loops and random data selections.

Sometimes it is useful to be able to use real data in your test scripts. For example, we might want to better simulate real user activity by displaying a wide range of different products from the database, with different users displaying different products. The easiest way to do this is to use a CSV-formatted file containing your test data. This file might look like this:

RP-LI-02,Iguana
RP-SN-01,Rattlesnake
K9-BD-01,Bulldog
K9-CW-01,Chihuahua
K9-DL-01,Dalmation
K9-PO-02,Poodle
...

You can import this data into your test plan using the CSV Data Set Configuration element (see Figure 16-15). You define variable names that you can use elsewhere to refer to the data read from this file. The data read from this file will be used to initialize these variables, one row at a time. Each new thread will get a new data row from the CSV file, so the data in the file will be dispatched evenly across all of your test threads.

Figure 16-15. Setting up a CSV data set

Within your test case elements, you can use these variable names to refer to this data. You can refer to these variables anywhere in your test case using the “${...}” notation. In Figure 16-16, for example, we use the ${PRODUCTID} expression to insert a different product ID each time when displaying the viewProduct page.

Figure 16-16. Using a variable from the CSV file

In addition, we can use a Response Assertion element to make sure the server is sending back the right details page. This is more functional testing than performance testing, but it’s always handy to know you are getting coherent responses from your application. In Figure 16-17, we use a Response Assertion to make sure that the returned HTML page contains the name (${PRODUCTNAME}) that we read from the CSV file.

Figure 16-17. Using assertions to verify the response data

For very heavy load testing, involving thousands of simultaneous users or more, you may find that one machine is not enough. Indeed, you can only simulate so many users on the same machine before coming up against CPU and memory constraints. It is more efficient and more realistic to use several machines to simulate large numbers of users. In JMeter, you can do just this. Using JMeter, you can run test cases on a battery of remote test machines via a central JMeter client. The good thing about this approach is that the machines don’t need to be high-powered workhorses—low-end desktops will do fine. Test results from the various machines are saved and displayed centrally.

The first thing that you need to do is to start the JMeter engine on the remote machines manually. JMeter needs to be installed on each remote machine. You start the JMeterEngine instance using the jmeter-server script, as follows:

D:>cd D:	oolsjmeterjakarta-jmeter-2.3in
D:	oolsjmeterjakarta-jmeter-2.3in>jmeter-server.bat

Or, in a Unix environment:

$cd /usr/local/jmeter/jakarta-jmeter-2.3/bin
$jmeter-server

Next you need to tell your main JMeter client which remote servers you will be controlling. You can do this in one of two ways. The first involves adding the hostnames of your remote machines to the remote_hosts variable in the bin/jmeter.properties file, as shown here:

# Remote Hosts - comma delimited
remote_hosts=testhost1,testhost2,testhost3

Alternatively, you can use the -J command-line option when you start JMeter, as shown here:

$jmeter -Jremote_hosts=testhost1,testhost2,testhost3

Now, when you run JMeter you can use the Run menu to manage your remote test machines (see Figure 16-18). You can start (and stop) remote machines either one by one, or all together using the Remote Start All option. The test results coming from the different servers will be aggregated and displayed on your JMeter client interface in the same way as for ordinary tests.

Figure 16-18. Managing remote test machines with JMeter