Your load generator might have one or two of the following features, but JMeter is the only one that has them all, touting a feature set robust enough to be shared by QA and developers. It would take many volumes to show you all of this material, so this chapter will showcase just a few of the more important ones in this list.

To be frank about it, I would have ditched JMeter long ago without Andrey Pohilko’s ¹ incredible jmeter-plugins.org, which is a set of additional tools and graphs and things that plug right into JMeter. I suppose I’m a little biased; I contributed a few hundred lines of code, myself.

The project started way back in 2009, so it’s pretty stable. Chapter 6, on the Scalability Yardstick, relied heavily on JMeter Plugins. The CPU consumption was from JMeter Plugins , as well as the thread groups that draw the nice stair-steps for an incremental load plan. Remember when I mentioned the two approaches to installing this? I talked about installation using the Plugin Manager or by unzipping jar files for a single plugin into JMETER_HOME/lib/ext.

Here are two other installation approaches that I think are just as important as the first two:

Please use the first technique (“old style”) to install JMeterPlugins on your own workstation to view the .jmx scripts for this chapter. Those .jmx files are in the sampleJMeterScripts/ folder of this repo:

https://github.com/eostermueller/javaPerformanceTroubleshooting

I refer to this as the jpt repo.

Consider this everyday problem: Let’s say you create a JMeter load script (a .jmx file) that uses a few jmeter-plugins. When your cube neighbor who does not have JMeter-Plugins installed tries to run your .jmx file, they will get cryptic errors that do very little to highlight the missing dependency.

The two install approaches above help with this. Maven is not always easy to use, but the above pom-load.xml is wonderful—it will download, on the fly, any dependent JMeter-Plugins specified as a <dependency> in the pom-load.xml file.

The moral of this story is that sharing your .jmx file, alone, is prone to cryptic dependency issues. Sharing both your .jmx file and your customized pom-load.xml file fixes all of that. But another way to discover JMeter is to use the JMeter Plugins Plugin Manager, one of the install techniques mentioned in Chapter 6. Here is the URL for quick reference:

https://jmeter-plugins.org/wiki/PluginsManager/

If you have the PluginsManager installed (see above), it will prompt you with “Your test plan requires the following plugins” when it discovers a .jmx file that references a JMeter Plugin you don’t have installed. Then, it will give you the option to actually install in or abort.

The load-pom.xml mentioned above does a full JMeter install and also let’s you specify the dependent Plugins in use. Alternatively, you can also use this script to install all plugins in a specific plan file:

This is really helpful for automated, lights-out perf testing, but keep in mind that even the PluginsManagerCMD has its own install process here: https://jmeter-plugins.org/wiki/PluginsManagerAutomated/

In Chapter 2, I showed you how to graph the CPU consumption of individual PIDs. To do that, download and install this plugin—this part enables just the graphs, not the collecting of CPU data:

https://jmeter-plugins.org/wiki/PerfMon/

Then use the ServerAgent to collect data that will display on the graphs: Download ServerAgent-N.N.N.zip from

https://jmeter-plugins.org/wiki/PerfMonAgent/#Download

Unzip it onto any machine (any platform) from which you want to collect metrics. When you start the startAgent.sh (or startAgent.bat), it listens for a connection on port 4444 from the JMeter GUI (or headless) process on your desktop. It is extremely well behaved, cross platform and easy to use.

The StartAgent captures RAM , network and other metrics—not just CPU. There are a number of other JMeterPlugins, detailed next.

In Chapter 3 I talked about how metrics answer three basic questions for us:

Load generators were specifically designed to answer the third question about meeting performance requirements. The jmeter-plugins.org PerfMon component also provides high-level answers to the first question. When I say high-level, I mean CPU, RAM, and network consumption and a little more than that ( https://jmeter-plugins.org/wiki/PerfMonMetrics/ ). Here is how it works.

For starters, no administrative or root permissions are required to run startAgent.bat (Listing 7-1). This process consumes very few resources, so it is safe to launch this process and forget about it for days or months.

Figure 7-1 shows how to add a JMeter graph that connects to the serverAgent and display a CPU graph.

So, JMeter helps answer two out of three of the metric questions. You can even make a case that it answers the remaining question, about which component is to blame for a particular performance problem. How? Check out these other metrics available that you can graph in JMeter.

The JMXMon plugin graphs JMX metrics in JMeter:

https://jmeter-plugins.org/wiki/JMXMon/

It is an unfortunate coincidence that the file extension for the JMeter load plan (.jmx) is the same three letters as the abbreviation for Java Management Extensions (JMX) , a monitoring API. JMX the monitoring API isn’t very popular right now, but Spring Boot’s Actuator is. It delivers great JVM health metrics in json-over-http messages. The following page shows how you can enable Actuator in Spring Boot by adding a simple dependency to your Maven pom.xml Spring Boot file:

http://www.baeldung.com/spring-boot-actuators

Then, you can use the following JMeterPlugin to graph Actuator data in JMeter:

https://jmeter-plugins.org/wiki/PageDataExtractor/

JMX and Actuator data can point to a number of problems inside your JVM, thus helping answer the “which component is to blame” question. So JMeter and JMeterPlugins help provide all three kinds of metrics. Impressive.

From time to time , I discover JMeter properties that tweak JMeter’s default configuration in some particular way. So that these property tweaks don’t get lost in the 45KB (as of JMeter version 2.13) of other properties in JMETER_HOME/bin/jmeter.properties, I segregate them into JMETER_HOME/bin/user.properties, where they override those in jmeter.properties.

Listing 7-2 shows my personal user.properties file with my favorite set of JMeter customizations that I recommend for all.

When you share your .jmx files with your friends, you also might want to share properties like the ones above. But you might instead want to use the pom-load.xml file, above, to specify any properties you want your friends to use. Check out the above pom-load.xml file for the syntax in Listing 7-3, which does the same thing as setting summariser.interval=5 in JMeter’s user.properties file .

You can use the technique in Listing 7-3 to specify any JMeter property. Here is a reference to all of JMeter’s properties:

http://jmeter.apache.org/usermanual/properties_reference.html

When I was starting out with JMeter, I wish I’d had a screenshot like Figure 7-2. An upscale framed copy on your wall is a bit overkill, but how about a photocopy taped to the bottom edge of one of your massive 42 inch monitors? The text I added to the screenshots shows basic load generation functionality, and the arrows point to the JMeter Test Element you will need to implement it. The first screenshot in Figure 7-2 is the blank slate you get when you start JMeter. To set up JMeter as you see in the second screenshot, you right-click on Test Plan and start adding things.

I must concede that the JMeter load plan tree, that tree-like structure on the left side of the main window, is a tough usability hurdle for new users. Users generally know the end result they’re after. For example, “apply 5 threads of load to two different SOA services over HTTP for 5 minutes, need response time and throughput times series graphs.” But to accomplish this seemingly straightforward goal, JMeter forces the uninitiated into an ugly little trial-and-error game that I call “Right-Clicking Education.” Intermediate users navigate this easily, but not beginners. In that left pane of the UI, you assemble a tree structure with funny named building blocks called Test Elements. (Thread Group, Dummy Sampler, View Results Tree, and so on). Each Test Element in the tree has a type, and it has to be precisely positioned according to a set of rules that a beginner knows nothing about. The Test Elements and the types (I work mostly with these types: Thread Group, Sampler, Listener, Assertion) and all of the individual Test Elements are documented here:

http://jmeter.apache.org/usermanual/component_reference.html#introduction

There is some nice high level documentation here:

http://jmeter.apache.org/usermanual/test_plan.html

If you right-click on a particular Test Element, you’ll see what kind of child Test Elements you can legally add from that spot. So if your right-clicker has enough gusto on that day, you can right-click every node on the tree (sometimes there are dozens or hundreds of blocks) and deduce and commit to memory the whole set of rules—which Test Element types can/should be the parents/children of other Test Element types. That’s a lot of right-clicking fun. Or not.

My point here is not to dish out a black eye to “the one I love.” Instead, it is to encourage you to avoid the pain of the Right-clicking Education. At first, start by relying on the help of others to structure your load plan tree. Specifically, you can use the screenshot in Figure 7-2 as a guide, or perhaps use the relatively new JMeter Template feature to start with a ready-made load plan tree, or just have a friend share their load plan (a .jmx² file of a proprietary .xml grammar). To use the Template feature, start JMeter with an empty load plan tree, and then choose File ➤ Templates. Even with a lot of experience, I still rely on the Record template to record a new HTTP load script, because the tree structure is a little unique. You will soon become comfortable with which Test Elements go where, and you can then branch out (ha) and get more creative with your load plan tree structure .

The quiverful of big red arrows in Figure 7-2 all point to individual items, the Test Elements. Navigating, nay surfing, around all of those Test Elements is a breeze by simply using the arrow keys on your keyboard. Each Test Element has a particular Test Element type. All graphs, for example, are of the Test Element type Listener. Test Elements for submitting HTTP requests are called Samplers . To make them easy to find, I stick all my graphs at the highest level (as children of Test Plan, the root). But let’s say I have inadvertently placed some graphs further down into the tree. There are a number of features you could use to quickly relocate them, and do other things as well:

All of this cut/copy/paste functionality is great for:

The .jmx script for this example is in the jpt github.com repo in this location:

In Chapter 5, I mentioned a half dozen situations where you should bite the bullet and discard the results of a test and rerun the entire thing. One of those potential reasons was lots of SUT errors. You’ll have to decide for yourself exactly what percentage of errors triggers a redo.

But if your SUT fails silently or important client data is quietly omitted from an SUT web page, you might end up basing all kinds of decisions on really faulty data. To avoid that situation, please take the time to tell JMeter what constitutes a successful HTML response by adding JMeter Assertions to check the HTML responses for key pieces of text, like the title of a web page, or perhaps HTML text labels for a few key fields. Figure 7-3 shows how to add an Assertion.

Figure 7-4 shows how to enter the text to be verified. Before entering the text, you’ll need to first click the ‘Add’ button—bottom, center. If JMeter finds that the text you enter is missing from the corresponding HTTP response, it will tally an error. If your entire business process culminates in an “Order Submitted” message, this is your chance to validate that the system coughs up this all-important message. Don’t miss this chance.

It would be bittersweet, to say the least, if your application displayed “Order Submitted” immediately followed by “Fatal Exception / Your code failed.” You can add a second Assertion as shown in Figure 7-5 that flips the logic. Just mark the ‘Not’ flag and JMeter will tally an error if the given text, like ‘Fatal Exception’ is present.

I have prematurely claimed performance victory before, only to later discover that 100% of the requests failed. Its pretty embarrassing; I don't recommend it.

Not only should you take the time to add Assertions, you also need to watch for error counts/percentages, where all Assertion failures and other errors (like HTTP errors) are recorded. Here are three components (there are probably others) that will help you track errors:

The Assertion discussed here (Figures 7-4 and 7-5) only checks for preordained list of strings. There are a number of other more sophisticated Assertions detailed here: http://jmeter.apache.org/usermanual/component_reference.html#assertions

For example, the JSR223 Assertion lets you write code to validate any Sampler response.

Figure 7-6 shows a “Synthesis Report” with failed Assertions. The Synthesis Report from JMeter Plugins performs much better JMeter’s “Summary Report”, which lacks a few key columns like “like 90% response time.” The Sumary Report has less functionality and worse performance.

Almost every script I create has a few script variables, making it easier to run the script in different hardware environments. The MY_DIR variable shown in Figure 7-7 will be used in the following examples to indicate where to store JMeter output files on the machine where JMeter is running.

Figure 7-8 shows how to reference a variable. Practically all input boxes in JMeter, for either numeric or text data, like the one displayed here, will correctly resolve the value of a script variable—just don’t forget to wrap the variable name in the ${} syntax.

You can save/persist JMeter results (response time, throughput, and so on) to disk in a few different ways. You will need this for many reasons, like comparing current performance to that of the last release, documenting performance defects, or showing throughput process to your peeps.

To save JMeter UI graphs to .png files, start by adding a graph to the test plan tree, as shown in Figure 7-9. For a nice “response time over time” graph, right-click on the root test plan and choose Add ➤ Listener ➤ jp@gc - Response Times Over Time. After you run the test, you can just right-click to copy the image of the graph to the Clipboard or to the file system.

The second way to save your test results is to have JMeter save the raw data to the disk. The .jtl file extension is a nice convention, although it is not enforced. For saving data to .jtl files, you can use the same graph as Figure 7-9 by specifying it as shown in Figure 7-10. All of the graphs are Test Elements with the type Listener, and they each have this panel.

Yes, the screenshots in Figures 7-8 and 7-10 are identical, but only because this is important functionality. Even though the UI plainly says, “Write results to file / Read from file,” most people, including me, have trouble seeing that this box is used to render a graph from a .jtl file.

Note that Run ➤ Clear All (Ctrl+E) will clear out all the results in graphs and most Test Elements in the UI, but it will neither delete nor zero out the .jtl results file. It appends data. You have to manually delete the .jtl file to start from scratch. This can be a pain when you start what you thought was a new test, but you were really appending data to the test you ran yesterday at the same time of day. If you try to render the data in the .jtl to a graph, say a throughput graph, you will see throughput from yesterday’s test on the left, a huge, vacuous 24 hour gap in the middle, followed by today’s throughput on the right. Not what you expected. So, remember to clear out your .jtl files when restarting to avoid this huge gap.

To avoid playing “file system hide-and-seek” to find your output files, I strongly recommend specifying a full path with the Filename input box; Figure 7-7 shown earlier shows how to use a script variable to say which folder the file will land in.

That shows how to write the .jtl files. To read a .jtl file and display the graph, for example when you restart the JMeter UI, you need to load each Listener Test Element one at a time. There is an interesting usability quirk here to get JMeter to read the .jtl file: make sure the .jtl file name is in place. Then put or keep focus on the input box, and press the Enter key. That starts the process to paint the graph from text data in the .jtl file. Alternatively, using the Browse button to select the file will paint the graph based on the .jtl data.

These instructions show how to record an HTTP script:

http://jmeter.apache.org/usermanual/jmeter_proxy_step_by_step.pdf

Rather than duplicate the instructions, I think it is important to highlight a few key steps in the recording process.

Configuring a .jmx file just right for recording used to be very difficult. Now, there is an easy way to create it: start with .jmx configuration from a JMeter Template.

Choose the File ➤ Templates menu option. Then choose the Recording template in the combo box at the top center. Finally, click Create, as shown in Figure 7-14.

Once the Recording template is selected, the JMeter plan tree should look like Figure 7-15.

Figure 7-16 details a critical part of the process: your browser’s HTTP proxy configuration. With no HTTP proxy configured, your browser makes requests directly to the Internet. With a proxy configured, all requests are forwarded to a specific host (labeled as Address) and port specified as shown.

Note that the IE Address input box on the left must point to the local machine and the port number in IE (left) and the port number on the JMeter panel (right) must match. The value in Figure 7-16 is 8888. Also note the JMeter Start button is disabled, meaning that it has already been clicked to start the JMeter HTTP proxy program.

When the IE proxy is correctly configured and you click the big Start button on the HTTP(S) Test Script Recorder, all browser network traffic will be forwarded to the JMeter HTTP proxy, and JMeter proxy will immediately forward all traffic to the SUT. JMeter will record all traffic as you navigate the SUT.

Paste your SUT’s URL into the browser and start navigating the business processes you want to record. When finished navigating, log out of the SUT and click Stop on the JMeter’s HTTP(S) JMeter Test Script Recorder on the right of Figure 7-16.

JMeter will display all the recorded traffic in the Recording Controller, as shown in Figure 7-17.

Working with XML Schema validators can be a pain. The validation errors are so cryptic, so not-quite-English. It reminds me of being on a technical support call with the dunce of the English-as-a-second-language (ESL) class.³ But the only thing worse than working with a tightly governed XML Schema is a system without any schema at all, where there are no good rules for right/wrong input.

This is exactly like the load generator’s task of properly assembling the HTTP GET to pass to a web application. What URL parameters are required on the HTTP GET? What kind of web authentication is used? Are CSRF tokens or JSESSIONID required? Any other cookies? What HTML “hidden” variables are pulled from the previous HTML response by the SUT’s browser code and used to build the HTTP GET parameters for a subsequent page? All this uncertainty makes me pine for some kind of tightly governed schema that would plainly spell out the requirements . In fact, answering any of these questions wrong generally results in an incomplete HTML response and/or an exception on the server side, and getting this kind of an error is a likely sign that your .jmx load script is broken—perhaps out of sync with SUT requirements.

One way of dealing with this uncertainty is to keep a backup of the verbose HTTP log of the initial recording, along with the initially recorded load script itself. This creates a tidy little vault of all the required data items; it isn’t a schema, but it is close enough. But then load script maintenance happens. Bugs are easily introduced later on when enhancing the script as detailed in Chapter 4. Remember? These are the changes required to make your load resemble the actual production load.

When a load script bug shows, you can find whether you have all the required data items by performing a detailed text file “diff” between the original verbose HTTP log and the one from the test run with the bug. The following JMeter instructions show how to capture those verbose HTTP logs.

When you look at the differences between those logs, you most often find that you did not fully understand the HTTP GET requirements (two paragraphs above). Alternatively, perhaps you have deployed a different version of the SUT with different requirements for an HTTP GET.

To repeat, the “diff” you will need to perform is between two different logs: one of the HTTP recording of the business process and the other from playing back the script. I use the JMeter View Results Tree to create these logs. The one for the recording must be under the Workbench (see Figure 7-18). The one for the playback must be configured to record in XML format like the first one, but instead it must be an immediate child of Test Plan, with a different output file name (obviously). You can use the JMeter copy/paste to copy the one created in the Workbench (as in the following instructions) to the Test Plan.

To capture that very detailed log of the HTTP recording, do the following:

Once upon a time I was load-testing the reporting section of a loan system. When a user requested a small 1K report, it had to be extracted from specific byte offset into a single multi-megabyte file that was periodically downloaded from a back-end system. Response time under load for this component got as slow as 5 seconds; server-side concurrency was about five threads.

I knew how to launch my code five separate threads, but I thought collecting the results from those separate threads would take too much time to code, and then I’d have to graph the data somehow. So instead of launching the five threads myself, I had JMeter do it for me.

http://openjdk.java.net/projects/code-tools/jmh/

The following shows how I got JMeter to launch five threads, all running my RandomAccessFile implementation of the reporting code; it took about 15 minutes to figure it all out. I actually ended up with JMeter graphs that compared the performance of both implementations, and the RandomAccessFile approach was tons faster—about 100-400ms for 5 threads of load, zero think time.

The JMeter .jmx file for this section and the next section is located in the jpt repo in this location:

Are you a little hesitant about mastering the art of load generation? You should be—it takes work. But fortunately, learning JMeter is easier than other load generators, because you can test just about every Test Element in its own little sandbox without the complexity of your SUT. So turn off the Wi-Fi (don’t do that) and let’s get started:

With this little sandbox technique, you can:

Once we cover items 1 and 2 from this list of sandbox tests, you will be primed and ready to play (yes play) with 3, 4, and 5 for homework. It is just plain fun. Here is how the sandbox works: close out any script you’re working on and add a Thread Group and a child jp@gc - Dummy Sampler ( https://jmeter-plugins.org/wiki/DummySampler/ ). Here are the steps:

This will leave you with the screen shown in Figure 7-19.

The JMeter .jmx file for this section is the same file used in the previous section:

With the sandbox foundation in place, just add one of the dozens of Test Elements you want to play with! In Figure 7-20, I added jp@gc Response Times Over Time ( https://jmeter-plugins.org/wiki/ResponseTimesOverTime/ )and started the test (Ctrl+R or Cmd+R) and voila, live-looking graphs. Use Ctrl+Comma or Cmd+Comma to stop.

This section shows how to put two or more metrics on a single JMeterPlugins graph.

To see this in action yourself, look in the jpt repository for this .jmx file. Open it in JMeter:

Of course taking guesses, educated ones, is a key part of performance troubleshooting. But if a guess cannot be supported by data, we all need to be big enough to fall on our swords and verbally disown the unproven assertion. In fact, in Chapter 3 I called metrics the antidote to guesswork. So, I think developers need to hone their skills a little bit at finding, collecting and efficiently presenting performance data so we can substantiate more of our guesses or cleanly divorce them.

To see the impact of one metric on another, I think it’s important to put two different metrics on the exact same graph. In Chapter 6, on the Scalability Yardstick, I showed you one example of that. Remember that we had to check to see how many thread of load were required to push the SUT CPU to 25%? The two metrics were CPU and count of JMeter threads of load. In this case, the increase in threads of load impacted two other metrics: both CPU and throughput were impacted—they both increased.

Actually, my favorite four metrics for a single start are CPU, response time, throughput, and load generator thread count.

To help you get up-to-speed on displaying graphs like this yourself, Figure 7-21 shows my test plan, still using the jp@gc - Dummy Sampler sandbox, to put two metrics on the same graph: jp@gc - Response Times Over Time and jp@gc - Transactions per Second .

Once you have added them to your load plan tree as shown in Figure 7-21, you need to tell the new jp@gc - Composite Graph which metrics to display. So I selected the jp@gc - Composite Graph and clicked the Graphs tab. Whoops, but there was nothing there, as shown in Figure 7-22.

To fix this little problem , you just need to run the test for a few seconds and stop the test. This will populate the Available Sources box so you can select the metrics you want to see (Figure 7-23).

Once the Available Sources box is populated, you can double-click the metrics you want to move to the Composed Graph box .

The JMeter .jmx files for the next two examples are located in the jpt repo in this location:

In Chapter 4, I showed a number of circumstances where the load generator script has to step in and do small data movement chores that in a live situation are done by your SUT’s JavaScript in the browser. But in that chapter, I tried to show everything in a load-generator–agnostic way. This section jumps into the JMeter specifics on how to implement this.

Sometimes with the right SUT functionality, a unique ID generated on the server SUT gets recorded into your load script. Adding the right script variable, one that I’m calling an “Output-Input” variable, will retrieve the freshly generated value, instead of relying on the single, static value that was initially recorded into the script. That way, the generated ID can be submitted somehow (you have to figure that part out) in a subsequent request, perhaps as a query parameter or somewhere in the body of an HTTP post.

But before sticking a generated ID like this into a script variable, your JMeter script first has to find that particular ID from its location embedded hopefully a predictable place in lots of text in the output HTML stream.

To conclude this chapter, I’ll present a to-do list of things you should know how to do:

Load gen features:

Graphing features:

Must know:

This chapter provided a tour of some JMeter functionality that is rarely if ever documented, and other functionality that is documented but still causes trouble for the many technicians that I have taught JMeter to.

Don’t forget that if you can’t figure out how to get some JMeter feature to work, create a little “Hello World” sandbox-like .jmx script to experiment with the feature. Copy and paste a snippet of HTML (or json or XML or whatever) into the Dummy Sampler so that your test plan can operate on it. Just about every single JMeter Test Element can be tested like this. You can experiment with logging, load plans, Correlation Variables, JMeter Assertions, Composite Graphs (multiple metrics on the same graph), you name it.

The next chapter provides an overview of the two sample applications that come with this book. These applications reproduce the write-once-run-anywhere (WORA) performance defects that are detectable in both large environments and small ones. These examples will give you hands-on experiences hooking up plug-it-in-now performance and observability tools, where available, that will help you find and fix performance defects quickly.