JRockit, as a Java runtime, is required to constantly monitor the running Java application. As has been explained in previous chapters, JRockit must, among other things, be able to find out in which methods the Java application is spending the most time executing code. JRockit is also responsible for keeping track of the memory usage and memory allocation behavior of the application—it would be fairly upsetting if JRockit forgot a few objects, or if memory was not reclaimed when objects were no longer referenced.
The wealth of data that JRockit already collects is, of course, a very good source of information when profiling your Java program or when doing diagnostics.
In this, the second part of the book, the JRockit tools suite is presented. The following chapters introduce four of the tools that are included in the JRockit distribution—the JRockit Mission Control Console, the JRockit Runtime Analyzer (which was superseded by the JRockit Flight Recorder in R28), the JRockit Memory Leak Detector, and JRCMD.
The first three tools are included in the JRockit Mission Control tools suite. The last one, JRCMD, is a command-line tool that ships as part of the JRockit JDK. A common denominator for the tools is their ability to interact with an already running JVM. There is no need to pre-configure the JVM or the application to be able to later attach the tools. Also, the tools add virtually no overhead, making them suitable for use in live production environments.
In this chapter you will learn:
- How to start JRockit Mission Control, both as a standalone application and inside the Eclipse IDE
- How to configure JRockit JVM instances so that they can be managed by JRockit Mission Control from remote locations
- How to make JRockit Mission Control automatically discover remotely running JRockit instances
- How to configure the management agent that is part of the JRockit JVM
- How to use JRockit Mission Control and the JRockit Management Agent in a secure environment
- How to troubleshoot connections between JRockit Mission Control and JRockit
- Various ways to get more debug information from JRockit Mission Control
- About the Experimental Update Site for JRockit Mission Control and how to extend JRockit Mission Control with plug-ins from the Experimental Update Site
While not necessary for the bigger picture, parts of this chapter assume various degrees of familiarity with Eclipse IDE. For more information about Eclipse, see the Eclipse homepage at http://www.eclipse.org.
JRockit Mission Control started out as a set of internal tools used by the JRockit team to monitor and tune the JRockit JVM in order to make it a better Java runtime. The early analysis tools were initially not available to end customers. However, after we used the tools to resolve a couple of high profile customer issues, the word started to spread. We realized that customers found the tools useful when tuning and profiling their Java applications, and consequently the tools were made more user friendly, packaged together, and released as the Java tools suite known as JRockit Mission Control.
Today, the JRockit Mission Control tools suite is a set of tools for monitoring, managing and profiling Java applications running on JRockit. It also includes a powerful tool for tracking down memory leaks. The JRockit Mission Control tools can, with very little overhead, retrieve a profile that is a fair approximation of the actual runtime behavior of an application. Most other profilers cause significant slowdowns which, in turn, can alter the behavior of the running application. As mentioned in Chapter 5, Benchmarking and Tuning, if a lot of overhead is incurred from the actual profiler, it is fair to say that what is being observed is no longer a true representation of the application behavior. It is rather the behavior of the application, plus that of the profiler itself, that is being profiled.
Note
Behavior change in an application due to profiling is sometimes referred to as the observer effect—when the act of observation alters what is being observed. The term Heisenberg effect is also sometimes used. The Heisenberg Uncertainty Principle is related to, and often conflated with, the observer effect.
In a BEA internal study, before the Oracle acquisition, the BEA performance team was benchmarking different profilers. They were looking for a low overhead tool to profile a J2EE benchmark on WebLogic Server. Several different profilers were examined, and the overhead as a function of change in response time was calculated. For this particular benchmark, the overhead for Mission Control was 0.5 percent. The second best tool, a leading Java profiler, incurred 93.8 percent overhead on that very same benchmark.
Naturally, different tools have different strengths and weaknesses. The data captured by JRockit Mission Control aims to provide a statistically accurate representation of what is going on in the JRockit JVM. It will not always be exact, but it will usually provide the information required to solve important problems. We call this sampling-based profiling. Sampling-based profiling lends itself well to when a statistical approach can be used to periodically record a state. The most common variants in JRockit Mission Control are time-based sampling and sampling based on a subset of state changes. The JVM, being a large state machine, can cheaply provide a large amount of samples and events. The data can be readily exposed by the JVM to a consumer, for example Mission Control. Sampling-based profiling has the additional benefit that it is much easier to estimate the profiling overhead.
For example, the best way to find out where an application is spending the most execution time is to use the JRockit Flight Recorder tool to look at the hot methods list. The list provides a statistically representative view of what methods JRockit spent the most time executing, using information from the code profiling thread in the JRockit code generator. It will, however, not provide information about every single method call or exactly how much wall clock time was spent executing the method.
There are some exact profiling alternatives in JRockit Mission Control, as well, that can provide such metrics. These, however, may incur a much larger overhead if enabled. It is, for instance, possible (but please don't try this at home) to connect the JRockit Management Console to a running application and enable exact timing and invocation counters for each and every method in the system. Doing exact profiling always incurs extra runtime overhead. Enabling exact profiling for every method in the system will require the JVM to both generate and execute a large amount of extra profiling code. This will not only adversely affect performance, but it is also quite difficult to determine the exact performance implications. Some of the JRockit code is written in Java. Instrumenting all allocation and locking code would almost certainly bring the application to its knees. If the aim is to determine what parts of the application would benefit most from improvements and optimization, the sampling-based approach is superior.
One might think that the cost of overhead for exact profiling is paid for in better measurements. For example, given exact data, the timing values for all the methods in the system can then be measured, and the one with the highest invocation count multiplied by execution time would be the natural one to target for optimization first, wouldn't it?
It might, but in a sufficiently complex system the data will be misleading and distorted. The overhead of the methods on the critical paths in the system will quite likely increase drastically and the overall system performance will degrade. Furthermore, applying the exact method profiler to every single method in the system would most likely severely change the behavior of the application, rendering the measurements inaccurate.
Note
Using the JRockit Mission Control Management Console to retrieve exact method timings and invocation counts is one of the rare cases where the overhead of running the JRockit tools suite is very difficult to estimate. The time it takes to calculate the profiling data can be considered to be constant. Invocation counting code is mainly placed at method entries and method exits. So, the distortion caused by the profiling is proportional to how often the method is invoked and inversely proportional to the time the method takes to execute. If the method is frequently called, the overhead will add up. If the method already takes a long time to execute, the overhead may be less noticeable. In multi-threaded scenarios, the non-determinism of exact profiling with the Management Console is even greater.
JRockit Mission Control is used in very different ways by different people. Outside Oracle, it is most commonly used as a very capable profiling tool.
Some people use JRockit Mission Control to track down application problems. This is the main use case for Oracle Support. The authors also find it enlightening to turn JRockit Mission Control on itself during the development phase to find both performance issues and bugs.
There are also the ultra nerds, the authors included, who use the JRockit Mission Control tools suite to squeeze out that extra percentage of performance from JRockit by doing benchmarking and by tuning the JVM for a very specific application.
Here are a few typical JRockit Mission Control use cases:
- Finding hotspots: Where should we start optimizing an application? Is the application even computationally bound or is it experiencing latency related issues? Which method is being executed the most and would thus be the most important to improve for better performance?
- Tracking latencies: Finding out why application throughput is worse than expected or why response times are too long. In applications with latency-related throughput problems, the CPU is typically not saturated. Over-reliance on synchronization may cause thread stalls. Why are all the worker threads blocking most of the time? Why is the system starting to time out on requests while utilizing only 20 percent of the CPU?
- Memory profiling: Useful when trying to find out what is causing all those garbage collections. Where is the pressure on the memory system created? What part of the program is causing all those garbage collections? How much time is spent doing garbage collections? What phase of a garbage collection is making them take so long? How badly fragmented is the heap?
- Exception profiling: Throwing and handling a lot of unnecessary exceptions can be taxing on a system. How many exceptions are being thrown? Where do they originate? Tracking down and eliminating the sources of unnecessary exceptions often leads to significant increases in runtime performance.
- Heap analysis: Finding out what type of data is on the heap in various phases of a running program can provide insight on how to choose, for example, garbage collector strategies or compaction strategies. It can also identify and help get rid of situations where too many
Referenceobjects are used."George, why does your ticker quote cache contain a
HashMapholding on to 96 percent of the heap? It is still growing by the way, and I expect the server will throw anOutOfMemoryErrorin two hours at this rate. We need to restart our server twice a week. Otherwise we will get anOutOfMemoryError. But that's okay. Everyone does that..." - Tuning the memory system: The out of the box behavior for the memory system is usually good enough, but some extra performance can usually be gained by tuning it a little. For some corner case applications, tuning may even be necessary to get the required performance. In those cases, JRockit Mission Control can quickly reveal why the garbage collector fails to keep up.