4. The Serviceability Agent
Debugging and troubleshooting tools play an important role in the development of well-performing Java applications. There are many tools available to debug applications at the Java level, but very few are available for troubleshooting problems in Java applications at the JVM level. This chapter talks in detail about a powerful set of debugging tools called the Serviceability Agent (SA) that can help in debugging Java applications at the Java as well as the JVM level. The SA has the ability to debug live Java processes as well as core files, also called crash dump files.
In this chapter, we will take a look at what the Serviceability Agent is, where we can get it, and what we can do with its set of tools.
For building high-performance Java applications, it is a must that those applications be free from various problems that an application can encounter. Those problems could be application hangs, memory leaks, unexpected application behavior, and crashes. The Serviceability Agent’s set of tools can greatly help in troubleshooting and nailing down those problems, resulting in stable, reliable, scalable, and efficient Java applications.
The Serviceability Agent is very useful in debugging core files or crash dump files. As we all know, core or crash dump files contain the complete state of a program at a specific time such as the state of memory, processor registers, stack information, and other processor and operating system flag information. The Serviceability Agent has the capability to read the core files of the Java HotSpot Virtual Machine (HotSpot VM) and present the details from it in a human-readable format.
What Is the Serviceability Agent?
The HotSpot Serviceability Agent is a hidden treasure present in the JDK that very few people know about. It is a set of Java APIs and tools that can be used to debug live Java processes and core files (aka crash dumps on Windows). The SA can examine Java processes or core files, which makes it suitable for debugging Java programs as well as the HotSpot VM. It is a snapshot debugger that lets us look at the state of a frozen Java process or a core file. When the SA is attached to a Java process, it stops the process at that point so that we can explore the Java heap, look at the threads that were running in the process at that point, examine internal data structures of the HotSpot VM, and look at the loaded classes, compiled code of methods, and so on. The process resumes after the SA is detached from it.
As mentioned before, the SA is a snapshot debugger, which means that it works with the snapshot of a process, unlike the usual debuggers that provide the facility to step through a running program.
Note that the SA runs in a separate process from the target process and executes no code in the target process. However, the target process is halted while the SA observes it.
Why Do We Need the SA?
Why use the SA when we have native debugging tools like dbx, gdb, WinDbg, and many others available on all the platforms?
First, the SA is a Java-based platform-independent tool, so it can be used to debug Java processes/cores on all the platforms where Java is supported. Additionally, debugging a Java process or the Java HotSpot Virtual Machine with native debuggers is very limiting as they can help us examine the native OS process state but not the Java or the JVM state of the process. For example, if we need to view the objects in the Java heap, native debuggers would show us the raw hex numbers, whereas the SA has the ability to interpret those hex numbers and present the object view instead. The SA has the knowledge about the Java heap, its boundaries, objects in the Java heap, loaded classes, thread objects, and internal data structures of the Java HotSpot Virtual Machine. Empowered with this knowledge, the SA makes it very easy for us to examine the Java/JVM-level details of the Java process or core.
SA Components
The SA consists mostly of Java classes, but it also contains a small amount of native code to read raw bits from the processes and the core files:
On Solaris, SA uses libproc to read bits from a process or a core file.
On Linux, SA uses a mix of /proc and ptrace (mostly the latter) to read bits from a process. For core files, SA parses Executable and Linkable Format (ELF) files directly.
On Windows, SA uses the Windows Debugger Engine interface (dbgeng.dll library) to read the raw bits from the processes and core files.
SA Binaries in the JDK
There are two binaries present in the JDK that make up the SA. These binaries are shipped with the JDK:
sa-jdi.jar
sawindbg.dll on Windows, libsaproc.so on Solaris/Linux
SA components are built as part of the standard build of the HotSpot repository. The native code component of SA is placed in libsaproc.so or sawindbg.dll, and the Java classes are placed in sa-jdi.jar.
The binary sa-jdi.jar provides the SA Java APIs and also includes useful debugging tools implemented using these APIs. It also includes an implementation of the Java Debug Interface (JDI), which allows JDI clients to do read-only debugging on core files and hung processes.
JDK Versions with Complete SA Binaries
SA binaries are shipped with the following JDK releases:
JDK 7, and later releases on all platforms
JDK 6u17+ on Solaris and Linux
JDK 6u31+ on Windows
Prior to these versions, the SA was not shipped with the JDK on Windows, and only a subset of SA classes was shipped with JDKs on Solaris and Linux. The JDK versions listed make the complete set of SA classes available on all of these platforms.
How the SA Understands HotSpot VM Data Structures
There is a file src/share/vm/runtime/vmStructs.cpp in the HotSpot source base that defines a VMStructs class that contains declarations of each HotSpot class and its fields, as well as the declarations of the processor-dependent items such as registers, sizeof types, and so on. The SA understands the Java objects and HotSpot data structures with the help of the declarations in this file.
As an example, in the file src/share/vm/oops/cpCacheOop.hpp we have the following:
:
class constantPoolCacheOopDesc: public arrayOopDesc {
friend class VMStructs;
private:
// the corresponding constant pool
constantPoolOop _constant_pool;
:
In vmStructs.cpp, the _constant_pool field is declared as
nonstatic_field(constantPoolCacheOopDesc, _constant_pool,
constantPoolOop)
From the _constant_pool field in the file vmStructs.cpp, the SA knows there is a class named constantPoolCacheOopDesc that has a field with the name _constant_pool of type constantPoolOop in the Java HotSpot VM.
Note that VMStructs is declared as a friend class. Most of the classes in HotSpot declare VMStructs to be a friend so that the private fields of that class can be accessed in VMStructs.
During the HotSpot build, vmStructs.cpp is compiled into vmStructs.o, which is included in the shared library libjvm.so or jvm.dll. vmStructs.o contains all the data that the SA needs to read the HotSpot data internal representations. And at runtime, the SA can read this data from the target VM.
The structure and field names declared in vmStructs.cpp are used by the corresponding Java code in the SA. Thus, if a field named in vmStructs.cpp is deleted or renamed, the corresponding Java code that accesses that field also needs to be modified. If declarations in VMStructs and the Java code in SA are not in sync, SA will fail when it tries to examine a process/core file.
SA Version Matching
As we saw in the previous section, the Java code in SA is a mirror of the C++ code in HotSpot. If some data structures or algorithms are changed, added, or removed in HotSpot, the same changes have to be made in the SA Java code. Due to this tight coupling between the SA Java code and the HotSpot implementation, an SA instance can reliably debug only the HotSpot VM that was built from the same repository as that of the SA instance. In order to detect the version mismatch between the SA and the target HotSpot, we place a file named sa.properties into sa-jdi.jar during the HotSpot build process. This file contains an SA version property, for example,
sun.jvm.hotspot.runtime.VM.saBuildVersion=24.0-b56
At runtime, SA reads this property, compares it with the version of the target HotSpot VM being analyzed, and throws a VMVersionMismatchException if the versions do not match. This check can be disabled by running the SA tools with the following system property:
-Dsun.jvm.hotspot.runtime.VM.disableVersionCheck
With this option, SA does not complain if its version does not match the version of the target VM and attempts to attach to it. This option is useful if you want to attach the SA to a non-matching version of the target HotSpot VM.
The Serviceability Agent Debugging Tools
There are two main SA debugging tools that are implemented using the Serviceability Agent APIs: HotSpot Debugger (HSDB) and Command-Line HotSpot Debugger (CLHSDB). HSDB is the top-level GUI program offering many visual utilities that help in debugging the HotSpot VM. CLHSDB is the command-line variant of HSDB. When debugging remote core dumps, it is much easier to work with the command-line HSDB than with the GUI HSDB tool.
HSDB
HotSpot Debugger is the main GUI tool. It facilitates examining a Java process, core file, and also a remote Java process. Let’s see how we can launch and use it on a Windows machine.
First, let’s set the JAVA_HOME environment variable to the folder where the JDK that we want to use is installed so that we can use this variable wherever we need to access files/folders in that JDK:
set JAVA_HOME=d:Java7u40
On Windows, the PATH environment variable should contain the location of the JVM binary used by the target process/core and also the folder where Microsoft Debugging Tools for Windows is installed on the machine, for example:
set PATH=%JAVA_HOME%inserver;d:windbg;%PATH%
Now we can launch HSDB using the following command:
%JAVA_HOME%injava –classpath%JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.HSDB
On Solaris and Linux, we just need to set JAVA_HOME to point to the installed JDK and then launch the tool as in the following:
$JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.
HSDB
These launch commands bring up the HSDB GUI tool as shown in Figure 4.1.
Beginning with JDK 9, there are two different ways to launch HSDB. For instance, here is an example of how to launch the SA with JDK 9:
set PATH=d:Javajdk9-b102in;%PATH%
java sun.jvm.hotspot.HSDB
This brings up HSDB.
JDK 9 also contains a jhsdb executable in JDK 9’s bin directory that can be used to launch various SA tools. For example:
set PATH=d:Javajdk9-b102in;%PATH%
jhsdb.exe
clhsdb command line debugger
hsdb ui debugger
jstack --help to get more information
jmap --help to get more information
jinfo --help to get more information
jhsdb.exe with no arguments offers help by showing additional arguments that can be passed to jhsdb to launch various SA tools. For example:
jhsdb.exe hsdb
This brings up the HSDB GUI debugger.
The following will start the command-line debugger:
jhsdb.exe clhsdb
hsdb>
And the following shows how to start the jstack tool with additional options that can be passed to the jstack tool:
jhsdb.exe jstack
--locks to print java.util.concurrent locks
--mixed to print both java and native frames (mixed mode)
--exe executable image name
--core path to coredump
--pid pid of process to attach
Serviceability Agent Debugging Modes
There are three ways in which the HSDB can attach to the target HotSpot VM:
Attach to a local HotSpot process
Attach to a core file
Attach to a remote debug server
Attach to a HotSpot Process
HSDB provides a facility to attach to a running Java process. Click File >Attach to HotSpot process and it brings up a dialog box that asks for the ID of the process to which we want to attach the SA, as shown in Figure 4.2.
After typing the process ID, click OK and SA gets attached to the process. The process is paused at this point and remains paused as long as SA is attached to it. Recall that SA is a snapshot debugger and provides a snapshot view of the process at the point in time when it is attached to the process. Figure 4.3 shows the HSDB main window after the SA gets attached to the process.
After analyzing the process, when we detach SA from the process, the process resumes its execution normally. The action of detaching the SA from the process is shown in Figure 4.4.
Attach to HotSpot Core Files
SA is a very useful tool for postmortem analysis. With SA, it is possible to attach to HotSpot “core” dump files, also known as crash dump files on the Windows platform. A core or crash dump file is a binary file that consists of the state of a running program at a specific time. Core files may be generated due to process crashes, or they can be dumped from a running application for offline debugging. More details on core/crash dump files are covered in the section “Core Dump or Crash Dump Files” later in the chapter.
Tip
Note that the HotSpot core/crash dump file can be very large depending on the amount of state information it contains at the time of core/crash dump file generation. You may have to configure the operating system to generate large core files and also ensure that the file system where the file is generated has sufficient space.
To open a core file with the SA, launch HSDB and click File > Open HotSpot core file as shown in Figure 4.5. Then, enter the path to the core and the path to the Java executable as shown in Figure 4.6.
After attaching to the core file, HSDB provides the same set of tools to explore the core as it does with the live process.
Connect to Debug Server
This feature is very useful for the remote debugging of processes or core files when it is not possible to do so on the same machine as the SA. With this feature, we can run the debug server on the remote machine where the process/core is present and then connect the SA tools to this remotely running debug server. We need to perform the following steps on the remote machine to start the debug server:
1. Start the RMI registry with sa-jdi.jar in the classpath:
$JAVA_HOME/bin/rmiregistry -J-Xbootclasspath/p:${JAVA_HOME}/lib/sa-jdi.jar
This command creates and starts a remote object registry. One can specify a port number at which this registry should be started. If no port number is specified, the registry is started on port 1099.
2. Start the debug server on the remote machine, specifying the process or core file to be debugged:
${JAVA_HOME}/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.DebugServer <pid> [uniqueID]
or
${JAVA_HOME}/bin/java -classpath ${JAVA_HOME}/lib/sa-jdi.jar
sun.jvm.hotspot.DebugServer
<pathname to the java executable that produced the core file>
<pathname of the corefile to debug> [uniqueID]
uniqueID is an optional string. If we want to run more than one debug server at the same time on the same machine, we must specify a different uniqueID string for each debug server. The debug server starts the RMI registry at the default port 1099 if the RMI registry was not already started.
Now, let’s start a Java process on a Solaris SPARC machine, attach a debug server to it, and then connect to that debug server from a Windows machine.
1. Start the Java process:
myserver% java TestClass &
27813
2. Start rmiregistry and the debug server:
myserver% rmiregistry -J-Xbootclasspath/p:$JAVA_HOME/lib/sa-jdi.jar
3. Start the debug server, passing it the HotSpot process ID and the unique name we want to assign to this debuggee:
myserver% $JAVA_HOME/bin/java -cp $JAVA_HOME/lib/sa-jdi.jar
-Djava.rmi.server.codebase=file:/$JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.DebugServer 27813 1234
Attaching to process ID 27813 and starting RMI services, please wait...
Debugger attached and RMI services started.
From the Windows machine, we can now connect to this specific debug server using the unique identifier and the hostname in [uniqueID@]hostname format as shown in Figure 4.7. Once the unique ID and hostname have been entered and the OK button has been pressed, the SA will display a status window saying it is trying to connect to the debug server, as shown in Figure 4.8.
After connecting the HSDB to the debug server, we can use all the utilities available under the Tools menu and debug the process as if it were running on our local machine.
Java Threads
The first window that appears in HSDB when it is connected to a Java process or core file is the panel that displays all the Java threads in the target JVM. Figure 4.9 shows all the Java threads in the attached Java process.
This panel has some interesting icons at the top left to show information on the selected Java thread:
Inspect Thread: This icon brings up the Object Inspector window showing the VM representation of the thread object. See Figure 4.10.
Show Stack Memory: This shows the stack data with symbol information at the stack memory of the selected thread as in Figure 4.11.
Show Java Stack Trace: This shows the Java stack trace of a thread. The method names and addresses are hyperlinks in the displayed stack trace, and clicking these method links shows the method details in the lower part of the window. See Figure 4.12.
Show Thread Information: This shows detailed information about the selected thread, as shown in Figure 4.13.
Find Crashes: This last icon on the Java Threads panel searches for whether any of the threads encountered a crash and, if so, shows details about the crash.
Now let’s take a quick look at the utilities available in this GUI tool under the Tools menu, as shown in Figure 4.14.
Some of these tools are helpful in debugging the Java-level issues, and some are very useful in troubleshooting the JVM-level problems.
Class Browser
With the Class Browser (see the example in Figure 4.15), we can see all the classes loaded by the target VM. It also allows us to dump the class files for all or some selective classes loaded in the target VM. This tool is very useful when we do not have access to the source code of the application and just have the core dump file and we need to investigate some issue with the loaded classes. For example, if some loaded class is not behaving as expected, we can dump that class, look at its code, and try to figure out the problem. Or perhaps there are too many loaded classes and we are getting out-of-memory errors; with the Class Browser we can look through the classes and see if some unneeded classes are also getting loaded or whether the classes are getting unloaded as expected.
Deadlock Detection
This feature helps us detect the Java-level deadlock among the threads. If a Java-level deadlock exists among the threads in the target VM, this tool prints information about the threads involved in the deadlock and also the monitors they are waiting to acquire. Figure 4.16 shows an example where no deadlocks were found.
Object Inspector
We can inspect the Java objects as well as the VM internal C++ structures using the Object Inspector tool.
To inspect a Java object, we need to provide the object’s address in the Java heap; this tool shows the internal fields of the object as in Figure 4.17.
Object Inspector can also show the VM internal C++ structures that are described by the VMStructs database in the target VM. See Figure 4.18.
Object Histogram
With the Object Histogram tool, an example of which is shown in Figure 4.19, we can get a histogram of the objects present in the Java heap. This tool helps in diagnosing memory leaks or out-of-memory-related problems in Java programs.
This tool also has a feature to show the instances of a particular class. Clicking on the search icon in the top right corner brings up the window showing all the instances of the selected class. See Figure 4.20.
We can select any instance from the list of instances and get the liveness path of the object through which that object is reachable and is considered alive by the garbage collector, as shown in Figure 4.21.
Compute Reverse Pointers
This utility computes the reference paths through which an object is reachable from the GC roots—the reference paths that are responsible for keeping that object alive in the Java heap. Once we have computed the reverse pointers, the <<Reverse pointers>> field in the Object Inspector window for an object shows the reverse pointers through which that object is reachable, as shown in Figure 4.22.
We can also see the liveness path of the object in another window by using the Compute Liveness and Show Liveness buttons available in the Inspector window. See Figure 4.23.
Find Object by Query
This tool provides an SQL-based query language to query Java objects from the Java heap; for example, to find all the thread objects in the Java heap, we can execute a query like this:
select t from java.lang.Thread t
Figure 4.24 shows the result of executing this query.
Find Pointer
This tool can help us find where a particular address lies in the Java process address space. This is particularly useful when we are dealing with JVM crashes and want to know the details of the memory address being accessed when the JVM crashed. For instance, where in the JVM does the address reside? Is the address at a location in the Java heap, in the eden space, in the old generation space? Figure 4.25 shows an example of an address that has been entered and the resulting information about the address that is displayed.
Find Address in Heap
This tool reports all the locations where a particular value is located in the Java heap. This is very useful when we want to find all the addresses from where an object is being referenced in the Java heap.
This utility greatly helps in nailing down garbage-collector-related issues, such as in the case of a GC crash where an object was collected prematurely and the JVM crashes while accessing the memory location of that object. By transitively tracking the references pointing to that object, we can get clues to why the object did not get marked as live by the garbage collector and was collected prematurely.
Figure 4.26 shows an example address entered and the resulting output.
Find Value in Code Cache
This tool shows all the locations from where a given address is being accessed in the cache of compiled methods. This helps in troubleshooting just-in-time (JIT) compiler-related problems. An example is shown in Figure 4.27.
Memory Viewer
Memory Viewer shows the raw memory contents in hexadecimal format at any given heap address. See Figure 4.28.
Monitor Cache Dump
This utility dumps the cache of object monitors as shown in Figure 4.29. This is useful in troubleshooting synchronization-related issues.
Code Viewer
Code Viewer (see the example in Figure 4.30) can show us the bytecodes of a method and the JIT compiler-generated machine code of the method if the method has been compiled. This tool is very useful in troubleshooting JIT compiler issues. Many times we encounter problems where a certain method compiled by the server/client compiler is either producing unexpected results or is causing a JVM crash. By looking at the disassembled generated compiled code (see Figure 4.31), we can get to the root of such issues.
Heap Parameters
The Heap Parameters utility shows us the heap boundaries of various generations of the Java heap. It is helpful in finding out the heap region in which a particular address lies. See Figure 4.32 for an example.
System Properties
We can get the system properties used by the target VM using the System Properties tool. An example is shown in Figure 4.33.
VM Version Info
This utility shows the detailed JVM version of the target process or core file. An example is shown in Figure 4.34.
Command Line Flags
This utility shows the values set for all the command-line -XX JVM options. An example is shown in Figure 4.35.
CLHSDB
Command-Line HotSpot Debugger is the command-line variant of the HSDB. To launch CLHSDB, we need to set the same environment variables as we did for the HSDB. Use the following command to launch this tool:
$JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.CLHSDB
It offers almost all the features that the UI version of the tool does; for example, to examine any Java object or VM data structure there is a command called inspect:
hsdb> inspect 0x23f50a20
instance of Oop for java/lang/Thread @ 0x23f50a20 @ 0x23f50a20 (size = 104)
_mark: 1
_metadata._klass: InstanceKlass for java/lang/Thread @ 0x38966700 Oop @
0x38966700
name: [C @ 0x23f50ac0 Oop for [C @ 0x23f50ac0
priority: 5
threadQ: null null
eetop: 4758528
single_step: false
daemon: false
stillborn: false
target: null null
group: Oop for java/lang/ThreadGroup @ 0x23f50840 Oop for java/lang/
ThreadGroup @ 0x23f50840
contextClassLoader: Oop for sun/misc/Launcher$AppClassLoader @ 0x23f7b398
Oop for sun/misc/Launcher$AppClassLoader @ 0x23f7b398
inheritedAccessControlContext: Oop for java/security/AccessControlContext
@ 0x23f50ad8 Oop for java/security/AccessControlContext @ 0x23f50ad8
threadLocals: Oop for java/lang/ThreadLocal$ThreadLocalMap @ 0x23f7c960
Oop for java/lang/ThreadLocal$ThreadLocalMap @ 0x23f7c960
inheritableThreadLocals: null null
stackSize: 0
nativeParkEventPointer: 0
tid: 1
threadStatus: 5
parkBlocker: null null
blocker: null null
blockerLock: Oop for java/lang/Object @ 0x23f50ab8 Oop for java/lang/
Object @ 0x23f50ab8
uncaughtExceptionHandler: null nullCheck heap boundaries
To get the heap boundaries, we can use the command universe:
hsdb> universe
Heap Parameters:
Gen 0: eden [0x23f50000,0x23fae5a0,0x243a0000) space capacity = 4521984, 8.546337182971014 used
from [0x243a0000,0x243a0000,0x24420000) space capacity = 524288, 0.0
used
to [0x24420000,0x24420000,0x244a0000) space capacity = 524288, 0.0 usedInvocations: 0
Gen 1: old [0x294a0000,0x294a0000,0x29f50000) space capacity =
11206656, 0.0 usedInvocations: 0
perm [0x33f50000,0x33f68700,0x34b50000) space capacity = 12582912,
0.7954915364583334 used ro space: [0x37f50000,0x383d2e40,0x38950000)
space capacity = 10485760, 45.1129150390625 used, rw space:
[0x38950000,0x38fd67b8,0x39550000) space capacity = 12582912,
54.37768300374349 usedInvocations: 0
hsdb> help
Available commands:
assert true | false
attach pid | exec core
detach
dumpcfg { -a | id }
dumpcodecache
dumpideal { -a | id }
dumpilt { -a | id }
echo [ true | false ]
examine [ address/count ] | [ address,address]
field [ type [ name fieldtype isStatic offset address ] ]
findpc address
flags [ flag | -nd ]
help [ command ]
history
inspect expression
intConstant [ name [ value ] ]
jhisto
jstack [-v]
livenmethods
longConstant [ name [ value ] ]
pmap
print expression
printas type expression
printmdo [ -a | expression ]
printstatics [ type ]
pstack [-v]
quit
reattach
revptrs address
scanoops start end [ type ]
search [ heap | perm | rawheap | codecache | threads ] value
source filename
symbol address
symboldump
symboltable name
thread { -a | id }
threads
tokenize ...
type [ type [ name super isOop isInteger isUnsigned size ] ]
universe
verbose true | false
versioncheck [ true | false ]
vmstructsdump
where { -a | id }
Some Other Tools
Some more very handy, small utilities are bundled with the SA. These tools can be attached to a process, to a core file, or to a remote debug server. Let’s take a look at how we can use them and what useful information they provide.
FinalizerInfo
This tool prints details on the finalizable objects in the target VM:
java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.FinalizerInfo 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
Number of objects pending for finalization: 0
HeapDumper
This tool can dump the Java heap to a file in the hprof format:
java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.HeapDumper 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
Dumping heap to heap.bin ...
Heap dump file created
PermStat
This tool prints the statistics of the permanent generation of the attached process or the core file:
java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.PermStat 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
10713 intern Strings occupying 802608 bytes.
finding class loader instances ..
done.
computing per loader stat ..done.
please wait.. computing
liveness..............................................done.
class_loader classes bytes parent_loader alive? type
<bootstrap> 342 1539808 null live <internal>
0x23f7b398 3 28016 0x23f762e0 live
sun/misc/Launcher$AppClassLoader@0x38a0e9c0
0x23f762e0 0 0 null live
sun/misc/Launcher$ExtClassLoader@0x389eb420
total = 3 345 1567824 N/A alive=3, dead=0 N/A
PMap
This tool prints the process map of the target process/core much like the Solaris pmap tool:
java -classpath %JAVA_HOME%libsa-jdi.jar sun.jvm.hotspot.tools.PMap 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
0x011c0000 184K d:java7u40injava.exe
0x6c0f0000 3232K d:java7u40jreinclientjvm.dll
0x6f5c0000 200K C:windowssystem32WINMM.dll
0x70900000 760K d:java7u40jreinmsvcr100.dll
0x74220000 1656K C:windowsWinSxSx86_microsoft.windows.common-
controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2COMCTL32.dll
0x74990000 28K C:windowssystem32WSOCK32.dll
0x74b20000 76K d:java7u40jreinzip.dll
0x74b40000 128K d:java7u40jreinjava.dll
0x74b60000 48K d:java7u40jreinverify.dll
0x74cc0000 48K C:windowssyswow64CRYPTBASE.dll
0x74cd0000 384K C:windowssyswow64SspiCli.dll
0x74d30000 212K C:windowssyswow64WS2_32.dll
0x74d70000 1088K C:windowssyswow64kernel32.dll
0x74fa0000 628K C:windowssyswow64USP10.dll
0x75cc0000 640K C:windowssyswow64ADVAPI32.dll
0x75ff0000 960K C:windowssyswow64RPCRT4.dll
0x76280000 384K C:windowssystem32IMM32.DLL
0x762e0000 816K C:windowssyswow64MSCTF.dll
0x763e0000 280K C:windowssyswow64KERNELBASE.dll
0x76430000 1024K C:windowssyswow64USER32.dll
0x76530000 40K C:windowssyswow64LPK.dll
0x76540000 688K C:windowssyswow64msvcrt.dll
0x76870000 20K C:windowssyswow64PSAPI.DLL
0x76880000 348K C:windowssyswow64SHLWAPI.dll
0x768e0000 576K C:windowssyswow64GDI32.dll
0x76b50000 100K C:windowsSysWOW64sechost.dll
0x77140000 24K C:windowssyswow64NSI.dll
0x77170000 1536K C:windowsSysWOW64
tdll.dll
Object Histogram
Object histograms can be collected using the utilities available in HSDB and CLHSDB. A standalone tool is also available that can be used to dump object histograms from the target VM:
D: >java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.ObjectHistogram 6468
Attaching to process ID 6468, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
Iterating over heap. This may take a while...
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 5659 546080 * ConstMethodKlass
2: 5659 455376 * MethodKlass
3: 385 221880 * ConstantPoolKlass
4: 385 154496 * InstanceKlassKlass
5: 1687 149824 char[]
6: 353 139264 * ConstantPoolCacheKlass
7: 477 122920 byte[]
8: 445 54320 java.lang.Class
9: 644 37600 * System ObjArray
10: 583 37008 short[]
11: 1364 32736 java.lang.String
12: 377 21064 int[]
13: 43 14104 * ObjArrayKlassKlass
14: 328 13544 java.lang.Object[]
15: 120 7680 java.lang.reflect.Field
16: 30 3232 java.util.HashMap$Entry[]
17: 125 3000 java.util.HashMap$Entry
18: 118 2832 java.util.Hashtable$Entry
19: 8 2624 * TypeArrayKlassKlass
20: 73 2472 java.lang.String[]
21: 12 1680 java.util.Hashtable$Entry[]
22: 96 1536 java.lang.StringBuilder
23: 38 1520 sun.util.locale.LocaleObjectCache$CacheEntry
.........
........
228: 1 8 java.lang.Terminator$1
229: 1 8 java.security.ProtectionDomain$1
230: 1 8 sun.net.www.protocol.file.Handler
Total : 20102 2059376
Heap traversal took 0.153 seconds.
Structured Object Query Language—SOQL
This tool provides a shell-like command-line interface for executing SOQL queries. SOQL is an SQL-like language that can be used to query the Java heap:
java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.soql.SOQL 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
soql> select mirror(t) from java.lang.Thread t
> go
Thread[main,5,main]
Thread[Signal Dispatcher,9,system]
Thread[Attach Listener,5,system]
Thread[C1 CompilerThread0,9,system]
Thread[Service Thread,9,system]
jhat also provides an interface to use this language. Good documentation on this language is available in the jhat tool. That help documentation can also be accessed from here: https://blogs.oracle.com/poonam/entry/object_query_language_help.
ClassDump
Using this tool, we can dump the loaded classes from the target VM. It is possible to dump a single class or multiple classes from the selected packages. A few system properties are available that can be set to specify the name of the class that we want to dump or the list of packages from which we want to dump the classes. These are listed in Table 4.1.
Here is how we can attach the ClassDump utility to a running process and dump the loaded classes in a folder specified using the -Dsun.jvm.hotspot.tools.jcore.outputDir property:
java -classpath $JAVA_HOME/lib/sa-jdi.jar
-Dsun.jvm.hotspot.tools.jcore.outputDir=
classes sun.jvm.hotspot.tools.jcore.ClassDump 2402
Attaching to process ID 2402, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.0-b56
myserver 20 % ls classes
./ ../ TestClass.class java/ sun/
As we can see, the ClassDump utility has dumped the classes loaded in the process in the classes/folder. Similarly, this tool can attach to a core file or to a remote debug server and dump the classes loaded in the target VM.
DumpJFR
DumpJFR is an SA-based tool that can be used to extract Java Flight Recorder (JFR) information from the core files and live HotSpot processes.
Java Flight Recorder and Mission Control tools are shipped with the JDK since JDK 7u40. As we know, the Java Flight Recorder is a tool for collecting diagnostic and profiling data about a running Java application. It is integrated into the JVM, and its usage causes almost no performance overhead. Java Mission Control can be used to analyze the data collected by the Java Flight Recorder.
DumpJFR provides the capability to extract the JFR data from the core files of crashes, or hung Java processes, which otherwise is not possible to access. This tool is shipped with the JDK since JDK 8u60.
DumpJFR tool sources are present under hotspot/src/closed/agent/ in the repository. During the build process of HotSpot sources, DumpJFR class files get included into sa-jdi.jar when the closed part of the HotSpot repository gets built.
Please note that Java Flight Recorder and Mission Control, and hence this tool, require a commercial license for use in production.
This is how we can attach DumpJFR to a live process and dump the JFR data into a recording file:
java -cp $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.tools.DumpJFR <pid>
This attaches DumpJFR to a core file to extract the Java Flight Recorder information:
java -cp $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.tools.DumpJFR
<java> <core>
The DumpJFR tool dumps the JFR data to a file called recording.jfr in the current working folder. This recording file can be analyzed using Java Mission Control.
JSDB
JavaScript Debugger provides a JavaScript interface to the SA. It is a command-line JavaScript shell based on Mozilla’s Rhino JavaScript Engine.
java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.soql.JSDB 5684
Attaching to process ID 5684, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
jsdb>
More details on this utility can be found in the open-source HotSpot repository in the file hotspot/agent/doc/jsdb.html.
Core Dump or Crash Dump Files
We have been talking about the core files on Unix systems and the crash dump files on Windows systems. Here is a quick briefing on what these dump files are.
A core or crash dump file is a binary file that consists of the state of a running program at a specific time. A core file is generally created when a program terminates unexpectedly, due to a violation of the operating system’s or hardware’s protection mechanisms. The operating system kills the program and creates a core file that programmers can later use to figure out what went wrong. It contains a detailed description of the state that the program was in when it died.
It is also possible to create a core/crash dump file at will from a running (non-crashing) program. On Solaris/Linux, there is a tool called gcore that can dump the state of a running or hung process into a core file. On Windows, we can use WinDbg, userdump, or ADPlus to collect the crash dump files.
Core dump files are very useful for diagnosing and debugging problems in Java programs whenever it is difficult to debug the live processes.
Debugging Transported Core Files
Sometimes a core file is created on one system (core host) and we need to debug it on some other system (debugger host). Native debuggers (dbx, gdb) as well as the SA are not always able to open the transported core files successfully, the reason being a mismatch in the kernel and shared libraries between the system where the core was produced and the one where we are trying to debug it. Debuggers may face problems due to the mismatch of the following two types of libraries:
The shared libraries used by the program on the core host may not be the same as those on the debugger host. So we need to make the original libraries from the core host available on the debugger system.
Debuggers use the system libraries (e.g., libraries in /usr/lib, /lib64, etc.) to understand the implementation details of the runtime linker and the threads library on the system. To load the core files successfully we also need to get the system libraries from the core host and make them available on the debugger host.
If the core file was produced from a crash and a crash log hs_err file was also generated, we can get the list of the shared libraries (system and program) loaded by the process from the hs_err log file.
Shared Library Problems with the SA
When using SA with transported core files, we may get failures related to rtld_db or libthread_db mismatch, or SA may throw errors that some HotSpot symbol is missing in the target process. For example, in the following we are trying to open a core file with SA on a Linux/x64 machine transported from another Linux/x64 system:
-bash-3.2$ $JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.CLHSDB $JAVA_HOME/bin/java core.16963
Opening core file, please wait...
Unable to open core file
core.16963:
Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes"
in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot
VM (could not find symbol "gHotSpotVMTypes" in remote process)
at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:405)
at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:314)
at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:173)
at sun.jvm.hotspot.CLHSDB.attachDebugger(CLHSDB.java:188)
at sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:55)
at sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:35)
hsdb> Input stream closed.
SA is not able to open this transported core file.
Eliminate Shared Library Problems
So, how do we fix shared library problems encountered when opening transported core files with the SA? Here is what we can do to eliminate these problems:
1. Copy all the libraries used by the program from the core host to the debugger host, say, to a folder /space/corelibs/. Note that the libraries can be copied either directly under /space/corelibs/ or to a full directory path under /space/corelibs/. For example, /local/java/jre/lib/sparc/server/libjvm.so from the core host can be copied directly either under /space/corelibs/ or under /space/corelibs/local/java/jre/lib/sparc/server/ on the debugger host. Similarly, /usr/lib/libthread_db.so from the core host can be copied either to /space/corelibs/ or to /space/corelibs/usr/lib/ on the debugger host.
The list of required library files can be obtained either from the hs_err log file under the section “Dynamic Libraries” or by using the native debuggers such as gdb, dbx, and WinDbg.
2. Then set the SA environment variable SA_ALTROOT to the folder containing the shared libraries on the debugger host, that is, setenv SA_ALTROOT /space/corelibs/.
Now, for the core file core.16963 that we tried to open in the previous section, we copied all the required libraries from the core host to the system where we want to open the core with SA and then set the environment variable SA_ALTROOT:
-bash-3.2$ export SA_ALTROOT=/space/corelibs/
-bash-3.2$ $JAVA_HOME/bin/java -classpath $JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.CLHSDB $JAVA_HOME/bin/java core.16963
Opening core file, please wait...
hsdb> universe
Heap Parameters:
ParallelScavengeHeap [ PSYoungGen [ eden =
[0x00000000eaa00000,0x00000000eaa00000,0x00000000faa80000] , from =
[0x00000000faa80000,0x00000000fb64975
0,0x00000000fd540000] , to =
[0x00000000fd540000,0x00000000fd540000,0x0000000100000000] ] PSOldGen [
[0x00000000bfe00000,0x00000000bfe12010,0x00000
000eaa00000] ] PSPermGen [
[0x00000000bac00000,0x00000000bc0942c0,0x00000000bc200000] ] ]
hsdb>
With SA_ALTROOT, SA picks up the required libraries from /space/corelibs/ and thus can open the core file successfully.
System Properties for the Serviceability Agent
There are some system properties that can be used to define the runtime behavior of the SA tools. These are shown in Table 4.2.
Environment Variables for the Serviceability Agent
There are some environment variables that can be set to define the runtime behavior of the SA tools. These are shown in Table 4.3.
JDI Implementation
The SA binary sa-jdi.jar also has an implementation of the Java Debug Interface (JDI) that makes it possible for any JDI client (e.g., JDB) to attach to the core files and also Java processes using the JDI Connectors provided by this implementation.
The VM object returned by the attach() method of these connectors is read-only. This means that the obtained VM object cannot be modified. So, JDI clients using these connectors should not call any JDI methods that are defined to throw a VMCannotBeModifiedException.
SA Core Attaching Connector
This connector can be used by a debugger application to debug a core file. It allows multiple debugging sessions on the same core file. It is uniquely identified by the name sun.jvm.hotspot.jdi.SACoreAttachingConnector and can be used as in the following:
$JAVA_HOME/bin/jdb
-J-Xbootclasspath/a:$JAVA_HOME/lib/sa-jdi.jar:$JAVA_HOME/lib/tools.jar
–connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:core=
${CORE_FILE},javaExecutable=${EXEC_FILE}
SAPID Attaching Connector
This connector can be used by a debugger application to debug a process. The process to be debugged need not have been started in the debug mode (i.e., with -agentlib:jdwp or -Xrunjdwp), and it is permissible for the process to be hung. Using this connector, the debugger gets a read-only view of the process, and the VM object obtained after attaching to the process cannot be modified.
The process is suspended when this connector attaches and is resumed when the connector detaches. More than one SAPID Attaching Connector cannot attach to a single process simultaneously. This connector is uniquely identified by the name sun.jvm.hotspot.jdi.SAPIDAttachingConnector.
Debuggers can use this connector as in the following to connect to and debug a process:
$JAVA_HOME/bin/jdb
-J-Xbootclasspath/a:$JAVA_HOME/lib/sa-jdi.jar:$JAVA_HOME/lib/tools.jar
-connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=2402
SA Debug Server Attaching Connector
This connector can be used by a debugger application to debug a process or core file on a machine other than the machine on which the debugger is running. This connector uses Remote Method Invocation (RMI) to communicate with a debug server running on the remote machine. Before the attach() method on this connector is called, the debug server must be started on the remote machine and told what process or core file is to be debugged. The following needs to be done on the remote machine:
1. Start the RMI registry with sa-jdi.jar in the classpath:
${JAVA_HOME}/bin/rmiregistry
-J-Xbootclasspath/p:${JAVA_HOME}/lib/sa-jdi.jar
This command creates and starts a remote object registry. An optional port number may be specified as the the port number at which the registry should be started. If no optional port number is specified, the registry is started on port 1099.
2. Start the debug server on the remote machine, specifying the process or core file to be debugged:
${JAVA_HOME}/bin/java -classpath ${JAVA_HOME}/lib/sa-jdi.jar
sun.jvm.hotspot.jdi.SADebugServer <pid> [uniqueID]
or
${JAVA_HOME}/bin/java -classpath ${JAVA_HOME}/lib/sa-jdi.jar
sun.jvm.hotspot.jdi.SADebugServer
<pathname to the java executable that produced the core file>
<pathname of the core file to debug> [uniqueID]
SA Debug Server starts the RMI registry at port 1099 if the registry is not already running.
An alternative to these two steps is to use the jsadebugd utility that is shipped with the JDK to start the RMI registry and the debug server on the remote machine.
uniqueID is an optional string. If more than one debug server is to run at the same time on the same machine, each must have a different uniqueID string.
Extending Serviceability Agent Tools
The Serviceability Agent APIs provide a class sun.jvm.hotspot.tools.Tool which can be extended to write our own troubleshooting tools.
Here is a simple tool extending the sun.jvm.hotspot.tools.Tool class that prints the system properties of the target VM. This tool is part of the sa-jdi.jar file:
package sun.jvm.hotspot.tools;
import java.io.PrintStream;
import java.util.*;
import sun.jvm.hotspot.runtime.*;
public class SysPropsDumper extends Tool {
public void run() {
Properties sysProps = VM.getVM().getSystemProperties();
PrintStream out = System.out;
if (sysProps != null) {
Enumeration keys = sysProps.keys();
while (keys.hasMoreElements()) {
Object key = keys.nextElement();
out.print(key);
out.print(" = ");
out.println(sysProps.get(key));
}
} else {
out.println("System Properties info not available!");
}
}
public static void main(String[] args) {
SysPropsDumper pd = new SysPropsDumper();
pd.start(args);
pd.stop();
}
}
To write our own tool, we need to extend it from the Tool class and implement the run() method where we add the main functionality of the tool. In the preceding example, using the VM class we obtain the system properties and then print those properties to the standard output.
Let’s compile and run this tool against a running process and see what the output looks like:
D:>javac -classpath %JAVA_HOME%libsa-jdi.jar
sun/jvm/hotspot/tools/SysPropsDumper.java
D:>java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.SysPropsDumper 5880
Attaching to process ID 5880, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
java.runtime.name = Java(TM) SE Runtime Environment
java.vm.version = 24.0-b56
sun.boot.library.path = D:Java7u40jrein
java.vendor.url = http://java.oracle.com/
java.vm.vendor = Oracle Corporation
path.separator = ;
file.encoding.pkg = sun.io
java.vm.name = Java HotSpot(TM) Client VM
sun.os.patch.level = Service Pack 1
sun.java.launcher = SUN_STANDARD
user.script =
user.country = US
user.dir = D: ests
java.vm.specification.name = Java Virtual Machine Specification
java.runtime.version = 1.7.0_40-b43
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
os.arch = x86
java.endorsed.dirs = D:Java7u40jrelibendorsed
line.separator =
java.io.tmpdir = C:UserspobajajAppDataLocalTemp
java.vm.specification.vendor = Oracle Corporation
user.variant =
os.name = Windows 7
sun.jnu.encoding = Cp1252
java.library.path = D:Java7u40in;C:windowsSunJavain;C:windows
system32;C:windows;D:Java7u40in;C:windowssystem32;C:windows;C:
windowsSystem32Wbem;C:windowsSystem32WindowsPowerShellv1.0;C:Program
Files (x86)Microsoft SQL Server100ToolsBinn;C:Program FilesMicros
oft SQL Server100ToolsBinn;C:Program FilesMicrosoft SQL Server100
DTSBinn;D:Program FilesPerforce;.
java.specification.name = Java Platform API Specification
java.class.version = 51.0
sun.management.compiler = HotSpot Client Compiler
os.version = 6.1
user.home = C:Userspobajaj
user.timezone =
java.awt.printerjob = sun.awt.windows.WPrinterJob
file.encoding = Cp1252
java.specification.version = 1.7
user.name = pobajaj
java.class.path = .
java.vm.specification.version = 1.7
sun.arch.data.model = 32
sun.java.command = TestClass
java.home = D:Java7u40jre
user.language = en
java.specification.vendor = Oracle Corporation
awt.toolkit = sun.awt.windows.WToolkit
java.vm.info = mixed mode, sharing
java.version = 1.7.0_40
java.ext.dirs = D:Java7u40jrelibext;C:windowsSunJavalibext
sun.boot.class.path = D:Java7u40jrelib
esources.jar;D:Java7u40jre
lib
t.jar;D:Java7u40jrelibsunrsasign.jar;D:Java7u40jrelibjsse
.jar;D:Java7u40jrelibjce.jar;D:Java7u40jrelibcharsets.jar;D:
Java7u40jrelibjfr.jar;D:Java7u40jreclasses
java.vendor = Oracle Corporation
file.separator =
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
sun.io.unicode.encoding = UnicodeLittle
sun.cpu.endian = little
sun.desktop = windows
sun.cpu.isalist = pentium_pro+mmx pentium_pro pentium+mmx pentium i486
i386 i86
The SA code is located under the <hotspot-repo>/agent/src/ folder, and the implementation of a few tools that get bundled with sa-jdi.jar exists under sun/jvm/hotspot/tools/ under that folder.
The JavaDoc API for these classes is available for download from http://docs.oracle.com/javase/8/docs/serviceabilityagent/.
Serviceability Agent Plugin for VisualVM
Before we delve into the details on the Serviceability Agent Plugin for Java VisualVM (VisualVM), let’s first talk briefly about VisualVM.
VisualVM is a visual tool for troubleshooting, monitoring, and observing Java applications. It is positioned as an all-in-one monitoring and troubleshooting tool for Java applications. Most of the standalone tools shipped with the JDK such as jstack, jmap, and so on are also available in VisualVM. Starting with JDK 6u7, VisualVM is also shipped with the JDK and can be launched using the command jvisualvm. Figure 4.36 shows an example of what VisualVM looks like when it has been launched.
One great thing about VisualVM is that it is possible to extend it using plugins. One can write plugins for VisualVM using its APIs and develop plugins for others to use. The plugins then can be made available in the VisualVM Plugins Center.
Serviceability Agent Plugin for VisualVM is a VisualVM plugin that brings key features of SA to VisualVM. This plugin provides the ability to look at the VM representation of Java objects in the heap, Java threads in the application, and compiled/interpreted code of methods and to find references for a given address in the Java heap, all from within the VisualVM GUI.
How to Install the SA-Plugin in VisualVM
VisualVM plugins are available at the VisualVM Plugins Center. The latest VisualVM Plugins Center is at http://visualvm.java.net/pluginscenters.html. To install the SA-Plugin in VisualVM, click on Tools > Plugins. This will bring up the Plugins dialog box. Click on the Available Plugins tab, select SAPlugin from the list, and then click Install. This will install the SA-Plugin in your VisualVM installation.
How to Use the SA-Plugin
Let’s take a look how we can use the SA-Plugin to explore a running process or a core file.
To explore a live process with SA-Plugin, run the process with a JDK that is shipped along with the SA binaries. Open this process in VisualVM. When SA-Plugin is installed in VisualVM, the SAPlugin tab appears in the Application view when the process is opened in VisualVM. Now, click Attach to Process in the SAPlugin tab; this will attach SA to the running process, and the process will be paused at that point in time. Now we can explore the snapshot of the process. We can look at its Java threads, inspect the Java objects, look at the bytecodes/compiled code of methods, and so on. After we are done, we can detach the process from the SA by clicking the Detach from Process button. The process will resume.
Opening and diagnosing core files is similar to exploring processes with SA-Plugin. First we need to open the core file in VisualVM and then attach to that core from the SA Plugin view. As in the case of a process, we can explore objects in the Java heap, look at thread objects, search for values in the Java heap, and look at the compiled code of methods from the core file. SA-Plugin’s capability of exploring the core files helps a great deal in the postmortem of dead processes.
SA-Plugin Utilities
This plugin makes some of the Serviceability Agent utilities available in the VisualVM in four panels described in the following sections.
Java Threads/Java Stack Trace
This panel, shown in Figure 4.37, shows all the Java threads in the attached process or core file. Double-clicking on any thread or clicking on the Open Inspector icon on the Java Threads panel shows internal details of the thread object in another Oop Inspector panel. And clicking on the Show Stack Trace icon on the Java Threads panel shows the stack trace of the thread. An example of the Show Stack Trace panel is shown in Figure 4.38.
Oop Inspector
We can inspect the Java objects as well as the VM internal C++ structures using this panel. All the Java objects in the HotSpot VM are represented as oops—ordinary object pointers. The Oop Inspector panel shows details of oops. The ability to see all the internal fields of any Java object provides great debugging help. This panel also provides the ability to compute the liveness of any oop in the Java heap. An example of the Oop Inspector panel is shown in Figure 4.39.
Code Viewer
This panel shows the bytecodes of a method and the JIT compiler-generated machine code of the method if the method has been compiled. See Figures 4.40 and 4.41 for examples of the Code Viewer panel. Figure 4.40 shows the bytecodes of a method, and Figure 4.41 shows the JIT compiler-generated machine code for the compiled method.
Find Panel
This panel, seen in Figure 4.42, has three utilities:
Find Pointer helps us find where a particular address lies in the Java process address space.
Find Value in Heap helps us find all the locations in the Java heap where a particular value is present.
Find Value in CodeCache helps us find all the locations in the compiler code cache where a particular value is present.
Troubleshooting Problems Using the SA
In the previous sections we looked at what the SA is and how it works. In this section we will talk a bit about the real-world problems of Java applications and then see in detail how SA can help us get to the root of these issues.
A Java application can face multiple problems: slowness, huge memory consumption, unexpected behavior of the program, and even crashes. SA tools can greatly help in figuring out the root cause of these and also help improve the performance of applications.
Diagnosing OutOfMemoryError
java.lang.OutOfMemoryError is thrown when there is insufficient space to allocate new objects in the Java heap or in the permanent generation space or metaspace. (Please note that in JDK 8, permanent generation space has been removed in favor of the metaspace.) At that point, the garbage collector cannot make any more space available and the heap cannot be expanded further.
The possible reasons for this error could be that the footprint of the application is large and cannot be accommodated in the specified Java heap, or there is a memory leak in the application.
Here is a Java program that throws java.lang.OutOfMemoryError in the Java heap space:
D: ests>java MemoryError
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Vector.<init>(Vector.java:131)
at java.util.Vector.<init>(Vector.java:144)
at java.util.Vector.<init>(Vector.java:153)
at Employee.<init>(MemoryError.java:42)
at MemoryError.main(MemoryError.java:14)
There are many ways to approach this problem:
Collect the heap dump from the process using the jmap utility, the JConsole utility, or using the -XX:+HeapDumpOnOutOfMemoryError JVM option, and then analyze the heap dump using jhat or VisualVM.
Collect the heap histogram from the running process using the SA tools.
Collect the core or crash dump file at the occurrence of OutOfMemoryError using the -XX:OnOutOfMemoryError option, and then obtain the heap histogram from that core/crash dump file using the SA tools.
Since our program does not run for long and does not give us enough time to attach any tool to the running process, we run the process with -XX:OnOutOfMemoryError to produce a crash dump file when the OutOfMemoryError occurs. We are running this program on a Windows machine.
D: ests>java -XX:OnOutOfMemoryError="D:Toolsuserdump8.1x64userdump
%p" MemoryError
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="D:Toolsuserdump8.1x64userdump %p"
# Executing "D:Toolsuserdump8.1x64userdump 4768"...
User Mode Process Dumper (Version 8.1.2929.5)
Copyright (c) Microsoft Corp. All rights reserved.
Dumping process 4768 (java.exe) to
d: estsjava.dmp...
The process was dumped successfully.
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Vector.<init>(Vector.java:131)
at java.util.Vector.<init>(Vector.java:144)
at java.util.Vector.<init>(Vector.java:153)
at Employee.<init>(MemoryError.java:42)
at MemoryError.main(MemoryError.java:14)
To generate the core file on Solaris or Linux platforms, the program can be run as
java -XX:OnOutOfMemoryError="gcore %p" MemoryError
Now, let’s open this crash dump file in the HSDB tool:
D: ests>java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.HSDB %JAVA_HOME%injava.exe java.dmp
Get the object histogram of the heap by clicking Tools > Object Histogram, as shown in Figure 4.43.
This histogram shows that apart from the system classes, Address and Employee classes appear near the top of the histogram and occupy a lot of space in the heap. This tells us that the instances of these classes are the main culprits for the OutOfMemoryError.
In the Object Histogram utility, we can find out all the instances of a particular class that are present in the Java heap. To do this, there is an icon at the top right corner of the Object Histogram window. Let’s find all the instances for the Address class. Figure 4.44 shows the Object Histogram window actively searching for all address instances.
Finding all the instances may take some time as the tool has to traverse the whole heap. This will bring up the Show Objects of Type window for the Address class as shown in Figure 4.45.
Now, by clicking on the Compute Liveness button we can get the liveness path (the reference path by which an instance is reachable from the GC roots) for the instances of the Address class. See Figure 4.46.
Then we can see the liveness path of the selected instance by clicking on the Show Liveness Path button.
The liveness path of the Address instance at 0x2564b698 shows that it is reachable from a Vector field employeesList.
Here is the Java code of the test program:
import java.util.*;
public class MemoryError{
static Vector employeesList;
public static void main(String arg[]) throws Exception{
HashMap employeesMap = new HashMap();
employeesList = new Vector();
int i = 0;
while(true){
Employee emp1 =
new Employee("Ram",
new Address("MG Road", "Bangalore", 123, India"));
Employee emp2 =
new Employee("Bob",
new Address("House No. 4", "SCA", 234, "USA"));
Employee emp3 =
new Employee("John",
new Address("Church Street", "Bangalore",
569, "India"));
employeesMap.put(new Integer(i++), emp1);
employeesList.add(emp1);
employeesMap.put(new Integer(i++), emp2);
employeesList.add(emp2);
employeesMap.put(new Integer(i++), emp3);
employeesList.add(emp3);
emp2.addReports(emp1);
emp3.addReports(emp1);
emp3.addReports(emp2);
}
}
}
class Employee{
public String name;
public Address address;
public Vector directReports;
public Employee(String nm, Address addr){
name = nm;
address = addr;
directReports = new Vector();
}
public void addReports(Employee emp){
directReports.add((Object)emp);
}
}
class Address{
public String addr;
public String city;
public int zip;
public String country;
public Address(String a, String ci, int z, String co){
addr = a;
city = ci;
zip = z;
city = co;
}
}
We can see that this Java code maintains records of employees along with their address details. The employee records are added to two collection objects: to a hash map and to a vector. Vector employeesList and hash map employeeMap hold references to the Employee and Address instances and thus prevent them from getting collected by the garbage collector. Therefore, these instances keep increasing in number in the Java heap.
This test program demonstrates a mistake that is commonly made in real applications. Often, applications inadvertently maintain multiple caches and collections of objects that are not required for the program logic, and that mistake increases the footprint of the process, thus leading to out-of-memory errors.
Please note that to demonstrate the problem and to produce the out-of-memory error, the employee records are added in a while loop in the test program, and that may not be the case in real-world applications.
We can also obtain the object histograms from the crash dump file using the Object Histogram standalone utility:
D: ests>java -classpath %JAVA_HOME%libsa-jdi.jar
sun.jvm.hotspot.tools.ObjectHistogram
%JAVA_HOME%injava.exe java.dmp
Attaching to core java.dmp from executable D:Java7u40injava.exe,
please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.0-b56
Iterating over heap. This may take a while...
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 1431913 90668040 java.lang.Object[]
2: 1431625 34359000 java.util.HashMap$Entry
3: 1431606 34358544 java.util.Vector
4: 1431599 34358376 Employee
5: 1431599 34358376 Address
6: 1431599 22905584 java.lang.Integer
7: 13 8389584 java.util.HashMap$Entry[]
8: 29398 3287200 * ConstMethodKlass
9: 29398 2122528 * MethodKlass
10: 2278 1424456 * ConstantPoolKlass
11: 2278 1074240 * InstanceKlassKlass
12: 2278 998272 * ConstantPoolCacheKlass
13: 14552 846096 byte[]
14: 11086 606008 char[]
.....
.....
166: 1 8 java.nio.Bits$1
167: 1 8 java.lang.Runtime
168: 1 8 java.security.ProtectionDomain$1
169: 1 8 java.lang.ref.Reference$Lock
170: 1 8 java.security.ProtectionDomain$3
Total : 8705181 270932344
Heap traversal took 125.427 seconds.
Diagnosing a Java-Level Deadlock
Let’s look at a small Java program that gets locked up in a deadlock soon after the start of its execution:
public class DeadLockTest extends Thread {
public static Object lock1 = new Object();
public static Object lock2 = new Object();
private int index;
public static void main(String[] a) {
Thread t1 = new Thread0();
Thread t2 = new Thread1();
t1.start();
t2.start();
}
private static class Thread0 extends Thread {
public void run() {
synchronized (lock1) {
System.out.println("Thread 0: Holding lock 1...");
try { Thread.sleep(10); }
catch (InterruptedException e) {}
System.out.println("Thread 0: Waiting for lock 2...");
synchronized (lock2) {
System.out.println("Thread 0: Holding lock 1 & 2...");
}
}
}
}
private static class Thread1 extends Thread {
public void run() {
synchronized (lock2) {
System.out.println("Thread 1: Holding lock 2...");
try { Thread.sleep(10); }
catch (InterruptedException e) {}
System.out.println("Thread 1: Waiting for lock 1...");
synchronized (lock1) {
System.out.println("Thread 1: Holding lock 2 & 1...");
}
}
}
}
}
We start the execution of this program and then let it run for some time. The program enters deadlock soon after. Then we attach HSDB to the hung process. HSDB has a utility called Deadlock Detection that is available under the Tools menu. Upon launching the Deadlock Detection tool, we get the message window shown in Figure 4.47.
This shows that the program has one Java-level deadlock involving two threads—Thread-0 and Thread-1. Thread-0 is waiting to lock Monitor 0x00ef2aac, which is held by Thread-1, and Thread-1 is waiting to lock Monitor 0x00ef0f8c, which is already held by Thread-0; hence the deadlock.
To get more details on these monitors, we can use the Monitor Cache Dump utility. See Figure 4.48.
Sometimes, it is interesting to look at the state of the threads. We can do this by inspecting the thread object in the Oop Inspector utility. Let’s take a look at the Thread-0 and Thread-1 thread objects in the Oop Inspector. Double-clicking on these threads in the Java Threads panel will bring up the Oop Inspector windows for these thread objects. Figure 4.49 shows the Oop Inspector for Thread-0, and Figure 4.50 shows the Oop Inspector window for Thread-1.
In these two snapshots, both threads have the thread status of 1025. Let’s look at the code in Java HotSpot VM that computes the thread state of the Java thread from its threadStatus field:
public static Thread.State toThreadState(int threadStatus) {
if ((threadStatus & JVMTI_THREAD_STATE_RUNNABLE) != 0) {
return RUNNABLE;
} else if ((threadStatus &
JVMTI_THREAD_STATE_BLOCKED_ON_MONITOR_ENTER) != 0) {
return BLOCKED;
private final static int
JVMTI_THREAD_STATE_BLOCKED_ON_MONITOR_ENTER = 0x0400;
This computation would return the thread state as BLOCKED when the threadStatus is 1025. This means that both threads are in the BLOCKED state.
So, using the Deadlock Detection tool, we can easily nail down the Java-level deadlocks in applications.
Postmortem Analysis of a HotSpot VM Crash
SA can help in diagnosing JVM crashes with its capability to do postmortem analysis of core files produced with these crashes. I have a simple program that uses JNI. It writes to a byte array beyond its size limit, which results in overwriting and corrupting the object that follows it in the Java heap. This causes the program to crash when the garbage collector tries to scan the corrupted Java heap.
Test Program
Here is the test program that has the Java classes TestCrash and Object1. Object1 has a native method (nativeMethod()) which is implemented in C code in the file test.c. The TestCrash class creates Object1 instances and then calls nativeMethod() on one of these Object1 instances.
import java.util.ArrayList;
public class TestCrash {
static {
System.loadLibrary(“test”);
}
ArrayList a2 = new ArrayList();
ArrayList a3 = new ArrayList();
public static void main(String[] args) {
TestCrash t = new TestCrash();
t.method1();
}
void method1() {
/* Fill the heap with ArrayLists */
ArrayList a1 = new ArrayList();
for (int i=0; i<10000; i++) {
Object1 obj = new Object1();
a1.add(obj);
}
/* corrupt an object with native method*/
((Object1)a1.get(1)).nativeMethod();
/* Invoke garbage collection */
for (int p=0; p<10; p++) {
System.gc();
}
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
}
}
}
class Object1 {
byte[] array = {1, 2, 3};
native void nativeMethod();
}
Here is the native part of the test case implementing method nativeMethod() using JNI. This source code is placed in a file named test.cpp following the JNI source file naming conventions.
#include "test.h"
#include <stdlib.h>
#include <memory.h>
/*
* Class: Object1
* Method: nativeMethod
* Signature: ()V
*/
JNIEXPORT void JNICALL Java_Object1_nativeMethod
(JNIEnv *env, jobject obj) {
jclass cls = env->GetObjectClass(obj);
jfieldID fldID = env->GetFieldID(cls, "array", "[B");
jbyteArray arrayObject =
(jbyteArray)env->GetObjectField(obj, fldID);
jbyte* array =
(jbyte*)env->GetPrimitiveArrayCritical(arrayObject, 0);
memcpy(array, "Hello Java.Hello
Java.Hello Java.Hello Java.Hello Java.Hello
Java.Hello Java.Hello Java.Hello Java.Hello Java.", 100);
env->ReleasePrimitiveArrayCritical(arrayObject, array, 0);
env->ExceptionDescribe();
env->ExceptionClear();
}
Now, we need to compile this program on a Solaris machine with the following instructions:
export JAVA_HOME=/java/jdk1.7.0_40/
$JAVA_HOME/bin/javac TestCrash.java
CC -m32 -I$JAVA_HOME/include -I$JAVA_HOME/include/solaris -G
test.cpp -o libtest.so
and then execute it as shown here:
oracle@solaris_11X:/demo# java -Djava.library.path=. TestCrash
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0xfe20af22, pid=1860, tid=3
#
# JRE version: Java(TM) SE Runtime Environment (7.0_40-b43) (build
1.7.0_40-b43)
# Java VM: Java HotSpot(TM) Client VM (24.0-b56 mixed mode solaris-x86 )
# Problematic frame:
# V [libjvm.so+0x20af22] void
objArrayKlass::oop_follow_contents(oopDesc*)+0x1d2
#
# Core dump written. Default location: /demo/core or core.1860
#
# An error report file with more information is saved as:
# /demo/hs_err_pid1860.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
#
Abort (core dumped)
The program crashes with the following stack trace:
Stack: [0xe77c1000,0xe7841000], sp=0xe783ff90, free space=507k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0x20af22] void
objArrayKlass::oop_follow_contents(oopDesc*)+0x1d2
V [libjvm.so+0x208f40] void MarkSweep::follow_stack()+0xc0
V [libjvm.so+0x1ef4a1] void InterpreterFrameClosure::offset_do(int)+0x59
V [libjvm.so+0x1ef3d1] void
InterpreterOopMap::iterate_oop(OffsetClosure*)+0xd9
V [libjvm.so+0x50af15] void
frame::oops_do_
internal(OopClosure*,CodeBlobClosure*,RegisterMap*,bool)+0x901
V [libjvm.so+0x9805c1] void
JavaThread::oops_do(OopClosure*,CodeBlobClosure*)+0x1f5
V [libjvm.so+0x98619a] void
Threads::oops_do(OopClosure*,CodeBlobClosure*)+0x3a
V [libjvm.so+0x8db11a] void
SharedHeap::process_strong_
roots(bool,bool,SharedHeap::ScanningOption,OopClosure*,
CodeBlobClosure*,OopsInGenClosure*)+0xc2
V [libjvm.so+0x5718db] void
GenCollectedHeap::gen_process_strong_
roots(int,bool,bool,bool,SharedHeap::ScanningOption,
OopsInGenClosure*,bool,OopsInGenClosure*)+0x5b
V [libjvm.so+0x574594] void
GenMarkSweep::mark_sweep_phase1(int,bool)+0x88
V [libjvm.so+0x573e5d] void
GenMarkSweep::invoke_at_safepoint(int,ReferenceProcessor*,bool)+0x179
V [libjvm.so+0x579d9e] void
OneContigSpaceCardGeneration::collect(bool,bool,unsigned,bool)+0x8a
V [libjvm.so+0x57116a] void
GenCollectedHeap::do_collection(bool,bool,
unsigned,bool,int)+0x676
V [libjvm.so+0x572743] void
GenCollectedHeap::do_full_collection
(bool,int)+0x4f
V [libjvm.so+0x217f6a] void VM_GenCollectFull::doit()+0xa6
V [libjvm.so+0x1d05df] void VM_Operation::evaluate()+0x77
V [libjvm.so+0x14aa82] void VMThread::loop()+0x496
V [libjvm.so+0x14a4d0] void VMThread::run()+0x98
V [libjvm.so+0x85a40d] java_start+0xaf5
C [libc.so.1+0xbd673] _thrp_setup+0x9b
C [libc.so.1+0xbd920] _lwp_start+0x0
The crash happened in objArrayKlass::oop_follow_contents(oopDesc*) at program counter (pc) 0xfe20af22. The preceding listing taken from the hs_err file shows the stack trace of the crash.
When the running process crashes, a core file gets generated. Let’s debug this core file with HSDB and try to find the cause of this crash.
Launch HSDB to open the core file using the following command, and then, as shown in Figure 4.51, specify the location of the core file and location of the Java executable that was used to run the program:
oracle@solaris_11X $ export PATH=$JAVA_HOME/bin/:$PATH
oracle@solaris_11X $ java -classpath $JAVA_HOME/lib/sa-jdi.jar
sun.jvm.hotspot.HSDB
Figure 4.52 shows the disassembly of the code that was being executed around PC 0xfe20af22 when the crash happened.
The program counters and corresponding assembly instructions shown in Figure 4.52 indicate that the process crashed when trying to access the value at address eax+100, which is at program counter 0xfe20af22. From the hs_err file, we can see the contents of the registers, and the value in the EAX register was:
EAX=0x2e617661, EBX=0xe7ab6e00, ECX=0x4a206f6c, EDX=0x00000004
ESP=0xe783ff90, EBP=0xe783ffc8, ESI=0xe7a4ea68, EDI=0x00000000
EIP=0xfe20af22, EFLAGS=0x00010297
To find out what was at 0x2e617661 and determine why the crash happened while attempting to read the value at 0x2e617661+100 we can use HSDB’s Find Pointer panel, shown in Figure 4.53, to see that this address does not lie in the Java heap.
Using the Find Address in Heap tool, we can find all the locations in the Java heap from where this particular address is referenced. See Figure 4.54.
Now, examine these found locations in the Object Inspector (see Figure 4.55) to see if the addresses lie within some valid objects.
All the found addresses bring up the byte array object at 0xe7a4ea38 in the Object Inspector, which means the object at 0xe7a4ea38 is the closest valid object just before these locations. If we look carefully, these locations actually go beyond the limits of the byte array object, which should have ended at 0xe7a4ea48, and from address 0xe7a4ea48 the next object should have started. See the raw contents at memory location 0xe7a4ea38 in Figure 4.56.
We can look at the raw bits as characters in the dbx debugger (shown below). This clearly shows that the object at 0xe7a4ea38 has a byte stream that goes beyond its size limit of three elements and overwrites the object starting at 0xe7a4ea48.
(dbx) x 0xe7a4ea38/100c
0xe7a4ea38: ' 01' ' ' ' ' ' ' ' 30' ' 07' '' '' ' 03' ' '
' ' ' ' 'H' 'e' 'l' 'l'
0xe7a4ea48: 'o' ' ' ' 'J' 'a' 'v' 'a' '.' 'H' 'e' 'l' 'l' 'o' ' ' 'J'
'a' 'v'
0xe7a4ea58: 'a' '.' 'H' 'e' 'l' 'l' 'o' ' ' 'J' 'a' 'v' 'a' '.' 'H'
'e' 'l'
0xe7a4ea68: ' 03' ' ' ' ' ' ' 'a' 'v' 'a' '.' 'H' 'e' 'l' 'l' 'o'
' ' 'J' 'a'
0xe7a4ea78: 'v' 'a' '.' 'H' 'e' 'l' 'l' 'o' ' ' 'J' 'a' 'v' 'a' '.'
'H' 'e'
0xe7a4ea88: 'l' 'l' 'o' ' ' 'J' 'a' 'v' 'a' '.' 'H' 'e' 'l' 'l' 'o' ' '
'J'
0xe7a4ea98: 'a' 'v' 'a' '.'
This gives us a big clue. Now, we can easily search in the code where the bytes “Hello Java.Hello Java. . .” are being written and find the buggy part of the code that overflows a byte array. The following shows the faulty lines in our JNI code:
jclass cls = env->GetObjectClass(obj);
jfieldID fldID = env->GetFieldID(cls, "array", "[B");
jbyteArray arrayObject = (jbyteArray)env->GetObjectField(obj, fldID);
jbyte* array = (jbyte*)env->GetPrimitiveArrayCritical(arrayObject, 0);
memcpy(array, "Hello Java.Hello
Java.Hello Java.Hello Java.Hello Java.Hello
Java.Hello Java.Hello Java.Hello Java.Hello Java.", 100);
env->ReleasePrimitiveArrayCritical(arrayObject, array, 0);
The line memcpy(array, "Hello Java.Hello Java.Hello Java.Hello Java.Hello Java.Hello Java.Hello Java.Hello Java.”, 100); is the culprit, which is writing to the byte array beyond its size limit.
Hmm . . . wasn’t this fun? Debugging a crash with the SA tools!