The overhead of using JRA proved so low that we started considering the notion of always having it enabled when the JVM is running. The project to implement such a recording engine was internally first known as continuous JRA. In the R28 version of JRockit, this has finally been productized and named JRockit Flight Recorder.
As near-zero overhead data is continuously stored about JVM behavior, the JRockit Flight Recorder allows us to go back in time and analyze the behavior of the application and the JVM even after something has gone wrong. This is a very powerful feature for JVM and application forensics—the recording being the "black box" that contains information on all events leading up to a problem. Naturally, the framework still works well as a profiling and instrumentation tool, which will likely remain the most common use case.
In this chapter, you will learn:
- How JRockit Flight Recorder works
- About the Flight Recorder event model
- How to start a continuous recording
- How to start a JRA-style recording
- How recordings interact in JRockit Flight Recorder
- How to configure the various aspects of JRockit Flight Recorder
- The main differences between JRockit Flight Recorder and the JRockit Runtime Analyzer
- How to record custom events
- How to design custom extensions to the JRockit Flight Recorder client
- About future tools, APIs, and projects around JRockit Flight Recorder
Just like the JRockit Runtime Analyzer, the JRockit Flight Recorder consists of two parts—a recording engine built into the JRockit JVM and an analysis tool built into the JRockit Mission Control client. The recording engine produces a recording file that can be analyzed. The file does not require an active connection; the format is self describing, that is, all metadata that the recording needs is part of the recording and can be saved for later or sent to a third party for further analysis.
Note
Throughout this chapter, we will use the terms JRockit Flight Recorder and Flight Recorder interchangeably.
With JRockit Flight Recorder, recordings are no longer in XML format. Everything is recorded as time-stamped events in internal memory buffers and written to a binary file, which constitutes the recording. There is also a public Java API available for providing custom events to the JRockit recording engine, and a design mode that allows the creation of custom designed user interfaces from within the analysis tool itself.
Note
Recall from Chapter 8, The Runtime Analyzer, that a JRA recording is in XML format and the files are suffixed .jra. With Flight Recorder, the files are suffixed .jfr and are in a binary format. Because of the immense amount of events produced, there is a need to avoid unnecessary overhead and consequently to store events in a more compact way. JRA recordings are not forward-compatible and cannot be opened with JRockit Flight Recorder.
As mentioned, data is recorded as events. An event is simply data recorded at a specific time.
There are four different types of events:
- Duration events: Duration events are events that last over a duration of time or, in other words, events with a start time and an end time. The Garbage Collection event is an example of a duration event.
- Timed events: The timed events are duration events for which a threshold can be set. The concept of a threshold is discussed in detail later in this chapter. The Java Wait and Java Sleep events are examples of timed events.
- Instant events: Instant events have no duration, only a start time. The Exception and the Event Settings Changed events are examples of instant events.
- Requestable events: Requestable events can be configured to be polled periodically by the recording engine. The event implements a callback that a separate thread in the recording engine will call at specified intervals. An example of a requestable event is the CPU Load Sample event.
Events are produced by event producers. An event producer defines the types of events being produced, also known as event types, as well as the actual events. An event type contains metadata that describes how the events of that type will look. The metadata contains information such as what attributes (also known as fields) the event contains, of what types the attributes are, and human-readable descriptions of the attributes. Every recording file contains information about its event producers.
An event producer with the imaginative and thought-provoking name "JRockit JVM" is already built into JRockit. The main advantage of the JRockit JVM producer, just as with JRA, is that it cheaply records information that the runtime already needs to collect as part of doing its job. Using the Java API, which will be discussed later, anyone can contribute events to bring additional context to the lower level events created by the JRockit JVM producer.
The recording engine, also known as the recording agent, is part of the JVM itself and provides highly optimized services for the event producers. A few examples are:
- A recording facility: Of course, the main purpose of the recording engine is to record events. It does this highly efficiently by providing thread local buffers where the events are recorded and a scheme to transfer these to a global buffer when the thread local buffers are full. Once the global buffer is full, it is either emitted to disk in the previously mentioned compact binary format, or reused in a circular fashion depending on the configuration. See the next figure.
- Stack traces: If the event producer sets the appropriate options, the stack trace that generated the event will be recorded along with the event. This is useful for finding where, in the source code of a Java application, an event originated.
- Threshold: The recording engine can be configured to only include duration events that last longer than a specified amount of time. This is used to both limit the amount of data in the recording, and to filter out data that would be of no interest anyway. The threshold can be configured per event type.
- Highly optimized time stamping of events: As was briefly discussed in Chapter 5, Benchmarking and Tuning, using
System.currentTimeMillis()for getting the system time can be a much more expensive operation than one would think. The recording engine provides a highly optimized native implementation for timestamping events.
Note
As JRockit R28 is released as part of a patch release of the application stack, it was decided that enabling the default recording was too big a change. The recording engine is enabled in R28, but there is no recording running from the start of the JVM. Both the recording engine and the JRockit JVM event producer have been designed and tested for always being enabled. As a matter of fact, all of the testing, including the stress testing of JRockit R28, was performed with the default recording enabled. The default recording will most likely be enabled out of the box in future releases of JRockit.
Ongoing recordings in the JVM have an associated recording ID, which is unique, and a recording name, which does not have to be. The recording ID is automatically assigned to the recording when it is created, and can be used for identifying a recording. The recording ID can, for example, be used when referring to a recording from JRCMD.
To enable the Flight Recorder and to configure it to continuously record data with the default settings, simply start JRockit with the following option:
-XX:FlightRecorderOptions=defaultrecording=true
This will create a recording with recording ID 0 and the name JRockit default.
There can be an arbitrary amount of recordings running in JRockit Flight Recorder at any given time. If more than one recording is active, the recorded data will contain events from the union of the enabled event types and use the threshold found for each type. For users new to the Flight Recorder, this can be quite confusing, as there may actually be more events in the recording than asked for.
To configure the engine to record more detailed information, you can either change the event settings of the default recording, or start a new recording with different settings. To record less information, the event settings of any ongoing recording must be changed.
There are various ways to configure the different aspects of the recording engine, some of which are only available from the command line when starting the JVM.
There are two main command-line parameters. As previously mentioned, the first one turns on (+) or off (-) the Flight Recorder altogether:
-XX:[+|-]FlightRecorder
The second is for controlling the Flight Recorder:
The available parameters are:
|
Parameter |
Description |
|---|---|
|
|
Loads additional event settings from this server-side template. The default templates available under |
|
|
The base directory where the Flight Recorder will emit chunks of data. This can be seen as the temporary directory of the Flight Recorder. The default is a directory under |
|
|
The size to use for the thread local buffers. The default is 5 KB. |
|
|
The size to use for a single global buffer. The default is 64 KB. |
|
|
There may be more than one global buffer. This sets the number of global buffers to use. The default is 8. |
|
|
The maximum size of a single data chunk in the repository. The default is 12 MB. |
|
|
Enable the default continuous recording, that is the built-in continuous recording with recording ID 0. This does not enable or disable continuous recording as a concept. No matter what value this attribute has, continuous recordings can still be started from JRockit Mission Control as long as the Flight Recorder engine itself is enabled. As mentioned, the default recording is disabled in JRockit R28.0, but is likely to be enabled by default in future versions of JRockit. |
|
|
Emit data to disk. This is disabled by default, meaning that circular in-memory buffers will be used instead. The contents of the buffers can be dumped to disk, either from Mission Control or by using JRCMD. |
|
|
Defines the maximum age of the data kept on disk. Data younger than this is retained. The time is specified in nanoseconds by default. The default value is 0, which means the age check is ignored and that all data is retained. |
|
|
The maximum size of the data to keep on disk. The default value is 0, which means the size check is ignored. |
When using either JRCMD or the command-line options, specifying what information will actually be recorded is done through JSON-based template files. The JRockit distribution comes with several example templates available under JROCKIT_HOME/jre/lib/jfr. These templates are also known as server-side templates, as they differ from the ones used by the JRockit Mission Control client. The templates in the JRockit distribution serve as good examples, should you want to create your own server-side template.
Note
An in-depth discussion on the server-side templates is beyond the scope of this book. See the example templates for further information on server-side templates and Chapter 11, JRCMD, for information about using JRCMD to control the recording life cycle.
Just like with JRA, it is possible to use command-line options to start time-limited recordings. In the Flight Recorder, the parameter is called -XX:StartFlightRecording. This parameter is useful when wanting to do several recordings and compare them. It is for instance possible to delay a recording to give the application and the JVM time to warm up. The following example starts a recording after two minutes. The recording will last for one minute and will be named MyRecording. The resulting recording file ends up as C: mpmyrecording.jfr. Just like with -XX:FlightRecorderOptions, a server-side template can be referred to by name. The example uses the profile.jfs template.
-XX:StartFlightRecording=delay=120s,duration=60s, name=MyRecording,filename=C: mpmyrecording.jfr,settings=profile
Note
See the command-line reference for JRockit R28 for more information about the available parameters to the StartFlightRecording option.
The rest of this chapter focuses on using the JRockit Mission Control client for controlling the Flight Recorder and for viewing recordings.