Though the details differ somewhat, all garbage collectors work by splitting the heap into different generations. These are called the old (or tenured) generation, and the young generation. The young generation is further divided into sections known as eden and the survivor spaces (though sometimes, eden is incorrectly used to refer to the entire young generation).

The rational for having separate generations is that many objects are used for a very short period of time. Take, for example, the loop in the stock price calculation where it sums the square of the difference of price from the average price (part of the calculation of standard deviation):

sum = new BigDecimal(0);
for (StockPrice sp : prices.values()) {
    BigDecimal diff = sp.getClosingPrice().subtract(averagePrice);
    diff = diff.multiply(diff);
    sum = sum.add(diff);
}

Like many Java classes, the BigDecimal class is immutable: the object represents a particular number and cannot be changed. When arithmetic is performed on the object, a new object is created (and often, the previous object with the previous value is then discarded). When this simple loop is executed for a year’s worth of stock prices (roughly 250 iterations), 750 BigDecimal objects are created to store the intermediate values just in this loop. Those objects are discarded on the next iteration of the loop. Within the add() and other methods, the JDK library code creates even more intermediate BigDecimal (and other) objects. In the end, a lot of objects are created and discarded very quickly in this very small amount of code.

This kind of operation is quite common in Java, and so the garbage collector is designed to take advantage of the fact that many (and sometimes most) objects are only used temporarily. This is where the generational design comes in. Objects are first allocated in the young generation, which is some subset of the entire heap. When the young generation fills up, the garbage collector will stop all the application threads and empty out the young generation. Objects that are no longer in use are discarded, and objects that are still in use are moved elsewhere. This operation is called a minor GC.

There are two performance advantages to this design. First, because the young generation is only a portion of the entire heap, processing it is faster than processing the entire heap. This means that the application threads are stopped for a much shorter period of time than if the entire heap were processed at once. You probably see a trade-off there, since it also means that the application threads are stopped more frequently than they would be if the JVM waited to perform GC until the entire heap were full; that trade-off will be explored in more detail later in this chapter. For now, though, it is almost always a big advantage to have the shorter pauses even though they will be more frequent.

The second advantage arises from the way objects are allocated in the young generation. Objects are allocated in eden (which comprises the vast majority of the young generation). When the young generation is cleared during a collection, all objects in eden are either moved or discarded: all live objects are moved either to one of the survivor spaces or to the old generation. Since all objects are moved, the young generation is automatically compacted when it is collected.

All GC algorithms have stop-the-world pauses during collection of the young generation.

As objects are moved to the old generation, eventually it too will fill up, and the JVM will need to find any objects within the old generation that are no longer in use and discard those objects. This is the area where GC algorithms have their biggest differences. The simpler algorithms stop all application threads, find the unused objects and free their memory, and then compact the heap. This process is called a full GC, and it generally causes a long pause for the application threads.

On the other hand, it is possible—though more computationally complex—to find unused objects while application threads are running; CMS and G1 both take that approach. Because the phase where they scan for unused objects can occur without stopping application threads, CMS and G1 are called concurrent collectors. They are also called low-pause (and sometimes—incorrectly—pauseless) collectors, since they minimize the need to stop all the application threads. Concurrent collectors also take different approaches to compacting the old generation.

When using the CMS or G1 collector, an application will typically experience fewer (and much shorter) pauses. The trade-off is that the application will use more CPU overall. CMS and G1 may also perform a long, full GC pause (and avoiding those is one of the key factors to consider when tuning those algorithms).

As you consider which garbage collector is appropriate for your situation, think about the overall performance goals that must be met. There are trade-offs in every situation. In an application (such as a Java EE server) measuring the response time of individual requests, consider these points:

Similarly, the choice of garbage collector in a batch application is guided by the following trade-off:

The JVM provides four different algorithms for performing GC.

The choice of a GC algorithm depends in part what the application looks like, and in part on the performance goals for the application.

The serial collector is best used only in those cases where the the application uses less than 100 MB. That allows the application to request a small heap, one that isn’t going to be helped by the parallel collections of the throughput collector, nor by the background collections of CMS or G1.

That sizing guideline limits the usefulness of the serial collector. Most programs will need to choose between the throughput collector and a concurrent collector; that choice is most influenced by the performance goals of the application.

An overview of this topic was discussed in Chapter 2: there is a difference in measuring an application’s elapsed time, throughput, or average (or 90th%) response times.

GC Algorithms and Batch Jobs

For batch jobs, the pauses introduced by the throughput collector—and particularly the pauses of the full GC cycles—are a big concern. Each one adds a certain amount of elapsed time to the overall execution time. If each full GC cycle takes 0.5 seconds and there are 20 such cycles during the five-minute execution of a program, then the pauses have added a 3.4% performance penalty: without the pauses, the program would have completed in 290 rather than 300 seconds.

If extra CPU is available (and that might be a big if), then using a concurrent collector will give the application a nice performance boost. The key here is whether adequate CPU is available for the background processing of the concurrent GC threads. Take the simple case of a single-CPU machine where there is a single application thread that consumes 100% of the CPU. When that application is run with the throughput collector, then GC will periodically run, causing the application thread to pause. When the same application is run with a concurrent collector, the operating system will sometimes run the application thread on the CPU, and sometimes run the background GC thread. The net effect is the same: the application thread is effectively paused (albeit for much shorter times) while the OS is running other threads.

The same principle applies in the general case when there are multiple application threads, multiple background GC threads, and multiple CPUs. If the operating system can’t run all the application threads at the same time as the background GC threads, then the competition for the CPU has effectively introduced pauses into the behavior of the application threads.

Table 5-1 shows how this trade-off works. The batch application calculating stock data has been run in a mode that saves each set results in memory for a few minutes (to fill up the heap); the test was run with a CMS and throughput GC algorithm.

Table 5-1. Batch processing time with different GC Algorithms

GC Algorithm	4 CPUs (CPU utilization)	1 CPU (CPU utilization)
CMS	78.09 (30.7%)	120.0 (100%)
Throughput	81.00 (27.7%)	111.6 (100%)

The times in this table are the number of seconds required to run the test, and the CPU utilization of the machine is shown in parenthesis. When four CPUs are available, CMS runs a batch of operations about three seconds faster than the throughput collector—but notice the amount of CPU utilized in each case. There is a single application thread which will continually run, so with four CPUs, that application thread will consume 25% of the available CPU.

The extra CPU reported in the table comes from the extra processing introduced by the GC threads. In the case of CMS, the background thread periodically consumes an entire CPU, or 25% of the total CPU available on the machine. The background thread here runs periodically—it turned out to run about 5% of the time—leaving an average CPU utilization of 30%.

Similarly, the throughput collector runs four garbage collection threads. During GC cycles, those threads consume 100% of the four available CPUs, leaving a 28% average usage during the entire test. During minor collections, CMS will also run four GC threads that consume 100% of the four CPUs.

When only a single CPU is available, the CPU is always busy—whether running the application or GC threads. Now the extra overhead of the CMS background thread is a drawback, and the throughput collector finishes nine seconds sooner.

Average CPU usage and GC

Looking at only the average CPU during a test misses the interesting picture of what happens during GC cycles. The throughput collector will (by default) consume 100% of CPU available on the machine while it runs, so a more accurate representation of the CPU usage during this test is shown in Figure 5-2.

Figure 5-2. Actual vs. Average CPU usage (Throughput)

Most of the time, only the application thread is running, consuming 25% of total cpu. When GC kicks in, 100% of CPU is consumed. Hence, the actual CPU usage resembles the saw-tooth pattern in the graph, even though the average during the test is reported as the value of the straight dashed line.

The effect is different in a concurrent collector, when there are background thread(s) running concurrently with the application threads. In that case, a graph of the CPU might look like Figure 5-3.

Figure 5-3. Actual vs. Average CPU usage (CMS)

The application thread starts by using 25% of total CPU. Eventually it has created enough garbage for the CMS background thread to kick in; that thread also consumes an entire CPU bringing the total up to 50%. When the CMS thread finishes, CPU usage drops to 25%, and so on. Note that there are no 100% peak CPU periods, which is a little bit of a simplification: there will be very short spikes to 100% CPU usage during the CMS young generation collections, but those are short enough that we can ignore them for this discussion.

There can be multiple background threads in a concurrent collector, but the effect is similar: when those background threads run, they will consume CPU and drive up the long-term CPU average.

This can be quite important when you have a monitoring system triggered by CPU usage rules: you want to make sure CPU alerts are not triggered by the 100% CPU usage spikes in a full GC, or the much longer (but lower) spikes from background concurrent processing threads. These spikes are normal occurrences for Java programs.

Quick Summary

1. Batch jobs with application threads that utilize most of the CPU will typically get better performance from the throughput collector.
2. Batch jobs that do not consume all the available CPU on a machine will typically get better performance with a concurrent collector.

The first basic tuning for GC is the size of the application’s heap. There are advanced tunings that affect the size of the heap’s generations; as a first step, this section will discuss setting the overall heap size.

Like most performance issues, choosing a heap size is a matter of balance. If the heap is too small, the program will spend too much time performing GC and not enough time performing application logic. But simply specifying a very large heap isn’t necessarily the answer either. The time spent in GC pauses is dependent on the size of the heap, so as the size of the heap increases, the duration of those pauses also increases. The pauses will occur less frequently, but their duration will make the overall performance lag.

A second danger arises when very large heaps are used. Computer operating systems use virtual memory to manage the physical memory of the machine. A machine may have 8 GB of physical RAM, but the OS will make it appear as if there is much more memory available. The amount of virtual memory is subject to the OS configuration, but say the OS makes it look like there is 16 GB of memory. The OS manages that by a process called swapping.^[24] You can load programs that use up to 16 GB of memory, and the OS will copy inactive portions of those programs to disk. When those memory areas are needed, the OS will copy them from disk to RAM (and usually, it will first need to copy something from RAM to disk to make room).

This process works well for a system running lots of different applications, since most of the applications are not active at the same time. It does not work so well for Java applications. If a Java program with a 12 GB heap is run on this system, the OS can handle it by keeping 8 GB of the heap in RAM and 4 GB on disk (that simplifies the actual situation a little, since other programs will use part of RAM). The JVM won’t know about this; the swapping is transparently handled by the OS. Hence, the JVM will happily fill up all 12GB of heap it has been told to use. This causes a severe performance penalty as the OS swaps data from disk to RAM (which is an expensive operation to begin with).

Worse, the one time this swapping is guaranteed to occur is during a full GC, when the JVM must access the entire heap. If the system is swapping during a full GC, pauses will be an order of magnitude longer than they would otherwise be. Similarly, if a concurrent collector is used, when the background thread sweeps through the heap, it will likely fall behind due to the long waits for data to be copied from disk to main memory—resulting in an expensive concurrent mode failure.

Hence, the first rule is sizing a heap is never to specify a heap that is larger than the amount of physical memory on the machine—and if there are multiple JVMs running, that applies to the sum of all their heaps. You also need to leave some room for the JVM itself as well as some memory for other applications—typically, at least 1GB of space for common OS profiles.

The size of the heap is controlled by two values: an initial value (specified with -XmsN) and a maximum value (-XmxN). The defaults for these vary depending on the operating system, the amount of system RAM, and the JVM in use. The defaults can also be affected by other flags on the command line as well; heap sizing is one of the JVM’s core ergonomic tunings.

The goal of the JVM is to find a “reasonable” default initial value for the heap based on the system resources available to it, and to grow the heap up to a “reasonable” maximum if (and only if) the application needs more memory (based on how much time it spends performing GC). Absent some of the advanced tunings flags and details discussed later in this and the next chapters, the default values for the initial and maximum sizes are given in Table 5-4.^[25]

On a machine with less than 192MB of physical memory, the maximum heap size will be half of the physical memory (96MB or less).

Having an initial and maximum size for the heap allows the JVM to tune its behavior depending on the workload. If the JVM sees that it is doing too much GC with the initial heap size, it will continually increase the heap until the JVM is doing the “correct” amount of GC, or until the heap hits its maximum size.

For many applications, that means a heap size doesn’t need to set at all. Instead, you specify the performance goals for the GC algorithm: the pause times you are willing to tolerate, the percentage of time you want to spend in GC, and so on. The details of that will depend on the GC algorithm used and are discussed in the next chapter (though even then, the defaults are chosen such that for a wide range of applications, those values needn’t be tuned either).

That approach usually works fine if an application does not need a larger heap than the default maximum for the platform it is running on. However, if the application is spending too much time in GC, then the heap size will need to be increased by setting the -Xmx flag. There is no hard-and-fast rule on what size to choose (other than not specifying a size larger than the machine can support). A good rule of thumb is to size the heap so that it is 30% occupied after a full GC. To calculate this, run the application until it has reached a steady-state configuration: a point at which it has loaded anything it caches, has created a maximum number of client connections, and so on. Then connect to the application with jconsole force a full GC and observe how much memory is used when the full GC completes.^[26]

Be aware that the automatic sizing of the heap occurs even if you explicitly set the maximum size: the heap will start at its default initial size, and the JVM will grow the heap in order to meet the performance goals of the GC algorithm. There isn’t necessarily a memory penalty for specifying a larger heap than is needed: it will only grow enough to meet the GC performance goals.

On the other hand, if you know exactly what size heap the application needs, then you may as well set both the initial and maximum values of the heap to that value (e.g., -Xms4096m -Xmx4096m). That makes GC slightly more efficient, because it never needs to figure out whether the heap should be resized.

Once the heap size has been determined, you (or the JVM) must decide how much of the heap to allocate to the young generation, and how much to allocate to the old generation. The performance implication of this should be clear: if there is a relatively larger young generation, then it will be collected less often, and fewer objects will be promoted into the old generation. But on the other hand, because the old generation is relatively smaller, it will fill up more frequently and do more full GCs. Striking a balance here is the key.

Different GC algorithms attempt to strike this balance in a different way. However, all GC algorithms use the same set of flags to set the sizes of the generations; this section covers those common flags.

The command-line flags to tune the generation sizes all adjust the size of the young generation; the old generation gets everything that is left over. There are a variety of flags that can be used to size the young generation:

The young generation is first sized by the NewRatio, which has a default value of 2. Parameters that affect the sizing of heap spaces are generally specified as as ratios; the value is used in an equation to determine the percentage of space affected. The NewRatio value is used in this formula:

Initial Young Gen Size = Initial Heap Size / (1 + NewRatio)

Plugging in the initial size of the heap and the NewRatio yields the value that becomes the setting for the young generation. By default, then, the young generation starts out at 33% of the initial heap size.

Alternately, the size of the young generation can be set explicitly by specifying the NewSize flag. If that option is set, it will take precedence over the value calculated from the NewRatio, There is no default for this flag (though PrintFlagsFinal will report a value of 1 MB). If the flag isn’t set, the initial young generation size will be based on the NewRatio calculation.

As the heap expands, the young generation size will expand as well, up to the maximum size specified by the MaxNewSize flag. By default, that maximum is also set using the NewRatio value, though it is based on the maximum (rather than initial) heap size.

Tuning the young generation by specifying a range for its minimum and maximum sizes ends up being fairly difficult. When a heap size is fixed (by setting -Xms equal to -Xmx), it is usually preferable to use -Xmn to specify a fixed size for the young generation as well. If an application needs a dynamically-sized heap and requires a larger (or smaller) young generation, then focus on setting the NewRatio value.

When the JVM loads classes, it must keep track of certain metadata about those classes. From the perspective of an end-user, this is all just bookkeeping information. This data is held in a separate heap space. In Java 7, this is called the permgen (or permanent generation), and in Java 8, this is called the metaspace.

Permgen and metaspace are not exactly the same thing. In Java 7, permgen contains some miscellaneous objects that are unrelated to class data; these are moved into the regular heap in Java 8. Java 8 also fundamentally changes the kind of metadata that is held in this special region—though since end users don’t know what that data is in the first place, that change doesn’t really affect us. As an end user, all we need to know is that permgen/metaspace holds a bunch of class-related data, and that there are certain circumstances where the size of that region needs to be tuned.

Note that permgen/metaspace does not hold the actual instance of the class (the Class objects), nor reflection objects (e.g., Method objects); those are held in the regular heap. Information in permgen/metaspace is really only used by the compiler and JVM runtime, and the data it holds is referred to as class metadata.

There isn’t a good way to calculate in advance how much space a particular program needs for its permgen/metaspace. The size will be proportional to the number of classes it uses, so bigger applications will need bigger areas. One of the advantages to phasing out permgen is that the metaspace rarely needs to be sized—because (unlike permgen), metaspace will by default use as much space as it needs. Table 5-5 lists the default initial and maximum sizes for permgen and metaspace.

These memory regions behave just like a separate instance of the regular heap. They are sized dynamically based on an initial size and will increase as needed to a maximum size. For permgen, the sizes are specified via these flags: -XX:PermSize=N and -XX:MaxPermSize=N. Metaspace is sized with these flags: -XX:MetaspaceSize=N and -XX:MaxMetaspaceSize=N.

Resizing these regions requires a full GC, so it is an expensive operation. If there are a lot of full GCs during the startup of a program (as it is loading classes), it is often because permgen or metaspace is being resized, so increasing the initial size is a good idea to improve startup in that case. Java 7 applications that define a lot of classes should increase the maximum size as well. Application servers, for example, typically specify a maximum permgen size of 128 MB, 192 MB, or more.

Contrary to its name, data stored in permgen is not permanent (metaspace, then, is a much better name). In particular, classes can be eligible for GC just like anything else. This is a very common occurrence in an application server, which creates new class loaders every time an application is deployed (or redeployed). The old class loaders are then unreferenced and eligible for GC, as are any classes which they defined. In a long development cycle in an application server, it is not unusual to see full GCs triggered during deployment: permgen or metaspace has filled up with the new class information, but the old class metadata can be freed.

Heap dumps (see Chapter 7) can be used to diagnose what classloaders exist, which in turn can help determine if a classloader leak is filling up permgen (or metaspace). Otherwise, jmap can be used with the argument -permstat (in Java 7) or -clstats (in Java 8) to print out information about the class loaders. That particular command isn’t the most stable, though, and it cannot be recommended.

All GC algorithms except for the serial collector use multiple threads. The number of these threads is controlled by the -XX:ParallelGCThreads=N flag. The value of this flag affects the number of threads used for the following operations:

Because these GC operations stop the application threads from executing, the JVM attempts to use as many CPU resources as it can in order to minimize the pause time. By default, that means the JVM will run one thread for each CPU on a machine, up to eight. Once that threshold has been reached, the JVM only adds a new thread for every 5/8ths of a CPU. So the total number of threads (where N is the number of CPUs on the machine) on a machine with more than eight CPUs is:

ParallelGCThreads = 8 + ((N - 8) * 5 / 8)

There are times when this number is too large. An application using a small heap (say, 1 GB) on a machine with eight CPUs will be slightly more efficient with four or six threads dividing up that heap. On a 128-CPU machine, 83 GC threads is too many for all but the largest heaps.

Additionally, if more than one JVM are running on the machine, it is a good idea to limit the total number of GC threads among all JVMs. When they run, the GC threads are quite efficient and each will consume 100% of a single CPU (this is why the CPU usage for the throughput collector was higher than expected in previous examples). In machines with eight or fewer CPUs, GC will consume 100% of the CPU on the machine. On machines with more CPUs and multiple JVMs, there will still be too many GC threads running in parallel.

Take the example of a 16-CPU machine running four JVMs; each JVM will have by default 13 GC threads. If all four JVMs execute GC at the same time, the machine will have 52 CPU-hungry threads contending for CPU time. That results in a fair amount of contention; it will be more efficient if each JVM is limited to four GC threads. Even though it may be unlikely for all four JVMs to perform a GC operation at the same time, one JVM executing GC with 13 threads means that the application threads in the remaining JVMs now have to compete for CPU resources on a machine where 13 of 16 CPUs are 100% busy executing GC tasks. Giving each JVM four GC threads provides a better balance in this case.

Note that this flag does not set the number of background threads used by CMS or G1 (though it does affect that). Details on that are given in the next chapter.

The sizes of the heap, the generations, and the survivor spaces can vary during execution as the JVM attempts to find the optimal performance according to its policies and tunings.

This is a best-effort solution, and it relies on past performance: the assumption is that future GC cycles will look similar to the GC cycles in the recent past. That turns out to be a reasonable assumption for many workloads, and even if the allocation rate suddenly changes, the JVM will re-adapt its sizes based on the new information.

Adaptive sizing provides benefits in two important ways. First, it means that small applications don’t need to worry about over-specifying the size of their heap. Consider the administrative command-line programs used to adjust the operations of things like an application server—those programs are usually very short-lived and use minimal memory resources. These applications will use 16 (or 64) MB of heap even though the default heap could potentially grow to 1 GB. Because of adaptive sizing, applications like that don’t need to be specifically tuned; the platform defaults ensure that they will not use a large amount of memory.

Second, it means that many applications don’t really need to worry about tuning their heap size at all—or if they need a larger heap than the platform default, they can just specify that larger heap and forget about the other details. The JVM can auto-tune the heap and generation sizes to use an optimal amount of memory given the GC algorithm’s performance goals. Adaptive sizing is what allows that auto-tuning to work.

Still, doing the adjustment of the sizes takes a small amount of time—time which occurs for the most part during a GC pause. If you have taken the time to finely-tune GC parameters and the size constraints of the application’s heap, adaptive sizing can be disabled. Disabling adaptive sizing is also useful for applications that go through markedly different phases, and you want to optimally tune GC for one of those phases.

At a global level, adaptive sizing is disabled by turning off the -XX:-UseAdaptiveSizePolicy flag (which is true by default). With the exception of the survivor spaces (which are examined in detail in the next chapter), adaptive sizing is also effectively turned off if the minimum and maximum heap sizes are set to the same value, and the initial and maximum sizes of the new generation are set to the same value.

In order to see how the JVM is resizing the spaces in an application, set the -XX:+PrintAdaptiveSizePolicy flag. When a GC is performed, the GC log will contain information detailing how the various generations were resized during a collection.