7
Recurring challenges when running on Java 9 or later

This chapter covers

• Distinguishing standardized, supported, and internal JDK APIs
• Finding dependencies on JDK-internal APIs with JDeps
• Compiling and running code that depends on internal APIs
• Why a split package can make classes invisible
• Mending split packages

Chapter 6 discusses some problems you may come up against when migrating a project to Java 9+. Once you’re done with that, though, you aren’t going to encounter those issues again unless you pick up pre-Java 9 dependencies. This chapter explores two challenges you might still need to deal with:

• Relying on internal APIs leads to compile errors (section 7.1). This is true for JDK-internal APIs, such as classes from sun.* packages, but also for code internal to the libraries or frameworks you depend on.
• Splitting packages across artifacts causes compile-time and run-time errors (section 7.2). Again, this can happen between your code and JDK modules as well as between any other two artifacts: for example, your code and a third-party dependency.

Just like the problems we’ve discussed so far, you’ll also have to work through these two issues when getting your project to work on Java 9+, but it doesn’t stop there: you’ll occasionally encounter them, even after migration, when working on code or pulling in new dependencies. Dependencies on module internals and split packages cause trouble regardless of the kinds of modules involved. You’re just as likely to encounter them with class-path code and platform modules (the migration scenario) as with application modules (a scenario in which you’re already running on Java 9 or later and are using modules).

This chapter shows how to break a module’s encapsulation and how to mend package splits, regardless of the context in which these situations occur. Together with chapter 6, this prepares you for most things that could go wrong during a migration.

7.1 Encapsulation of internal APIs

One of the module system’s biggest selling points is strong encapsulation. As section 3.3 explains in depth, we can finally make sure only supported APIs are accessible to outside code while keeping implementation details hidden.

The inaccessibility of internal APIs applies to the platform modules shipped with the JDK, where only java.* andjavax.* packages are fully supported. As an example, this happens when you try to compile a class with a static dependency (meaning an import or a fully qualified class name, as opposed to reflective access) on NimbusLookAndFeel in the now-encapsulated package com.sun.java.swing.plaf.nimbus:

> error: package com.sun.java.swing.plaf.nimbus is not visible
> import com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel;
>                               ^
>     (package com.sun.java.swing.plaf.nimbus is declared
>      in module java.desktop, which does not export it)
> 1 error

Surprisingly, many libraries and frameworks, but also application code (often the more important parts), use classes from sun.* or com.sun.* packages, most of which are inaccessible from Java 9 on. In this section, I’ll show you how to find such dependencies and what to do about them.

But why discuss that? If internal APIs are inaccessible, there’s nothing to talk about, right? Well, it’s time to let you in on something: they’re not totally inaccessible. At run time, everything will continue to work until the next major Java release (although you may get some undesired warning messages); and with control over the command line, any package can be made accessible at compile time. (I think I just heard a sigh of relief—was that you?)

Section 9.1.4 discusses the broader implications of using command-line options to configure the module system; here we focus on solving the immediate problem. We’ll distinguish between static and reflective and between compile-time and run-time access (sections 7.1.3 and 7.1.4) because there are some critical differences. But before we get to that, you need to know exactly what constitutes an internal API and how the Java Dependency Analysis Tool (JDeps) can help find problematic code in your project and your dependencies.

When you’re done with this section, it will be an easy task for you to break open modules to benefit from APIs their maintainers didn’t want you to use. More important, you’ll be able to evaluate the benefits and drawbacks of that strategy, so you can make an informed decision about whether it’s worth going down that road.

7.1.1 Internal APIs under the microscope

Which APIs are internal? In general, every class that’s not public or not in an exported package—and this rule fully applies to application modules. Regarding the JDK, the answer isn’t that simple, though. On top of the already historically complicated situation with standardized, supported, and internal APIs, Java 9+ adds a layer of complexity by making a special case for some APIs and removing others. Let’s unravel the situation step by step.

Three kinds of JDK APIs: Standardized, supported, and internal

Historically speaking, the Java Runtime Environment (JRE) has three kinds of APIs:

• The public classes found in java.* and javax.* packages are standardized and fully supported across all JREs. Using only these makes for the most portable code.
• Some com.sun.* and jdk.* packages and some classes they contain are marked with the jdk.Exported annotation, in which case they’re supported by Oracle but not necessarily present in non-Oracle JREs. Depending on these binds code to specific JREs.
• Most com.sun.* packages and all sun.* packages as well as all non-public classes are internal and can change between different versions and JREs. Depending on these is the most unstable, because such code could theoretically stop working on any minor update.

With Java 9+ and the module system in play, these three kinds of APIs—standardized, supported, and internal—still exist. Whether a module exports a package is a key indicator but obviously doesn’t suffice to demarcate three categories. The other indicator is the module’s name. As you may recall from section 3.1.4, platform modules are split into those defined by the Java specification (prefixed with java.*) and JDK-specific ones (prefixed with jdk.*):

• The public classes found in packages exported by java.* modules (these can be java.* and javax.* packages) are standardized.
• The public classes found in packages exported by jdk.* modules aren’t standardized but are supported on Oracle’s and OpenJDK’s JDK.
• All other classes are internal APIs.

Which specific classes are standardized, supported, or internal is largely unchanged from Java 8 to Java 9+. As a consequence, many classes in com.sun.* and all classes in sun.* are internal APIs just as they were before. The difference is that the module system turns this convention into an actively enforced distinction. Figure 7.1 shows the split where internal APIs are not exported.

That the jdk.* modules aren’t standardized is only a convention, and the module system is unaware of it. So although it may not be wise to depend on their exported APIs, the JPMS won’t encapsulate them, and none of the command-line options we’ll discuss are necessary. Here, when I talk about internal APIs, I mean those the module system makes inaccessible because classes aren’t public or packages aren’t exported.

A special case for the infamous sun.misc.Unsafe

As you might imagine, the original idea was to encapsulate every API that was internal prior to Java 9. That caused a ruckus when the larger Java community realized it in 2015. Although the average Java developer may only occasionally use internal APIs, many of the best-known libraries and frameworks do so frequently, and some of their most-critical features depend on it.

The poster child for this situation is sun.misc.Unsafe, a class that, given its package name, is obviously internal. It offers functionality that’s uncommon for Java and, as the class name suggests, unsafe. (Talk about expressive names!) Maybe the best example is direct memory access, which the JDK has to perform occasionally.

But it went beyond the JDK. With Unsafe readily available, some libraries, particularly those focused on high performance, started using it; over time, large parts of the ecosystem ended up directly or indirectly depending on it. The prospect of that class and others like it getting encapsulated led to the community uproar.

Following that, the team working on Project Jigsaw decided to allow a smoother migration path. A survey of the existing internal APIs and their use outside the JDK yielded this result:

• Most affected APIs are rarely or never used.
• Some affected APIs are occasionally used, but standardized alternatives existed before Java 9. A prime example is the BASE64Encoder/BASE64Decoder pair in sun.misc, which can be replaced with java.util.Base64.
• Some affected APIs are used occasionally but deliver critical functionality, for which no alternatives exist. This is where sun.misc.Unsafe can be found.

The decision was made to encapsulate the first two kinds but leave the third accessible for at least another major Java version. Exporting them from their respective modules would be confusing, though, because it would make them look like supported or even standardized APIs, which they’re most definitely not. How better to make that point than by creating a suitably named module?

The critical APIs, for which no replacements existed before Java 9, are exported by the module jdk.unsupported. As the name suggests, it’s JDK-specific (only guaranteed to be present on Oracle JDK and OpenJDK) and unsupported (content may change in the next release). In Java 9 to 11, it contains the following classes:

• From sun.misc: Signal, SignalHandler, and Unsafe
• From sun.reflect: Reflection and ReflectionFactory
• From com.sun.nio.file: ExtendedCopyOption, ExtendedOpenOption, ExtendedWatchEventModifier, and SensitivityWatchEventModifier

If your code or dependencies depend on these classes (section 7.1.2 shows how to find out), then even though they were internal API before Java 9, you don’t need to do anything to keep using them. For now. As standardized alternatives for their functionality are released (like variable handles, which replace parts of Unsafe), they will be encapsulated. I strongly recommend you have a close look at your use of these classes and prepare for their eventual disappearance.

Removed APIs

Although some internal APIs remain available for a few more years and most have been encapsulated, a few met an even harsher fate and were removed or renamed. This breaks code that uses them beyond the reach of any transition period and command-line option. Here they are:

• Everything in sun.misc and sun.reflect that isn’t part of jdk.unsupported: for example, sun.misc.BASE64Encoder, sun.misc.BASE64Decoder, sun.misc.Cleaner, and sun.misc.Service
• com.sun.image.codec.jpeg and sun.awt.image.codec
• com.apple.concurrent
• com.sun.security.auth.callback.DialogCallbackHandler
• Methods addPropertyChangeListener and removePropertyChangeListener on java.util.logging.LogManager, java.util.jar.Pack200.Packer, and java.util.jar.Pack200.Unpacker (deprecated in Java 8)
• Methods with parameters or return types from java.awt.peer and java.awt.dnd.peer (these packages were never standardized and are internal in Java 9 and later)

Most of these classes and packages have alternatives, and you can use JDeps to learn about them.

7.1.2 Analyzing dependencies with JDeps

Now that we’ve discussed the distinction between standardized, supported, and internal APIs and the special case of jdk.unsupported, it’s time to apply that knowledge to a real-life project. For it to be compatible with Java 9+, you need to figure out which internal APIs it depends on.

Just going through the project’s code base won’t cut it—you’re in trouble if the libraries and frameworks it depends on cause problems, so you need to analyze them as well. This sounds like horrible manual work, sifting through a lot of code in the search for references to such APIs. Fortunately, there’s no need to do that.

Since Java 8, the JDK ships with the command-line Java Dependency Analysis Tool (JDeps). It analyses Java bytecode, meaning .class files and JARs, and records all statically declared dependencies between classes, which can then be filtered or aggregated. It’s a neat tool for visualizing and exploring the various dependency graphs I’ve been talking about. Appendix D provides a JDeps primer; you may want to read it if you’ve never used JDeps. It isn’t strictly necessary to understand this section, though.

One feature is particularly interesting in the context of internal APIs: the option --jdk-internals makes JDeps list all internal APIs that the referenced JARs depend on, including those exported by jdk.unsupported. The output contains the following:

• The analyzed JAR and the module containing the problematic API
• The specific classes involved
• The reason that dependency is problematic

I’m going to use JDeps on Scaffold Hunter, “a Java-based open source tool for the visual analysis of data sets.” The following command analyzes internal dependencies:

$ jdeps --jdk-internals    ①  

    -R --class-path 'libs/*'    ②  

    scaffold-hunter-2.6.3.jar    ③  

The output begins with mentions of split packages, which we’ll look at in section 7.2. It then reports on problematic dependencies, of which a few are shown next. The output is detailed and gives all the information you need to examine the code in question or open issues in the respective projects:

> batik-codec.jar -> JDK removed internal API    ①  
>     JPEGImageWriter -> com.sun.image.codec.jpeg.JPEGCodec    ②  
>         JDK internal API (JDK removed internal API)    ③  
>     JPEGImageWriter -> com.sun.image.codec.jpeg.JPEGEncodeParam
>         JDK internal API (JDK removed internal API)
>     JPEGImageWriter -> com.sun.image.codec.jpeg.JPEGImageEncoder
>         JDK internal API (JDK removed internal API)
# [...]
> guava-18.0.jar -> jdk.unsupported    ④  
>     Striped64 -> sun.misc.Unsafe    ⑤  
>         JDK internal API (jdk.unsupported)
>     Striped64$1 -> sun.misc.Unsafe
>         JDK internal API (jdk.unsupported)
>     Striped64$Cell -> sun.misc.Unsafe
>         JDK internal API (jdk.unsupported)
# [...]
> scaffold-hunter-2.6.3.jar -> java.desktop    ⑥  
>     SteppedComboBox -> com.sun.java.swing.plaf.windows.WindowsComboBoxUI
>         JDK internal API (java.desktop)
>     SteppedComboBox$1 -> com.sun.java.swing.plaf.windows.WindowsComboBoxUI
>         JDK internal API (java.desktop)

JDeps ends with the following note, which gives useful background information and suggestions for some of the discovered problems:

> Warning: JDK internal APIs are unsupported and private to JDK
> implementation that are subject to be removed or changed incompatibly
> and could break your application. Please modify your code to eliminate
> dependence on any JDK internal APIs. For the most recent update on JDK
> internal API replacements, please check:
> https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool
>
> JDK Internal API                           Suggested Replacement
> ----------------                           ---------------------
> com.sun.image.codec.jpeg.JPEGCodec         Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.JPEGDecodeParam   Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.JPEGEncodeParam   Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.JPEGImageDecoder  Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.JPEGImageEncoder  Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.JPEGQTable        Use javax.imageio @since 1.4
> com.sun.image.codec.jpeg.TruncatedFileException
>                                            Use javax.imageio @since 1.4
> sun.misc.Unsafe                            See JEP 260
> sun.reflect.ReflectionFactory              See JEP 260

7.1.3 Compiling against internal APIs

The purpose of strong encapsulation is that the module system by default doesn’t let you use internal APIs. This affects the compilation and run-time behavior of any Java version starting with 9. Here we discuss compilation—section 7.1.4 addresses run-time behavior. In the beginning, strong encapsulation will mostly be relevant for platform modules, but as your dependencies are modularized, you’ll see the same barrier around their code.

Sometimes, though, you may be in a situation where you absolutely have to use a public class in a non-exported package to solve the problem at hand. Fortunately, that’s possible even with the module system in place. (I’m stating the obvious, but I want to point out that this is only a problem for your code, because your dependencies are already compiled—they will still be impacted by strong encapsulation, but only at run time.)

Until now, exports were always untargeted, so being able to export to specific modules is a new aspect. This feature is available for module descriptors as well, as section 11.3 explains. Also, I’m being a little handwavy about what ALL-UNNAMED means. It’s connected to the unnamed module, which section 8.2 discusses in detail, but for now “all code from the class path” is a good approximation.

Let’s return to the code that caused the following compile error:

> error: package com.sun.java.swing.plaf.nimbus is not visible
> import com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel;
>                               ^
>     (package com.sun.java.swing.plaf.nimbus is declared
>      in module java.desktop, which does not export it)
> 1 error

Here, some class (which I omitted from the output because it’s irrelevant) imports NimbusLookAndFeel from the encapsulated package com.sun.java.swing.plaf.nimbus. Note how the error message points out the specific problem, including the module that contains the class.

This clearly doesn’t work out of the box on Java 9, but what if you want to keep using it? Then you’d likely be making a mistake, because there’s a standardized alternative in javax.swing.plaf.nimbus; on Java 10, only that version remains, because the internal version is removed. But for the sake of this example, let’s say you still want to use the internal version—maybe to interact with legacy code that can’t be changed.

All you have to do to successfully compile against com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel is to add --add-exports java.desktop/com.sun.java.swing.plaf.nimbus=ALL-UNNAMED to the compiler command. If you do that manually, it will look similar to the following (all placeholders would have to be replaced with concrete values):

$ javac
    --add-exports java.desktop/com.sun.java.swing.plaf.nimbus=ALL-UNNAMED
    --class-path ${dependencies}
    -d ${target-folder}
    ${source-files}

With a build tool, you’ll have to put the option somewhere in the build descriptor. Check your tool’s documentation to find out how to add command-line options for the compiler.

This way, code happily compiles against encapsulated classes. But it’s important to realize that you’ve only pushed the problem to run time! Adding this export on the command line only changes the one compilation—no information is put into the resulting bytecode that would allow that class to access the package during execution. You still have to figure out how to make it work at run time.

7.1.4 Executing against internal APIs

I mentioned that, at least in Java 9, 10, and 11, JDK-internal dependencies are still available at run time. With everything else I’ve been telling you, that should be a little surprising. Throughout the book, I’ve been touting the benefits of strong encapsulation and said it’s as important as visibility modifiers—so why isn’t it enforced at run time?

Like many other Java quirks, this one was born from a dedication to backward compatibility: strong encapsulation of JDK internals will break a lot of applications. Even if it’s just the outdated use of the Nimbus look and feel, the application will crash. How many end users or IT departments would install Java 9+ if legacy apps stopped working? How many teams would develop against Java 9+ if few users had it available?

To make sure the module system doesn’t split the ecosystem in “pre Java 9” and “post Java 9,” the decision was made to grant code on the class path illegal access to JDK-internal APIs until at least Java 11. Each of those aspects was chosen deliberately:

• Code on the class path … —Running code from the module path expresses that it has been prepared for the module system, in which case there’s no need to make an exception. It’s hence limited to class-path code.
• … to JDK-internal APIs —From a compatibility perspective, there’s no reason to grant access to application modules, because they didn’t exist before Java 9. So the exception is limited to platform modules.
• … at least Java 11—If the exception were permanent, the incentive to update troublesome code would be much lower.

As you saw in chapter 6, this doesn’t solve all problems an application may run into when being executed on Java 9, 10, or 11, but it will be more likely to run successfully.

Managing blanket illegal access to JDK-internal APIs

For a successful migration, it’s important to understand the details behind the blanket illegal access to JDK-internal APIs; but exploring it will make your mental model of the module system more complicated. It helps to keep the big picture in mind: strong encapsulation disallows access to all internal APIs at compile time and run time. On top of that, a big exception was built, whose specific design was driven by compatibility concerns. It will disappear over time, though, bringing us back to the much more clear-cut behavior.

When allowing class-path code to access JDK-internal APIs, a distinction is made between code that statically depends on them and code that accesses them reflectively:

• Reflective access results in warnings. Because it’s impossible to exactly identify all such calls by static analysis, execution is the only time to reliably report them.
• Static access results in no warning. It can easily be discovered during compilation or with JDeps. Due to the omnipresence of static access, it’s also a performance-sensitive area, where checking for and occasionally emitting log messages is problematic.

The exact behavior can be configured with a command-line option. The java option --illegal-access=${value} manages how illegal access to JDK-internal APIs is handled, where ${value} is one of the following:

• permit —Access to all JDK-internal APIs is permitted to code on the class path. For reflective access, a single warning is issued for the first access to each package.
• warn —Behaves like permit, but a warning is issued for each reflective access.
• debug —Behaves like warn, but a stack trace is included in each warning.
• deny —The option for those who believe in strong encapsulation: all illegal access is forbidden by default.

On Java 9 to 11, permit is the default value. In some future Java version, deny will become the default; and at some point the entire option may disappear, but I’m sure that will take a few more years.

It looks like once you get troubling code past the compiler, either by using the Java 8 version or by adding the required options to the Java 9+ version, the Java 9+ runtime will begrudgingly execute it. To see --illegal-access in action, it’s time to finally look at the class that plays around with the internal Nimbus look and feel:

import com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel;

public class Nimbus {

    public static void main(String[] args) throws Exception {
        NimbusLookAndFeel nimbus = new NimbusLookAndFeel();
        System.out.println("Static access to " + nimbus);

        Object nimbusByReflection = Class
                .forName("com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel")
                .getConstructor()
                .newInstance();
        System.out.println("Reflective access to " + nimbusByReflection);
    }

}

It doesn’t do anything particularly useful, but it clearly tries to access NimbusLookAndFeel both statically and reflectively. To compile it, you need to use --add-exports, as described in the previous section. Running it is simpler:

$ java --class-path ${class} j9ms.internal.Nimbus

> Static access to "Nimbus Look and Feel"
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by j9ms.internal.Nimbus
>     (file:...) to constructor NimbusLookAndFeel()
> WARNING: Please consider reporting this to the maintainers
>     of j9ms.internal.Nimbus
> WARNING: Use --illegal-access=warn to enable warnings of
>     further illegal reflective access operations
> WARNING: All illegal access operations will be denied in a
>     future release
> Reflective access to "Nimbus Look and Feel"

You can observe the behavior defined by the default option --illegal-access=permit: static access succeeds without comments, but reflective access results in a lengthy warning. Setting the option to warn would change nothing, because there’s only one access, and debug adds the stack trace for the troublesome call. With deny, you get the same messages you saw in section 3.3.3 when you tested the accessibility requirements:

$ java
    --class-path ${class}
    --illegal-access=deny
    j9ms.internal.Nimbus

> Exception in thread "main" java.lang.IllegalAccessError:
>     class j9ms.internal.Nimbus (in unnamed module @0x6bc168e5) cannot
>     access class com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel (in
>     module java.desktop) because module java.desktop does not export
>     com.sun.java.swing.plaf.nimbus to unnamed module @0x6bc168e5

There’s one more detail to discuss: what happens with illegal access to JDK internals introduced in Java 9? Because the --illegal-access option was introduced to ease migration, it would be a shame if it made the eventual transition harder by giving you a few years to start depending on new internal APIs. That’s indeed a risk!

The things that are done for compatibility—I told you it would get more complex. And I’m not done yet, because we can also manage illegal access more specifically (see the next section). Table 7.1 in section 7.1.5 then compares the different variants.

Managing specific illegal access to selected APIs

The illegal-access option is characterized by three central properties:

• It manages illegal access in a wholesale manner.
• It’s a transitional option that will eventually disappear.
• It bugs you with warnings.

What happens when it’s gone? Will strong encapsulation be insurmountable? The answer is no, it won’t be. There will always be edge cases that require access to internal APIs (of platform and application modules), and hence some mechanism (maybe not an overly comfortable one) should exist to make that possible. Once again, we turn to command-line options.

The class NimbusLookAndFeel is public, so all you need to do to properly access it is export the package that contains it. To make sure you observe the effect of --add-exports, deactivate the default permission of illegal access with --illegal-access=deny:

$ java
    --class-path ${class}
    --illegal-access=deny
    --add-exports java.desktop/com.sun.java.swing.plaf.nimbus=ALL-UNNAMED
    j9ms.internal.Nimbus

> Static access to ${Nimbus Look and Feel}
> Reflective access to ${Nimbus Look and Feel}

The reflective access goes through. Also notice that you don’t get a warning—more on that in a minute.

This covers access to public members of public types, but reflection can do more than that: with the generous use of setAccessible(true), it allows interaction with nonpublic classes as well as nonpublic fields, constructors, and methods. Even in an exported package, these members are encapsulated, though, so to successfully reflect over them, you need something else.

The option --add-opens uses the same syntax as --add-exports and opens the package to deep reflection, meaning all of its types and their members are accessible regardless of their visibility modifiers. Because of its primary relation to reflection, the option is more formally introduced in section 12.2.2.

Still, its use case is to access internal APIs, so it makes sense to look at an example here. A fairly common one is provided by tools generating instances of classes from other representations, for example JAXB creating a Customer instance from an XML file. Many such libraries rely on internals of the class-loading mechanism, for which they reflectively accessed nonpublic members of the JDK class ClassLoader. Note that there are plans to remove the –illegal-access option in a future version of Java, but Oracle has not yet decided which version.

If you run such code with --illegal-access=deny, you’ll get an error:

> Caused by: java.lang.reflect.InaccessibleObjectException:
>   Unable to make ClassLoader.defineClass accessible:
>   module java.base does not "opens java.lang" to unnamed module

The message is pretty clear—the solution is to use --add-opens when launching the application:

$ java
    --class-path ${jars}
    --illegal-access=deny
    --add-opens java.base/java.lang=ALL-UNNAMED
    ${main-class}

Unlike --illegal-access and its current default value permit, the options --add-exports and --add-opens can be seen as “the proper way” (or rather, “the least shady way”) to access internal APIs. Developers deliberately formulate them based on their project requirements, and the JDK supports them in the long term. Accordingly, the module system emits no warnings for access permitted by these options.

More than that, they keep illegal-access from emitting warnings for packages that are made accessible by them. If these warnings bug you but you can’t solve the underlying problem, exporting and opening packages this way makes the warnings go away. If even that won’t work for you (maybe you don’t have access to the command line), take a look at this on Stack Overflow: http://mng.bz/Bx6s. But don’t tell anyone where you got that link.

7.1.5 Compiler and JVM options for accessing internal APIs

After working through this section, you’ve earned a pat on the back. The whole problem of internal APIs may look simple on the surface, but once you factor in the ecosystem’s legacy and compatibility concerns, it gets a little complicated. Table 7.1 gives an overview of the options and how they behave.

Static access
Class or member	Public		Nonpublic
Package	Exported	Non-exported	Exported	non-exported
Strong encapsulation	✔	✘	✘	✘
Default in Java 9 due to --illegal-access=permit	✔	✔	✘	✘
--illegal-access=warn	✔	✔	✘	✘
--illegal-access=debug	✔	✔	✘	✘
--illegal-access=deny	✔	✘	✘	✘
--add-exports	✔	✔	✘	✘
--add-opens	✔	✔	✘	✘

Reflective access
Class or member	Public		Nonpublic
Package	Exported	Non-exported	Exported	Non-exported
Strong encapsulation	✔	✘	✘	✘
Default in Java 9 due to --illegal-access=permit	✔	✘	Pre Java 9 ⚠ on first / else ✘	✘
--illegal-access=warn	✔	✘	Pre Java 9 ⚠ on all / else ✘	✘
--illegal-access=debug	✔	✘	Pre Java 9: ⚠ on all, and stack trace / else ✘	✘
--illegal-access=deny	✔	✘	✘	✘
--add-exports	✔	✔	✘	✘
--add-opens	✔	✔	✔	✔

Beyond technical details, it’s important to look at possible strategies that bind these and other options together in a path to Java 9 compatibility. That’s what section 9.1 does. If you’re not looking forward to specifying options on the command line (for example, because you’re building an executable JAR), take an especially close look at section 9.1.4—it shows three alternatives to that approach.

7.2 Mending split packages

Problems with illegal access to internal APIs, with unresolved JEE modules, or with most of the other changes discussed so far, as annoying as they may be, have something going for them: the underlying concepts are fairly easy to grasp; and thanks to precise error messages, the problems are easy to recognize. Neither can be said about split packages. In the worst case, the only symptom you’ll see is the compiler or JVM throwing errors because a class that’s clearly in a JAR on the class path can’t be found.

As an example, let’s take the class MonitorServer, which, among other annotations, uses JSR 305’s @Nonnull. (Don’t worry if you’ve never seen it—I explain in a minute.) Here’s what happens when I try to compile it:

> error: cannot find symbol
>     symbol:   class javax.annotation.Nonnull
>     location: class monitor.MonitorServer

That’s even though jsr305-3.0.2.jar is on the class path.

What’s happening? Why are some types not loaded even though the class path contains them? The critical observation is that those types are in a package that’s also contained in a module. Now let’s see why that makes a difference and leads to classes not being loaded.

When different artifacts contain classes in the same package (exported or not), they’re said to split the package. If at least one of the modular JARs doesn’t export the package, this is also called a concealed package conflict. The artifacts may contain classes with the same fully qualified name, in which case the splits overlap; or the classes may have different names and only share the package name prefix. Regardless of whether split packages are concealed and whether they overlap, the effects discussed in this section are the same. Figure 7.2 shows a split and concealed package.

Abundant sources for split-package examples are application servers, which typically run various JDK technologies. Take, for example, the JBoss application server and the artifact jboss-jaxb-api_2.2_spec. It contains classes like javax.xml.bind.Marshaller, javax.xml.bind.JAXB, and javax.xml.bind.JAXBException. This clearly overlaps with and thus splits the javax.xml.bind package, contained in the java.xml.bind module. (By the way, JBoss is doing nothing wrong—JAXB is a standalone JEE technology, as explained in section 6.1.1 and the artifact contains a full implementation of it.)

An example for a non-overlapping and generally more questionable split package comes from JSR 305. The Java Specification Request (JSR) 305 wanted to bring “annotations for software defect detection” into the JDK. It decided on a few annotations like @Nonnull and @Nullable that it wanted to add to the javax.annotation package, created a reference implementation, was successfully reviewed according to the Java Community Process (JCP), and then—went silent. That was 2006.

The community, on the other hand, liked the annotations, so static analysis tools like FindBugs supported them and many projects adopted them. Although not exactly standard practice, they’re commonly used throughout the Java ecosystem. Even in Java 9, they aren’t part of the JDK, and unfortunately the reference implementation places most of the annotations in the javax.annotation package. This creates a non-overlapping split with the java.xml.ws.annotation module.

7.2.1 What’s the problem with split packages?

What’s wrong with split packages? Why would they lead to classes not being found even though they’re obviously present? The answer isn’t straightforward.

A strictly technical aspect of split packages is that Java’s entire class-loading mechanism was implemented on the assumption that any fully qualified class name is unique—at least, within the same class loader, but because there’s by default only one class loader for the entire application code, this is no meaningful way to relax this requirement. Unless Java’s class loading is redesigned and reimplemented from the ground up, this forbids overlapping package splits. (Section 13.3 shows how to tackle that problem by creating multiple class loaders.)

Another technical aspect is that the JDK team wanted to use the module system to improve class-loading performance. Section 6.2.1 describes the details, but the gist is that it relies on knowing for each package which module it belongs to. This is simpler and more performant if every package only belongs to a single module.

Then, split packages collide with an important goal of the module system: strong encapsulation across module boundaries. What happens when different modules split a package? Shouldn’t they be able to access each other’s package-visible classes and members? Allowing that would seriously undermine encapsulation—but disallowing that would collide head-on with your understanding of visibility modifiers. Not a design decision I’d want to make.

Maybe the most important aspect is conceptual, though. A package is supposed to contain a coherent set of classes with a single purpose, and a module is supposed to contain a coherent set of packages with a single, although somewhat larger, purpose. In that sense, two modules containing the same package have overlapping purposes. Maybe they should be one module, then … ?

Although there’s no single killer argument against split packages, they have a lot of properties that are undesired and would foster inconsistencies and ambiguity. The module system hence views them with suspicion and wants to prevent them.

7.2.2 The effects of split packages

Given the inconsistencies and ambiguities split packages can incur, the module system practically forbids them:

• A module isn’t allowed to read the same package from two different modules.
• No two modules in the same layer are allowed to contain the same package (exported or not).

What’s a layer? As section 12.4 explains, it’s a container bundling a class loader with an entire graph of modules. So far, you’ve always implicitly been in the single-layer case, in which the second bullet wholly includes the first one. So unless different layers are involved, split packages are forbidden.

As you’ll see next, the module system behaves differently, though, depending on where the split occurs. After we’ve covered that, we can finally turn to mending the split.

Splits between modules

When two modules, such as a platform module and an application module, split a package, the module system will detect that and throw an error. This can happen at compile time or run time.

As an example, let’s fiddle with the ServiceMonitor application. As you may recall, the monitor.statistics module contains a package monitor.statistics. Let’s create a package with the same name (and the class SimpleStatistician) in monitor. When compiling that module, I get the following error:

> monitor/src/main/java/monitor/statistics/SimpleStatistician.java:1:
>     error: package exists in another module: monitor.statistics
>         package monitor.statistics;
>         ^
> 1 error

When trying to compile a module with a package that’s also exported from a required module, the compiler notices the error. But what happens when the package isn’t exported, meaning you have a concealed package conflict?

To find out, I added a class monitor.Utils to monitor.statistics, which means I split the monitor package between monitor and monitor.statistics. The split is concealed, because monitor.statistics doesn’t export monitor.

In that situation—and I found this a little surprising—compiling monitor works. It’s up to the runtime to report the error, which it dutifully does, immediately when launching the application:

> Error occurred during initialization of boot layer
> java.lang.reflect.LayerInstantiationException:
>     Package monitor in both module monitor.statistics and module monitor

The same is true if two modules (where neither requires the other) contain the same package: not the compiler but the runtime will find the error.

Splits between a module and the class path

This chapter is focused on compiling and running a class-path application on Java 9 or later, so let’s turn back to that use case. Interestingly, the module system’s behavior is different. All code from the class path ends up in the unnamed module (more on that in section 8.2); to maximize compatibility, it is, generally speaking, not scrutinized, and no module-related checks are applied to it. As a consequence, the module system won’t discover split packages and lets you compile and launch the application.

At first that may sound great: one less thing to worry about. Alas, the problem is still there, it just got less obvious. And arguably worse.

The module system knows for each named module (as opposed to the unnamed module), which packages it contains and that each package belongs to only one module. As I explained in section 6.2.1, the new class-loading strategy benefits from that knowledge; whenever it loads a class, it looks up the module containing the package and tries to load from there. If it contains the class, great; if it doesn’t, the result is a NoClassDefFoundError.

If a package is split between a module and the class path, the class loader will always and only look into the module when loading classes from that package (see figure 7.3). Classes in the class-path portion of the package are effectively invisible! This is true for splits between platform modules and the class path and just the same for application modules (meaning JARs loaded from the module path) and the class path.

Yes, you got that right. If some code contains a class from, say, the javax.annotation package, then the class loader will look into the only module that contains that package: java.xml.ws.annotation. If the class isn’t found there, you get a NoClassDefFoundError, even if the class is present on the class path!

As you may imagine, arbitrarily missing classes can lead to some head-scratching. This is the precise reason JEE modules, which foster package splits, aren’t resolved by default, as section 6.1 explains. Still, these modules can make for the weirdest split-package case.

Consider a project that uses the annotations @Generated and @Nonnull. The first is present in Java 8, and the second comes from a JSR 305 implementation the project has on its class path. Both are in the javax.annotation package. What happens when you compile that on Java 9 or later?

> error: cannot find symbol
>     symbol:   class Generated
>     location: package javax.annotation

So the Java class is missing? Yes, because it comes from the JEE module java.xml.ws.annotation, which isn’t resolved by default. But the error message is different here: it doesn’t hint toward the solution. Fortunately, you paid attention earlier and know that you can fix this by adding the containing module with --add-modules java.xml.ws.annotation. Then you get the following:

> error: cannot find symbol
>     symbol:   class Nonnull
>     location: class MonitorServer

The compiler found that class a minute ago—why doesn’t it now? Because now there’s a module containing the javax.annotation package, so the class-path portion becomes invisible.

To repeat (you can also see this in figure 7.4):

• The first error was caused by JEE modules not being resolved by default.
• The second error was caused by the module system ignoring the class-path part of a split package.

Makes perfect sense (right?). Now that you thoroughly understand what’s going on, let’s turn toward fixing the situation.

7.2.3 Many ways to handle split packages

There are quite a few ways to make a split package work. Here they are, in the general order I recommend considering them:

• Rename one of the packages.
• Move all parts of the split package into the same artifact.
• Merge the artifacts.
• Leave both artifacts on the class path.
• Upgrade the JDK module with the artifact.
• Patch a module with the artifact’s content.

The first approach works when the package-name collision was accidental—it should be the most obvious choice and be used whenever possible. When the split was made on purpose, this is unlikely to work, though. In that case, you could try to mend the split by moving a few classes or by merging the artifacts. These first three options are proper, long-term solutions to the problem, but obviously they only work when you have control over the splitting artifacts.

If the splitting code doesn’t belong to you, or the solutions aren’t applicable, you need other options that make the module system work even though the package remains split. A straightforward fix is to leave both artifacts on the class path, where they will be bundled into the same unnamed module and behave as they did before Java 9. This is a valid intermediate strategy while you wait for the project(s) to hash out the collision and fix it.

Unfortunately, none of the solutions discussed so far apply when part of the split belongs to a JDK module, because you have no direct control over it—to overcome that split, you need bigger guns. If you’re lucky, the splitting artifact consists of more than just a few classes that go into a random JDK package and a replacement for an entire, upgradeable JDK module is provided. In that case, see section 6.1.3, which explains how to use --upgrade-module-path.

If none of that helped, you’re stuck with the final and most hacky approach: patching modules.

7.2.4 Patching modules: Last resort for handling split packages

One technique can fix pretty much every split package but should always be the last resort: making the module system pretend the troublesome classes on the class path belonged into the split package’s module. The compiler and run-time option --patch-module ${module}=${artifact} merges all classes from ${artifact} into ${module}. There are a few things to look out for, but let’s see an example before we get to them.

Earlier, we looked at the example of a project that uses the annotations @Generated (from the java.xml.ws.annotation module) and @Nonnull (from a JSR 305 implementation). We discovered three things:

• Both annotations are in the javax.annotation package, thus creating a split.
• You need to add the module manually, because it’s a JEE module.
• Doing so makes the JSR 305 portion of the split package invisible.

Now you know that you can use --patch-module to mend the split:

javac
    --add-modules java.xml.ws.annotation
    --patch-module java.xml.ws.annotation=jsr305-3.0.2.jar
    --class-path 'libs/*'
    -d classes/monitor.rest
    ${source-files}

This way, all classes in jsr305-3.0.2.jar become part of the module java.xml.ws.annotation and can be loaded for a successful compilation (or, on java, execution). Yay!

There are a few things to look out for. First, patching a module doesn’t automatically add it to the module graph. If it isn’t required explicitly, it may still need to be added with --add-modules (see section 3.4.3).

Next, classes added to a module with --patch-module are subject to normal accessibility rules (see section 3.3 and figure7.5):

• Code that depends on such classes needs to read the patched module, which must export the necessary packages.
• Likewise, these classes’ dependencies need to be in exported packages in modules read by the patched one.

This may require manipulating the module graph with command-line options like --add-reads (see section 3.4.4) and --add-exports (see section 11.3.4). Because named modules can’t access code from the class path, it may also be necessary to create some automatic modules (see section 8.3).

7.2.5 Finding split packages with JDeps

Finding split packages by trial and error is unnerving. Fortunately, JDeps reports them. Appendix D gives a general introduction to the tool; you don’t need to know much more than that, because split packages are included in pretty much any output.

Let’s see what JDeps reports for the application that uses javax.annotation.Generated from java.xml.ws.annotationand javax.annotation.Nonnull from JSR 305. After copying all dependencies into the lib folder, you can execute JDeps as follows:

$ jdeps -summary
    -recursive --class-path 'libs/*' project.jar

> split package: javax.annotation
>     [jrt:/java.xml.ws.annotation, libs/jsr305-3.0.2.jar]
>
# lots of project dependencies truncated

That’s unambiguous, right? If you’re curious what depends on the split package, you can use --package and -verbose:class:

$ jdeps -verbose:class
    --package javax.annotation
    -recursive --class-path 'libs/*' project.jar

# split packages truncated
# dependencies *from* javax.annotation truncated

> rest-1.0-SNAPSHOT.jar -> libs/jsr305-3.0.2.jar
>     monitor.rest.MonitorServer -> Nonnull jsr305-3.0.2.jar

7.2.6 A note on dependency version conflicts

You saw in section 1.3.3 how Java 8 has no out-of-the-box support for running multiple versions of the same JAR—for example, if an application transitively depends on both Guava 19 and 20. Just a few pages later, in section 1.5.6, you learned that, unfortunately, the module system won’t change that. With what we just discussed about split packages, it should be clear why that’s the case.

The Java module system changed the class-loading strategy (looking into specific modules instead of scanning the class path) but didn’t change underlying assumptions and mechanisms. For each class loader, there can still be only one class with the same fully qualified name, which makes multiple versions of the same artifact impossible. For more details on the module system’s support for versions, check out chapter 13.

Summary

• To know how the classes your project may depend on can be accessed under the module system, it’s important to understand how they’re categorized in the era of the module system:

• All public classes in java.* or javax.* packages are standardized. These packages are exported by java.* modules and are safe to depend on, so no changes are required.
• Public classes in some com.sun.* packages are supported by Oracle. Such packages are exported by jdk.* modules, and depending on them limits the code base to specific JDK vendors.
• A few select classes in sun.* packages are temporarily supported by Oracle until replacements are introduced in future Java versions. They’re exported by jdk-unsupported.
• All other classes are unsupported and inaccessible. Using them is possible with command-line flags, but code that does so can break on JVMs with different minor versions or from different vendors; thus it’s generally inadvisable.

• Some internal APIs have been removed, so there’s no way to continue using them even with command-line options.
• Although strong encapsulation generally forbids access to internal APIs, an exception is made for code on the class path accessing JDK-internal APIs. This will ease migration considerably but also complicates the module system’s behavior:

• During compilation, strong encapsulation is fully active and prevents access to JDK-internal APIs. If some APIs are required nevertheless, it’s possible to grant access with --add-exports.
• At run time, static access to public classes in non-exported JDK packages is allowed by default on Java 9 to 11. This makes it more likely that existing applications will work out of the box, but that will change with future releases.
• Reflective access to all JDK-internal APIs is permitted by default but will result in a warning either on first access to a package (default) or on each access (with --illegal-access=warn). The best way to analyze this is --illegal-access=debug, which includes a stack trace in each warning.
• Stricter behavior for static and reflective access is possible with --illegal-access=deny, using --add-exports and --add-opens where necessary to access critically required packages. Working toward that target early on makes migration to future Java updates easier.

• The module system forbids two modules (in the same layer) to contain the same package—exported or not. This isn’t checked for code on the class path, though, so an undiscovered package split between a platform module and class-path code is possible.
• If a package is split between a module and the class path, the class-path portion is essentially invisible, leading to surprising compile-time and run-time errors. The best fix is to remove the split, but if that isn’t possible, the platform module in question can either be replaced with the splitting artifact with --upgrade-module-path (if it’s an upgradeable module) or patched with its content with --patch-module.