13
Module versions: What’s possible and what’s not

This chapter covers

• Why the module system doesn’t act on version information
• Recording version information
• Analyzing version information at run time
• Loading multiple module versions

As briefly mentioned in section 1.5.6, the JPMS doesn’t support module versions. But then what is jar --module-version good for? And didn’t section 12.3.3 show that ModuleDescriptor can at least report a module’s version? This chapter clears things up and looks at module versions from a few different angles.

We’ll first discuss in what ways the module system could support versions and why it doesn’t do that (section 13.1). It at least allows you to record and evaluate version information, though, and we’ll explore that next (section 13.2). Last on the list is the Holy Grail: running different versions of the same module (section 13.3). Although there’s no native support for that, there are ways to make it happen with some effort.

By the end of this chapter, you’ll have a clear understanding of the module system’s limited support for versions. This will help you analyze your application and can even be used to proactively report possible problems. Maybe more important, you’ll also know the reasons for the limitations and whether you can expect them to change. You’ll also learn how to run multiple versions of the same module—but as you’ll see, it will rarely be worth the effort.

13.1 The lack of version support in the JPMS

Java 8 and earlier have no concept of versions. As described in section 1.3.3, that can result in unexpected run-time behavior where the only solution may be to pick different versions of your dependencies than you’d like. That’s unfortunate, and when the module system was first conceived, one of its goals was to remedy this situation.

That didn’t happen, though. The module system that’s now operating in Java is still comparatively blind to versions. It’s limited to recording a module’s or dependency’s version (see section 13.2).

But why is that? Couldn’t the module system support having several versions of the same module (section 13.1.1)? If not, couldn’t it at least take a bunch of modules and version requirements as input and select a single version for each module (section 13.1.2)? The answer to both questions is “no,” and I want to explain why.

13.1.1 No support for multiple versions

A seemingly simple solution to version conflicts would be to allow running two versions of the same JAR. Straightforward. So why can’t the module system just do that? To answer that question, you have to know how Java loads classes.

How class loading prevents multiple versions

As discussed when we looked at shadowing in section 1.3.2, the JVM—or, more precisely, its class loaders—identify classes by their fully qualified name, such as java.util.List or monitor.observer.ServiceObserver. To load a class from the class path, the application class loader scans all JARs until it encounters a class with the specific name it’s looking for, which it then loads.

Turning back to our desire to run multiple versions of the same module, the roadblock is apparent: such modules are bound to contain classes with the same fully qualified name, and without any changes, the JVM would only ever see one of them. What could those changes look like?

Changes to class loading that would allow multiple versions

The first option for allowing several classes with the same name would be to rewrite the entire class-loading mechanism so that an individual class loader could handle that case. That would be a huge engineering task because the assumption that each class loader has at most one class of any given name permeates the entire JVM. In addition to the massive effort, it would also carry a lot of risk: it would be an invasive change and hence would be almost guaranteed to be backward incompatible.

The second option would be to allow multiple classes with the same name to do what, for example, OSGi does: use a separate class loader for each module (see figure 13.1). That would be comparatively straightforward but would also probably cause compatibility issues.

One potential source of problems is that some tools, frameworks, and even applications make specific assumptions about the exact class-loader hierarchy. (By default, there are three class loaders that reference one another—this didn’t change in Java 9. The details are explained in the description of the boot layer in section 12.4.1.) Putting each module in its own class loader would considerably change that hierarchy and would probably break most of these projects.

There’s another devious detail hidden in changing the hierarchy. Even if you were willing to require projects to adapt to that change to run from the module path, what would happen if they ran from the class path? Would JARs from the class path also each get a separate class loader?

• If so, projects that had trouble with the changed class-loader hierarchy not only wouldn’t run as modules, they also wouldn’t even run on Java 9+.
• If not, they would need to be aware of two different class-loading hierarchies and correctly interact with each of them, depending on which path they landed on.

None of these impacts on compatibility or migration paths are acceptable if applied to the entire ecosystem.

Well, if you have an instance of each class and compare the two, what’s one of the first things that happen in the equals comparison? It’s this.getClass() == other.getClass() or an instanceof check. In this case, that will always be false because the two classes aren’t equal.

That means with two versions of Guava, for example, mutimap1.equals(multimap2) would always be false, no matter what elements the two Multimap instances contained. You also couldn’t cast an instance of the class from one class loader to the same class loaded from the other, so (Multimap) multimap2 could fail:

static boolean equalsImpl(
        Multimap<?, ?> multimap,    ①  
        @NullableDecl Object object) {    ②  
    if (object == multimap) {
        return true;
    }
    if (object instanceof Multimap) {    ③  
        Multimap<?, ?> that = (Multimap<?, ?>) object;
        return multimap.asMap().equals(that.asMap());
    }
    return false;
}

It would be nice to know how many projects would be tripped up just by that detail. There’s no way to know, but my guess is a lot. Compared to that, chapters 6 and 7 are outright benign.

What we determined so far only means that the module system doesn’t allow multiple versions of the same module out of the box. There’s no native support, but that doesn’t mean it’s categorically impossible. Take a look at section 13.3 for ways to make it work.

13.1.2 No support for version selection

If the module system can’t load several versions of the same module, why can’t it at least select the correct versions for us? That, too, is, of course, theoretically possible, but unfortunately isn’t feasible—let me explain why.

How build tools handle versions

Build tools like Maven and Gradle work with versioned JARs all the time. They know for each JAR which version it has and which versions its dependencies have. Considering the shoulders of giants on which so many projects stand, it’s only natural that they have deep dependency trees that contain the same JARs several times, possibly with different versions.

Although it’s nice to know how many different versions require a JAR, that doesn’t change the fact that they better not all end up on the class path. If they do, you’ll run into problems like shadowing (see section 1.3.2) and outright version conflicts (see section 1.3.3), which will threaten your project’s stability.

Why the module system doesn’t select versions

Now let’s leave build tools behind and talk about the module system. As you’ll see in section 13.2, modules can record their own version and those of their dependencies. Assuming the module system can’t run several instances of the same module, couldn’t it select a single version of each?

Let’s play this through. In this hypothetical scenario, the JPMS would accept several versions of the same module on the module path. When building the module graph, it would decide for each module which version to pick.

On top of that would come the effort to implement and maintain the version-selection algorithm. The final nail in the coffin is performance: if the compiler and JVM had to run that algorithm before they could start their actual work, which would measurably increase compile and launch times. As you can see, version selection isn’t a cheap feature, and it makes sense that Java isn’t adopting it.

13.1.3 What the future may bring

In summary, the module system is version agnostic, meaning version information doesn’t impact its behavior. That’s today. Many developers hope Java will support either of these features in the future. If you’re one of them, I don’t want to rain on your parade, and however the future looks today doesn’t mean it won’t happen. I don’t see it, though.

Imagine you had no incentive to do that. How many more JARs would your project drag onto the class or module path? How much larger would it get, and how much more complicated would debugging be? No, I think allowing conflicting versions to work out of the box would be a horrible idea.

That said, the fact remains that there are cases where a version conflict stops important work dead in its tracks or makes critical updates impossible without having to update tons of other dependencies at the same time. To that end, it would be nice to have a command-line switch like java --one-class-loader-per-module that you could try on a rainy day. Alas, it doesn’t exist (yet?).

13.2 Recording version information

As we’ve just covered in detail, the module system doesn’t process version information. Interestingly enough, it does allow us to record and access that information. That may seem a little weird at first, but it turns out to be helpful when debugging an application.

Let’s first look at how to record version information during compilation and packaging (section 13.2.1) before discussing where you see that information and what benefits it provides (section 13.2.2). Recording and evaluating version information is demonstrated in ServiceMonitor’s feature-versions branch.

13.2.1 Recording versions while building modules

The jar command overrides the module’s version if it was present before. So, if --module-version is used on both jar and javac, only the value given to jar matters.

Listing 2.5 showed how to compile and package the monitor module, but you don’t need to flip back. Updating the jar command to record the version is trivial:

$ jar --create
    --file mods/monitor.jar
    --module-version 1.0
    --main-class monitor.Monitor
    -C monitor/target/classes .

As you can see, it’s as simple as slipping in --module-version 1.0. Because the script compiles and immediately packages the module, there’s no need to also add it to javac.

To see whether you succeeded, all you need to do is ask jar --describe-module (see section 4.5.2):

$ jar --describe-module --file mods/monitor.jar

> [email protected] jar:.../monitor.jar/!module-info.class
> requires java.base mandated
> requires monitor.observer
# truncated requires
> contains monitor
> main-class monitor.Main

The version is right there in the first line: [email protected]. Why don’t the dependencies’ versions show up, though? In this specific case, I didn’t record them, but java.base definitely has one, and it doesn’t appear, either. Indeed, --describe-module doesn’t print this information—neither the jar nor the java variant.

To access the versions of a module’s dependencies, you need a different approach. Let’s look at where the version information appears and how you can access it.

13.2.2 Accessing module versions

The versions recorded during compilation and packaging show up in various places. As you’ve just seen, jar --describe-module and java --describe-module both print the module’s version.

Version information in stack traces

Stack traces are also important locations. If code runs in a module, the module’s name is printed for each stack frame together with the package, class, and method names. The good news is that the version is included, too:

> Exception in thread "main" java.lang.IllegalArgumentException
>     at [email protected]/monitor.Main.outputVersions(Main.java:46)
>     at [email protected]/monitor.Main.main(Main.java:24)

Not revolutionary, but definitely a nice addition. If your code misbehaves for seemingly mysterious reasons, problems with versions are a possible cause, and seeing them in such a prominent position makes it easier to notice them if they’re suspicious.

Module version information in the reflection API

Arguably the most interesting place to handle version information is the reflection API. (Going forward, you need to know about java.lang.ModuleDescriptor. Check out section 12.3.3 if you haven’t already.)

Listing 13.1 Accessing a module’s raw and parsed version

ModuleDescriptor descriptor = getClass()
    .getModule()
    .getDescriptor();
String raw = descriptor
    .rawVersion()    ①  
    .orElse("unknown version");
String parsed = descriptor
    .version()    ②  
    .map(Version::toString)
    .orElse("unknown or unparsable version");

Dependency version information in the reflection API

That settles the module’s own version. You still didn’t see how to access the versions that were recorded for the dependencies, though. Or did you? Listing 12.8, which shows the output of printing pretty much everything a ModuleDescriptor has to offer, contains this snippet:

[] module monitor.persistence @ [] {
    requires [
        hibernate.jpa,
        mandated java.base (@9.0.4),
        monitor.statistics]
    [...]
}

See @9.0.4 in there? That’s part of the output of Requires::toString. Requires is another inner class of ModuleDescriptor and represents a requires directive in a module descriptor.

Listing 13.2 Printing dependency version information

module
    .getDescriptor()
    .requires().stream()
    .map(requires -> String.format("	-> %s @ %s",
            requires.name(),
            requires.rawCompiledVersion().orElse("unknown")))
    .forEach(System.out::println);

This code produces output like the following:

> monitor @ 1.0
>     -> monitor.persistence @ 1.0
>     -> monitor.statistics @ 1.0
>     -> java.base @ 9.0.4
# more dependencies truncated

And here they are: the versions of the dependencies against which monitor was compiled. Neat.

It’s fairly straightforward to write a class that uses this information to compare the version against which a module was compiled with the dependency’s actual version at run time. It could warn about potential problems, for example, if the actual version is lower, or log all this information for later analysis in case of problems.

13.3 Running multiple versions of a module in separate layers

Section 13.1.1 states that the module system has no native support for running multiple versions of the same module. But as I already hinted, that doesn’t mean it’s impossible. Here’s how people did it before the JPMS arrived on the scene:

• Build tools can shade dependencies into a JAR, which means all the class files from the dependency are copied into the target JAR, but under a new package name. References to those classes are also updated to use the new class names. This way, the standalone Guava JAR with package com.google.collect is no longer needed, because its code was moved to org.library.com.google.collection. If each project does that, different versions of Guava can never conflict.
• Some projects use OSGi or another module system that supports multiple versions out of the box.
• Other projects create their own class-loader hierarchy to keep the different instances from conflicting. (This is also what OSGi does.)

Each of these approaches has its own disadvantages, which I’m not going to go into here. If you absolutely have to run multiple versions of the same JAR, you need to find a solution that makes the effort worth it for your project.

13.3.1 Why you need a starter to spin up additional layers

As discussed in section 12.4, the module system introduces the concept of layers, which essentially pair a module graph with class loaders. There’s always at least one layer in play: the boot layer, which the module system creates at launch time based on the module path content.

Beyond that, layers can be created at run time and need a set of modules as input: for example, from a directory in the filesystem, which they then evaluate according to the readability rules to guarantee a reliable configuration. Because a layer can’t be created if it contains multiple versions of the same module, the only way to make that work is to arrange them in different layers.

• Paths to all application modules
• The module’s relations, which must consider their different version

Developing such a starter as a general solution is a considerable engineering task and effectively means reimplementing existing third-party module systems. Creating a starter that solves only your specific problem is easier, though, so we’ll focus on that. By the end of the section, you’ll know how to create a simple layer structure that allows you to run two versions of the same module.

13.3.2 Spinning up layers for your application, Apache Twill, and Cassandra Java Driver

Say you depend on two projects, Apache Twill and Cassandra Java Driver. They have conflicting version requirements for Guava: Apache Twill breaks on any version after 13, and Cassandra Java Driver breaks on any version before 16. You’ve tried everything you can think of to work around the problem, but nothing has worked, and now you want to solve the problem by using layers.

That means the base layer contains only your application starter. The starter needs to create one layer with Guava 13 and another with Guava 16—they need to reference the base layer to have access to platform modules. Then comes a fourth layer with the rest of the application and dependencies—it references both of the other layers the starter creates, so it can look up dependencies in them.

It won’t work exactly like that, though. As soon as Apache Twill’s dependencies are resolved, the module system will see Guava twice: once in each of the layers the top layer references. But a module isn’t allowed to read another module more than once because it would be unclear which version classes should be loaded from.

So you pull these two modules and all of their dependencies into their respective Guava layer, and you’re good to go. Almost. Both modules expose their dependency on Guava, so your code needs to see Guava, too; and if that code is in the top layer, you end up in the same situation as before, with the module system complaining about code seeing two versions of Guava.

If you pull your Twill- and Cassandra-specific code into the respective layers, too, you get the layer graph shown in figure 13.3. Now let’s create those layers. To do so, assume that you’ve organized the application modules into three directories:

• mods/twill contains Apache Twill with all its dependencies and your modules that directly interact with it (in this example, app.twill).
• mods/cassandra contains Cassandra Java Driver with all its dependencies and your modules that directly interact with it (in this example, app.cassandra).
• mods/app contains the rest of your application and its dependencies (in this example, the main module is app).

Your starter can then proceed as shown in listing 13.3:

1. Create a layer with the modules in mods/cassandra. Be careful to pick the right module as root for the resolution process. Pick the boot layer as the parent layer.
2. Do the same for modules in mods/twill.
3. Create a layer with the modules in mods/app, and pick your main module as root. Use the other two layers as parents; this way, your application’s dependency on the modules in mods/cassandra and mods/twill can be resolved.
4. When that’s all finished, get the class loader for the upper layer’s main module, and call its main method.

Listing 13.3 Starter that creates layers for Cassandra, Apache Twill, and the app

public static void main(String[] args)
        throws ReflectiveOperationException {
    createApplicationLayers()
        .findLoader("app")
        .loadClass("app.Main")
        .getMethod("main", String[].class)
        .invoke(null, (Object) new String[0]);    ①  
}

private static ModuleLayer createApplicationLayers() {
    Path mods = Paths.get("mods");

    ModuleLayer cassandra = createLayer(
        List.of(ModuleLayer.boot()),
        mods.resolve("cassandra"),
        "app.cassandra");    ②  
    ModuleLayer twill = createLayer(
        List.of(ModuleLayer.boot()),
        mods.resolve("twill"),
        "app.twill");    ②  

    return createLayer(
        List.of(cassandra, twill),
        mods.resolve("app"),
        "app");    ③  
}

private static ModuleLayer createLayer(    ④  
        List<ModuleLayer> parentLayers,
        Path modulePath,
        String rootModule) {
    Configuration configuration = createConfiguration(
        parentLayers,
        modulePath,
        rootModule);
    return ModuleLayer
        .defineModulesWithOneLoader(
            configuration,
            parentLayers,
            ClassLoader.getSystemClassLoader())
        .layer();
}

private static Configuration createConfiguration(    ④  

        List<ModuleLayer> parentLayers,
        Path modulePath,
        String rootModule) {
    List<Configuration> configurations = parentLayers.stream()
        .map(ModuleLayer::configuration)
        .collect(toList());
    return Configuration.resolveAndBind(
        ModuleFinder.of(),
        configurations,
        ModuleFinder.of(modulePath),
        List.of(rootModule)
    );
}

And that’s it! I admit it takes some time, and you’ll likely have to fiddle a while to make it work (I had to), but if it’s the only solution you’re left with, it’s worth giving it a try.

Summary

• The javac and jar commands let you record a module’s versions with the --module-version ${version} option. It embeds the given version in the module declaration, where it can be read with command-like tools (for example, jar --describe-module) and the reflection API (ModuleDescriptor::rawVersion). Stack traces also show module versions.
• If a module knows its own version and another module is compiled against it, the compiler will record the version in the second module’s descriptor. This information is only available on the Requires instances returned by ModuleDescriptor::requires.
• The module system doesn’t act on version information in any way. Instead of trying to select a specific version for a module if the module path contains several, it quits with an error message. This keeps the expensive version-selection algorithm out of the JVM and the Java standard.
• The module system has no out-of-the-box support for running multiple versions of the same module. The underlying reason is the class-loading mechanism, which assumes that each class loader knows at most one class for any given name. If you need to run multiple versions, you need more than one class loader.
• OSGi does exactly that by creating a single class loader for every JAR. Creating a similarly general solution is a challenging task, but a simpler variant, customized to your exact problem, is feasible. To run multiple versions of the same module, create layers and associated class loaders so that conflicting modules are separated.