Chapter 10. Integrating Scala with Java

One of the biggest advantages of the Scala language is its ability to seamlessly interact with existing Java libraries and applications. Although this interaction isn’t completely seamless, Scala offers the tightest integration to Java of any JVM language.

The key to knowing how to integrate Scala and Java lies in the Java Virtual Machine specification and how each language encodes onto that specification. Scala does its best to translate simple language features directly onto JVM features. But complicated Scala features are implemented with some compiler tricks, and these tricks are usually the cause of issues when integrating with Java. For the most part, the Java language translates simply into JVM bytecode; however, it too has language features that use compiler tricks. These will also cause rough spots in Scala/ Java interaction.

Another benefit of understanding how to interface Scala with Java is that it helps to integrate Scala with every other JVM language. Because Java is king on the JVM, all alternative JVM languages provide means of using existing Java code. This means that communications from Scala to another JVM language can be accomplished through Java in the worst case. Scala is working on language features to integrate directly with dynamic languages, but even with the 2.9.0 release, these features are considered experimental.

This chapter focuses on four big issues in Scala/Java interaction. The first issue is that Scala treats all types as objects, and Java supports primitives within the language. This leads to issues that can be solved by creating appropriate interfaces for communication between Java and Scala. Other mismatches can be alleviated with judicious use of implicit conversions.

The second issue is implicit conversions, which tend to be overutilized. While extremely useful, they can cause subtle bugs in Scala/Java interaction. We’ll cover these in detail in section 10.2.

The third issue is Java serialization. Scala does a lot to support Java serialization seamlessly and succeeds for the most part. A few advanced Scala features can cause issues with Java serialization. We’ll discuss these in section 10.3.

The fourth issue is with annotations. Scala adheres to a uniform access principle—that is, Scala makes no distinction between methods and fields; they share the same namespace. Java does distinguish between fields and methods. Some Java libraries require specific methods or fields to have annotations. Scala provides some advanced annotation features that enable this to succeed. We’ll discuss these in section 10.4.

Let’s look into the mismatch between Java primitives and Scala objects.

10.1. The language mismatch between Scala and Java

The Scala and Java languages offer tight integration. Scala classes can be instantiated and extended within Java. Java interfaces and classes can be extended within Java. Scala trait can be extended within Java using a bit of trickery. But this seemingly tight integration runs afoul of three rough patches: primitive boxing, visibility differences, and inexpressible language features.

Primitive boxing is the (semi-)automatic conversion of primitive values on the JVM into objects. This is done because generic parameters are implemented through type-erasure. Type-erasure refers to the practice where, although the generic parameters are known to the compiler at compile time, they get erased to java.lang.Object at runtime. This was one of the means with which Java retained backwards compatibility when it introduced generics. Scala and Java implement this differently, which we’ll look into in section 10.1.1.

Visibility refers to using protected and private modifiers to change the access restrictions on classes and their members. Scala prefers to make everything visible at runtime (that is, in the bytecode) while Java prefers to enforce as much runtime visibility restrictions as the JVM allows. These competing goals can lead to runtime visibility problems. We’ll discuss these in section 10.1.2.

Inexpressible language features are features within the Scala language that can’t be expressed within the Java language. Things like curried methods, implicit parameters and higher-kinded types are examples. It’s best to avoid or hide these features in any code that needs to interface with Scala and Java. We’ll discuss these in more detail in section 10.1.3.

The first difference between Scala and Java is the special treatment of primitives, things created directly on the stack and passed by value, and objects, things created on the heap and passed by reference. In Java, primitives are isolated from objects. Specifically, code using generic type parameters in Java can’t use primitives. To get around this, Java defines a set of classes that mimic the types of primitives. When an object is required, the primitive can be placed into an object. This technique is known as boxing the primitive. The object makes a box in which to carry the primitive. Scala makes no distinction between primitives and objects, and performs boxing behind the scenes on behalf of the developer.

10.1.1. Differences in primitive boxing

In the Scala language, everything is an object and the compiler does its best to hide the fact that primitives aren’t objects. In the Java language, the programmer is forced to pay attention to the difference between a primitive and an object containing the same value. This means that java.util.List<int> isn’t a valid type in Java, but java.util.List<Integer> is valid.

To relieve the overhead of boxing, Java introduced auto-(un)boxing in version 1.5. Autoboxing is an implicit conversion from a primitive type to its boxed type. This allows you to write a for loop as the following:

In the example, the line int item : foo is a for expression that’s unboxing all integers in the list foo. Although not seen, this is the same code as the following:

This example is similar except that the int item is explicitly unboxed from the Integer returned from the list. Although boxing happens automatically in Java, it can be an expensive operation at runtime.

In Scala, there’s no distinction between primitives and objects. The language treats scala.Int as an object. The compiler tries to optimize the usage of scala.Int such that it remains in primitive form throughout the life of a program. For example, we’ll define the following Scala object:

object Test {
  def add(x: Int, y: Int) = x + y
}

This object defines one method, add. The add method takes two scala.Int values and returns a scala.Int. The bytecode emitted by the compiler is as follows:

public int add(int, int);
  Code:
   0: iload_1
   1: iload_2
   2: iadd
   3: ireturn

}

The signature for the add method uses the primitive int type. The bytecode emitted uses iload, iadd, and ireturn. These three bytecodes operate on primitive integers. What happens if we use a generic type with scala.Int? The compiler will generate boxing code as needed:

object Test {
  def add2(items: List[Int]) = {
    var sum = 0
    val it = x.iterator
    while (it.hasNext) {
      sum += it.next
    }
    sum
  }
}

The object Test defines a new method add2. This method take a generic List class parameterized to have scala.Int elements. The code creates a sum variable, grabs an iterator to the list, and iterates over the values in the list. Each of these values is added to the sum variable and the sum is returned. Let’s take a look at the bytecode in the following listing.

Listing 10.1. The add2 method

public int add2(scala.collection.immutable.List);
  Code:
   0: iconst_0
   1: istore_2
   2: aload_1
   3: invokeinterface #28, 1;
     //InterfaceMethod
  scala/collection/LinearSeqLike.iterator:()Lscala/collection/Iterator;
   8: astore_3
   9: aload_3
   10: invokeinterface #34, 1;
     //InterfaceMethod scala/collection/Iterator.hasNext:()Z
   15: ifeq 33
   18: iload_2
   19: aload_3
   20: invokeinterface #38, 1;
     //InterfaceMethod
  scala/collection/Iterator.next:()Ljava/lang/Object;
   25: invokestatic #44;
     //Method
  scala/runtime/BoxesRunTime.unboxToInt:(Ljava/lang/Object;)I
   28: iadd
   29: istore_2
   30: goto 9
   33: iload_2
   34: ireturn
}

The add2 method is compiled so that it takes the scala.collection.immutable.List type as a parameter and returns a primitive integer. The List class is generic and suffers from the same problem as Java generics. The implementation of Generic types in Java forces the use of Object at runtime; therefore, primitives can’t be generic type parameters. Label 20 in the byte code shows that invoking next against the List’s iterator returns the type Object. Label 25 shows Scala’s version of autoboxing: the Boxes-RunTime class. Scala uses the scala.runtime.BoxesRunTime class to implement all boxing/unboxing operations as efficiently as possible.

trait Iterator[@specialized(Int) T] {
 def hasNext: Boolean
 def next: T
}

public interface Iterator {
 public abstract boolean hasNext();
 public abstract java.lang.Object next();
 public abstract int next$mcI$sp();
}

The important point here is that both Scala and Java use boxing with generic classes. Scala hides boxing entirely behind scala.Int while Java promotes boxing into the language itself. This mismatch can cause issues when working with Scala from Java or Java from Scala. These issues can be solved using one simple rule: Use primitives in methods used from both Scala and Java.

This simple rule can avoid a few of the issues with Scala/Java interaction. We still have the issue of generic parameters. In Java, a list of integers has the type java.util.List<java.lang.Integer>. In Scala, a list of integers has the type java.util.List [scala.Int]. Although the runtime implementation of the two lists is the same, Scala’s type system does not automatically convert from Java’s boxed primitives to Scala’s unified object types—the Scala compiler won’t automatically convert a java.util.List[java.lang.Integer] into a java.util.List[scala.Int] even if such a conversion would be type-safe.

Two solutions to this issue exist. One is to perform a cast from java.util.List[java.lang.Integer] to java.util.List[scala.Int]. The other is to define an implicit conversion that will shim Java types into Scala types. Let’s look at the casting:

scala> val x = new java.util.ArrayList[java.lang.Integer]
x: java.util.ArrayList[java.lang.Integer] = []

scala> x.add(java.lang.Integer.valueOf(1))
res0: Boolean = true

scala> x.add(java.lang.Integer.valueOf(2))
res1: Boolean = true

scala> val z = x.asInstanceOf[java.util.List[Int]]
z: java.util.List[Int] = [1, 2]

scala> z.get(0)
res3: Int = 1

The first line constructs a new java.util.ArrayList with a generic parameter equal to java.lang.Integer. The next two lines add data to the list. The third line defines a new list z, which is a cast from java.util.ArrayList[java.lang.Integer] to java.util.List[scala.Int]. The REPL prints the values in the list when describing the return types. Notice that the correct values are shown and there are no runtime exceptions. The next retrieves the first value from the cast list. Notice that the return type is scala.Int and there are no ClassCastExceptions. The asInstanceOf cast was legal because Scala and Java box their primitive integers to the same type: java.lang.Integer.

These casts may be considered dangerous. They subvert the type system in Scala and prevent it from discovering future errors. For example, if a method is changed from taking a java.util.List[java.lang.Integer] to a java.util.List [MySpecialClass], the cast to java.util.List[scala.Int] will still compile and prevent other compile-time errors.

The second solution can avoid this pitfall by operating within the type system. The second solution is to create an implicit conversion from java.util.List [java.lang.Integer] to java.util.List[scala.Int]:

scala> implicit def convertToScala(
     |	   x: java.util.List[java.lang.Integer]) =
     |       x.asInstanceOf[java.util.List[Int]]
convertToScala:
  (x: java.util.List[java.lang.Integer])java.util.List[Int]

scala> def foo(x: java.util.List[Int]) = x.get(0)
foo: (x: java.util.List[Int])Int

scala> foo(x)
res4: Int = 1

The implicit convertToScala is defined to take a java.util.List[java.lang.Integer]. It performs the same cast from the previous example. The difference here is that the dangerous cast is hidden behind the method such that it can only be used in a type-safe fashion; the method can only take lists of java.lang.Integer types, so if the generic type parameter of the list is changed, the implicit view won’t be used at all and the compiler will issue the appropriate type error.

The scalaj-collections library provides primitive-safe implicit conversions between Scala and Java collection types. This offers the best mechanism to handle primitives in collections, but noncollection types may still require a hand-rolled implicit conversion.

The next big issue is the difference in visibility implementation.

10.1.2. Differences in visibility

Java enforces visibility both statically and dynamically. Visibility is enforced both by the Java compiler and by the JVM runtime. Java embeds visibility restrictions directly into the bytecode that the JVM uses to enforce at runtime.

Scala enforces visibility statically, and does its best to encode visibility constraints for the JVM. Scala’s visibility design is far more powerful than Java’s and can’t be directly encoded into bytecode for runtime enforcement. Scala tends to make methods publicly visible and enforces all constraints at compile time, unless the visibility rule in Scala lines up directly with one from Java.

Let’s look at a simple example. Java’s protected modifier differs from Scala’s. Specifically, in Scala, companion objects are allowed to access protected members of their companion classes. This means that Scala can’t encode protected members using the JVM’s protected bytecode because that would restrict companion classes from accessing protected members. Let’s look at an example.

class Test {
 protected val x = 10
}

The Test class is defined with a single member x. The val x is protected and holds the value 10. Let’s look at the generated bytecode for this class.

public class Test extends java.lang.Object implements scala.ScalaObject{
private final int x;

public int x();
  Code:
   0: aload_0
   1: getfield #11; //Field x:I
   4: ireturn


...

The Test class is defined with a private field x and a public accessor called x. This means that in Java an external user of the Test class could access the protected x method. Here’s an example:

The Test2 class is defined in Java. The main method is defined to construct a new Scala Test instance. The next line calls the protected x method and prints its value to the console. Even though the value is protected within Scala, the call succeeds in Java. Let’s run the Test2 class:

$ java -cp /usr/share/java/scala-library.jar:. Test2
10

The program outputs the value 10 with no runtime visibility exception. Java doesn’t see Scala’s visibility constraints. This means that Java clients of Scala classes need to be on their best behavior to prevent modifying or accessing values that they shouldn’t.

Visibility issues are a subset of a bigger issue with Java/Scala integration—that of inexpressible language features.

10.1.3. Inexpressible language features

Java and Scala both have features that are inexpressible in the other language.

Java has static values on classes. These are values that are constructed when a class is loaded and aren’t associated with any particular instance of the class. In Scala, everything is an object and there are no static values. We might argue that Scala’s objects are static values. Scala’s objects are implemented in terms of static values on the JVM but aren’t themselves static values. Consequently, Java libraries that require static values are hard to interact with from Scala.

Scala has many features unavailable in Java, such as traits, closures, named and default parameters, implicit parameters, and type declarations. When interacting with Scala, Java can’t use implicit resolution to find missing parameters to methods. Java can’t use Scala’s default parameter definitions.

For each of these issues, there’s usually a workaround somewhere, but it’s best to avoid these issues entirely. You can do this with a simple mechanism: Construct interfaces in Java that define all the types that will be passed between Java and Scala.

Scala/Java Integration Tip

Construct interfaces in Java that define all types that will be passed between Java and Scala. Place these interfaces into a project that can be shared between the Java portions of code and the Scala portions of code. By limiting the features used in the integration points, there won’t be any feature mismatch issues.

Because Java is more limited in features and compiles more directly to bytecode, it makes for a great integration language. Using Java interfaces ensures you avoid the corner case issues of integration, besides those of boxing.

One example where using Java is required is on the Android platform which has an interface called Parcelable. You can use this interface to allow objects to be passed between processes. Because this could involve serializing the data, the Parcelable interface requires a static field that the Android platform can use to instantiate a Parcelable.

For example, say that an application needs to pass addresses between processes on the Android platform. In Java, the Address class would look as shown in the following listing:

Listing 10.2. Parcelable Address for Android

public class Address implements Parcelable {
     public String street;
     public String city;
     public String state;
     public String zip;
     public void writeToParcel(Parcel out, int flags) {
         out.writeString(street);
         out.writeString(city);
         out.writeString(state);
         out.writeString(zip);
     }

     private Address(Parcel in) {
         street = in.readString();
         city = in.readString();
         state = in.readString();
         zip = in.readString();
     }
    public int describeContents() {
        return 0;
    }

    public static final Parcelable.Creator<Address> CREATOR
            = new Parcelable.Creator<MyParcelable>() {
        public Address createFromParcel(Parcel in) {
            return new Address(in);
        }

        public Address[] newArray(int size) {
            return new Address[size];
        }
    };
}

The Address class is composed of four members: street, city, state, and zip. It has a writeToParcel method that’s Android’s way of flattening or serializing the class to send to another process. The private constructor for Address is used to deserialize the values from the Parcel it was stored in. The describeContents method returns a bit-mask that tells the Android platform the types of data that are contained in the parcel, in case any need special treatment. Finally, there’s a public static instance called CREATOR defined on the class of type Parcelable.Creator<Address>. The Android system uses this type to create and parse incoming Addresses from other processes. This mechanism is also inexpressible in Scala.

The solution in this case is to create a split between the pieces that require Java and the pieces that require Scala. In the case of Address, it’s such a simple class, that writing it completely in Java could be a fine solution. But if Address were more complex, this splitting would be appropriate. Let’s pretend that Address uses some advanced Scala type features in some of its member functions. To get Address to still be Parcelable in Android and to keep the advanced Scala features, it must be split. The Scala features can stay in an abstract class that the Java statics can extend. The Scala class would look as follows:

abstract class AbstractAddress(
      val street: String,
      val city: String,
      val state: String,
      val zip: String) extends Parceable {
  override def writeToParcel(out: Parcel, flags: Int) {
    out.writeString(street)
    out.writeString(city)
    out.writeString(state)
    out.writeString(zip)
  }
  override def describeContents = 0
}

The AbstractAddress class is defined with street, city, state, and zip as constructors and as val members. The abstract class can also define all the methods required by the Parcelable interface: writeToParcel and describeContents. But the static CREATOR instance can’t be made in Scala. This can be done in Java. Let’s extend the AbstractAddress class in Java to allow for usage in Android:

public class Address extends AbstractAddress {
  private Address(Parcel in) {
    super(in.readString(),
          in.readString(),
          in.readString(),
          in.readString());
  }
  public static final Parcelable.Creator<Address> CREATOR
    = new Parcelable.Creator<MyParcelable>() {
        public Address createFromParcel(Parcel in) {
          return new Address(in);
        }
        public Address[] newArray(int size) {
          return new Address[size];
        }
     };
}

The Address class is defined with a private constructor that takes in a Parcel and delegates to the constructor defined in Scala. Then the static CREATOR instance is defined similarly to the Java-only version.

Due to Scala’s tight integration with Java, interfacing with constructors and extending abstract classes can be seamless. This simple Address Parcelable example highlights what to do when running into APIs developed for Java without Scala in mind.

Another area of concern when integrating with Java is the overuse of implicit conversions to adapt Java libraries into Scala idioms.

10.2. Be wary of implicit conversions

One common mechanism of supporting the Scala/Java interaction is to create implicit conversions within Scala that promote Java types into a more Scala-friendly form. This can help ease the pain of using classes not designed for Scala but comes at a cost. Implicit conversions carry a few dangers that developers need to be aware of:

• Object identity and equality
• Chaining implicits.

The most common example of using implicit conversions to ease integration between Java and Scala are found in the Scala object scala.collection.JavaConverters. This object contains a set of implicit conversions to convert collections from Java to their Scala equivalents and vice versa. These implicit conversions are immensely handy but also suffer from all the issues associated with this design. Let’s look into how object identity and equality can become a problem when using JavaConversions.

10.2.1. Object identity and equality

One of the dangers of using implicits to wrap Scala or Java objects for interoperability is that it can alter object identity. This breaks equality in any code that might require equality. Let’s look at a simple example of converting a Java collection into a Scala one:

scala> import collection.JavaConversions._
import collection.JavaConversions._

scala> val x = new java.util.ArrayList[String]
x: java.util.ArrayList[String] = []

scala> x.add("Hi"); x.add("You")

scala> val y : Iterable[String] = x
y: Iterable[String] = Buffer(Hi, You)

scala> x == y
res1: Boolean = false

The first line imports the JavaConversions implicit conversions. The next line creates the Java collection ArrayList. The values "Hi" and "You" are added to the array list. The val y is constructed with the type of scala.Iterable. This invokes an implicit conversion to adapt the Java ArrayList into a Scala Iterable. Finally, when testing equality of the two collections, the value is false. When wrapping a Java collection, the wrapped collection isn’t equal to the original.

The nuance of this issue can be subtle. For example, the implicit conversion from a Java collection to a Scala collection isn’t as obvious as in the previous example. Imagine there’s a Java class that looks as follows:

import java.util.ArrayList;

class JavaClass {
  public static ArrayList<String> CreateArray() {
    ArrayList<String> x = new ArrayList<String>();
    x.add("HI");
    return x;
  }
}

The class JavaClass has one method called CreateArray that returns an ArrayList containing the value "HI". Now imagine the following Scala class:

object ScalaClass {
  def areEqual(x : Iterable[String], y : AnyRef) = x == y
}

The object ScalaClass is defined with one method, areEqual. This method takes a scala.Iterable and an AnyRef and checks the equality. Now let’s use these two classes together.

scala> import collection.JavaConversions._
import collection.JavaConversions._

scala> val x = JavaClass.CreateArray()
x: java.util.ArrayList[String] = [HI]

scala> ScalaClass.areEqual(x,x)
res3: Boolean = false

The first line imports the implicit conversions for Collection. The next line calls the Java class and constructs the new ArrayList. Finally, the same variable is placed into both sides of the areEqual method. Because the compiler is running the implicit conversions behind the scenes, the fact that x is being wrapped is less apparent in this code. The result of areEqual is false.

Although this example is contrived, it demonstrates how the issue can become hidden behind method calls. In real-world programming, this issue can be difficult to track down when it occurs, as the method call chains are often more complex.

10.2.2. Chaining implicits

The second issue facing implicits as a means to ease Java integration is that of chaining implicits. Scala and Java both support generic types. Collections in both languages have one generic parameter. The implicits that convert from Java to Scala and back again will alter the collection type, but usually not the underlying generic parameter. This means that if the generic parameter type also needs to be converted for smooth Java/Scala integration, then it’s possible the implicit won’t be triggered.

Let’s look at a common example: boxed types and Java collections.

scala> val x = new java.util.ArrayList[java.lang.Integer]
x: java.util.ArrayList[java.lang.Integer] = []

scala> val y : Iterable[Int] = x
<console>:17: error: type mismatch;
 found   : java.util.ArrayList[java.lang.Integer]
 required: Iterable[Int]
       val y : Iterable[Int] = x

The first line constructs a new Java ArrayList collection with generic parameter set to java.lang.Integer. In Scala, because the compiler doesn’t differentiate between primitives and objects, the type scala.Int can be safely used for generic parameters. But Java’s boxed integer, java.lang.Integer, isn’t the same type as scala.Int, but the two can be converted seamlessly. Scala provides an implicit conversion from java.lang.Integer to scala.Int:

scala> val x : Int = new java.lang.Integer(1)
x: Int = 1

This line constructs a java.lang.Integer with the value 1 and assigns it to the value x with the type scala.Int. The implicit in scala.Predef kicks in here and automatically converts from the java.lang.Integer type into scala.Int. This implicit doesn’t kick in when looking for implicit conversions from Java to Scala.

Let’s naively try to construct an implicit that can convert from a collection type and modify its nested element all in one go.

implicit def naiveWrap[A,B](
  col: java.util.Collection[A])(implicit conv: A => B) =
    new Iterable[B] { ... }

The naiveWrap method is defined with two type parameters: one for the original type in the Java collection, A, and another for the Scala version of that type, B. The naive-Wrap method takes another implicit conversion from the Java type A to the Scala type B. The hope is that an implicit view will bind the type parameter A to java.lang.Integer and B to scala.Int and the conversion from java.util.ArrayList[java.lang.Integet] to scala.Iterable[Int] will succeed.

Let’s try this out in the REPL:

scala> val x = new java.util.ArrayList[java.lang.Integer]
x: java.util.ArrayList[java.lang.Integer] = []

scala> val y : Iterable[Int] = x
<console>:17: error: type mismatch;
 found   : java.util.ArrayList[java.lang.Integer]
 required: Iterable[Int]
       val y : Iterable[Int] = x

This is the same error as before. The Java list x isn’t able to be converted to an Iterable[Int] directly. This is the same problem we saw before where the type inferencer doesn’t like inferring the A and B types from the naiveWrap method.

The solution to this problem is one used from 7.2.3: We can defer the type inference of the parameters. Let’s try to implement the wrap method again.

trait CollectionConverter[A] {
  val col: java.util.Collection[A]
  def asScala[B](implicit fun: A => B) =
    new Iterable[B] { ... }
}
object Test {
  implicit def wrap[A](i: ju.Collection[A]) =
    new CollectionConverter[A] {
      override val col = i
    }
}

The CollectionConverter type is implemented to capture the original A type from the naiveWrap method. The Converter trait holds the Java collection that needs to be converted. The asScala method is defined to capture the B type from the naiveWrap method. This method takes an implicit argument that captures the conversion from A to B. The asScala method is what constructs the Scala Iterable. The Test object is defined with a new implicit wrap method. This method captures the original A type and constructs a new CollectionConverter.

The new implicit conversions requires the asScala method to be called directly. Let’s take a look:

scala> import Test.wrap
import Test.wrap

scala> val x = new java.util.ArrayList[java.lang.Integer]
x: java.util.ArrayList[java.lang.Integer] = []

scala> x.add(1); x.add(2);

scala> val y: Iterable[Int] = x.asScala
y : Iterable[Int] = CollectionConverter(1, 2)

First, the new implicit wrap method is imported. Next a Java ArrayList[java.lang.Integer] is constructed and values are added to it. Finally, the conversion is attempted using the asScala method, and this time it succeeds.

The downside to this approach is the requirement of the additional method call to ensure the types are inferred correctly. But as a general solution, this is more ideal. The explicit asScala method call denotes a transformation to a new object. This makes it easy to know when a collection is being converted between the Scala and Java libraries.

Scalaj-Collections

The scalaj-collections library from Jorge Ortiz provides collection conversions to and from Scala and Java collections. The library uses the same technique of having an asScala and asJava method implicitly added to collections of the respected types. The scalaj library offers a more robust solution than what’s available in the standard library.

Although using implicits to wrap Java libraries into Scala libraries can be dangerous, it’s still a helpful technique and is used throughout the standard library. It’s important to know when only simple implicit conversions won’t be enough and how to solve these issues. Chaining implicit conversions can solve a lot of the remaining issues.

The important point here is that implicits aren’t magic and can’t automatically convert between Scala and Java types for all situations. Implicits can and should be used to reduce the overhead of these interaction points.

The next potential issue with Java integration is that of serialization.

10.3. Be wary of Java serialization

For most applications, Java serialization works well within Scala. Scala’s closures are automatically made serializable and most of the classes are serialization friendly.

Scala 2.7.x and Serialization

The Scala 2.7.x series had a lot of issues with Java serialization that have been fixed in 2.8.x and beyond. When using Scala with Java serialization, it’s recommended you use one of the newer releases.

A corner case is where Scala’s generation of anonymous classes can cause issues with serialization. Let’s look at an example.

We’ll define a set of objects to model characters within a game. This game will be composed of different people. Each person could be in one of two states: alive or dead. Let’s define the person class.

object PlayerState {
  sealed trait PlayerStatus extends Serializable
  val ALIVE = new PlayerStatus { override def toString = "ALIVE" }
  val DEAD = new PlayerStatus { override def toString = "DEAD" }
}
case class Player(s : PlayerState.PlayerStatus)

The object PlayerState is used to encapsulate the status enumeration. The sealed trait PlayerStatus represents the status enumeration. Two status values are defined: ALIVE and DEAD. Finally, the Player class is constructed with a single member s that holds the player status.

Now, imagine a few of these players are created and stored in some semipermanent fashion using Java serialization. The game server is running smoothly and everyone’s happy, even those who have dead players. To simulate this, let’s serialize a single dead player to disk.

scala> val x = new Player(PlayerState.DEAD)
x: test.Player = Player(DEAD)

scala> val out = new ObjectOutputStream(
     | new FileOutputStream("player.out"))
out: java.io.ObjectOutputStream = java.io.ObjectOutputStream@5acac877

scala> out.writeObject(x); out.flush()

The value x is created with a player in the DEAD status. The value out is constructed as a Java ObjectOutputStream for the file player.out. The output stream is used to serialize the dead player to disk.

Around this time, there’s a new feature request to allow players to sleep during the game. The PlayerStatus enumeration is updated to have a new state: sleeping.

object PlayerState {
  sealed trait PlayerStatus extends Serializable
  val ALIVE = new PlayerStatus { override def toString = "ALIVE" }
  val SLEEPING = new PlayerStatus { override def toString = "SLEEPING"}
  val DEAD = new PlayerStatus { override def toString = "DEAD" }
}

The SLEEPING value is added between the ALIVE and DEAD status. Other than the new value, nothing in the original code has changed. But when trying to load dead players from disk, there’s an issue:

scala> val input =
     | new ObjectInputStream(new FileInputStream("player.out"))
input: java.io.ObjectInputStream = java.io.ObjectInputStream@7e98f9c2

scala> val x = input.readObject
java.io.InvalidClassException: PlayerState$$anon$2;
  local class incompatible: stream classdesc
    serialVersionUID = -1825168539657690740,
  local class serialVersionUID = 6026448029321119659

A new ObjectInputStream is constructed to deserialize the object using Java’s serialization. The next line attempts to read the serialized player object and throws an InvalidClassException. What’s happened is the class that used to represent the DEAD value has moved. The ALIVE, SLEEPING, and DEAD classes are constructed anonymously: they aren’t given named classes.

Scala generates anonymous class names using a simple formula: location in source code + current count of anonymously generated classes for this location. This means that the original ALIVE class is generated with the name PlayerState$$annon$1 and the original DEAD class is generated with the name PlayerState$$annon$2. But when adding the new SLEEPING status, the anonymous class names are changed. ALIVE stays the same, but SLEEPING is named PlayerState$$annon$2 and DEAD is moved to PlayerState$$annon$3.

The mistake here was using anonymous classes rather than named classes. This issue could prevent refactoring in the code. Let’s dig deeper into anonymous classes and their interaction with Java serialization.

10.3.1. Serializing anonymous classes

Scala will generate anonymous classes to express core language features. Here are the situations where anonymous classes are created:

• Anonymous type refinements
  new X { def refinement = ... }
• Anonymous mixin inheritance
  new X with Y with Z
• Closures and lambda functions.
  List(1,2,3).map(_.toString)

Each of these scenarios has the potential to create a serializable class that can become a refactoring burden. Let’s see what happens when compiling these three lines. First, let’s create a Scala file:

The X and Y traits are defined to illustrate the class generation. The Foo object contains all three scenarios. The test1 method creates an anonymous class for the type refinement. The test2 method creates an anonymous class from the mixin inheritance. The test3 method creates an anonymous class for the closure _.toString. Let’s look at the class files that are generated:

> ls
anon.scala         Foo$$anonfun$test3$1.class  X.class
Foo$$anon$1.class  Foo.class                   Y.class
Foo$$anon$2.class  Foo$.class

The test1 method generated the Foo$$anon$1.class file. The test2 method generated the Foo$$anon$2.class file and the test3 method created the Foo$$anonfun$test3$1.class file. Notice that anonymous classes are numbered on a per file basis and anonymous functions are numbered based on their class/method scope. This means that anonymous classes make it easier to break long-term serializability of data, because any anonymous class defined in the file can change the numbering.

For anonymous classes, the simple solution is to ensure that any long-term persisted objects define named objects or classes. Doing this, the preceding example becomes:

class One extends X { def foo = "HI" }
class Two extends Y with X

object Foo {
  def test1 = new One
  def test2 = new Two
  def test3 = List(1,2,3).map(_.toString)
}

The classes One and Two are created to correspond to the anonymous classes from the earlier test1 and test2 methods. The test1 and test2 methods are changed to use the new named classes. The benefit to this approach is that the generated classfiles are file-order independent. Let’s look at the generated classfile directory.

> ls
anon.scala Foo$$anonfun$test3$1.class Foo.class
Foo$.class One.class                  Two.class
X.class    Y.class

The result is that the only remaining anonymous class is the closure defined in the test3 method. The class One and Two are now explicitly named and can be moved around within the file or into other files. The only remaining issue is the long-term serializability of the anonymous function.

Avoid Long-Term Serialization of Closures

Scala’s closure syntax is highly convenient and used frequently in development. But because of the volatile nature of randomly generated class names, it’s best to avoid persisting closures for any long-running applications. When no other option is available, you should ensure that closure deserialization issues are properly handled.

When it comes to anonymous functions, it’s best to avoid long-term serialization. This grants the most amount of flexibility in syntax and usage. Sometimes this isn’t an option. For example, imagine the following scheduling service:

trait SchedulingService {
  def schedule( cron_schedule: String, work: () => Unit) : Unit
}

The trait SchedulingService defines the interface for a long-term scheduler. The single method schedule is used to schedule tasks to be performed at a later time. The schedule method takes two parameters, a configuration for when to run the task and an anonymous closure to run. The SchedulingService could leverage the fact that closures are serializable and store the task on the filesystem. This would let the SchedulingService allow persistent schedules in the face of restarts.

In the face of closure class name instability, this is a bad long-term strategy. The simple solution to fix the problem is to force users away from using closures, as best as possible. For example, the SchedulingService could use a Job trait instead of a closure.

trait Job extends java.io.Serializable {
  def doWork(): Unit
}
trait SchedulingService {
  def schedule(cron_schedule: String, work: Job): Unit
}

The Job trait is defined as Serializable and has one abstract method, doWork. The doWork method will contain the same implementation that used to be in the anonymous closure. The SchedulingService is updated to take Jobs instead of Function0[Unit]. Although this doesn’t prevent users from creating anonymous subclasses of Job, it does make it easier for them to explicitly name their Job classes and avoid volatile classnames.

The upside to serialization issues in Scala is that Java serialization is often not used for long-term serialization. Java’s serialization frequently gets related to remote method invocations and live machine-to-machine messaging or temporary data storage. Long-term persistence tends to take the form of SQL databases, NoSQL databases (using something like Protocol Buffers), XML, or JSON (JavaScript Serialized Object Notation). This means that in the general case, no special care needs to be taken around anonymous classes. But in those few situations that are troublesome, there are solutions you can use to avoid refactoring hell.

The next potential wart in Java integration is that of annotations.

10.4. Annotate your annotations

Many libraries use annotations for runtime code generation and inspection. Annotations are pieces of metadata that can be attached to expressions or types. Annotations can be used to accomplish many different goals, including the following:

• Ensuring or altering compiler warnings and errors (@tailrec, @switch, @implicitNotFound).
• Alter the bytecode output from compilation (@serializable, @scala.annotations.BeanProperty).
• Configure external services (the Java Persistence API uses annotations like @Column and @ManyToOne, to denote how to serialize classes into a relational database system [RDBMS]).
• Create and enforce additional type system constraints (the continuations plugin defines @cpsParam on types to create additional type-system checks for delimited continuations).

In the JVM ecosystem, many libraries rely on annotations to work properly. Scala prefers annotations instead of keywords for features like Java serialization. Understanding annotations in Scala and where they wind up within the bytecode of a class is important for interoperability with Java frameworks.

One of the largest issues facing Scala and Java interoperability is the mismatch of how Scala compiles class members and annotations compared to how Java compiles class members and annotations. In Java, there’s a separate namespace for class fields and class methods. Both of these can be created, named, and annotated separately. In Scala, there’s one namespace for all members of a type. The compiler takes responsibility for creating fields on a class as needed. Annotations on a member of a Scala class could compile to multiple methods and fields in the bytecode. Let’s look at an example:

class Simple {
   @Id
   var value = 5
}

The Simple class defines a single member value. The value member is of type Int and is variable. It is also annotated with the ID annotation. In Scala 2.9.0, this class is compiled approximately into the following Java class:

class Simple {
  @Id private int value = 5;
  public int value() { return value; }
  public void value_$eq(int value) { this.value = value; }
}

The Simple class has three members: A value field, a value method, and a value_$eq method. The methods are defined public and the field is defined private. The annotation is only placed on the field representing the value. Even though the single member var value compiles into three separate locations in a classfile, the annotation is being placed on only one of them.

class Simple {
 @reflect.BeanProperty
 var value = 5
}

In the best case, this mismatch where one definition can compile to several locations in a classfile can confuse annotation libraries designed to work with Java. In the worst case, the libraries are completely unusable. The solution to this is to use annotations targets.

10.4.1. Annotation targets

Annotation targets are used to assign where in the resulting class files annotations should be placed. Scala provides the annotation targets shown in table 10.1:

Table 10.1. Annotation target types

The different annotations each target a separate area of generated bytecode. These allow complete customization of where annotations are applied. To use one of these annotations, you must apply them against another annotation—that is, the target annotations annotate other annotations with the desired bytecode location. Here’s an example:

import javax.persistence.Id

class Data {
  @(Id @annotation.target.getter)
  var dataId = 1
}

The class Data is defined with a single member dataId. The annotation Id is applied against the dataId member. The annotation Id also has the annotation annotation.target.getter applied to it. Scala allows annotations to be placed on expressions, types, members, and classes. The annotation target classes need to be placed against the annotation type that they wish to change. The expression @(Id @annotation.target.getter) is an annotation of the type Id @annotation.target.getter, which is the annotated type Id. This can be simplified by creating a type alias for the annotated type.

object AnnotationHelpers {
  type Id = javax.persistence.Id @annotation.target.getter
}

import AnnotationHelpers._

class Data {
  @Id
  var dataId = 1
}

The AnnotationHelpers object defines a type alias Id. The type alias is the annotated type javax.persistence.Id @annotation.target.getter. The next line imports the type alias. The Data class is now modified to use the type alias for its annotation. This results in the same bytecode as the previous example.

When using a library or framework designed to annotate JavaBeans, it’s helpful to create a wrapper for Scala. This wrapper should consist of an object, similar to AnnotationHelpers, that has the Java framework’s annotations assigned to the appropriate generated code locations. This can ease usage within Scala. This technique is helpful for defining Scala classes that work with the Java Persistence API (JPA).

A second issue needs to be dealt with: some libraries require annotations in locations that Scala doesn’t generate.

10.4.2. Scala and static fields

As discussed in section 10.1, Scala doesn’t have a way to express static fields on classes. Although the JVM allows fields associated with class instances at runtime, the Scala language doesn’t support this notion. You might argue that you can annotate Scala’s objects because they are compiled to static values. But this doesn’t work in practice.

Let’s look at a quick example:

object Foo {}

This defines a simple object Foo in the raw namespace. Scala compiles to bytecode an equivalent to this Java class:

class Foo$ {
  public static Foo$ MODULE$ = null;

  private Foo$() {}

  static {
    MODULE$ = new Foo$
  }
}

The Foo$ class is defined with a single static member: MODULE$. The static block is run when the class is loaded into the JVM. This instantiates the Foo object and assigns it to the MODULE$ static field. Scala converts all objects to JVM classes with the same name as the object but with a $ appended to the name. This prevents trait/class/object name clashes.

In this example, note that there’s only one static field. You also have no way to provide an annotation on the static field. If a Java library requires static fields or annotations on static fields to work, this library is unusable against Scala classes.

But the Java library isn’t completely unusable. The solution here is the same as before: Use Java for the portion of code that needs to interact with Java.

This is the unfortunate reality of interacting with Java libraries. A few were designed in such a way as to not be usable from Scala.

10.5. Summary

Using Java from Scala is usually a painless process. This chapter covered the areas of concern and offered solutions to each.

First, is the mismatch between Java’s primitive with boxing and Scala’s unified AnyVal types. You can simplify this mismatch by preferring primitive types on the Java side. Because Scala always prefers using the primitive value at runtime, this reduces the total amount of boxing/unboxing overhead within a program.

The second area of concern is when there exists a solution to a problem in both Scala and Java. The canonical example is the differing collections libraries. The Scala collections API isn’t friendly to use from Java, and the Java collections API lacks many of the functional features found in the Scala version. To ease integration between Java portions of code and Scala portions, providing implicit conversions on the Scala side can be beneficial. It’s important to be careful here to ensure you don’t make assumptions about equality. Using explicit conversion functions can help highlight where object identities are changing. They can also be used to perform more than one implicit coercion.

The next area of concern is Java serialization. This works well in Scala. The downside is when Java serialization is used for long-term persistence. Scala allows the easy creation of anonymous classes, classes that could be serialized. If an object is intended for long-term serialization, the class should be formalized and named. Otherwise the source code structure may become locked for the lifetime of the serialized object, or worse. The persistent storage may need to be flushed and migrated.

Finally, when faced with a Java library that won’t work from Scala, it’s best to avoid such a library. If this isn’t possible, then constructing the portion of code required to interact with the library in Java and exposing a Scala-friendly interface is the only solution.

The next chapter covers functional programming, which is a way of writing programs that may be foreign to those of us coming from an object-oriented or imperative background. Let’s look into a world where no effects are side effects and operations are deferred as long as possible.