We have already encountered AST transformations in Chapter 3, Using Groovy Language Features, in the form of out-of-the-box annotations available in Groovy. In this recipe, we will show how to create a brand-new AST transformation to apply to your code.
But first, we'll see some theory as this is required before you dive into AST transformations. An AST transformation is a clever mechanism to modify the bytecode generation at compile time, hence the association with the broader term compile-time metaprogramming.
By modifying the bytecode, we can augment our code with additional features that are added transparently during compilation time; for example, adding getters and setters to a class.
In Java and other languages, it is relatively easy to generate source code, think of domain entity classes generated out of database tables.
But, compile-time metaprogramming goes to the next level and directly generates bytecode that is loaded directly into the JVM. Let me tell you that the last sentence is not completely correct; through AST transformation we don't generate bytecode, but the abstract syntax tree, which is turned into bytecode by the Groovy compiler.
So, what is AST exactly? AST is a tree-based representation of the code that will be eventually compiled into bytecode. Each node of the tree contains an instruction from the source code. The AST is abstract because it doesn't contain all the information of the program such as spaces, parenthesis, comments, and brackets. The AST transformations essentially provide a means of extending the Groovy language without requiring language grammar changes, by introducing a mechanism to manipulate the AST during compilation prior to bytecode generation.
In some ways, the AST is comparable to the DOM tree model of an XML file.
Here are some examples of ASTs. Each image contains an instruction and the AST representation of it:
Here's the first example:
Here's the second example:
In Groovy, there are two types of AST transformations: local and global.
A local transformation uses annotations to generate the code, and it's the easiest to write and debug.
Global transformations are applied to every compilation unit; so, no annotation is used to enforce the transformation.
Enough with the theory; let's get our hands dirty with an actual implementation of a local AST transformation. In this recipe, we will implement a caching annotation.
A cache is often used to store the results of an expensive operation. Typically, a somewhat simple (and unsafe code) would look like the following code snippet:
def cacheMap = [:] def expensiveMethod(Long a) { def cached = cacheMap.get(a) if (!cached) { // Very expensive operation Long res = service.veryLongCall(a) cacheMap.put(a, res) cached = res } cached }
The code simply checks if the argument of the function is a key of the Map, and it executes the expensive code if the key is missing. Otherwise, it returns the value associated to the key.
A more elegant and more idiomatic variation of the previous code can be expressed by using a closure:
def cacheMap = [:] Long expensiveMethod( Long a ) { withCache (a) { // Very expensive operation ... } } def withCache = { key, Closure operation -> if (!cacheMap.containsKey(key)) { cacheMap[key] = operation() } cacheMap.get(key) }
The caching AST transformation will be based on the less elegant code, except that it will be implemented as an annotation. Furthermore, in this recipe, we show different ways to manipulate the AST.
Before we begin, we will create a simple Gradle build (see the Integrating Groovy into the build process using Gradle recipe in Chapter 2, Using Groovy Ecosystem) to hold our code together and help with the test. In a new folder, touch a build.gradle
file and add the following content:
apply plugin: 'groovy' repositories { mavenCentral() } dependencies { compile localGroovy() testCompile 'junit:junit:4.+' }
Create the following folder structure in the same folder where the build file resides:
src/main/groovy/org/groovy/cookbook/
src/test/groovy/org/groovy/cookbook/
Groovy local AST transformations are based on Java annotations. The first step is to write the code for an annotation that we will use on any method for which the return value is cached:
src/main/groovy/org/groovy/cookbook
folder:package org.groovy.cookbook import org.codehaus.groovy. transform.GroovyASTTransformationClass import java.lang.annotation.* @Retention (RetentionPolicy.SOURCE) @Target ([ElementType.METHOD]) @GroovyASTTransformationClass ( ['org.groovy.cookbook.CacheableTransformation'] ) @interface Cacheable { }
package org.groovy.cookbook import org.codehaus.groovy.ast.* import org.codehaus.groovy.ast.builder.AstBuilder import org.codehaus.groovy.ast.expr.ArgumentListExpression import org.codehaus.groovy.ast.expr.ConstructorCallExpression import org.codehaus.groovy.ast.stmt.BlockStatement import org.codehaus.groovy.ast.stmt.Statement import org.codehaus.groovy.control.CompilePhase import org.codehaus.groovy.control.SourceUnit import org.codehaus.groovy.transform.ASTTransformation import org.codehaus.groovy.transform.GroovyASTTransformation import java.lang.reflect.Modifier @GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS) class CacheableTransformation implements ASTTransformation { @Override void visit(ASTNode[] astNodes, SourceUnit sourceUnit) { if (doTransform(astNodes)) { MethodNode annotatedMethod = astNodes[1] BlockStatement methodCode = annotatedMethod.code def methodStatements = annotatedMethod.code.statements def parameterName = annotatedMethod.parameters[0].name def cachedFieldName = annotatedMethod.name def declaringClass = annotatedMethod.declaringClass FieldNode cachedField = new FieldNode( "cache_${cachedFieldName}", Modifier.PRIVATE, new ClassNode(Map), new ClassNode(declaringClass.getClass()), new ConstructorCallExpression( new ClassNode(HashMap), new ArgumentListExpression() ) ) declaringClass.addField(cachedField) Statement oldReturnStatement = methodCode.statements.last() def ex = oldReturnStatement.expression def stats = """ def cached = cache_${cachedFieldName}. get(${parameterName}) if (cached) { return cached } """ List<ASTNode> checkMap = new AstBuilder(). buildFromString( CompilePhase.SEMANTIC_ANALYSIS, true, stats ) def putInMap = new AstBuilder().buildFromSpec { expression { declaration { variable "localCalculated_${cachedFieldName}" token '=' { -> delegate.expression << ex }() } } expression { methodCall { variable "cache_$cachedFieldName" constant 'put' argumentList { variable parameterName variable "localCalculated_${cachedFieldName}" } } } returnStatement { variable "localCalculated_${cachedFieldName}" } } methodStatements.remove(oldReturnStatement) methodStatements.add(0, checkMap[0]) methodStatements.add(putInMap[0]) methodStatements.add(putInMap[1]) methodStatements.add(putInMap[2]) } } boolean doTransform(ASTNode[] astNodes) { astNodes && astNodes[0] && astNodes[1] && (astNodes[0] instanceof AnnotationNode) && (astNodes[1] instanceof MethodNode) && (astNodes[1].parameters.length == 1) && (astNodes[1].returnType.name == 'void') } }
MyTestClass
in the usual src/main/groovy/org/groovy/cookbook/
folder:package org.groovy.cookbook class MyTestClass { def cacheMap1 = [:] @Cacheable Long veryExpensive(Long a) { sleep(rnd()) a * 20 } static rnd() { Math.abs(new Random().nextInt() % 5000 + 1000) } }
This class has only one veryExpensive
method that does nothing but sleep for a random amount of milliseconds. The method is annotated with the @Cacheable
annotation so that the result is cached after the first invocation. Note how the method always return the same result.
src/test/groovy/org/groovy/cookbook/
folder, create a simple unit test to verify that our transformation works:package org.groovy.cookbook import org.junit.* class TestAst { @Test void checkCacheWorks() { def myTest = new MyTestClass() (1..3).each { withTime { println myTest.veryExpensive(10) } } } def withTime = {Closure operation -> def start = System.currentTimeMillis() operation() def timeSpent = System.currentTimeMillis() - start println "TIME IS > ${timeSpent}ms" } }
The test doesn't really assert anything; but we can still use it to print out the timing of the method invocation.
gradle -i clean test
from the command line. The output should look as follows:... Gradle Worker 2 finished executing tests.org.groovy.cookbook.TestCacheableAST > testInvokeUnitTest STANDARD_OUT 50 TIME IS > 3642ms 50 TIME IS > 0ms 50 TIME IS > 0ms ...
From the output, it is clearly visible how the first method took almost four seconds to execute, while the second and third invocations executed instantaneously thanks to the caching annotation.
Let's start by taking a look at the annotation declared in step 1. The RetentionPolicy
is set to SOURCE
, which means that the compiler will discard the annotation; after all it is not needed at runtime. This annotation can be only used at the METHOD
level because we want only certain methods to cache the result and not the whole class.
The last annotation's attribute is @GroovyASTTransformationClass
. The value is set to the actual source code of the transformation we wish to implement.
The code for the AST transformation in step 2 is not very easy to follow, and we are going to analyze it step-by-step. The CacheableTransformation
class used in this recipe uses two different styles of abstract syntax tree modification:
ASTNode
subclasses approachASTBuilder
-based approach, which is easier to read and maintainAn AST transformation class must implement ASTTransformation
. The class itself must also be annotated with the @GroovyASTTransformation
.
Following the class declaration, we find some sanity checks, which will exit the transformation if some conditions are true. This should never happen, but it is better to exercise some defensive programming style with AST transformations. The expression assigned to a cacheField
variable does create a private Map
variable named after the annotated method. For example, consider the following snippet:
@Cacheable Long veryExpensive(Long a) { ... }
The cacheField
variable will be named cache_veryExpensive
, as the annotated method is named veryExpensive
.
As you can notice, the AST API expression, which is required to create a statement as simple as private Map cache_veryExpensive = [:]
, is indeed quite complex, and it requires a deep knowledge of the org.codehaus.groovy.ast
package. Imagine what the statement would look like for something more complex. There is no abstraction over the AST, and the code to create the AST doesn't remotely resemble the Groovy code we are trying to execute.
In order to facilitate manually writing AST code, we can take a little shortcut in the shape of groovyConsole
. Let's fire up groovyConsole
and type into the main window:
class MyClass { private final Map m = [:] }
Select the Script menu and click on Inspect AST. A new window should pop-up. Then select Semantic Analysis phase in the drop-down list and expand the Fields node and its sub-elements to show the classes required to define the Map
variable using AST.
The AST inspector can be used as a guide to instantiate the right classes. Even with the help of the AST inspector, building AST code by hand is not trivial, and the resulting code is not easy to maintain.
Let's move on to the next statements:
Statement oldReturnStatement = methodCode.statements.last() def ex = oldReturnStatement.expression
These two lines are required to extract the return statement in the annotated function. We are going to need it later.
def stats = """ def cached = cache_${cachedFieldName}.get(${parameterName}) if (cached) { return cached } """ List<ASTNode> checkMap = new AstBuilder(). buildFromString( CompilePhase.SEMANTIC_ANALYSIS, true, stats )
The second style of AST manipulation uses the org.code.codehaus.groovy.ast.builder.AstBuilder
class, written by Hamlet D'Arcy, to ease the pain of creating an AST. The class provides three ways of building an AST. The three methodologies allow you to create Groovy code:
buildFromSpec
methodbuildFromString
methodbuildFromCode
methodIn the previous code snippet, the buildFromString
method is used. With this approach, you can use any string containing valid Groovy code to build an AST. Alternatively, with buildFromCode
, you can pass the Groovy source code directly, instead of a string. The buildFromCode
method approach has some strengths and weaknesses. The code is automatically validated by the compiler when it is passed as a closure.
On the other hand, certain operations that are available by using buildFromString
are not possible with buildFromCode
; for example, creating a new class or binding data from the enclosing context. The stats
variable contains the first part of the code that we want to add to an annotated method before the method body is invoked. The buildFromString
method takes three arguments:
boolean
type named statementsOnly
, indicates that when true, only the script statements are returned. When false, a Script
class is returned.The second part of our caching transformation uses the buildFromSpec
method approach to finalize the caching behavior. The buildFromSpec
method uses a thin DSL on top of the ASTNode API. The reason we switched from buildFromString
to buildFromSpec
is the following code statement:
expression { declaration { variable "localCalculated_$cachedFieldName" token '=' { -> delegate.expression << ex }() } }
This DSL entry assigns the body of the return statement (the ex
variable declared earlier) to a local variable named localCalculated_$cachedFieldName
. This would be impossible to achieve using buildFromString
because the ex
variable would not be accessible from within the string.
The last lines of the transformation are used to assign the newly created statements to the function's body.
The statements in the last part of the visit
function are used to:
buildFromString
method in which we create the call to the Map used for cachingbuildFromSpec
methodLet's look at how a function annotated with the @Cacheable
annotation gets modified. Here, is the original function:
@Cacheable Long veryExpensive(Long a) { callVerySlowMethod(a) }
Here is the modified one:
private Map cache = [:] Long veryExpensive(Long a) { def cached = cache.get(a) if (cached) { return cached } localCalculated = callVerySlowMethod(a) localCalculated.put(localCalculated) localCalculated }
The caching done by the transformation is not very sophisticated, but for the sake of brevity and to alleviate an already intricate topic, we have simplified the code a bit.
3.145.179.59