The distinction between equality and identity is one of the first things you learn about Groovy. There are some consequences you should be aware of. Table 13.1 has the comparison between Java and Groovy idioms of equality and identity.

Groovy equality isn’t necessarily commutative; it isn’t guaranteed that a==b is the same as b==a. A programmer may choose to override equals and break this behavior, even though you shouldn’t do so.

Furthermore, Groovy allows null in equality checks:

null == null // is true
null == 1    // is false

You cannot do that in Java, because null.equals(b) would throw a NullPointerException.

When in doubt, you can use the Java style of using parentheses: always putting parentheses around method arguments. On the other hand, leaving out parentheses can often enhance readability by focusing the eye of the reader on the guts of the code. You have the choice between the following:

println 'hi'
println('hi')

However, if no arguments are given, the parentheses are mandatory to distinguish method calls from property access:

println()  // ok
println    // <- fails with MissingPropertyException

The MissingPropertyException is thrown because with no arguments and no parentheses given, Groovy assumes you are looking for the println property and would call getPrintln if there were such a method.

Note that this is different from other languages with optional elements of syntax, such as Ruby. Another difference is that parentheses can be omitted only for method calls that are top-level statements. In other words, parentheses are mandatory for method calls that are used in expressions:

'abc'.substring 1,3         // ok
x = 'abc'.substring 1,3     // assignment expression -> parser error
println 'abc'.substring 1,3 // argument expression -> parser error

Finally, putting symbols in parentheses forces the Groovy parser to resolve the symbol as an expression. This can be helpful when specifying keys in maps. Consider a map like

map = [x:1]

which is equivalent to

map = ['x':1]

Now, what if you have a variable x in scope, and you would like to use its content as a key in the map? You can enforce that with parentheses around x:

def x = 'a'
assert ['a':1] == [(x):1]

This trick is also described in section 14.3.

Remember that inside a closure, return returns from the closure, not from the method the closure was passed to as an argument, nor from any method surrounding the closure definition. Suppose you run a line like

[1,2,3].each { print it; return }

This prints 123, not 1 as some might expect, because return returns from the closure, not from the each method. The closure is called three times, and each time it is left via return. Compare this to

for (it in [1,2,3]) { print it; return }

which prints 1 because return now leaves the current block. With this difference in mind, you can guess what this snippet does:

def myMethod() {
    [1,2,3].each { print it; return }
}
myMethod()

Right, it prints 123. Again, the return keyword only leaves the closure, not the surrounding myMethod.

So, how can you write closure code that leaves a method prematurely? There currently is only one way—by throwing an exception:

def myMethod() {
    [1,2,3].each { print it; throw new RuntimeException() }
}
try {myMethod()} catch (Exception e){}

This prints 1. However, this code is really ugly. Alternatives are in the works but not yet available at the time of writing. See also section 5.6.

The groovier way to leave an iteration prematurely is different. If possible, you should attempt to iterate over the right set of elements to start with, rather than aborting the iteration early. The methods find, findAll, and grep and the subscript operator with indexes or ranges are your friends here. The following lines show some alternatives:

list[0..1]                .each { processing(it) }
list.find{ it == 2 }      .each { processing(it) }
list.findAll{ it % 2 == 0}.each { processing(it) }
list.grep(~/d/)          .each { processing(it) }

In essence, you’re using a GPath to restrict the work items declaratively rather than using control structures in a procedural way. This course of action isn’t always available, but it should be used where it is both possible and elegant. When you follow this style, you have the additional benefit of separating the concerns of selecting items and processing them.

Suppose you are going to build nodes with NodeBuilder such that you get an outer node containing a nested middle node with an inner node like this:

outer() {
  middle() {
    inner()
  }
}

The usual code for producing this structure with NodeBuilder is straightforward:

new NodeBuilder().outer {
    middle {
        inner()
    }
}

Now, suppose you would like to extract the production of the middle and inner nodes to a method. You might want to do this because the production is complicated or you use that production logic in multiple places. Let’s call the new method produce.

You cannot implement it as

def produce(){
    middle {    // fails - no such method!
        inner()
    }
}

and call it like this:

new NodeBuilder().outer {
    produce()
}

Groovy will complain because it can’t find the middle method. Within the scope of the produce method, you have to make the builder known to the first method call on the builder:

def builder = new NodeBuilder()

builder.outer {
    produce()
}
def produce(){
    builder.middle(){ // needs the builder reference
        inner()       // now it's known
    }
}

Alternatively, you can use the following to avoid using a shared variable—if your production code is in a different class to the declaration of the builder, for example:

def builder = new NodeBuilder()

builder.outer {
    produce(builder)
}
def produce(builderContext){
    builderContext.middle(){
        inner()
    }
}

In both cases, you make the reference to the NodeBuilder available in the produce method in order to get back into the context of the builder. Once the first method call has been made, the builder will set the delegate of the closure to the builder, which is why the call to inner doesn’t need to be made explicitly on the builder.

Apart the builder reference, there is another issue to keep in mind when using methods from within builder code: how the produce method is looked up. Why does the preceding code call the produce method we’ve defined rather than creating a new node called produce?

Before doing any builder-specific handling of a method call, the builder first tries calling the method on the owner. The builder handles the method call (by building nodes, for example) only if the owner doesn’t handle the method.

Consequently, in the preceding code, there would be a conflict if there were another method called inner within the script.

All builders that come with the Groovy distribution obey this rule. All builders that subclass BuilderSupport also have this behavior by default. However, the priority of local method lookup in builders cannot be guaranteed for all possible builders, because a pathological builder implementation may choose to override it, even though this is not advised.

When referring to fields or methods from within the same class, most of the time it’s optional to prefix the name of the field or method with the this. qualifier. This behavior is equivalent to that of Java.

MyClass (Object myField) {       // Java constructor example
    this.myField = myField;
}

void setField (Object myField) { // Java property setter example
    this.myField = myField;
}

Disambiguation in Groovy

In Groovy, the need for distinction goes beyond that. Listing 13.1 combines examples for using the this prefix to differentiate between local variables, fields, and properties.

Example 13.1. Using this to distinguish between property and field access

It goes without saying that it is always good practice to avoid such name clashes. But they sometimes occur accidentally, such as when performing a renaming refactoring.

If you ever find yourself unsure about what’s going on but do want to make a lookup against this, it’s worth qualifying it, even if you decide that would be the default behavior anyway. There’s no need to make a maintenance engineer go through the same hoops as you to work out behavior.

A reference prefix like this is always needed when denoting method closures. A reference like &myMethod will never work; only using a reference like this.&myMethod works. The same is true for field access with the @ sign, as in this.@zero, which cannot be used without a preceding reference.

Groovy shines at capturing business logic in a declarative style. In the financial business and in scientific research, this often means lots of formulas and calculations.

In this scenario, you need to remember that Groovy returns BigDecimal objects from the division operator, and any BigDecimal math is slow compared to other number types.

When calculations are used extensively, it is profitable to avoid full floatingpoint division operations where possible. For example, you may want to calculate monetary values with cents instead of dollars and use intdiv for division. This proved to be useful in the first big commercial project that was fully implemented in Groovy.

Although Groovy relieves the programmer of tinkering with number types in a lot of places, there are remaining areas that need attention. Suppose we need to print the sine values from zero to 2π at every increment of π/2. Expected values would be close to 0, 1, 0, -1. The following solution would be straightforward, but wrong:

0.step(Math.PI*2, Math.PI/2){       // wrong!
  println "$it : ${Math.sin(it)}"
}

The Integer.step method takes an Integer argument for the upper bound. The preceding code is like using 0.step (6,PI/2). The correct version needs to call the step method on a non-integer, such as the BigDecimal 0.0 :

0.0G. step(Math.PI*2, Math.PI/2){ println "$it : ${Math.sin(it)}" }

Note that the G suffix is optional, but helps to make it obvious which dot is part of the number and which dot is involved in the method call. Whenever you encounter unexpected values with your calculations, check the number types being used and the method signatures.

Groovy and Ant make a power duo. From within a Groovy script, all Ant capabilities are easily accessible via AntBuilder. From within an Ant script, all Groovy capabilities are easily accessible via the <groovy> task. You can take the best of both worlds, mixing and matching as needed.

def files = new AntBuilder().fileScanner {
    fileset(dir: '.') {
        include(name: 'Listing*.groovy')
    }
}
def scriptName = getClass().name + '.groovy'
assert files.collect{ it.name }.contains(scriptName)

new AntBuilder().ant(antfile:'build.xml')

<?xml version="1.0" ?>
<project name="groovy-test" default="test" >

  <taskdef name="groovy"
    classname="org.codehaus.groovy.ant.Groovy"
    classpath="groovy-all-1.0.jar"/>

  <target name="test">
    <groovy>
      println "Running in Groovy"
    </groovy>
  </target>

</project>

<groovy>
    def dirMap = ['old1': 'new1', 'old2': 'new2']
    dirMap.each {old, new -> ant.copy(dir: old, toDir: new) }
</groovy>

<groovy>
    class RulePrinter extends Task {
        def size = 40
        def symbol = '*'
        public void execute() { println symbol * size }
    }
    project.addTaskDefinition('ruler', RulePrinter)
</groovy>

<ruler/>
<ruler symbol="--8<-" size="10"/>

properties.'out' = properties.'user.dir'+System.currentTimeMillis()

<groovy><![CDATA[
  println (Math.random() < 0.5 ? "Lower" : "Higher")
]]></groovy>

One of the biggest misconceptions about Groovy is that a Groovy script will be interpreted line-by-line. This is not the case.

When a Groovy script gets executed, it is transformed into a class, and then the class is executed. This transformation happens transparently to the developer. However, the process has some consequences that are helpful to be aware of.

println x

class x

Book gina = new Book('Groovy in Action')

def builder = new NodeBuilder()
builder.prefs(name:'Dierk') {
     language('Groovy')
     conference('http://www.waterfall2006.com')
     for (i in 9..17) {
         workingHour(i)
    }
}

def prefs = evaluate(new File('Preferences.groovy'))

assert prefs.'@name' == 'Dierk'
assert prefs.workingHour*.value().contains(16)

Suppose you have a collection—a list, for example—and you would like to shuffle the content. For instance, you may have track numbers for your Groovy MP3 player and wish to create a random playlist. The Groovy variant of a solution that is often suggested for scripting languages is

[1, 2, 3, 4, 5].sort { Math.random() } // very questionable solution

This works the following way: when a closure that is passed to the sort method does not take two parameters (in which case it would have been used as a Comparator) then sort applies the closure to each element before comparing. Because we return random numbers each comparison has a random outcome.

Although this works, it is neither efficient nor guaranteed to be stable with all sort algorithms, nor does it deliver good results.

Programming in Groovy means you have the wealth of Java at your disposal, and thus you can use the shuffle method of java.util.Collections:

def list = [1,2,3,4,5]
Collections.shuffle(list)
println list

This solution is efficient and stable, and it leads leads to an even distribution of the shuffled object; each item has an equal probability of being shuffled to a given index.

We will reuse this functionality in the next example.

You may have heard about the experiment where text remains readable even though the words in the text are scrambled, as long as the first and last character don’t change. Look at the following scrambled text. Can you read what it means?

Sarbmlce the inner crharatces of words
laenvig the text sltil reabldae for poeple but
not for cutoermps.

Listing 13.6 implements this scrambling process in Groovy.

Example 13.6. Scrambling the inner character of words

We use a regular expression to find all inner word characters. Then, replaceAll replaces all occurrences with the result of a closure that is fed the corresponding match. The match is converted to a list, shuffled, converted to a string, and returned. The regular expression for finding the inner characters of a word models the first and last character as a non-word-boundary (B) with one or more word characters (w+) in between.

The ability to use a closure to build the replacement value for a regular expression match is often very useful.

We proceed with other helpful examples of closures.

Suppose you have a time-consuming task that you need to apply to every file in a directory. It would be helpful to get some information about the progress: how much has already been processed, how much is still left to do, and which file is currently being processed.

The output should not be longer than a single line on the console, showing updated information on-the-fly.

When started on the directory containing this book’s listings, this line may for example read

::::::::: AthleteDAO.groovy

in between be refreshed to

####::::: Mailman.groovy

and finally be

######### x.groovy

Note: This is all one single displayed line that is updated over time, like a normal progress bar. If you have used the wget command-line tool for fetching web content, you have seen the same kind of display there.

The processFiles method in listing 13.7 takes a closure argument called notify. This closure is notified whenever a new files starts being processed. This is equivalent to the Observer pattern.^[4]

The processFiles method is called with a closure that updates the progress bar whenever it receives a notification. For simplicity, our processing only consists of sleeping a little, and processing is done for files in the current directory only.

Example 13.7. Printing a progress bar on the console

Of course, this snippet could be extended in a number of ways. However, even running this simple version on the console is fun and worthwhile.

We will look into more cool things you can do with the console in the next example.

How about a snippet that reads a codebase and prints it to the console with an indication what each line evaluates to? Example output could look like this:

data = [0,1,2,3]         //-> [0, 1, 2, 3]
data[1..2]               //-> [1, 2]
data.collect { it / 2 }  //-> [0, 0.5, 1, 1.5]

Saving this output back to the original file would mean we have written a piece of code that is able to write comments about itself.

Listing 13.8 reveals how to achieve this. We split the code by line, ignore empty lines, print each line, and finally evaluate the line and print the result.

Example 13.8. Evaluating and printing line-by-line

But wait—didn’t we say that you cannot evaluate Groovy code line-by-line? Yes, and the example works only because data has no declaration, which Groovy takes as a hint to put it into the current binding. Each line is evaluated separately, but the binding is passed onto the GroovyShell that conducts the evaluation. The first line adds data to the binding; the second line reads data from the binding when getting the 1..2 range from it.

What would happen if the first line read List data = [0,1,2,3]? At that point, data would be a local variable in the script and so would not be added to the binding. The first line would still evaluate correctly, but the second line will fail because data would not be known in the scope of the GroovyShell that evaluates the second line.

That means that the applicability of our single-step printer is very restricted. However, it makes a good example to sharpen your understanding of scripts being classes rather than sequences of evaluated lines.

In the majority of cases, GStrings are used for simple formatting with the placeholders resolved immediately, as in

println "Now is ${new Date()}"

GStrings have a special way in which they resolve any contained placeholders. At the time of the GString creation, they evaluate each placeholder and store a reference to the result of that evaluation within the GString object. At the time of transformation into a java.lang.String, each reference is asked for its string representation in order to construct the fully concatenated result.

In other words: Although the placeholder resolution is eager, writing the references is lazy. The interesting point comes when a placeholder reference refers to an object that changes its string representation over time, especially after the GString was constructed. There are a number of objects that behave like this, such as lists and maps that base their string representation on their current content. Listing 13.9 uses a list to demonstrate this behavior and a typical Groovy object that writes itself lazily: a writable closure.

Example 13.9. Writing GString content lazily

Note how the stanza GString first works on the current values of count and data but changes its string representation when count and data change.

This behavior enables GStrings to be used as a lightweight alternative to Groovy’s template engines (see section 9.4).

One word of caution: You need to be extremely careful when using such dynamic GStrings as elements of a HashSet or as keys in a HashMap. In general, you should avoid doing so, because the hash code of the GString will change if its string representation changes. If the hash code changes after the GString has been inserted into a map, the map cannot find the entry again, even if you present it with the exact same GString reference.

Writing idiomatic Groovy is one side of working with the language instead of fighting against it. Another side is using the tools provided as effectively as possible. In the next section, we will give more information on the groovy tool used to run scripts and classes.

The -e option (e stands for evaluate) lets you pass one-line scripts to groovy on the command line as well as pipe output from one command or script as input to the groovy command. It is similar to the -e option in Perl, Ruby, and other languages.

A simple one-liner using -e follows. This script prints the vendor of the JVM in which Groovy is running, using the java.lang.System class to retrieve the java.vendor System property value:

> groovy -e "println System.properties.'java.vendor'"

Sun Microsystems Inc.

Note the enclosing quotes around the script: When using –e to pass scripts to groovy on the command line, make sure you enclose the script in single or double quotes so that the command or shell interpreter in which you are running (cmd on Windows or bash on UNIX, for example) does not interpret the contents of your Groovy script as commands or wildcards for itself.

Here is an example demonstrating piping the output of one Groovy script to another Groovy script that takes it as input and transforms the characters to uppercase. Enter the whole input in one line:

> groovy -e "println System.properties.'java.vendor'" |
  groovy -e "println System.in.text.toUpperCase()"

SUN MICROSYSTEMS INC.

You can also do this with native operating system commands, of course.

If you pass additional arguments on the command line, they are available to the script in the args variable. That means you can, for example, count lines in a file like so:

> groovy -e "println new File(args[0]).readLines().size()" jokes.txt

1024

Alternatively, you could print a random joke:

> groovy -e "lines = new File(args[0]).readLines();
  println lines[(int)(lines.size()*Math.random())]" jokes.txt

A horse goes into a bar ... "Hey buddy, why the long face?"

So far, so good—but we’re not making particularly extensive use of the piping feature of most shells, where the result of one operation can be the input to the next. That’s just one of the uses for the options we deal with next.

The -e option becomes more interesting when combined with other options. The -p (print) and –n (line) options tell groovy to create an implicit variable named line from each line of input the groovy command receives from standard input. Standard input may be sourced from a pipe or from files given as trailing command-line arguments.

The line variable is useful when you want to do something for each line of input rather than for the text of the input stream as a whole.

Assume there is a file example.txt in the subdirectory data containing

line one
line two
line three

You can cut off the line prefix with

> groovy -pe "line-'line '" dataexample.txt
one
two
three

The -p option is essentially the same as -n, except it ensures that the result of processing each line is printed to the console (it is an implicit println for each line processed), whereas with -n you need to explicitly specify print or println for anything you want to output.

This can be helpful when filtering, for example, the directory entries for a given date:

> dir | groovy -ne "if (line.contains('05.02.06')) println line"
05.02.06  17:48    <DIR>          .
05.02.06  17:48    <DIR>          ..
05.02.06  14:16               272 BraceCounter.groovy

Here’s a second example for system administrators, which uses the input redirection capabilities of your command shell with the < sign. In a cygwin shell, you might do something like this:

> groovy -ne 'if (line =~ /dierk/) println line' < /etc/passwd
dierk:unused_by_nt/2000/xp:...:/home/dierk:/bin/bash

Note how the examples read from different input sources: from a file given on the command line, from the piped output stream of the dir command, and from streams redirected by the shell. On the command line, you always have a close interaction with your command shell.

The –l (listen) option lets you run a Groovy script in client-server mode. You execute a script (using –e or specifying a file to execute), and Groovy starts a simple server on port 1960 (by default; you may override the port setting if you choose). You can then connect to that server via a telnet application, for example, and run the script or pass arguments to the script for it to process and return results to your client.

Here is an example of a tiny script that looks up and returns the IP address of any hostname it receives. You will need two console windows for this example, one for the server and one for the client. First start the server. By default, the server will start on port 1960, but you can specify any unused port on the command line after the –l option. We’re using port 5000 here:

> groovy -l 5000 -e "println 'ip address: ' +
  InetAddress.getByName(line).hostAddress"

groovy is listening on port 5000

Now the server is running, has opened a socket, and is listening for input on port 5000. Run a telnet client to connect to the server, and send it some hostnames to look up:

> telnet localhost 5000
Trying ::1...
Connected to localhost.
Escape character is '^]'.
localhost
ip address: 127.0.0.1
java.sun.com
ip address: 209.249.116.141
manning.com
ip address: 64.49.223.143

Line-oriented client-server programming could hardly be simpler.

Finally, the –i (in-place edit) option is used when you want your Groovy script to iterate over a file or list of files, modifying them in place and, optionally, saving backups of the original files. Here is an example that goes through all *.java files in the current directory and replaces author tags in the Javadoc such that Dierk’s full name appears instead of his nickname. For every file, a backup is generated with a .bak extension:

> groovy -p –i .bak -e
  "line.replaceAll('@author Mittie','@author Dierk Koenig')" *.java

If you do not provide a backup extension, no visible backup file will be generated. The “visible” part is necessary for accuracy’s sake because behind the scenes, Groovy creates a backup anyway in your personal temporary folder and deletes it when finished normally. So, in the worst case, such as when your power supply is interrupted in the middle of such an operation and your working file is corrupted, you can still recover it from the temporary folder. However, providing a backup extension is the safer choice.

That’s it for the numerous options that groovy can be started with. If you come from Ruby or Perl, they probably look familiar.

Now that you can write useful scripts, you can use them to handle minor chores you have to perform time and time again. Our next section helps to smooth the process of automating away annoyance.

Helper scripts are often started automatically from a scheduler such as cron or at, or as a service. Therefore, they have no graphical user interface but receive all necessary configuration on the command line. Starting a script generally looks like this:

> groovy MyScript –o value

where –o value stands for assigning value to the o option. This is a standard way of dealing with command-line options that users expect nowadays, and Groovy supports it in its libraries.

def cli = new CliBuilder()
cli.h(longOpt: 'help', 'usage information')

def options = cli.parse(args)

if (options.h) cli.usage()

The Mailman example

Assume we set out to provide a Groovy command-line script that sends a message via email on our behalf. Our Mailman script should be reusable, and therefore it cannot hard-wire all the details. On the command line, it expects to get information about the mail server, the mail addresses it should use, the text to send, and optionally the mail subject.

Here is how a casual user can request the information about the script and its options:

> groovy Mailman -h
error: sft
usage: groovy Mailman -sft[mh] "text"
 -f,--from <address>     from mail address (like [email protected])
 -h,--help               usage information
 -m,--subject <matter>   subject matter (default: no subject)
 -s,--smtp <host>        smtp host name
 -t,--to <address>       to address (like [email protected])

The user will also see this output whenever they pass options and arguments that are incomplete or otherwise insufficient.

Listing 13.10 implements the script starting with a specification of its command-line options. It proceeds with parsing the given arguments and using them for instrumenting the Ant task that finally delivers the mail.

Example 13.10. Mailman.groovy script using CliBuilder

There are multiple aspects to consider about listing 13.10. It shows how the compact declarative style of CliBuilder not only simplifies the code, but also improves the documentation as well: better for the user because of the instant availability of the usage statement, and better for the programmer because of the inherent self-documentation.

The multiple uses for documentation, parsing, and validation pay off after the initial investment in the specification. With this support in place, you are likely to produce professional command-line interfaces more often.

Providing command-line options is one part of starting a program, but you won’t get very far if the program can’t find all the classes it requires. Next, you will see how Groovy helps you with that perennial Java bugbear, the classpath.

Suppose you’d like to start a script using groovy MyScript but your script code depends on libraries that are not on the default classpath (<GROOVY_HOME>/ lib/*.jar and <USER_HOME>/.groovy/lib/*.jar).

In this case, you’d need to set the classpath before calling the script, just like you need to do for any Java program.

def loader = this.class.classLoader.rootLoader

loader.addURL(new File('lib/mylib.jar').toURL())

loader.URLs.each{ println it }

import org.jfugue.*

def darthVaderTheme = new Pattern('T160 I[Cello] '+
     'G3q G3q G3q Eb3q Bb3i G3qi Eb3q Bb3i G3hi')

new Player().play(darthVaderTheme)

def loader = this.class.classLoader.rootLoader

def dir = new File('lib')
dir.eachFileMatch(~/.*.jar$/) {
    loader.addURL(it.toURL())
}
evaluate(new File('StarWars.groovy'))

Automation scripts really shine when running unattended on a background schedule. As the saying goes, “They claim it’s automatic, but actually you have to press this button.”

There are numerous ways to schedule your automation scripts:

In a lot of scenarios, it is sufficient to schedule an execution like so:

while(true) {
    println "execution called at ${new Date().toGMTString()}"
    // call execution here
    sleep 1000
}

Remember that unlike in Java, the Groovy sleep method really sleeps at least a second, even if interrupted (see section 9.1.2).

Listing 13.13 extends this simple scheduling to a real-life^[9] scenario. A task should be scheduled to run all working days (Monday through Friday) at office hours (08:00 a.m. to 06:00 p.m.). Within this timeframe, the task is to be started every 10 minutes.

def workDays    = Calendar.MONDAY..Calendar.FRIDAY
def officeHours = 8..18

while(true) {
    def now = new Date()
    if (
        workDays.contains(now.day)       &&
        officeHours.contains(now.hours)  &&
        0 == now.minutes % 10
    ) {
        println "execution called at ${now.toGMTString()}"
        // call execution here
        sleep 31 * 1000
    }
    sleep 31 * 1000
}

The purpose of sleeping 31 seconds is to make sure the check is performed at least once per minute. The extra sleep after execution is needed to avoid a second execution within the same minute.

The solution in listing 13.13 is certainly not suited for scheduling at the granularity of milliseconds. It is also not perfect, because it uses deprecated Date methods.^[10] However, it is sufficient for the majority of scheduling tasks, such as checking the source code repository for changes every 10 minutes, generating a revenue report every night, or cleaning the database every Sunday at 4:00 a.m.

We’ve examined how to make scripts easy to run and easy to schedule, but we’ve said little about the kinds of things you might want such a script to do. Our next section gives a few examples to whet your appetite.

The web is not only full of endless information, but it is also full of interesting new and updated information. Regularly visiting your favorite pages for updated content is one of the plates you need to keep spinning. It’s easy to delegate this task to a Groovy script.

The script needs to

Finding the information of interest is the tricky part, because HTML source code can be complex. Also, our script should be forgiving in terms of whitespaces, attribute sequences, quoting of attribute values, and so on. In other words, we cannot use regular expressions to cut the information out of the source code.

If we could work in XML rather than HTML, we could use an XML parser and GPath or XPath expression to scrape off the interesting parts reliably.

The good news is that there are free open-source parsers that read HTML and expose the content as SAX events such that Groovy’s XML parsers can work with it. The popular NekoHTML parser can be found at http://people.apache.org/~andyc/neko/doc/index.html. Download it, and copy its jar file to the classpath.

As an example, consider analyzing the HTML page of http://java.sun.com as captured in figure 13.2. Let’s assume we’re interested in the news items, or everything that appears as links in bold type. For the screen shown in figure 13.2, our script should print

Developing Web Services Using JAX-WS
More Enhancements in Java SE 6 (Mustang)
"Get Java" Software Button Now Available
Gosling T-Shirt Hurling Contest

Figure 13.2. Screenshot of http://java.sun.com to scrape information off

The links in bold appear in the page’s HTML source like this (pretty-printed):

<B>
  <A href="http://logos.sun.com/spreadtheword/">
    "Get Java" Software Button Now Available
  </A>
</B>

Listing 13.14 shows the surprisingly compact solution to extract this data.

import org.cyberneko.html.parsers.SAXParser

def url = 'http://java.sun.com'

def html = new XmlSlurper(new SAXParser()).parse(url)

def bolded = html.'**'.findAll{ it.name() == 'B' }
def out = bolded.A*.text().collect{ it.trim() }
out.removeAll([''])
out[2..5].each{ println it }

We only need to wrap the NekoHTML SAXParser with the Groovy XmlSlurper. With the help of the slurper, we find all B nodes and their nested A nodes. Finally, we trim surrounding whitespace and remove empty links for nicer output.

Of course, if the web site offers XML datafeeds such as RSS or ATOM, or even as web services, then it’s more reliable to use those. See chapter 12 for more details. But think about all those web pages that have no such luxury, but still convey important information: webmail clients, web server administration pages, web-based planning tools, calendaring systems, conference pages, project build information, and so forth. The list is literally endless.

In combination with a task scheduler, you can use this approach to regularly check whether your server is alive and kicking. If it doesn’t respond in a timely manner or contains an error indication in the page, you can send a notification to the admin.

Reading HTML is nice, but how about clicking links and submitting forms? We’ll show that next.

HTML-based web applications are perfect candidates for automating all the actions that you would do manually otherwise. Think about the steps you repeatedly take in web applications: filling in your daily timesheet, updating the project plan, synchronizing with the address database, posting your current location to the corporate intranet, and so on.

To automate these steps, you can download HtmlUnit^[11] from http://htmlunit.sourceforge.net/ and put its jars on the classpath. HtmlUnit was originally designed for testing web applications and thus developed all the means to operate them. We will only use the operation controls here.

Our example of an interactive web interface is the ubiquitous Google search form, as shown in figure 13.3, with search results for “Groovy” in figure 13.4.

Figure 13.3. The Google search form when searching for “Groovy”

Figure 13.4. Top three Google search results for “Groovy”

Our example is a basic interaction, but nevertheless it contains all the steps for automated web actions:

From the results, we filter the top three hits and report them as follows:

http://groovy.codehaus.org/    : Groovy - Home
http://www.groovy.de/          : Groovy.de - Headshop Growshop...
http://www.jeronimogroovy.com/ : JERONIMO GROOVY RADIO

Listing 13.15 uses HtmlUnit to achieve this. With a newly constructed WebClient, it gets the starting page with the Google URL. From the page, it reads the form and input field by the names they are tagged with. The input field is filled and the form submitted. The form submission returns the result page with the main result anchors having the class attribute 'l'.

import com.gargoylesoftware.htmlunit.WebClient

def client = new WebClient()
def page   = client.getPage('http://www.google.com')
def input  = page.forms[0].getInputByName('q')
input.valueAttribute = 'Groovy'
page       = page.forms[0].submit()

def hits   = page.anchors.grep { it.classAttribute == 'l' } [0..2]
hits.each  { println it.hrefAttribute.padRight(30) + ' : ' +
             it.asText() }

HtmlUnit offers a lot of sophisticated features. It also includes NekoHTML and can deliver the current pages asXml, allowing Groovy to fully leverage its XML support. It can also deal with a wide range of JavaScript content and present the DOM for XPath processing. See its API documentation for details.

All this makes it an ideal companion to Groovy when implementing a remote control for web applications.

One nice feature of version-control systems such as Concurrent Versioning System (CVS) or Subversion (SVN) is that they come with command-line clients. This makes them ideal candidates for inspection by Groovy scripts.

Let’s go through a CVS example. CVS comes with a command-line client that supports a variety of options. You can achieve almost everything with these options, but sometimes you need a little more. For example, when trying to find out who accessed the repository since a certain date, you can use the history command:

cvs history -a -e -D 2006-02-04

But that prints too much and in a rather cryptic way, with countless lines such as

M 2006-02-03 23:52 +0000 denis 1.8 website_base.css ...

It would be nice if a Groovy script could consolidate the output into something that displays the information as a summary:

2006-02-04   cruise  update: delete
2006-02-04   denis   commit: modified
2006-02-04   marc    update: delete
2006-02-04   paul    commit: add
2006-02-04   paul    commit: modified

This summary tells you that the user named cruise^[12] updated from the repository, deleting a local file as a result. You can see who accessed the repository and the resulting operations.

The output is the result of running listing 13.16 against the CVS repository of the open-source Canoo WebTest project. It issues the cvs command and processes the output line by line. Each line is split on whitespace, and the fields of interest are extracted and joined to a string. Each string is put into a HashSet, which has the effect of removing duplicates. The result set is finally printed in a sorted order.

Example 13.16. Summarizing cvs command output for access surveillance

Some aspects in listing 13.16 are particularly Groovy in style. The command is first executed in an extremely simple and readable fashion. The output of the resulting process is processed line by line . Using list and string operations, including a literal list, we transform the raw output into the more readable format .

Note that we could have replaced the last two data extraction lines with a single line of code:

result << "${fields[1]}	${fields[4]}	${codes[fields[0]]}"

This would have been even shorter, because it saves one line. However, it doesn’t read as declaratively, because it mixes the concerns of field selection and presentation. Changing either the fields to be displayed or the formatting of those fields is a simple task in the original code, requiring no duplication.

In our consulting work, we’re asked to do code reviews every now and then. We even review and analyze code of our own projects regularly. In the course of this activity, we’ve learned to value pragmatic tools that work on any codebase.

When reviewing, you need a starting point. A good move is to assemble some statistical data such as the number of files per directory, files sizes, line count per file, and so on for that purpose. It’s amazing how much you can tell about a project from this data. Put it in a spreadsheet, and generate charts for the various dimensions. You will soon spot the hot candidates for review.

There are some helpful measures (we wouldn’t dare to call them metrics) that you can assemble with the help of Groovy. For example, it helps to know the revision number of each file in the version-control system. Unusually high revision numbers can indicate a problematic area, just like files with the most conflicts (see the previous section).

Listing 13.17 points to another interesting measure: maximum nesting depth of braces. It’s a pragmatic approach, because it doesn’t use a real parser for the language and may thus be slightly off when braces occur in comments or strings. However, the solution can be applied to a wide range of languages and gives a good indication of complexity.

def source = new File(args[0]).text

def nesting = 0
def maxnest = 0

for (c in source) {
    switch (c) {
        case '{' : nesting++
                   if (nesting > maxnest) maxnest++
                   break
        case '}' : nesting--
                   break
    }
}
println maxnest

When applying this measure to Groovy code, you can expect higher numbers than for Java, due to the usage of braces in builders, closures, and GPath expressions. On the other hand, the line count is likely to be significantly lower!

There is a huge list of external libraries that are specifically helpful when used together with Groovy scripts.

First, automation often sends notifications. There are Ant tasks for this purpose, but for fine-grained control, you can use the JavaMail^[13] API that comes as an external package of the JRE (javax.mail). Via mail gateways, you can also send text messages to a cell phone.

When automation is used for periodic reporting, libraries for producing graphs and charts are useful. You will find many of these on the Web, such as Snip-Graph, JCCKit, and JFreeChart. They all work from a textual representation of data, and using them with Groovy is therefore easy. Groovy templates can make the production of such text input files much simpler.

Reporting can also mean producing Microsoft Office documents. When running on a Windows platform, you can relay such tasks to Groovy’s Scriptom module, which we will describe in chapter 15. There also are platform-independent solutions with restricted functionality that may nonetheless be sufficient for your needs: POI (http://jakarta.apache.org/poi) for Office documents, and for spreadsheets in particular, JExcelApi (http://jexcelapi.sourceforge.net).

A variety of projects implement customized Groovy support. You can find the list at http://groovy.codehaus.org/Related+Projects. There is, for example, special support for the Lucene search engine. Running its indexer repeatedly would be a typical automation task.

When reports are to be published on the Web, using a Groovy-enabled Wiki can be handy, because the pages can contain Groovy code to update themselves. Currently, the known implementations^[14] are Biscuit, SnipSnap, and XWiki.

The Groovy developers provide specialized modules for making particularly interesting libraries more groovy. Have a look at the modules section at http://groovy.codehaus.org. For example, you will find Groovy support for Google’s calendaring package, allowing constructions such as

import org.codehaus.groovy.runtime.TimeCategory

use(TimeCategory) {
    Date reminder = 1.week.from.now
}

and expressions such as 2.days + 10.hours, basically allowing convenient definitions of dates, timestamps, and durations as usual Java objects. Over time, such a module may be promoted to the Groovy distribution.

Because you can use any Java library, there is an endless list of possibilities. The goal was to trigger your curiosity and make you think about the wide range of applicability.

Next, we will go through various aspects of making your life as a Groovy programmer easier.

As soon as you step into serious Groovy programming, you should look at the available IDE plug-ins and select the one of your choice.

Make sure you have your JDK configured to also include the JDK source and its API documentation. Unlike Java, Groovy plug-ins cannot always provide you with instant code completion. Therefore, you will look up JDK classes and methods more often than you are used to when programming Java. With a proper setup, such a lookup by name can still be efficient.

Most IDEs support the notion of a library that assembles jar files, classes, resources, source code, and API documentation of a common purpose. Create such a library for Groovy, including the Groovy source tree. This enables you to quickly look up important information such as GDK methods. For example, you could run a search for method definitions of the name eachFile*.

Note that Groovy comes with a comprehensive suite of unit tests. Most of these are written in Groovy. This is also a good source of information to have around when programming.

What is true for the JDK and the Groovy distribution is also true for any other external library. The better your setup and the more complete your local information, the less time you will spend scanning through external documentation.

If your IDE can include a Java decompiler such as JAD, get it. It helps a lot when decompiling class files that were generated by groovyc. Make sure the decompilation writes into a directory that will not be used to pick up source files for your next run or compile operation.

IDEs often support a mechanism to break the whole source tree into modules or projects with an option to define their dependencies. In case of a mixed Groovy/Java project, you can use this feature to avoid compile problems with mutual dependencies. For example, you can have three modules: a Groovy-only module, a Java-only module, and a module of shared Java interfaces.

Figure 13.5 illustrates the dependencies of the Groovy and Java modules to the shared interface module.

Figure 13.5. Groovy and Java source modules depending on a shared interface module

This setup ensures that you can compile either module and the whole project easily. Once you have your code compiling, you’ll want to run it sooner or later—and sometimes that will mean running it in a debugger.

Debugging is the act of removing bugs from the code. Some people claim that this implies that programming is the act of putting them in.

The best advice we can possibly give about debugging is advice on how to avoid it. The need for debugging is drastically reduced when solid unit testing is in place and when code is created in a test-first manner.

The next best approach is to make wise use of assertions throughout your code, making it fail early for obvious reasons.

The debugging tool that everybody uses every day is putting println statements in the code under development. This is certainly helpful, and Groovy makes it a workable way of debugging, because transparent compiling and instant class-reloading lead to quick coding cycles.

However, don’t fall into the trap of leaving println statements in the code after debugging is done. Don’t even put them in comments. Depending on the purpose of the line, you can change it into an assertion or into a log statement.

Generally, debugging offers a chance to learn something new about your code and improve it. After finding the bug, you can ask yourself how you could have found it earlier or could have avoided it altogether: what log statement would have helped, what assertion, what unit test, and how could the wrong behavior have been more visible in the code?

Until then, you first have to locate the bug.

Exploiting groovyc

When you get errors from Groovy scripts, precompilation with groovyc can provide more detailed error messages, especially when you’re working with multiple dependent scripts. In this case, use groovyc on all scripts.

When things get really tricky and you suspect Groovy is parsing your code incorrectly or producing bad constructions from it, you can use the properties listed in table 13.5 to make groovyc produce more artifacts. Set the environment variable JAVA_OPTS to the appropriate value before calling groovyc.

Table 13.5. System properties that groovyc is sensitive to

JAVA_OPTS	Purpose	Destination
-Dantlr.ast=groovy	Pretty printing	Foo.groovy.pretty.groovy
-Dantlr.ast=html	Writing a colored version of source, with each AST node as a rollover tooltip	Foo.groovy.html
-Dantlr.ast=mindmap	Writing the AST as a mind map (view with http://freemind.sf.net)	Foo.groovy.mm

The pretty-printer gives a first indication of possible misconceptions about the nesting structure in the code. This can easily occur when you’re using single-statement control statements without braces, such as

if (true)
    println "that's really true"

If you later add more lines to the if-block but forget to add the braces that are then needed, you end up with an error that you can spot by looking at the pretty-print.

The pretty-printer is an obvious candidate to integrate in the Groovy IDE plug-ins.

The HTML and mindmap options in table 13.5 are two other interesting views on the Abstract Syntax Tree (AST) that the Groovy parser creates from your source code. The HTML view is rather conventional, but the mindmap allows you to navigate and expand/collapse the AST nodes. Figure 13.6 shows the AST mindmap for the brace-matching analyzer in listing 13.17.

Figure 13.6. Mindmap AST of listing 13.17 expanded on a for loop

So far, we have assessed only the static aspects of the code. We will now look into the live execution of the code.

boolean isPrime(int x) {
    ! (2..<x).any { y ->  x % y == 0 }
}

for (i in 1..100) {
    println "$i : ${isPrime(i)}"
}

exclude groovy.*,java.*,org.codehaus.*,sun.*

> %JAVA_HOME%injdb groovy.lang.GroovyShell Primer
Initializing jdb ...
*** Reading commands from ...jdb.ini

> > stop at Primer:2
Deferring breakpoint Primer:2.
It will be set after the class is loaded.

> run
run groovy.lang.GroovyShell Primer
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
>
VM Started: Set deferred breakpoint Primer:2
Breakpoint hit: "thread=main", Primer.isPrime(), line=2 bci=13
2        ! (2..<x).any { y ->  x % y == 0 }

main[1] list
1    boolean isPrime(int x) {
2 =>     ! (2..<x).any { y ->  x % y == 0 }
3    }
4
5    for (i in 1..9) {
6        println "$i : ${isPrime(i)}"
7    }

main[1] locals
Method arguments:
Local variables:
x = instance of groovy.lang.Reference(id=726)

main[1] eval x.get()
 x.get() = "1"

main[1] step out
>
Step completed: "thread=main", Primer.run(), line=6 bci=76
6        println "$i : ${isPrime(i)}"

main[1] exit

Profiling is the task of analyzing a run of your program for memory and CPU time consumption. Groovy code can be profiled with any ordinary Java profiling tool. Profiling our Primer script as shown in listing 13.18 can easily be done with the profiling support that comes with the Java Runtime Environment. The JDK documentation comes with extensive documentation of this topic. In short, you can a run a compact command-line profiler with

java -Xprof groovy.lang.GroovyShell Primer

A more sophisticated solution is available with

java -agentlib:hprof groovy.lang.GroovyShell Primer

This second way of starting the JRE profiler writes extensive data to a file named java.hprof.txt. There are a lot of options that you can set when profiling this way. For a list of options, type

java -agentlib:hprof=help

That extensive output of the JRE profiler requires some time to understand. Therefore, commercial profiling solutions are used more often. Figure 13.7 shows profiling data from a Primer run as presented by the commercial YourKit profiler, which grants a free license to the Groovy committers.

From looking at the profiling analysis in figure 13.7, you can tell (line 2) that we started the profiling run from within the Intellij IDE. A special YourKit plug-in for that IDE allows easy profiling setup.

Figure 13.7. Profiling data from YourKit for the Primer script

Because scripts are created with main and run methods, you see these method calls in lines 4 and 6. Line 8 shows that calls to the isPrime method took almost no time. Almost all the time was used for writing the resulting GString to the console. This is not surprising, because I/O operations are always expensive compared to mere calculations.

In between the calls to Primer, you see calls to the Groovy runtime system. The icon indicates that these lines are filtered; in other words, they represent a series of hidden calls. Setting such filters is important to make the interesting parts of the stack stand out.

Profiling and debugging are expert activities, and it takes some time to get proficient with the tools and their usage. But this is not particular to Groovy. It is also true for Java.

What is particular to Groovy is that you will be faced with lots of the internals of the Groovy runtime system: how classes are constructed, what objects get created, how the method dispatch works, and so on.

Refactoring is the activity of improving the design of existing code. The internal structure of the code changes, but the external behavior remains unchanged.

The classic book on refactoring is Refactoring: Improving the Design of Existing Code by Martin Fowler. All the listed refactorings and mechanics can be applied to Groovy exactly as shown for Java code in the book. Where the mechanics suggest compiling the changed code, such a compilation check should be accompanied with running the unit tests for Groovy code.

For the Java world, lots of the standard refactorings such as Extract Method, Introduce Explaining Variable, Pull Members Up, and so on have been automated for use from inside the IDE.

For Groovy, refactoring support is currently not as complete. The future will show which IDE vendor or open source project will be able to provide a compelling solution.