Control flow is a sequence of execution of methods and instructions by a thread. The Java virtual machine (JVM) executes Java bytecode instructions in the order in which they are found in the class file. The control flow can be programmed using conditional statements such as if, else, and for or by invoking a method. Intercepting control flow includes the awareness of the executing instruction or method and the ability to alter the execution flow at runtime. For example, you might want to intercept a call to System.exit() to prevent the JVM from shutting down. Before you get too excited about the possibilities, let me set the expectations straight. There is no direct way of intercepting any instruction or method call in a running JVM unless it was started in profiling mode. The executing of methods is done by the JIT, and there is no standard Java API that can be used to add a listener, or hook, to the method calls. However, we will look at several indirect approaches to intercepting the control of common scenarios. We will also examine the JVM profiler interface that can be harnessed to intercept any call in debug mode.

System errors are reported by the JVM on abnormal conditions that are presumably outside the application control. They are thrown as subclasses of java.lang.Error and therefore are undeclared, meaning they can be thrown by any method even if its signature does not explicitly declare them. System errors include virtual machine errors such as OutOfMemoryError and StackOverflowError, linkage errors such as ClassFormatError and NoSuchMethodError, and other failures. Conventionally, application programmers are supposed to catch only instances of java.lang.Exception, which means that a condition such as out of memory goes undetected through the application error handling logic. For most real-life applications, this is not desirable because, even if nothing can be done when an error occurs, the application should generally log the error to a log file and attempt releasing held resources. A good design solution is to have a try-catch block at the top of the call stack on main application threads, catching java.lang.Error or java.lang.Throwable and delegating to a method that analyzes the error condition, logs it, and attempts a clean shutdown. Here's an example:

public static void main(String[] args) {
    try {
        // Execute application logic
        runApplication();
    }
    catch (Throwable x) {
        // Log error and attempt clean shutdown
        onFatalError(x);
    }
}

In situations where the JVM is out of memory or a class is not found, the application can attempt to mend it by freeing the contents of caches or disabling a feature affected by the missing classes. Anything is better than a disgraceful vanishing without a trace.

Before logging had become a de facto requirement for Java applications it was common to use System.out.println to output the debug traces. The disadvantages of this approach are abundant and obvious. Once written, such traces cannot be turned on or off without changing the code. Even though the application output stream can be redirected to a file for persistence, there is no rollover and because the file is kept open, it cannot be deleted until the application is shut down (hence, the file size can get exorbitant). When dealing with legacy Java code riddled with System.out.println() calls, a common problem is converting them to calls to a logging framework (see Chapter 6, “Using Effective Tracing,” for a discussion of logging and tracing). It is also important to capture the standard error stream, which receives output of methods such as Exception's printStackTrace(). One of the neat solutions to this is intercepting the output to System.out and System.err and sending it to the log file instead. The technique relies on the fact that the system output stream can be redirected to a custom PrintStream using the setOut method of java.lang.System. PrintStream is a decorator class around an instance of OutputStream, which is responsible for the actual output. The task at hand is therefore to develop a redirecting OutputStream that writes to a log file instead of the process standard output and to then assign the System.out to it.

We are going to develop a class called LogOutputStream that extends java.io.OutputStream and writes its output to a log file using Apache Log4J. The Java input/output framework is very well designed, and all methods of OutputStream eventually delegate to a single method—write()—that takes an integer parameter. LogOutputStream uses a StringBuffer to accumulate characters that it gets in the write(int) method and, when a line separator is detected, the whole buffer is written to disk using Log4J. The only tricky part about the implementation is detecting the end of a line. As you are undoubtedly aware, on Unix the end of a line is marked by a single character: (new line). On Windows, the end of a line is marked by a combination of two characters: and (carriage return and new line). To write truly cross-platform code in Java, you must rely on a system property called line.separator. Because the property is a string, the implementation has to rely on a substring search rather than character comparison. Our implementation is optimized to first use the character comparison to check for the possible end of a line and then use a substring search to ensure that it is the end of a line indeed. The overridden write() method is shown in Listing 16.1.

public void write(int b) throws IOException {
    char ch = (char)b;
    this.buffer.append(ch);
    if (ch == this.lineSeparatorEnd) {
        // Check on a char by char basis for speed
        String s = buffer.toString();
        if (s.indexOf(lineSeparator) != -1) {
            // The whole separator string is written
            logger.info(s.substring(0, s.length() - lineSeparator.length()));
            buffer.setLength(0);
        }
    }
}

The logger here is a reference to a static variable of type org.apache.log4j.Logger declared in LogOutputStream as follows:

static Logger logger = Logger.getLogger(LogOutputStream.class.getName());

Thus, the entire output to System.out is redirected to the Log4J framework as INFO-level messages from the LogOutputStream class. To see our class in action, we have to configure a file appender in log4j.properties and install the interceptor as shown in Listing 16.2.

public static void main(String[] args) {
    System.out.println("Installing the interceptor...");
    PrintStream out = new PrintStream(new LogOutputStream(), true);
    System.setOut(out);
    System.out.println("Hello, world");
    System.out.println("Done");
}

Running the main() method of LogOutputStream displays an Installing the interceptor... message on the console but writes Hello, world and Done messages to the log file. The same interceptor can be installed for the System.err stream. To make it flexible, it can be parameterized to take in the logging level and a stream name in the constructor. In a similar fashion, System.In stream can be programmatically set using System.setIn() to feed a desired input into an application.

The JVM process normally terminates when no active threads exist. Threads running as daemons (Thread.isDaemon() == true) do not prevent the JVM from being shut down. In multithreaded applications, which include Swing GUIs and RMI servers, it is not easy to achieve a clean shutdown by letting all threads end gracefully. Frequently, a call to System.exit() is made to forcefully shut down the JVM and terminate the process. Relying on System.exit() has become a common practice even in programs that are not very sophisticated; even though it makes the life of the application developer easier, it can present a problem for middle-tier products such as Web and application servers. An inadvertent call to System.exit() by a Web application, for example, can bring down the Web server process and possibly prevent users from accessing other Web applications and static HTML pages. This is no way to make friends with the system administrators, and every good developer knows the value of a healthy relationship with that team.

This section examines a simple way to intercept a call to System.exit() and prevent the shutdown of the JVM. This technique can be discovered by examining the source code of the exit() method in java.lang.System. The first thing the method does is check whether a security manager is installed. If it is, the method verifies that the caller has a permission to exit the JVM. Our task, therefore, is to install a custom security manager (or modify the security policy if a security manager is already installed) that disallows the exit until it is explicitly allowed. The InterceptingSecurityManager class located in the covertjava.intercept package extends the SecurityManager class and overrides the isExitAllowed() method to control the JVM shutdown. It uses an internal flag that can be set via the setExitAllowed() method to determine whether to allow the JVM to shut down. If the exit is not allowed, an unchecked SecurityException is thrown to alter the control flow. The main() method shown in Listing 16.3 shows how to install the intercepting security manager and how it affects the execution flow.

public static void main(String[] args) {
    InterceptingSecurityManager secManager = new InterceptingSecurityManager();
    System.setSecurityManager(secManager);
    try {
        System.out.println("Run some logic...");
        System.exit(1);
    }
    catch (Throwable x) {
        if (x instanceof SecurityException)
            System.out.println("Intercepted System.exit()");
        else
            x.printStackTrace();
    }
    System.out.println("Run more logic...");
    secManager.setExitAllowed(true);
    System.out.println("Finished");
}

To keep the example simple, the actual business logic that would normally be invoked inside the try block was replaced with a Run some logic... message. The key is to catch the Throwable class rather than the usual Exception because the intercepted System.exit() is reported as an unchecked exception. Running the main() method shown in Listing 16.3 produces the following output:

Run some logic...
Intercepted System.exit()
Run more logic...
Finished
Process terminated with exit code 0

Instead of terminating the JVM after a call to System.exit() inside the try block, the program continues to run until the exit is allowed.

The previous section has shown how to intercept a programmatic attempt to shut down the JVM by calling System.exit(). Sometimes the JVM shutdown is initiated by a user through a kill command on Unix or a Ctrl+C signal on Windows. The JVM can also be shut down because the user is logging off or the OS is being shut down. Can a Java program intercept the shutdown signal? The answer is no; it cannot intercept this signal, but it can react to it. Since JDK 1.3, an application can install a shutdown hook using the addShutdownHook() method of java.lang.Runtime. Shutdown hooks are instances of java.lang.Thread that are initialized but not started. When the JVM is being shut down, all shutdown hook threads are started to run concurrently with the other threads in the JVM. The hooks have access to the entire Java API, but they should be sensitive to the delicate JVM state. The hook threads should not perform any time-consuming operations and should be thread safe. No expectations should be made about the availability of the system services because they might be in the process of shutting down themselves. A good use for a shutdown hook is to write an entry into a log file before closing it and to release all other resources, such as open database connections and files. An example of installing a shutdown hook is shown in Listing 16.4.

public static void main(String[] args) {
    Runtime.getRuntime().addShutdownHook(new Thread() {
        public void run() {
            handleJVMShutdown();
        }
    });
}

public static void handleJVMShutdown() {
    // Record the shutdown and close all resources
}

Sometimes you need to do some preprocessing and post-processing for a method call. This can include tracing the method name and its parameter values, measuring the execution time, or even providing an alternative implementation. Assume you are developing a drawing editor application that uses interfaces such as Line, Circle, Rectangle, and Curve to represent the basic shapes. If you want to add tracing for all methods in those interfaces, you have several options. You can go through each function and meticulously insert tracing calls. Or you can code a proxy class, implementing every interface that prints the trace and then delegates to the original implementation. This is a cleaner approach because it keeps the debugging code separate from the implementation, but it requires a lot of mundane coding. An interesting and a somewhat unknown alternative is a dynamic proxy that uses reflection to intercept method calls. The java.lang.reflection package offers the interface InvocationHandler and a class (proxy) that together can be used to dynamically create an instance implementing multiple interfaces specified at runtime. This approach does not require compile-type definition of the interfaces that proxy implements. Once instantiated, the proxy can be cast to any of the interfaces that were specified during creation, and any call to a method defined by those interfaces is dispatched to a single method (invoke) of the proxy. The only requirement for the dynamic proxy class is that it implements the InvocationHandler interface that defines the invoke method.

Let's develop a dynamic proxy for Chat that traces out the invocations of the message listener. Recall that Chat relies on the MessageListener interface to associate the main frame with the RMI server. Even though MessageListener has only one method, it is good enough to illustrate the concept. We will place the dynamic proxy between the MainFrame instance and ChatServer instance to add tracing of the method calls. We'll create a TracingProxy class in the covertjava.intercept package and have it implement the InvocationHandler interface. The proxy will delegate the method invocations to the actual object, so we'll code the constructor to take the target object as a parameter. The TracingProxy class declaration and its constructor are shown in Listing 16.5.

public class TracingProxy  implements InvocationHandler {

    protected Object target;

    public TracingProxy(Object target) {
        this.target = target;
    }
    ...
}

Notice that the tracing proxy takes the target as a java.lang.Object type. This is the key point because the proxy class is not tied to MessageListener and therefore can be used on any interface.

We now have to code the invoke() method from the InvocationTarget interface. It takes three parameters—the proxy object itself, the method, and the array of method parameters. Our implementation prints the method name and then delegates the invocation to the target that was passed to the proxy constructor. Listing 16.6 shows that code.

public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
    Object result;
    try {
        System.out.println("Entering " + method.getName());
        result = method.invoke(target, args);
    }
    catch (InvocationTargetException e) {
        throw e.getTargetException();
    }
    finally {
        System.out.println("Leaving " + method.getName());
    }
    return result;
}

The proxy is now ready for a test drive. To see it in action, let's create an instance of TracingProxy initialized with an instance of Chat's MainFrame as the target. Then we'll create a java.lang.reflect.Proxy object that implements the MessageListener interface dynamically and delegates calls to the instance of the tracing proxy. Finally, we'll pass the reflection proxy to the Chat server, casting it to the MessageListener interface. Listing 16.7 shows the corresponding Java code.

public static void main(String[] args) throws Exception {
    ChatServer chatServer = ChatServer.getInstance();
    chatServer.setMessageListener(new MainFrame(false));

    TracingProxy listener = new TracingProxy(chatServer.getMessageListener());
    Object proxy = Proxy.newProxyInstance(
        chatServer.getClass().getClassLoader(),
        new Class[] {MessageListener.class},
        listener
        );
    chatServer.setMessageListener((MessageListener)proxy);
    MessageInfo messageInfo = new MessageInfo("localhost", "alex");
    chatServer.receiveMessage("Test message", messageInfo);
    System.exit(0);
}

Running the main() method of TracingProxy produces the output shown here:

C:ProjectsCovertJavaclasses>java covertjava.intercept.TracingProxy
Received message from host localhost
Entering messageReceived
Leaving messageReceived

Thus, we were able to intercept a call to the messageReceived method without having to implement the MessageInfo interface. Dynamic proxies can also come in handy for framework and tool development when you need to interface with classes whose types are unknown at compile time. Rather than having to generate and compile static Java proxy classes, the frameworks can rely on dynamic proxies as the glue between the components.

A promising development is the introduction of the Java Virtual Machine Profiler Interface (JVMPI), which standardizes the interaction between a profiler and the JVM. It was first exposed in JDK 1.2.2 and further extended in JDK 1.4. The API is a two-way interface specifying how a virtual machine should notify a profiler agent about the events inside the VM, such as thread starts, method calls, and memory allocations. It also specifies the means for a profiler to obtain the information about the state of the JVM and to configure which events it is interested in. The profiler agent runs inside the JVM and all API methods are C-style functions invoked via JNI. To access the API, the JVM has to be started with the -XrunProfilerLibrary parameter, where ProfilerLibrary is the name of the native library to be loaded. It is somewhat unfortunate that there is no Java-based interface to JVMPI, and going into the details of C implementations and JNI is outside the scope of this book. However, I have included a list of the most interesting events that can be intercepted:

The complete reference on JVMPI can be found at

http://java.sun.com/j2se/1.4.2/docs/guide/jvmpi/jvmpi.html