We have written a test with the JUnit test framework. Gradle also supports tests that are written with the TestNG test framework. Gradle scans the test classpath for all class files and checks if they have specific JUnit or TestNG annotations. If a test class or super class extends TestCase or GroovyTestCase or is annotated with the @RunWith annotation, the test class is also determined to be a JUnit test.

For Gradle to use either JUnit or TestNG tests when we run the test task, we invoke the useJUnit() or useTestNG() methods, respectively, to force Gradle to use the correct testing framework. Gradle uses JUnit as testing framework by default, so we don't have to use the useJUnit() method when we use JUnit or JUnit-compatible test frameworks to test our code.

Let's write a new test, but this time we will use TestNG annotations and classes. The following sample class is the same test as we saw before, but written with the TestNG framework:

// File: src/test/java/gradle/sample/SampleTestNG.java
package gradle.sample;

import org.testng.annotations.Test;
import org.testng.AssertJUnit;

public class SampleTestNG {

    @Test
    public void readWelcomeMessage() {
        final Sample sample = new Sample();
        final String realMessage = sample.getWelcomeMessage();

        final String expectedMessage = "Welcome to Gradle.";

        AssertJUnit.assertEquals("Get text from properties file", expectedMessage, realMessage);
    }

}

We need to add the TestNG dependency to the testCompile dependency configuration. Furthermore, we invoke the useTestNG() method on our test task, so Gradle will pick up our new test. We create a new build file and add the following:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)'
    testCompile 'org.testng:testng:6.5.1'
}

test.useTestNG()

Now we can run the test task again, but this time Gradle will use our TestNG test:

$ gradle test
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses
:test

BUILD SUCCESSFUL

Total time: 0.988 secs

The generated HTML test report is in the directory build/reports/tests. We can open the file index.html in our web browser and see the output that is generated by the TestNG framework. The following screenshot shows an example of the output that we can view:

Gradle cannot use the test task to run both the JUnit and TestNG tests at the same time. If we have both types of tests in our project and we want to run them, we must add a new task of type Test. This new task can run the specific tests for one of the frameworks.

We add a new task of type Test to run the TestNG tests in our build file:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)', 'org.testng:testng:6.5.1'
}

task testNG(type: Test) {
    useTestNG()
}

test.dependsOn testNG

To add configuration options for TestNG, we can pass a closure to the useTestNG() method. The closure has an argument of type org.gradle.api.tasks.testing.testng.TestNGOptions. The following table shows the options we can set:

The following sample build file uses some of these options to configure TestNG:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'org.testng:testng:6.5.1'
}

test {
    useTestNG { options ->
        options.excludeGroups = ['functional'] as Set
        options.parallel = 'method'
        options.threadCount = 4
    }
}

The tests that are executed by the test task run in a separate, isolated JVM process. We can use several properties to control this process. We can set system properties and JVM arguments, and we can configure the Java class that needs to be executed to run the tests.

To debug the tests, we can set the debug property of the test task. Gradle will start the test process in debug mode and will listen on port 5005 for a debug process to attach to. This way, we can run our tests and use an IDE debugger to step through the code.

By default, Gradle will fail the build if any test fails. If we want to change this setting, we must set the ignoreFailures property to true. Our build will then not fail, even if we have errors in our tests. The generated test reports will still have the errors. It is bad practice to ignore failures, but it is good to know the option is there if we need it.

The following build file configures the test task with the properties just discussed:

apply plugin: 'java'

repositories {
    mavenCentral()
}
dependencies {
    testCompile 'junit:junit:[4.8,)'
}

test {
    // Add System property to running tests.
    systemProperty 'sysProp', 'value'

    // Use the following JVM arguments for each test process.
    jvmArgs '-Xms256m', '-Xmx512m'
    
    // Enable debugging mode.
    debug = true

    // Ignore any test failues and don't fail the build.
    ignoreFailures = true

    // Enable assertions for test with the assert keyword.
    enableAssertions = true
}

Gradle can execute tests in parallel. This means Gradle will start multiple test processes concurrently. A test process only executes a single test at a time. By enabling parallel test execution, the total execution time of the test task can drastically decrease, if we have a lot of tests. We must use the maxParallelForks property to set how many test processes we want to run in parallel. The default value is 1, which means that the tests don't run in parallel.

Each test process sets a system property of the name org.gradle.test.worker with a unique value. We could use this value to generate unique files for a test process.

If we have a lot of tests that are executed by a single test process, we might get heap size or PermGen problems. With the property forkEvery, we can set how many tests need to run in a single test process, before a new test process is started to execute more tests. So, if Gradle sees that the number of tests exceeds the given number assigned to the forkEvery property, the test process is restarted and the following set of tests is executed.

Let's create a new build file and configure it such that we can run four test processes in parallel and relaunch the test process after 10 tests:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)'
}

test {
    forkEvery = 10
    maxParallelForks = 4
}

To determine which files are tests, Gradle will inspect the compiled class files. If a class or its methods have the @Test annotation, Gradle will treat it as a JUnit or TestNG test. If the class extends TestCase or GroovyTestCase or is annotated with @RunWith, Gradle will handle it as a JUnit test. Abstract classes are not inspected.

We can disable this automatic inspection with the scanForTestClasses property of the test task. If we set the property to false, Gradle will use the implicit include rules **/*Tests.class and **/*Test.class and the exclude rule **/Abstract*.class.

We can also set our own include and exclude rules to find tests. We use the include() method of the test task to define our own rule for test classes. If we want to exclude certain class files, we can use the exclude() method to define the exclude rules.

In the following build file, we disable the automatic class inspection for test classes and set the include and exclude rules for test classes, explicitly:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)'
}

test {
    // Disable automatic inspections.
    scanForTestClasses = false

    // Include test classes.
    include '**/*Test.class', '**/*Spec.class'

    // Exclude test classes.
    exclude '**/Abstract*.class', '**/Run*.class'
}

We already noticed that the output that is shown on the command line isn't much if we simply run the test task. We must set the logging level to info or debug, to get more information about the output that is generated by the tests. We can configure the test task to show more output with the testLogging property. This property is of type org.gradle.api.tasks.testing.logging.TestLoggingContainer. We can set different options for each log level. If we don't specify a log level, the lifecyle log level is implied. The property is marked as experimental, which means the features can change in future versions of Gradle.

TestLoggingContainer has the option showStandardStreams, which we can set to true or false. If we set the value of the property to true, we get the output from System.out and System.err when we run the test tasks.

We can also use the events() method to set which events are logged on the command-line output. For example, we can configure that we also want to see the passed tests with the String value passed as an argument. We can use the arguments standardOut and standardError to get the same effect as with the showStandardStreams property. Other valid arguments are failed, started, and skipped.

If a test fails, we only see the line number of the test that failed. To get more output for a failed test, we can set the option exceptionFormat to full. Then, we get the exception message with, say, the assertion failed message. The default value is short, which only shows the line number. With the property stackTraceFilters, we can determine how much of the stack trace is logged.

We can also set the maximum and minimum granularity of the log messages with the minGranularity and maxGranularity properties. We use the value 0 for the Gradle-generated test suite, 1 for the generated test suite per test JVM, 2 for a test class, and 3 for a test method.

The following sample build file sets some of the options that are available:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)'
}

test {
    // Set exception format to full 
    // instead of default value 'short'.
    testLogging.exceptionFormat 'full'

    // We can also a script block to configure
    // the testLogging property.
    testLogging {
        // No log level specified so the
        // property is set on LIFECYCLE log level.
        // We can pass arguments to determine
        // which test events we want to see in the
        // command-line output.
        events 'passed'
        
        // Show logging events for test methods.
        minGranularity = 3
        
        // All valid values for the stackTrace output.
        stackTraceFilters 'groovy', 'entry_point', 'truncate'
        
        // Show System.out and System.err output
        // from the tests.
        showStandardStreams = true

        // Configure options for DEBUG log level.
        debug {
            events 'started'
        }
    }
    
}

We have already seen the HTML reports that are generated when we run the tests, in the build/reports/tests directory. To change the directory name, we can set the testReportDir property as part of the test task.

Besides the generated HTML report, we have XML files that are generated by the test task, with the results of the tests. These XML files are actually the input for the generated HTML report. There are a lot of tools available that can use the XML files generated by JUnit or TestNG and perform an analysis on them. We can find the files in the build/test-results directory. To change this directory, we must change the testResultDir property of the test task.

To disable the generation of the test reports, we set the property testReport to false.

The following build file shows how we can change the report directories and disable the generation of the test reports:

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile 'junit:junit:[4.8,)'
}

test.testReportDir = file("$buildDir/test-reports")
test.testResultsDir = file("$buildDir/test-results")
test.testReport = false