System tests run as a step within the Continuous Delivery pipeline. They connect against a running application on a test environment, invoke business use cases, and verify the overall outcome.

System test cases usually don't impact the application's life cycle. The application is deployed upfront as part of the CD pipeline. If required, the system tests control the state and behavior of external mocks and contents of databases.

Generally speaking, it makes sense to develop system tests as separate build projects without any code dependency to the project. Since system tests access the application from the outside there should be no implications on how the system is being used. System tests are developed against the application's endpoint contracts. Similarly, the system tests should not use classes or functionality that is part of the application, such as using the application's JSON mapping classes. Defining technology and system access from the outside as separate build projects prevents unwanted side effects caused by existing functionality. The system test project can reside besides the application project in the same repository.

The following example will construct a system test from a top-down approach, defining test scenarios and appropriate abstraction layers.

The business use cases of the car manufacture application are accessed via HTTP. They involve external systems and database accesses. In order to verify the creation of a car, the system test will connect against the running application, as a real-world use case would.

In order to manage the test scenario, the case is crafted using logical steps with comments as placeholders first, and then implemented in several abstraction layers:

public class CarCreationTest {

    @Test
    public void testCarCreation() {

        // verify car 1234 is not included in list of cars

        // create car
        //   with ID 1234,
        //   diesel engine
        //   and red color

        // verify car 1234 has
        //   diesel engine
        //   and red color

        // verify car 1234 is included in list of cars

        // verify assembly line instruction for car 1234
    }
}

These comments represent the logical steps that are executed and verified when testing creation of a car. They are related to the business rather than the technical implementation.

We realize these comments in private methods, or better, own delegates. The delegates encapsulate technical concerns, as well as potential life cycle behavior:

We define CarManufacturer and AssemblyLine delegates that abstract the access and behavior of the applications and delegates. They are defined as part of the system test and have no relation to or knowledge of the managed beans with the same name in the application code. The system test project code is defined independently. It could also be implemented using a different technology, only depending on the communication interface of the application.

The following code snippet shows the integration of the delegates. The car creation system test only contains business logic relevant to implementation, with the delegates realizing the actual invocations. This leverages readable as well as maintainable test cases. Similar system tests will reuse the delegate functionality:

import javax.ws.rs.core.GenericType;

public class CarCreationTest {

    private CarManufacturer carManufacturer;
    private AssemblyLine assemblyLine;

    @Before
    public void setUp() {
        carManufacturer = new CarManufacturer();
        assemblyLine = new AssemblyLine();

        carManufacturer.verifyRunning();
        assemblyLine.initBehavior();
    }

    @Test
    public void testCarCreation() {
        String id = "X123A345";
        EngineType engine = EngineType.DIESEL;
        Color color = Color.RED;

        verifyCarNotExistent(id);

        String carId = carManufacturer.createCar(id, engine, color);
        assertThat(carId).isEqualTo(id);

        verifyCar(id, engine, color);

        verifyCarExistent(id);

        assemblyLine.verifyInstructions(id);
    }

    private void verifyCarExistent(String id) {
        List<Car> cars = carManufacturer.getCarList();
        if (cars.stream().noneMatch(c -> c.getId().equals(id)))
            fail("Car with ID '" + id + "' not existent");
    }

    private void verifyCarNotExistent(String id) {
        List<Car> cars = carManufacturer.getCarList();
        if (cars.stream().anyMatch(c -> c.getId().equals(id)))
            fail("Car with ID '" + id + "' existed before");
    }

    private void verifyCar(String carId, EngineType engine, Color color) {
        Car car = carManufacturer.getCar(carId);
        assertThat(car.getEngine()).isEqualTo(engine);
        assertThat(car.getColor()).isEqualTo(color);
    }
}

This serves as a basic example for an application system test. The delegates such as CarManufacturer handle the lower-level communication and validation:

public class CarManufacturer {

    private static final int STARTUP_TIMEOUT = 30;
    private static final String CARS_URI = "http://test.car-manufacture.example.com/" +
            "car-manufacture/resources/cars";

    private WebTarget carsTarget;
    private Client client;

    public CarManufacturer() {
        client = ClientBuilder.newClient();
        carsTarget = client.target(URI.create(CARS_URI));
    }

    public void verifyRunning() {
        long timeout = System.currentTimeMillis() + STARTUP_TIMEOUT * 1000;

        while (!isSuccessful(carsTarget.request().head())) {
            // waiting until STARTUP_TIMEOUT, then fail
            ...
        }
    }

    private boolean isSuccessful(Response response) {
        return response.getStatusInfo().getFamily() == Response.Status.Family.SUCCESSFUL;
    }

    public Car getCar(String carId) {
        Response response = carsTarget.path(carId).request(APPLICATION_JSON_TYPE).get();
        assertStatus(response, Response.Status.OK);
        return response.readEntity(Car.class);
    }

    public List<Car> getCarList() {
        Response response = carsTarget.request(APPLICATION_JSON_TYPE).get();
        assertStatus(response, Response.Status.OK);
        return response.readEntity(new GenericType<List<Car>>() {
        });
    }

    public String createCar(String id, EngineType engine, Color color) {
        JsonObject json = Json.createObjectBuilder()
                .add("identifier", id)
                .add("engine-type", engine.name())
                .add("color", color.name());

        Response response = carsTarget.request()
                .post(Entity.json(json));

        assertStatus(response, Response.Status.CREATED);

        return extractId(response.getLocation());
    }

    private void assertStatus(Response response, Response.Status expectedStatus) {
        assertThat(response.getStatus()).isEqualTo(expectedStatus.getStatusCode());
    }

    ...
}

The test delegate is configured against the car manufacture test environment. This configuration could be made configurable, for example, by a Java system property or environment variable in order to make the test reusable against several environments.

If the delegate needs to hook up into the test case life cycle, it can be defined as a JUnit 4 rule or JUnit 5 extension model.

This example connects against a running car manufacture application via HTTP. It can create and read cars, mapping and verifying the responses. The reader may have noted how the delegate encapsulates communication internals, such as HTTP URLs, status codes, or JSON mapping. Its public interface only comprises classes that are relevant to the business domain of the test scenario, such as Car or EngineType. The domain entity types used in system tests don't have to match the ones defined in the application. For reasons of simplicity, system tests can use different, simpler types that are sufficient for the given scenario.