Continuous Delivery and the Need for Automating Acceptance Tests

About continuous integration and continuous delivery, a lot has been already said and written. My favorite resource is the book by Jez Humble and David Farley Continuous Delivery.¹

The practice of continuous delivery is all about shortening the feedback loop by automating all the necessary steps to provide quality software to end users. And among these steps, there are of course the tests.

Continuous integration on the other hand is about frequently integrating the developments into a main branch, but not necessarily deploying this into production.

In these four domains, several constraints and maturity levels can be present in an organization.

In our project, a COTS integration type of project, some steps were possible and others not, either due to technological or organization constraints or by choice.

For example, until the 1.0 release, it was not possible to deliver frequently some functionalities or part of it. This is mainly to avoid having the business work with two systems.

The development work was done by the editor; therefore the continuous integration and code versioning was not under our control.

In terms of testing, unit testing was not under our control either, and our focus was on Acceptance Testing and automation of the regression tests.

The deployment automation was possible: the Jenkins pipeline was provided by the editor and adapted to our environments.

Acceptance testing can be defined as the last level of testing in most cases. Automating acceptance tests is important to catch issues that other test levels don’t catch.² Once automated, acceptance tests can be run often and early on, unlike manual tests.

Regression testing must be part of the acceptance test suite. The challenge to cover new features can be addressed by adopting a BDD approach, as explained in Chapter 13, “Gherkin and Behavior Driven Development.”

In the next section we dive into the steps necessary to automate the regression tests.

Puppeteer and the Boozang Runner

Puppeteer ³ is a node.js library used for Web UI automation. Much like Selenium , it provides an API to control the browser, but compared to Selenium has some significant differences. First of all the age; the project was developed in 2017 (Selenium in 2004) by Google, hence focused on Chrome support (Selenium instead supports all the most common browsers). In terms of supported languages, Puppeteer is full JavaScript, while Selenium supports many different languages.

Compared to Selenium, Puppeteer is simpler to install and use. The protocol used is also different (DevTools⁴ vs WebDriver⁵), providing important advantages in terms of testing capabilities.

From a goal point of view, Puppeteer’s focus is not on testing, but on pure Web UI automation. It therefore lacks all those extra features that are useful for testing, like recording, but this is what Boozang does.

The Boozang runner is a node-js package⁶ built on top of Puppeteer that allows you to execute Boozang tests from the command line.

Another (beta) version of the Boozang runner⁷ is based on Microsoft Playwright.⁸ Both versions of the runner are freely available on GitHub under the MIT license.

When you run tests through Jenkins however, you will not use the CLI runner, but rather a docker container that encapsulates the CLI and makes it easy to run tests. This is the topic of the next section.

How to Use the Docker Runner

Let’s see how to run a simple test via the Docker runner.

First of all, make sure to have Docker installed on the machine where you will run the tests.

To execute a test, you need a “tokenized” URL, that is, the URL of the Test with your personal token (linked to your Boozang account). To get this URL, from any Test in the IDE simply open the Advanced Operations menu, select the link “Generate Tokenized URL ,” and input your Boozang account password. You will then be able to copy the URL.

A screenshot of an advanced operations webpage on Boozang I D E. An arrow points to a link on the page for generating tokenized U R L.

Generate tokenized URL in the Advanced Operations menu

A screenshot of a password entry window. It contains a test case link and a password entry field for the token execution link.

Enter password screen

A screenshot of a window with a tokenized U R L. The page contains a test case link and a token execute link that is selected.

Tokenized URL to copy

The URL format is the following:

https://eu.boozang.com/extension?token=<your personal token>&env=<environment ID>#<project ID>/<branch>/<module ID>/<test ID>/run

To see all the available options of the Docker runner, you can head to the github repository page.¹⁰

In the previous example we used a single Test, but of course this process can be followed to execute a Test Suite that contains Tests or Scenarios. It won’t be available however on Modules and Features since these two objects are mainly to group Tests and Scenarios.

Keeping Execution Time Low with Parallel Runs

Execution time of your test suite is crucial. You want your full suite, which might include hundreds or even thousands of scenarios to run easily overnight for regression testing. And you want a subset of them, critical, to run in 15 or 30 minutes so that, while implementing new tests, you can check their impact on existing tests.

Executing tests or scenarios in parallel is the main solution, combined with other good practices like using APIs to setup the initial state, and keeping the scenarios focused on a single objective.

Parallel Execution and Workers Setup

When your acceptance test suite starts growing, the execution time might soon become an issue. In this case, executing tests in parallel is your main ally.

In the NIS project we started with parallel runs early on, around the third implementation Sprint.

We initially used GNU Parallel , then used plain Jenkins pipeline, and finally moved to the new feature developed by the Boozang team, the distributed master-slave setup. In the next sections, we will go through these three options. Clearly, the latest option is the best one, and we will spend more time on it.

Boozang Workers Concepts

The workers approach is similar to an agile team. Each worker is a team member, and each team is a group. Each worker can have a scope that is like a specialization (like BA, Developer, Tester). There is finally a master that is like a PO or the backlog itself.

The master can dispatch tests to workers, based on their scope. Normally master and workers belong to the same group.

An illustration of the Boozang work concept. It presents the components of the agile team and the Boozang workers in a similar format.

Boozang distributed execution approach

Each worker must have a specific ID, and a worker can be associated with only one group.

This organization offers a lot of flexibility for running your tests in a distributed way. We will focus mainly on parallel runs, which is just one of the possible use cases for distributed runs.

In this case all the workers have the same scope, no specialization. The master dispatches scenarios to the workers as soon as a worker is available.

But it gets even better; the parallelism is at scenario iteration level: this means that when you have Examples with Scenario Outlines, examples can be run in parallel. As far as I know, Boozang is the only Test Automation tool that offers this level of parallelism.

One interesting use case for having several groups is to reserve a group for nightly run, and another for debugging. Another one is in the case of a large organization with several automation teams; each team could have its own group of workers that can run independently of others.

Boozang Workers Setup in the Jenkins Pipeline

When using a declarative pipeline in Jenkins , there are some key steps to follow to run tests in parallel.

1. 1.

   Launch n workers (the number will be limited by the resources available on your server¹¹). Workers will stay idle until they receive some execution tasks.

    
2. 2.

   Launch the master with the Test Suite you want to run. The master will dispatch the execution tasks to workers as they are available.

    

Launching the Workers

In order to launch the worker containers, we have two steps: creation of the worker URL and execution of the docker run command. To define the worker URL, let’s see a snapshot taken from our pipeline with variables defined for each URL element.¹²

worker_url = "http://${params.BZBASEURL}/extension?token=${params.BZTOKEN}&env=${env}&group=${group}&key=$worker_key&self=${self}#${params.BZPROJECT}/${params.BRANCH}"

This URL is like the previous tokenized URL already seen, with the following differences:

• There is no Test or Suite ID at the end, which means the docker runner will run and listen for tasks from the master.

• There are three additional fields:

  • Group: An identifier to group workers (we currently use the Jenkins build number for this). This can be the counter of the loop mentioned below.

  • Key: An identifier unique for each worker.

  • Self: It can be either 1 or 0. 1 means that the worker is private to the user linked with the token specified in the URL. 0 means that the worker can be shared among different users.

Create a loop to launch the container in detached mode to allow the pipeline to continue.

sh "nohup docker run --name bg-worker$counter -e TZ=Europe/Paris --rm -v "$/var/boozang:/var/boozang/" -v "/var/common:/var/common/" styrman/boozang-runner "$worker_url" >  ${reportsDir}/boozang-worker_${worker_key}.log &"

Let’s see the elements of this command:

• nohup: Used to detach the command from the user session.

• name: To name each worker container.

• -e: Set an environment variable for the timezone, so that timestamps in the logs are aligned with your time.

• rm: Already mentioned, it will remove the containers after run.

• -v: Set some volumes for logs and reports.

• The output of the command is redirected to a worker log file, that can be useful when debugging.

• Finally the whole command is run in detached mode.

Launching the Master

To launch the master we follow two similar steps: URL creation and docker run command.

master_URL = "http://${params.BZBASEURL}/extension?token=${params.BZTOKEN}&env=${env}&group=${group}&key=$master_key&self=${self}#${params.BZPROJECT}/${params.BRANCH}/${params.TEST_ID}/run"

This is a very similar URL to the one used for the workers, the only differences are

• The key is the one of the master, another unique ID.

• We have the information of the suite to run (TEST_ID parameter) with the “/run” to launch the execution.

sh "docker run --name bg-master -e TZ=Europe/Paris --rm -v "/var/boozang:/var/boozang/" -v "/var/common:/var/common/" styrman/boozang-runner "$master_URL" >  ${reportsDir}/boozang-master.log"

This is a very similar command to the previous one used to run the worker containers. The only difference is that now we do not use the `nohup` nor the `&` so that the pipeline will wait for this command to finish before passing to the following stages.

Reports and Log Files

When the pipeline is executed, you want to make sure to have all the necessary log and report files generated by Boozang. What are those files located and what do they contain?

Execution Reports

Test execution reports are stored, within the container, in /var/boozang. In this folder you will find the Cucumber JSON reports. These reports can be used in two ways:

• With the Jenkins cucumber reports plugin¹³

• By sending these reports to XRay (as mentioned in the previous chapter, section “Pushing Test Results Back to XRay”)

It must be noted that if you do not use the Gherkin layer, you still have access to the Boozang standard reports, but these are by default stored on the server side. If you want to download them, you need to specify the “auto-download” option in the Boozang settings.

Log Files and the Log Formatter

Each Boozang container will produce some execution logs, which will be stored as well in /var/boozang.

When running several workers, each worker will produce a log file.

As we will see later, in our Jenkins pipeline, we have a stage to aggregate the worker log files into a single one.

A screenshot of a Jenkins webpage with multiple lines of code for log execution. The page highlights the worker logs and the aggregated log.

Log files published on Jenkins

Having an aggregated log file allows us to use the log formatter that is available from the add-on icon:

A screenshot of a log formatter tab. It has tick boxes for auto format, retrieving test worker logs, and fields for performance thresholds.

Log formatter

Using the log formatter will give you the possibility to better troubleshoot the scenarios, adding a structure, colors, and useful performance information directly from your Jenkins page:

A screenshot of a Jenkins window with the formatted logs. Create and cancel policy under policy cancellation in the completed list is selected.

Log files formatted

Root Cause Analysis Report

The Root Cause Analysis report (see Chapter 11, “Reporting and Troubleshooting”) is a special type of report that is produced by a script that you can download from GitHub at the URL:

https://raw.githubusercontent.com/ljunggren/bz-utils/main/scripts/generate_summary.sh ".

Here is the pipeline stage used to produce the RCA report :

Twenty-three lines of code. The code performs a root cause analysis and publishes the result.

RCA report generation stage

The main steps are

• Install prerequisites packages: jq and bc

• Download the shell script, adapting the rights if needed

• Run the script giving as input the JSON reports files

• Copy the html reports produced in a report folder of later use and display

Publishing Reports on Jenkins and JIRA

To publish reports on Jenkins , we rely on the Cucumber reports plugin¹⁴ and on the HTML Publisher plugin.¹⁵

We have four main sections to publish for each Jenkins build:

• Cucumber reports → Cucumber report plugin

• Log files → HTML publisher

• Cucumber reports → HTML publisher

• RCA report → HTML publisher

A screenshot of cucumber reports. It lists worker logs, cucumber J son files, and R C A report.

Reports sections

Here are the two stages used to publish these reports.

Cucumber Reports

The following picture shows the pipeline stage for producing the Cucumber reports .

Seventeen lines of code. The code publishes the Cucumber report after collecting the results with the cucumber report publisher plugin.

Stage to publish cucumber reports

Boozang when running Gherkin scenarios will produce a JSON result file for each Scenario, and in case of Scenario Outline, for each iteration. You will then have a lot of JSON files in the report folder.

To send the execution reports to XRay, all those files need to be aggregated into a single one (more on this in the next section), while for the Cucumber report plugin we let the plugin take care of it (mergeFeatureById set to true).

The main thing to note in this stage is that we give all the reports to use (*.json) via the fileIncludePattern property, and these files will be looked for in the jsonReportDirectory.

We use the fileExcludePattern to avoid picking up a JSON file that is the aggregation of all the JSON files.

The results will then be shown as in the following picture.

A screenshot of the cucumber report with a pie chart. The pie chart represents the passing and failing statistics for the features from the report.

Cucumber reports

These reports are interesting and quite nice for communicating the results. For troubleshooting they allow you to zoom in the failures and see the details.

HTML Reports

The following step for the HTML reports is shown. We execute this step in the post section, so that it will be produced even if some of the previous stage fails.

Multiple lines of code in 3 sections. The code publishes worker logs, J son reports, and a root cause report.

Each section creates a menu in the left bar of Jenkins with the name mentioned in the “reportName” property. The options are quite self explanatory; you need to specify the files you want to include in the page and the folder containing those files, always relative to the Jenkins workspace.

A screenshot of the dashboard on the Jenkins window. The worker logs, cucumber J son files, and R C A report on the left panel are highlighted.

HTML publisher menus

JIRA Reports

The stage to import test results in JIRA has been already covered in Chapter 14, “JIRA with Boozang via XRay.”

For simplicity, we report it again here as follows.

Twenty-five lines of code. The code uploads cucumber results to Jira.

Complete Working Pipeline

In this section we review our pipeline which has now reached a good level of maturity and stability.

The following picture shows the main stages, and then the following sections will review the code in each stage.

The exact code was reported to help the reader and see a real working example.¹⁶

A flowchart of a working pipeline in Jenkins. The flowchart has 3 stages namely initial settings, pipeline stages, and post.

Jenkins pipeline

The first step to prepare for a parallel execution with different credentials is to set up the test users. This is done directly in the application and then aligned in Boozang as explained in the next section.

Docker in Docker Setup

In our Jenkins setup, we have a dedicated linux host on a virtual machine, with a docker engine running on it. Jobs run on this slave but with a docker agent created on the fly,¹⁹ and destroyed when the build is finished. The image is an Ubuntu image with a few more packages installed, hence a dedicated boozang-image.

In the dockerized Jenkins boozang-agent , we then execute the pipeline, which then launches the actual boozang master and workers.

In this “docker in docker” setup, a correct setup of the volumes is critical. The following picture summarizes the setup.

A flow diagram of the docker setup in Jenkins. The job is executed on Jenkins master and slave with volume setup for agent, master, and worker.

Jenkins Docker in Docker setup

The Boozang runner containers store report files in /var/boozang. We mapped this folder to /tmp/boozang on the Jenkins boozang-agent . In order to save those reports permanently, we then map the /tmp/boozang folder to a local folder on the physical host, where enough space is available.

It is worth mentioning that during remote executions via Jenkins, the currently active workers are visible in the UI. The following two screenshots have been taken at two different times during the execution of a scenario outline.

A screenshot of a popup Boozang window with an execution worker list. The window has configuration items and a tab to generate C I, U R L.

CI view at the beginning of the job

A screenshot of a popup Boozang window during worker execution. It lists the workers and provides updates on success, failure, testing, and waiting.

CI view during workers execution

While setting up your Jenkins pipeline, you might need to check directly on the containers what is happening. Especially to make sure that volumes are correctly mapped and that access permissions are correctly set.

If you are not too familiar with the Docker CLI, a tool like Portainer²⁰ can help. Without going into too much detail, Portainers allow you to see the images available, volumes and networks, and especially the running containers.

A screenshot of a Portainer window. The dashboard provides an endpoint summary of stacks, containers, images, volumes, and networks.

Portainer UI

When the containers are running, you can list them, enter the container shell, and see what is going on in case of issues .

A screenshot of the container section on Portainer dot i o. The section lists container I D, images, commands, dates, status, ports, and names.

Boozang workers view in Portainer