Back Matter

Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

Conclusions

I hope you enjoyed this read and most of all learned some new things for your professional activity.

My initial goal was to write a book “from the trenches” and not a plain user guide or theoretical testing book. I believe this goal was achieved by not only presenting Boozang but most of all how we used it in a real project. Of course, the recipes presented in this book might not apply to different contexts, but I believe that what was presented can still be useful to other testing folks.

It was not possible to show in detail all the features that Boozang offers; for this, you can always refer to the official Boozang documentation and their support channel. Whenever possible, I updated the screenshots to the latest IDE version.

Every feedback is always welcome, so feel free to reach out via my LinkedIn profile or via the book blog:

Appendix: ROI Example for a Test Automation Initiative

Let’s look at an example of a fictitious automation initiative where the SUT is in place and automated tests need to be developed to replace part of the manually executed regression tests.

Let’s assume the following for our example:

Licenses for the Test Automation tool cost 10K¹ per year.
Training for the automation team costs 10K for 2 weeks.
It will take 6 months to automate 80% of the regression test suite.
Two FTEs working on the automation project, with an average daily cost of 1K per FTE.
There are monthly releases, with 1 regression test cycle per release.
Each regression test cycle run manually takes 3 FTEs for 5 days.
The average cost of one FTE running manual tests is 1K per day.
There are 20 working days in a month.
Once the automated suite is in place, you will need 3 man-days² of maintenance effort per month.

The ROI formula we use is the following:

$$ ROI= gain from investmentdiv cost of investment $$

This calculation will not consider the time value of money; therefore, ROI calculation is done without incorporating the net present value concept.

The gain from investment is going to be the cost savings achieved on the testing activity.

To calculate this gain we will take a look at the costs with automation and costs without automation, over a period of 3 years, which is a reasonable time for most initiatives of this type and scale.

In case of no automation, the monthly costs are equal to running the entire suite manually; 3 FTE * 5 days * 1K = 15K.

In case of automation , the first month you will have the manual testing costs (15K), the licenses (10K), the training (10K), and the automation effort (20days*2FTEs*1K=40K) (which sums up to 75K).

To make the computation simple but realistic, let’s say that for the first month there are no benefits since you are learning and setting up the tool. The following 2 months you manage to automate 10% of the tests each month, and then your efficiency increases to 20% per month.

So the cost with automation will be the automation effort (40K), plus the remaining manual effort (15K-10% for month 2 and 15K-20% for month 3, 15K-40% for month 4, 15K-60% for month 5, and 15K-80% at month 6).

On month 6 your automation work is complete, and for the following months you just have 3 man-days of effort per month needed for maintenance. Therefore after the sixth month, each additional month you are left with 20% of manual testing (3K) plus the automation maintenance (3K), for a total of 6K per month.

Month	1	2	3	4	5	6	7	8	9	10	11	12	...	24	...	36
Without automation	15	15	15	15	15	15	15	15	15	15	15	15	...	15	...	15
With automation	75	53.5	52	49	46	43	6	6	6	6	6	6	...	6	...	6

To correctly draw a graph you need to cumulate the costs.

Also, at month 13 and 25 the license renewal costs must be considered (remember it was 10K per year).

Month	1	2	3	4	5	6	7	8	9	10	11	12	...	24	...	36
Cum without automation	15	30	45	60	75	90	105	120	135	150	165	180	...	360	...	540
Cum with automation	75	128.5	180.5	229.5	275.5	318.5	324.5	330.5	336.5	342.5	348.5	354.5		436	...	518.5

Using the last table with the cumulative costs , we can draw the following graph.

A line graph of the cost versus month. 2 curves representing cumulative costs with or without automation for 40 months. Both the lines display an increasing trend with an increase in months. The lines cross at about month 16.

As you can see from the graph, the costs with automation start to be lower than the costs without automation around month 34. This is called the break-even point .

In this case, almost 3 years are needed for the automation initiative to be profitable.

To calculate the ROI over 3 years, let’s look at the costs with and without automation :

Costs without automation: 540K
Costs with automation: 518.5K
→ ROI = gain from investment / costs of investment = (540-518)/518 = 4.15%

This was a simple example, and one could argue that the business case is weak.

But this is like comparing apples to oranges. With automation, you have added benefits. First of all, you can run all the regression tests nightly, rather than just once per release as you would do if you were to run them manually.

Therefore to make it more realistic and to have a similar quality objective, you could assume a scenario where you would run the manual tests 2 times for each release.

In that case, the costs without automation are 30K instead of 15K for each monthly release.

Adapting the table and resulting graph would look like:

A line graph represents the variations in the cumulative costs with and without automation for 40 months. Both the lines depict an increasing trend with an increase in months.

In this case, the break-even point is sooner, around month 16.

And the ROI over 3 years is

Costs without automation: 1080K
Costs with automation: 677K
→ ROI = (1080-677)/677 = 62%

As you see, the business case changes considerably, and we can push the scenario even further.

When using automation you find issues early on, while the fixing cost is lower.

To account for this you can assume that a certain percentage of issues discovered during UAT are now discovered during development, with little to none additional cost for fixing it.

Let’s assume that, without Test Automation, 1 regression defect is spotted on UAT every other release. Over 1 year it makes 6 defects. Considering 2 days of effort for fixing a defect found in UAT (bug reporting, analysis, fix implementation, deployment, and retest), this means 12K saved each year, and 36K over 3 years.

Let’s consider this and update the ROI :

Costs without automation: 1080K + 36K = 1116K
Costs with automation: unchanged, 677K
→ ROI = (1116 - 677) / 677 = 83%

Moreover, the manual test efforts you saved (12 man-days each month) will be reinvested in usability testing, or used on other value-add tasks.

Last but not least, the two team members that went from 15 boring days of manual regression tests to just 3 days will be much happier!

With these examples, I just wanted to illustrate how an investment that may initially seem worthless, in reality proves to be beneficial. Test Automation , both at the unit testing level and system level, is rarely a poor investment.

Sure, this type of analysis can and should be done in specific cases like

Legacy applications with a limited lifetime
Systems technologically complex where Test Automation might be challenging
Systems where releases are not frequent and where regressions can be tolerated and quickly fixed

Aside from these specific cases, Test Automation should not be doubted: the question should be not if, but when, how, and to what degree.

In my experience, I have seen contexts where Test Automation was not applied from the beginning and manual testing effort was one of the reasons that slowed down release cycles and limited system evolutions. Other times, the Test Automation effort had to be justified and a formal ROI deeply questioned.

Luckily, in the NIS project , I was afforded a “good” context, where we do not question whether to apply Test Automation or not, but rather, how to do it in the most efficient way.

Index

Acceptance testing

ACP environment

API/GUI scenarios

actions

API calls

considerations

meaning

PCAS application

product acquisition step

Automation team

Behavior Driven Development (BDD)

COTS

discovery

formulation

success factor

Bind data

Bookmarks

Boozang

API calls

API tests

creation window

empty module

Google APIs

main items

NISTA module

parsing/validating, response object

architecture

branch

details

develop

display

Jenkins

management

master

milestone

naming convention

Test Automation

Chrome Extension

client-server communication

concurrent development

confirmation email

data management

ease of use

edit project details

feature branch approach

Gherkin/BDD

GIT

Git-Flow

home page

HTML fragment

language-based element recognition

location strategy

merge process

Advanced operations menu

branch selection, no conflicts

changes

options

possibilities

projects dashboard

projects profile page

registration page

support/vision

team access

Test Automation

test results

import options

input format

JIRA instance

pipeline stage

plugin

post build action

specification

Xray

useful links

welcome email

Boozang runner

Business User

Client-server communications

Commercial Off-The-Shelf (COTS)

Conditional flows

back-dated policies

FOR/WHILE group

Group action

group type

IF group

If-Then-Else groups

PCAS logic

products

dependent products

ETI

TCS membership

TCS Membership

Content Policy

Continuous delivery (CD)

COTS integration

deployment automation

development work

domains

INVEST principles

Continuous integration (CI)

Cucumber reports

Cucumber website

Customer Relationship Management (CRM)

Data management

defectsCreatedDuringTestExecution() function

Details panel

bottom

tabs

top

Digital Services team

Docker in docker setup

boozang-agent

Boozang runner

Boozang workers view

CI view

containers

Jenkins

Portainers

Docker runner

Generate Tokenized URL

options

password screen

run command

URL format

Dom Picker

action config

autocheck

element

index

method

speed

text

uniqueness

Enterprise Resource Planning (ERP)

Environments page

Environment URL

Executing tests/scenarios

Execution modes

Execution reports

Execution time

log files

execution reports

log formatter

RCA report

parallel execution/workers setup

GNU parallel

Jenkins

NIS project

workers concepts

publishing reports

Cucumber

HTML

Jenkins

JIRA

Exit conditions

Action settings tab

default behavior

exit action

Goto setting

mastering

overridden

report

Test action

test flow

Explicit lock

Features

Fictitious automation

Full regression suite

Functional testing

Functions details log

Function summary log

G, H

“General Ledger” (GL)

Generate Test

Gherkin

automated test

Cucumber tool

definition process

discovery process

endorsement management

policy cancellation

policy numbering

post-implementation phase

practices

scenarios

Gherkin format

Gherkin tests

GIT

Global suites

Integrated Development Environment (IDE)

Integration

INVEST principles

J, K

Jenkins configuration

setup input

setup output

Jenkins Cucumber reports

Jenkins pipeline

Boozang workers and master

build description

Cucumber reports

HTML publisher

import results, XRay

job parameters

JSON report/log files

RCA script

smoke tests

test users setup

variables

version

JIRA

XRay

automated tests

Coverage chart

Coverage configuration

features

issue types

JQL functions

project/release

typical test process

UAT dashboard

JSON Server

Linux Kernel

Locating elements

dropdown list

fixed column

non-unique

Loops

financial data

group

iteration

loop condition

while loop

PCAS

data table

parameters

scenario

step implementation

steps

Module data ($module)

Modules

create new

page

parts

N, O

Natural language

Navigation tree panel

NIS project

NIS Test Automation

Non-functional tests

Page Object Pattern

Pilot phase

JIRA

Mats

results

Plugged Test

Policy and Claims Administration System (PCAS)

Project data ($project)

Project Settings page

content policy

element definition

environments

integration

Proof of Concept (PoC) phase

Puppeteer

Quality control

Check Test Case

errors

data error

missing elements

missing expectations

missing test case target

issue list tab

issues summary

levels

warnings

empty input values

launch page issue

non-unique element selector

not implementation step

unreferenced test cases

Recorded test

Recording

Regression testing

Reporting execution

Representational State Transfer (REST)

Request for Information (RFI)

Request for Proposal (RFP)

REST APIs

HTTP methods

JSON/YAML files

path parameters

status codes

Return on Investment (ROI)

automation work

break-even point

business case

case of automation

costs with/without automation

cumulative costs

formula

gain from investment

license renewal costs

manual test efforts

NIS project

regression tests

Test Automation

UAT

updation

Root Cause Analysis (RCA)

failure details/hash/scope

issue type

known issues

module.test.action id

reports

Root Cause window

test suite

URL

SaaS solutions

Scenarios

Search engine

Selenium

browsers

development skills

locating elements

reporting/script

timing and waits

WebDriver

Selenium-based framework

code repository

Jenkins Cucumber reports

Server side mock

Single suite

Slack

Software Development Engineer in Test (SDET)

Software Engineer in Test (SET)

Solid automated test suite

Submodules

Summary report

System Under Test (SUT)

Team collaboration

Test actions

Test Automation

Agile development

challenges

culture

long-term maintainability

skills/resources

technology

environments

ATCS

DevInt

JIRA

pre-production

production

test/quality assurance

ROI

testing roles

test-levels

test types

Test data ($test)

Test Driven Development (TDD)

Test map

file type selection

popup test details

relationships map

generation

upstream/downstream tests

use cases

Test parameters

automapping

Console tab

JSON editor

key-value editor

section parameter

uses

testPlanTestExecutions() function

Test reference

Test reports

Cucumber Scenarios

failure details

failure view

reports

IDE

End-to-End scenario

execution screen

Failed Actions

Gherkin scenario

Results list

reports menu

Boozang reports

execution reports

RCA

reports page

Test Suites

Test URL

Timers

Touring Club Suisse (TCS)

Trenches

actions

business logic

contact creation process

exit rules

membership acquisition

product acquisition

renewal operation

API Tests

payment slip

use cases

confluence pages

JIRA Agile board

JIRA Dashboard

JIRA structure

Test

definition

parameters

Trend reports

example search

iteration

results, scenario

search criteria

test page

test results

Troubleshooting failures

UI customization

Unit testing

User Acceptance Tests (UATs)

Validations

exists

innerText

Is Checked

not exists

script

Version control systems (VCS)

Weak locators

Web automation tool

X, Y, Z

XRay-Boozang integration

Advanced Operations

documentation

import method

import process

JIRA filter

list of features

not implemented step

progress view

setting

structure view, tests

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for Back Matter

Create new playlist

Sign In

Sign Up

Table of Contents for
Back Matter