Before playing, let’s set the table with a few rules.
A cartoon of 2 opponents at a chess table.
This chapter introduces basic Test Automation concepts that will be used throughout the book. Besides ISTQB1 standard terminology, I’ll also cover the constraints typical of an enterprise environment that might not be the same for small startups and IT giants.
Common Test Automation Challenges
If you are reading this book, you know why we automate tests, and I don’t need to convince you of the value of moving from manual testing to Test Automation. If you still have doubts, there are lots of great resources on the Web. My favorite read on this topic is Continuous Delivery by Dave Farley and Jez Humble. It was published in 2010 but is still, in my opinion, one of the must-reads for IT professionals trying to improve their delivery process. That book will certainly convince you that you should automate many more manual activities than just tests.
While automation is great, common sense should always prevail. Some automation initiatives are more profitable than others (see section “Test Automation ROI, Theory and Practice”).
My personal view is that manual testing, besides being error-prone and time-consuming, is boring. Test Automation can be fun and we should always try to have fun at work.
Certainly we can agree on the fact that you will never be able to (and should not aim to) automate all testing activities. There are still situations where human beings do a better job than robots. For example, exploratory testing and testing non-functional requirements like usability are better suited to manual tests.
I’d also like to clarify that testing, especially Test Automation, is very context-dependent. In Chapter 4, “The TCS Project,” I will spend time clarifying the context for the project in this book. Organizational structure, business domain, technologies, culture, and background are to be considered when applying Test Automation and selecting a tool.
The next section lists the four main challenges that organizations can face when trying to get management buy-in or implementing Test Automation. This is not a comprehensive list, but one that shines a light on the most common, all of which we will provide practical solutions for in this book.
Challenge N1: Culture
The challenges encountered when promoting or applying Test Automation might be less about technologies and tools than you would expect.
In my professional experience, for example, I’ve worked in two types of environments: one where Test Automation was part of the company’s culture and one where every cent spent in Test Automation was challenged and had to be justified.
In an enterprise environment, projects and budget approvals have key gates where Test Automation is either promoted or killed before it starts.
You might benefit from learning how to advocate Test Automation and prove its benefits (see section “Test Automation ROI, Theory and Practice”).
Another challenge you may face requires soft skills to manage opposing views on Test Automation. You might work in a context where developers and QA are on opposite sides. One is responsible for implementing features, and the other is responsible for ensuring that features work as expected. Thankfully this situation is less and less common these days since agile and lean practices are being adopted.
These two cultural challenges are especially important when Test Automation is done by an off-shore team, or by a team which is not part of the main organization.
Challenge N2: Skills and Resources
Let’s assume your organization uses agile principles, and you overcome the first challenge. The next challenge you might face is about people skills and resource availability.
Even if your team supports Test Automation, you may lack the resources and skills necessary to conduct conventional coded Test Automation.
I was in this situation, and here a tool can help. As long as the tool is easy to use, more people can use it. Better yet, if the tool allows you to be efficient without needing developer skills, you can get more work done, making it easier to scale up your automation initiative.
If your tool and the process driving it fit well into your organization, you will not need to waste time begging for buy-in, convincing stakeholders, and training expensive resources.
Challenge N3: Technology
Less often than you think, technology will represent a challenge for Test Automation in today’s world. Web, API, desktop, and mobile applications are the four main categories supported by functional Test Automation tools. Some tools are better than others in each category. A few commercial tools try to support all of the above, but they end up being hard to use and difficult to integrate into your delivery process.
In my case, which I believe is quite representative, our main objective is to automate web applications (GUI and API), for the critical End-to-End business processes. No mobile and no desktop applications. Therefore a functional web automation tool would cover our primary requirements.
Additionally, the project affords us the luxury of ignoring some edge cases that are difficult to automate. For those cases we still rely on manual testing as part of User Acceptance Tests (the section on “Test Levels” later in this chapter covers UAT and other test levels and types).
Challenge N4: Long-Term Maintainability
A last common challenge worth mentioning is an automation framework that is hard to maintain, with flaky tests which rarely provide the expected value of Test Automation.
Even when you have the right organizational culture, in-house skills, and no specific technological challenges, you might still have problems in the long run, where Test Automation initiatives often tend to be abandoned. This normally happens when Test Automation is made without any previous experience and tests are built with little modularity and reusability. In these cases, a simple UI change might become very costly to address if it impacts many tests. Test Automation projects built without modularity can be more costly to maintain than manual testing.
Even with a great automation tool, if best practices are not applied, success is less than guaranteed.
If any of these challenges sound familiar to you, then this book is for you, keep reading!
Test Automation can be applied at several test levels. Therefore, it is important to understand the concepts of test level and also test type. We will discuss these two concepts at length in the next sections.
Test Levels, Test Types, and Testing Roles
Since the early 2000s, the Standard Software Testing Qualification Board (ISTQB) provides a common vocabulary and certification path for professional testers and test managers, as PMI does for Project Management and ITIL for Service Management.
An advantage to following these standards is the adoption of a common vocabulary. When dealing with partners located in different countries, referring to the same thing when discussing “unit testing” or “UAT” is fundamental.
Let’s briefly review some important taxonomy that will help you better understand the later chapters of this book. We will start with test levels, test types, and test environments. You will then see how they apply to Test Automation and Boozang.
Test Levels
Unit and component tests: These tests are normally implemented by developers for small software units, like classes and functions. xUnit frameworks are used to implement and run these tests. They are dependent on the development language. Boozang does not apply here.
System tests: Here we test a single system or application. Boozang can certainly play an important role at this level.
Integration tests: Testing multiple systems to validate how they interact and provide a certain user functionality. Boozang definitely helps here with its GUI- and APIs-based End-to-End capabilities.
User Acceptance Tests (UAT): At this level, Boozang can replace manual tests. This is especially true when automated tests are part of a fully automated delivery pipeline.
In reality, few organizations can deliver to production environments without the validation of real users. Therefore, in most cases, I say Boozang can reduce the manual effort needed. This allows end users to focus on usability issues rather than regression tests.
Test Types
Functional and non-functional testing: Functional testing is definitely where Boozang shines. For non-functional tests , Boozang provides some interesting features for performance testings, while for things like usability and security2 testing, other tools are certainly better suited. Boozang can measure the performance of any actions (UI and API), but it is not made to generate load with different patterns.
Testing related to changes: In a classic release management process, for each new release, we need to test new features while making sure existing ones didn’t break. Regression testing is essential, and automating regression tests is key to keep down application maintenance costs and provide confidence with frequent releases. Regression testing is definitely where Boozang fits, and, depending on the process, it can also be used to test new features before they are shipped.
White vs. black box testing: Boozang can be used in a black box approach,3 meaning that you do not need to know how your System Under Test (SUT) is built to be able to test it. White box testing is usually done at the unit test level, often using tools like Sonar Cube ( www.sonarqube.org/ ).
Testing Roles
Since the early 2000s , the profession of test engineers has gained popularity, and certification paths are more available than ever for professional testers. Besides the advantage of having a common dictionary, as I already mentioned, testing and test management techniques have become widely consolidated and adopted.
In 2012 the book How Google Tests Software was published, and with its gain in momentum, so did the profession of Software Engineer in Test (SET) or Software Development Engineer in Test (SDET) . Today SDET professionals are highly sought after and an integral part of performant agile teams.
As the name suggests, an SDET is a software developer that has basic development skills and is able to apply these skills specifically to software testing. Besides knowing Test Automation tools, an SDET should also be able to set up and use tools like GIT, Maven, Jenkins, and Sonar to build CI/CD pipelines.
They should also have the necessary soft and hard skills to build effective Gherkin4 scenarios with business users and/or product owners. Most of all, they should stay curious, with the quizzical attitude and desire for quality typical of test engineers.
In agile contexts, however, the separation between developers and testers can be fuzzy. In an ideal world, a cross-functional team can assign implementation and testing tasks to any member, so that efficiency is maximized.
We covered test levels, test types, and testing roles. To wrap up, the next section will cover test environments and discuss which environment category is suitable for each test level, test type, and type of users .
Test Environments
Production: This is the environment used by business users to perform operational work. Despite current trends to shift-right, that is, testing in production, most enterprise companies are not comfortable allowing tests in production. Hence, no Boozang tests here, unless you know what you are doing. The only testing activity you may want to perform in a production environment is normally called a “sanity check.” Sanity checks are used to validate that deployments have been executed correctly and the environment is up and running.
Pre-production/acceptance/staging: This is the environment used by business users to perform User Acceptance Tests. This environment should be feature equivalent to production and similar in terms of sizing. You can certainly run Boozang non regression tests on this environment, and let business users focus on validating new features.
Test/quality assurance: This is an environment where most of the testing activity should take place. It can be functionally equivalent to production and pre-production but less “powerful” in terms of sizing.5 This is the main environment where we design, run, and troubleshoot Boozang tests before running them in pre-production.
Development and integration/DevInt: One or more environments can be used to develop and integrate code. Normally, in these environments, developers run unit and component tests. Nothing prevents you from running Boozang tests here; in our project we wanted to implement tests in a more stable environment, like the previous two.
This covers the most common environments you will find, but others are possible depending on the specific needs of each project or organization (e.g., training, migration purposes, or for executing load and performance tests).
How We Manage Test Environments
Effectively managing test environments is important for every IT project. In our project (see Chapter 4, “The TCS Project”), we put in a considerable effort early on to outline the delivery chain and make sure the entire team was aligned on which environment must be used for each type of activity. This work is crucial when working with remote teams and integrating COTS6 solutions.
A flow diagram of the delivery process in local and remote environments. The steps include feature, development, sprint, build delivered, and go live.
A TCS model for test environments
We also noted which test activity is performed where, and the frequency of deliveries from one environment to the next. In this example, developers work on a local machine while developing on a feature branch. A pull-request based approach integrates all the developers work into the main branch, deployed on the DevInt on which unit tests are run. On a nightly basis the code is deployed on a CI environment where the automated regression suite is run. With a frequency that can be nightly or ad hoc, the build tested in CI with success is promoted to the QA environments, where in sprint testing activity can be performed. At the end of the sprint, the validated Stories are delivered on the ACP environment (pre-production) for end users validation.
The QA, ACP , and production environments are fully integrated with other systems via an API management layer that we called NIP.
In Chapter 4, “The TCS Project,” where the project scope and organization are described, we will see how we implemented this model in more detail.
Managing Test Environment in JIRA with the GoLive Add-on
At TCS, we use a JIRA add-on called GoLive from Apwide7 that is very handy for managing test environments. Each application environment has an entry in JIRA, and deployments are recorded so that each stakeholder knows, at any time, which build is deployed on any given environment. This information is key for testing since it should be clear whether a certain Story or Bug fix is present and can be tested.
A screenshot of the environment search window. The page lists the name, U R L, attributes, versions, status, watchers, and actions.
Environments page
A common challenge is that environments are often shared resources, used for different kinds of activities: security scans, load testing, infrastructure changes, etc. Since you don’t want someone to reboot your application in the middle of a business demo, you need to schedule those activities carefully. Apwide GoLive helps us avoid booking conflicts, thanks to a powerful timeline editable featuring a convenient drag-and-drop interface.
A screenshot of the timeline C R M window. The timeline of the scheduled activities is on the left and the calendar is on the right.
Timeline environments
Apwide GoLive goes way further than our basic use case, by allowing the integration of deployment events and environment statuses into a Continuous Delivery pipeline using the GoLive RestAPI (a Jenkins Shared Lib is also available). Thanks to Webhooks, JIRA users can also trigger deployments and/or environment provisioning.
In the “Application Environments” section, we will cover how Boozang manages Environment Categories and application URLs.
At the time of this writing , the entries in JIRA are duplicated with the ones in Boozang since no direct integration is available. Given the frequency this information changes, this is not too bad after all.
Test Automation in Agile Development
Agile development is widely adopted these days. Even the most conservative enterprises are adopting some of the principles found in the Agile Manifesto ( https://agilemanifesto.org/ ). On the other end of the spectrum are the organizations that adopt agility at the enterprise-scale with frameworks like SAFe (Scaled Agile Framework www.scaledagileframework.com/ ).
Most small and large enterprises today are still in a period of transition toward Agility, and they tend to implement their own brand of agile. In most cases, it has resulted in a sort of Water-SCRUM-fall, where the requirements gathering and final UAT still take a considerable effort. The TCS project falls into this category, as we will see in Chapter 4, “The TCS Project.”
About Agility and Testing, much has been said, and in my opinion, sometimes too much. Testing in an agile project requires additional skills, but overall the transition to agile should not scare test engineers that are used to waterfall approaches. Good testing practices remain good regardless of methodology.
Framework knowledge: If you are part of a SCRUM team, you should know the basics of SCRUM, specifically the roles and ceremonies.
Practices like TDD (Test Driven Development 8) and BDD (Behavior Driven Development 9) are often used in agile projects. One should learn the basics of them.
Soft skills: Collaboration and communications are more important than ever in an agile team. If you are used to receiving detailed scenarios and working alone at your desk, you should prepare to work directly with developers and product owners while tolerating not having a detailed long-term project plan.
One thing you will face while testing in an agile project is the constant evolution of your SUT (System Under Test ). Automated tests developed in one sprint might need to be adjusted in the following sprints. Even worse, some might need to be completely redone or thrown away.
In Chapter 13, “Gherkin and Behavior Driven Development,” we will cover Gherkin and BDD which are key parts of testing in agile contexts.
In most agile initiatives automated tests are included from the very beginning, ideally with practices like TDD and BDD. There will be times where your pursuit of Test Automation might be challenged by managers questioning the effort and the benefits. We all know that Test Automation projects have a high failure rate, and many have bad memories of their last automation effort that cost a fortune and was abandoned before producing any real benefits.
In the next section, you will see how to calculate an ROI analysis for a Test Automation initiative, if you really have to. ;)
Test Automation ROI, Theory and Practice
Test Automation, like any initiative that requires investment, must prove to be profitable.
ROI analysis is one of the common methods used when quantitative data needs to be provided to justify the budget needed to acquire a Test Automation tool. Luckily, the pricing for the Boozang tool is very reasonable, and most medium to large organizations can easily afford it.
A note on legacy applications. It mentions living documentation that makes software become legacy.
From my experience, objective ROI analysis for Test Automation initiatives is a minefield. Assumptions must be clearly stated, and numbers can easily be changed to make a good initiative look like a bad investment. The opposite is also true. Without an honest discussion about the assumptions, you can present a very bad initiative as a worthy investment.
A timeline of R O I analysis. T 1 to T 4 indicates manual testing, manual plus partially increasing automation, and maintenance plus manual residual.
Timeline for ROI calculation
There are some peculiarities with ROI estimations for Test Automation.
First of all, between t1 and t2, you should consider both the cost of the initiative and the cost of manual testing. From t2 on, the benefits can be added, but without forgetting that manual testing costs are reduced although still present. Moreover, automated tests need to be maintained, and this is a cost you should not forget to include. Maintenance cost estimation is very important and difficult to do. A solid and modular approach can be easy to maintain, while a poorly designed monolith, with lots of duplication can be very costly to maintain.
Another point to consider is that, once automated, tests will run way more frequently than if they were to run manually. Hence the comparison has to be “fair,” and consider quality independently of test execution time. This is the case for defect counts and defect fixing costs. Having a comprehensive set of automated regression tests normally prevents major bugs from being shipped into UAT phases or even worse into production.
Finally, you should not forget to include other benefits of automating tests that can be hard to quantify like employee satisfaction. Implementing and maintaining automated tests, when done correctly, is much more fun and rewarding than running manual tests over and over again.
If you are interested in knowing more on how to perform an ROI analysis for a Test Automation initiative, in the Appendix section “ROI Example for a Test Automation Initiative,” you find a detailed example with further considerations on this topic.