Chapter 4: Matching Personas and Use Cases to Testing Frameworks

Choosing a test automation framework is a fundamental aspect of the software development life cycle. A test automation framework serves so many different objectives and, as such, must be suited to cover different needs of both developers as well as test engineers. Specifically, in web application testing, the variety of frameworks that are available is huge; hence, there needs to be a prescriptive methodology that considers all the relevant pillars, as well as meeting the current and future-looking needs of users. Within a typical web application software release, there are a couple of personas that contribute to the overall quality of the product, including frontend developers and test automation engineers (also known as SDETs – software developer engineers in testing). A test automation framework and, in many cases, a combination of more than a single framework ought to match these personas' needs, skill sets, and expected velocity around scale and feedback loops.

In this chapter, we will provide a set of considerations to help these two main personas to choose the best test automation framework for their needs and suggest an innovative Venn diagram that illustrates these requirements and differences between the personas, their use cases, and the top-four test automation frameworks (Selenium, Cypress, Puppeteer, and Playwright).

The chapter is designed to cover the following:

• Introduction to the key personas within a web application development project
• Specifying the various use cases and considerations that need to be top of the mind when picking a test framework
• The test automation frameworks evaluation matrix – a table comparing the features across leading test frameworks

Upon completion of the chapter, you will understand the differences between each persona, the main objectives of various persona within a web application development project, and the differences in each test automation framework by category.

Technical requirements

The code files for this chapter can be found here: https://github.com/PacktPublishing/A-Frontend-Web-Developers-Guide-to-Testing.

Web testing personas overview

As already stated earlier in this book, it takes a village to build high-quality web applications, and specifically, testing such advanced apps is quite a challenge.

To cover all the testing types, including functional and non-functional, in a short amount of time, feature teams or squads consist of both developers and SDETs. Such high-skilled and technical resources are tasked with creating new test code upon any new product requirements, as well as maintaining older regression testing suites.

The frontend developer usually cares about the following objectives:

• Ease of creation of test code.
• High-scale and parallel testing to expedite feedback from their test runs.
• The creation of unit and API tests are typically top of mind.
• Having a framework with debugging and mocking abilities
• Integration with CI/CD tools, such as Jenkins and Azure DevOps.
• Integration with defect management and task management tools, such as Jira.

Since the developer persona commits multiple code changes a day, they need to develop very specific test code to cover these changes. In addition, developers unfortunately also create defects; hence, having the ability to properly reproduce bugs, analyze logs and network traffic, and monitor APIs is key for fast resolution of these defects (MTTR – mean time to resolution).

Some of the test automation frameworks cover the preceding objectives more than others, as we will specify later in this chapter. However, these are some of the top-of-mind objectives developers care about when evaluating different test automation frameworks. As Indeed, which is a hiring website, defines, a frontend web application developer is a developer that is responsible for all user-/client-side web application creations, from UI elements, buttons, and menus to compatibility across web and mobile platforms, high-quality code creation, and much more (https://www.indeed.com/hire/job-description/front-end-developer).

For frontend developers to be successful in what they create, they get tremendous help and support from SDETs.

SDETs are responsible for complementing the quality objectives that the developer doesn't. This includes higher coverage of testing types, multiple-platform testing, performance, accessibility, and different functionality testing that isn't in the scope of the developer to cover.

The SDET usually cares about the following objectives:

• Test automation scenario coverage (including the most advanced user flows within the app)
• Cross-platform support (web and mobile)
• Support and documentation within the community
• Support for multiple testing methodologies (functional, non-functional)
• Ease of use and ramping up
• Reporting and debugging abilities
• Integration with CI/CD tools, such as Jenkins
• Integration with defect management and task management tools, such as Jira

As identified, there are a few commonalities between the two persona types of objectives, which makes sense. Test code is code like any other software development project; hence, it needs to be managed, maintained, and used like production code.

Now that we've listed, at a very high level, some of the persona types and their objectives, let's identify what constitutes a solid test automation framework.

Use cases and considerations for picking a solid test automation framework

When considering the long list of activities that both developers and SDETs are responsible for, considering the tool stack that's available for them to accomplish these activities is imperative for success.

The following diagram presents a set of very important considerations that should be part of any proof of concept when choosing a web application testing framework:

Figure 4.1 – Key pillars of a high-value test automation framework

When starting to investigate a web application test automation framework, looking at the previously illustrated pillars is imperative. Let's look at each of them in the following sections. We will not address C.E.L.A as a standalone section since this refers to the overall capabilities together and it sums the software testing life cycle from test creation through its report at the end of execution.

Community considerations

Having a strong community behind the test automation framework is a key requirement for many reasons. Getting ongoing support, innovation, best practices, online discussions, code samples, and more are just a subset of the benefits a strong community can bring to any technology. A small community, an inactive community, and lack of contributions to an open source technology can be a negative sign that this solution is not really ramping up and adopted, and it can suggest that developers should look into other alternatives.

Scale testing capabilities

Building web applications these days with the ever-growing prevalence of digital transformation isn't just about covering a large range of desktop browsers and OS versions; it is also about mobile devices and OS versions. A test automation framework that can support high-scale parallel testing across these platforms is obviously important. Among the various test automation frameworks, there are a few frameworks that do not support all browsers, including mobile devices, which poses a limitation for developers and SDETs. Setting a coverage platform matrix during the test automation framework evaluation phase can help surface such limitations.

Tool stack integrations and plugins

Both developers and SDETs work with a wide range of technologies and tools when building their applications. Such tools include CI/CD tools, such as Jenkins and TeamCity, as well as defect management and user story management tools, such as Jira. In addition, these personas also work with Source Control Management (SCM) tools, such as GitHub, and other tools, such as Perforce's Helix Core, API management tools, security scanning tools, and network analysis tools. Test automation frameworks that can create and execute various testing types should also integrate properly with the existing tools within squad teams to ensure a fluent process as well as continuous productivity. In addition, such personas also rely on different plugins for accessibility, visual testing, code coverage analysis, and more; hence, working nicely with these plugins is a great advantage to consider and evaluate.

Ease of use and adoption

Getting started with any technology can make the difference between the successful evaluation and adoption of a tool and a failure. To get started quickly and have a positive experience, a test automation framework should have an easy-to-use guide with basic code samples and environment setup and dependency instructions, as well as documentation and examples of more advanced capabilities. Since the personas that are evaluating these frameworks have very technical knowledge and skills but are also short on time, it's important to inform them upfront what the unsupported features and limitations are so they do not waste time chasing something that is simply unavailable. A good example that I found was from Cypress. The Cypress documentation clearly states what's never going to be supported by the tool, as well as providing a dynamic web page with its upcoming roadmap items (https://docs.cypress.io/guides/references/roadmap).

For example, the iFrames testing capability is not supported at the time of writing this book (https://docs.cypress.io/api/commands/clock#iframes), which is clearly stated in the documentation, but appears as a future roadmap item.

Reusability and maintainability

Both developers and SDETs do not try to reinvent the wheel when it's not required. Since their time is short within the software sprints, they love to use framework capabilities that enable them to optimize their workloads and be more efficient.

Things such as test reusability, page object model support, and working with advanced framework capabilities (for example, the new Selenium 4 introduced one called Relative Locators) are great examples of productivity features developers would appreciate.

When evaluating a test automation framework, it's important to look for things that can optimize our workload and make our daily tasks more efficient. Also, thinking about maintaining the test code over time is very important, since the product and the web landscape evolve over time and the tests should also be resilient to these changes.

Reporting, test analysis, and intelligence

Test execution cycles happen multiple times a day, and this results in huge volumes of test data across different environments and from target web and mobile platforms. Having the ability to analyze test reports properly, either through intelligence (AI/ML) or built-in capabilities, is key for agility. When evaluating test automation frameworks, developers and SDETs should investigate the reporting layer and see what they can get out of the box from the framework, and if it's not directly built into the framework, whether there can be solid integrations with reporting frameworks such as Allure (https://github.com/allure-framework/allure-js) or the Perfecto cloud solution reporting SDK (https://help.perfecto.io/perfecto-help/content/perfecto/test-analysis/test_analysis_with_smart_reporting.htm).

Figure 4.2 – Cypress test automation flakiness dashboard

A nice example of a reporting feature Cypress provides within its dashboard is the automated classification of flaky and inconsistent testing scenarios (https://docs.cypress.io/guides/dashboard/flaky-test-management#Flaky-Test-Analytics). With such an ability, frontend developers can filter out noise and keep their testing pipeline more stable and reliable.

Now that we've looked at some of the core pillars and considerations of a test automation framework that both developers and SDETs should care about, let's look at a methodology that can help to connect all these dots and serve as a tool to evaluate the different frameworks continuously.

Testing automation evaluation matrix

To realize continuous value from your selected test automation framework, you need to get support for the core elements that you need, as highlighted previously. From language support through to community, advanced testing scenarios support, and testing types of support, those in charge must have a clear picture that they can analyze prior to, as well as during, the adoption and usage of these technologies. The continuous part here is very important, since, for example, as described previously, Cypress currently does not cover iFrames and cannot test WebKit Safari browsers, but these are on their roadmap, so that soon might change. Note that picking your tool stack is good for a given time frame, hopefully long enough, up until you need to re-evaluate your choices and either keep on using what you have or make changes.

In the following table, we will look at the differences and commonalities across the four leading test automation frameworks. This tool can help us visualize the benefits and disadvantages of each.

Other resources and comparisons between the frameworks are constantly being done by individuals, and such comparisons might change over time. A nice resource I was able to find was from BlazeMeter (https://www.blazemeter.com/blog/selenium-vs-cypress-a-complete-comparison-between-the-two-testing-frameworks?utm_content=180888216&utm_medium=social&utm_source=linkedin&hss_channel=lcp-2364558), which is worth reading.

Any given web application project has unique requirements, and these requirements should be compared across the available tools. While not supporting iFrames can seem like a limitation of Cypress, if this is not something that your app uses, it's not really a relevant limitation.

The following is a categorized table that breaks down each of the test automation frameworks according to the core capabilities test engineers would mostly care about. As you will observe, there are a few capabilities that are supported by more than a single framework; however, when considering the wider range of capabilities and matching them to the persona requirements, the decision around which framework to use becomes easier:

While the preceding table is not a complete deep dive and does not go through, API by API, everything that is supported by the various frameworks, it can be used to understand some of the core commonalities as well as unique strengths of each.

In many projects, a combination of more than one framework might prove to be the best approach since all the preceding have JavaScript support as a common entry point with similar test runners driving the test creations (Mocha, Jest, and so on). In addition, specific frameworks offer built-in unique features that frontend developers and SDETs would benefit a lot from.

Test coverage across web and mobile platforms is also among the strong requirements of each web application; hence, Selenium in such cases is the most mature framework to support real and virtual device testing either through its parallel Appium framework or its integration with mobile cloud vendors such as Perfecto (http://perfecto.io).

The preceding table should help frontend developers to better look at the different frameworks. In Chapter 7, Consideration Matrix between JavaScript Test Automation Frameworks, we will dive deeper and provide a breakdown of the key use cases mentioned previously.

It is important that frontend developers consider that as part of a single pipeline, there are various quality-check activities to perform, ranging from API testing, visual testing, accessibility testing, and performance testing to network control, functional testing, and other types of integration testing. As we will learn later, while almost every capability can be implemented by developers, there are some capabilities that are built into some frameworks that need to be integrated by others.

As an example, visual testing could be advanced through built-in capabilities within the Playwright framework, while for other frameworks, frontend developers will need to rely on third-party integrations.

Summary

In this chapter, we went deeper into the high-level considerations frontend web application developers have when it comes to selecting a test automation framework.

As outlined in the chapter, there are a few generic considerations, such as ease of use and reporting, but also some specific requirements around test coverage, API and mock testing abilities, visual testing, and others that would vary from one project to another.

We provided a useful table that can be used as a baseline for a comparison of the various test automation frameworks.

Later in the book, in chapters 9-12, we will look into each of the capabilities (supported languages, community engagement, and so on) represented in the table, and provide a deep-dive analysis and comparison of each of these categories.

That concludes this chapter!

In the following chapter, we will advance the preceding test consideration matrix and provide a way to match each test automation framework with a testing methodology, such as API testing, performance testing, and functional and accessibility testing.