Chapter 3: Top Web Test Automation Frameworks

When it comes to end-to-end test automation frameworks, frontend developers have a variety of choices to pick from. The open source community continues to innovate, as well as leverage, existing technologies to enhance testing frameworks and offer more coverage and depth to practitioners.

This chapter focuses on the top four leading open source frameworks in the market and provides an intermediate tutorial on how to get started with each of these. Since web developers have so many choices around testing their application code, it is very important that they understand the architecture and fundamentals of the top frameworks out there, so they can make guided decisions.

After reading this chapter, you will know how to get started with the four leading JavaScript open source test automation frameworks and run a basic test scenario on a web browser.

This chapter is specifically designed to cover the following:

• An overview of the web testing market
• Getting started with the Selenium WebDriver framework
• Getting started with the Cypress framework
• Getting started with the Google Puppeteer framework
• Getting started with the Microsoft Playwright framework

An overview of the web testing market

The web testing market constantly changes, and newer versions of existing frameworks are being developed with brand new solutions. A great resource to learn about what's trending, what's declining, and how big the community is behind leading open source technologies is the npm trends website (https://www.npmtrends.com/site). When focusing on the top four frameworks available for frontend developers that support JavaScript, at the time of writing this book, Cypress emerges as the leading framework. As highlighted in an npm trends report (https://www.npmtrends.com/cypress-vs-playwright-vs-selenium-webdriver-vs-puppeteer), Cypress has over 3 million weekly downloads compared to Selenium WebDriver, which has just over 2 million downloads. Keep in mind that we are only showing in this resource the JavaScript flavor of these frameworks. For frameworks such as Selenium and Playwright that support more language bindings, the market share and downloads will probably be higher; however, here we are focusing only on JavaScript testing:

Figure 3.1 – npmtrends frameworks comparison – Cypress, Playwright, Selenium, and Puppeteer (source: https://www.npmtrends.com/cypress-vs-playwright-vs-selenium-webdriver-vs-puppeteer)

It is not only the number of downloads that helps determine the popularity of one framework over another, but also the number of ratings (known as stars) each repository on GitHub receives, the last change made to the framework, the number of versions, and other factors. The decision of a frontend developer to go with one framework over another is larger than just the stats behind the framework on GitHub and the number of downloads, but such stats do help us to realize whether a technology is backed by a community, and whether it's being adopted. In addition to these stats, a user would obviously need to perform proofs of concept, understand the best fit for their needs, and other requirements. In Chapter 7, Core Capabilities of the Leading JavaScript Test Automation Frameworks, we will dive deeper into ways to better compare frameworks and match each one based on relevant considerations.

While in the previous statistics and this book we are only focusing on the four leading open source frameworks, it's important to note that the market is flooded with dozens of additional open source frameworks, as well as many commercial tools that can be used for cross-browser testing, such as WebDriver.io, TestCafe, and NightWatch.js. These are not part of the scope of this book.

As Gleb Bahmutov classifies the four different frameworks in his blog (https://glebbahmutov.com/blog/cypress-vs-other-test-runners/), it is important to understand some of the fundamental differences between the four technologies.

While Selenium and its underlying frameworks are built on the WebDriver protocol and provide maximum cross-browser platform coverage and various language bindings, this framework is known to be flakier than Cypress, as well as slower from a test execution performance perspective. On the other hand, Playwright and Puppeteer are built on top of the Chrome DevTools Protocol (CDP), which gives them the advantage of gaining deep coverage of inner features of browsers, as well as enhanced debugging abilities. Cypress is unique in its architecture since it is a JavaScript framework that runs on the browser itself, making the execution performance extremely fast, and the debugging, including the document object model (DOM) access, great. Cypress is also known for its test reliability and low flakiness – it is also a framework that utilizes a flakiness test filter that comes with the framework itself:

Figure 3.2 – High-level classification of cross-browser testing frameworks (sourced from Gleb Bahmutov: https://glebbahmutov.com/blog/cypress-vs-other-test-runners/)

Now that we've classified the web testing landscape at a high level and examined the high-level architecture of each framework, let's dive into the frameworks and learn how to install and get started with them.

Getting started with the Selenium WebDriver framework

To get started with Selenium, you'll need to follow a number of steps that include the installation of the framework, as well as setting up the grid for the specific browser drivers on which you would like to run your tests (Chrome, Firefox, Safari, or Edge).

Your main website to learn from and get started with Selenium is https://www.selenium.dev/. From this link, you will be able to download the relevant driver, see the latest release notes, obtain documentation and code samples, and more.

As you'll learn in this chapter, and even more so later in this book, the Selenium framework consists of three main components:

• Selenium WebDriver: This is a collection of several language bindings to drive different browsers for testing purposes, as well as the individual browsers controlling code.
• Selenium Grid: This is a Selenium framework component that is designed to distribute and scale test automation by running tests in parallel across different combinations of browsers/OS versions.
• Selenium IDE: This is a quick ramp-up solution that through recording helps generate, the first basic Selenium scripts through a browser plugin, with no lines of code needing to be written.

In this chapter, we will only cover Selenium WebDriver and Selenium Grid. Selenium IDE is a basic entry-level browser plugin that allows beginners to record their first Selenium script without writing a single line of code. Once the script is recorded, they can copy the code to their IDE and continue building on top of it using code.

Setting up Selenium WebDriver

To install the Selenium libraries, please run the following command from your desktop command-line interface (CLI):

npm install selenium-webdriver

Selenium WebDriver is a recommended W3C testing technology that is intended to drive native browsers effectively and in adherence to the W3C standards. This conformance ensures that all scripts across different browsers are written in the same way and are simple to use.

As illustrated in the Selenium provided architecture (https://www.selenium.dev/documentation/webdriver/understanding_the_components/), a test framework that implements the WebDriver technology would drive the tests through an installed Selenium server that communicates with a specific browser driver (for example, a Chrome or Firefox driver):

Figure 3.3 – Selenium WebDriver framework components and architecture (sourced from selenium.dev)

Now that we have installed the Selenium framework locally, let's move on to the next steps.

Download any of the WebDriver that you're interested in using locally, for example, the Chrome driver from the following URL: (https://www.selenium.dev/documentation/getting_started/installing_browser_drivers/). Once downloaded and installed, make sure to set the path to where the driver is placed to ease the execution from the CLI, as indicated here:

Setx PATH "%PATH";c:usersekinsbrunerWebDriverin"

Now that we've installed the Selenium package and a specific browser driver, let's run the first basic test written in JavaScript. Next, we will enhance the project to run within a grid across more than one browser.

From your CLI (Windows machine), launch the Google chromedriver:

chromedriver

You should see a prompt in your CLI with the following text:

"Started ChromeDriver (v2.0) on port 9515"

The version number in the output would obviously match the version you downloaded from the previous URL.

Now that chromedriver is running, run the following JavaScript Selenium scenario from any of your preferred environments. It can be from Visual Studio Code, Eclipse, IntelliJ Idea, or from a simple command line. Note that you would also need to have installed node.js on your local machine. I named the file GoogleSelenium.js:

const {By,Key,Builder} = require("selenium-webdriver");

require("chromedriver");

async function example(){

       var searchString = "packt publishing";

       //To wait for browser to build and launch properly

       let driver =

         await new Builder().forBrowser("chrome").build();

        //To fetch http://google.com from the browser with

        //our code.

        await driver.get("http://google.com");

            

        //To send a search query by passing the value in

        //searchString.

        await driver.findElement(By.name("q")).sendKeys(

          searchString,Key.RETURN);

        //Verify the page title and print it

        var title = await driver.getTitle();

        console.log('Title is:',title);

        //It is always a safe practice to quit the browser

        //after execution

        await driver.quit();

}

example()

To run the preceding test that open a Chrome browser and search for Packt publishing, simply run the following command:

node GoogleSelenium.js

If there are no issues within your environment, you should see the Chrome browser launched and the preceding test running. At the end of the test, you will also get the following prompt to your CLI:

Figure 3.4 – First Selenium test execution example in JavaScript

Note that all the previous scenarios could have been similarly executed against Mozilla geckodriver upon installation of the driver.

Node.js

Based on the Wikipedia definition (https://en.wikipedia.org/wiki/Node.js), Node.js is an open source, cross-platform, backend JavaScript runtime environment that executes JavaScript code outside of a web browser. In the previous example, we used the node command to run the Selenium script from a CLI outside of the browser.

Selenium Grid

Let's scale the previous execution example to run using Selenium Grid's capabilities

The Grid option can be set up as standalone or as a hub with nodes. We will look at the standalone option. On a simple setup of a Selenium local Grid, the Selenium server will listen by default at http://localhost:4444 and detect any installed drivers that the developer downloaded and added to their machine PATH.

After downloading the selenium-server JAR file from the Selenium website, simply run the following command to launch the server:

java -jar WebDriverinselenium-server-4.1.0.jar standalone

Do make sure prior to running the command, that the relevant drivers you wish to run against are launched in the background (for example, Chrome, Gecko, or Edge). Please also ensure you have Java installed on your machine prior to running the preceding command.

You should see the following prompt in your CLI:

Figure 3.5 – Launching output of Selenium Grid command-line execution

Once the Grid command has been executed and assuming all local drivers were installed and launched successfully, you should be able to see the following output on your browser when navigating to localhost:4444:

Figure 3.6 – Local Selenium Grid output within a browser

To run a code sample against a Grid of browsers, you will need to modify your code and point the WebdDriver URL to the local host Selenium grid:

let driver = await new webdriver.builder()

     .forBrowser(' chrome')

     .usingServer('http://localhost:4444/wd/hub')

     .build();

When running the new project when the grid is set as the target driver, the test would run against the browsers that are registered in that grid, as shown in Figure 3.6.

Lastly, developers can easily configure their desired capabilities when using larger grids and trying to run their Selenium tests in parallel using the supported W3C capabilities (https://w3c.github.io/webdriver/). Selenium supports a wide range of capabilities that allow the test to run across different and complex configurations. To learn about the supported driver capabilities, please visit https://www.selenium.dev/documentation/webdriver/capabilities/driver_specific_capabilities/.

In the below subset table taken from the W3C documentation (https://w3c.github.io/webdriver/#capabilities), you can see some of the common WebDriver capability that a Selenium test would need to use in order to launch a specific browser with a specific version:

Figure 3.7 – W3C WebDriver supported capabilities (sourced from W3C - https://w3c.github.io/webdriver/#capabilities)

Selenium is a rich framework and while we are focusing on JavaScript in this chapter (and book), the framework fully supports multiple language bindings, including C#, Python, Ruby, and Java. It also supports all the different browsers that are out there.

To better develop Selenium code, developers should leverage best practices including the page object model (POM), the use of relative locators that were introduced in Selenium 4, the use of reliable locators out of the eight types that are supported by Selenium, and more. Later in the book, we will cover advanced practices with Selenium, and I expand on these items.

Selenium-Supported Element Locators

Selenium supports the use of finding elements across the following element types:

ID, name, tagName, className, linkText, partialLinkText, xpath, and cssSelector

For example, to search for an element on the website under test, a developer would use the following command:

driver.find_elements(By.XPATH, //button)

Now that we've gone through the basics of Selenium WebDriver and Selenium Grid, let's continue with an introduction to the Cypress framework.

Getting started with the Cypress framework

As noted earlier in the chapter, Cypress (https://www.cypress.io/) is by far the fastest and most adopted cross-browser frontend JavaScript testing framework. It is a developer-friendly, fast execution solution by design, and runs on the browser. In this section, we will learn how to install, set up, and run the first Cypress test in JavaScript. Note that Cypress also supports TypeScript and can be configured to run with the Cucumber BDD framework as well.

To get started with Cypress, please run the following command to install the node package on your machine:

npm install cypress --save-dev

Similar to Selenium and JavaScript, Cypress also requires Node.js to be installed on the local machine to run the Cypress tests. If you do not have Node.js installed, please make sure that, in addition to the installation of Cypress, you install it as a dependency as well.

Once the Cypress framework is installed, users can drive the tests either through an IDE such as IntelliJ or Visual Studio Code or through the CLI/Jenkins.

From the CLI, users can launch the GUI version of Cypress and launch any tests that are available in the testing directory or connect the Cypress framework with the online Cypress web dashboard.

We can launch the Cypress GUI by using the following command:

npx cypress open

Upon execution of this command, the Cypress GUI will launch as follows:

Figure 3.8 – Cypress GUI interface when launched from the CLI

When a user clicks on any of the available JavaScript tests on the left panel under INTEGRATION TESTS, the execution will immediately start on a local browser. Users who have multiple browsers installed locally can define which of these will be used for the target executions.

The following is a screenshot of a live execution on a local browser using Cypress:

Figure 3.9 – Cypress live execution of a JavaScript test scenario on a local browser

To run the exact same test spec, you can also use the following command line:

Cypress run -–spec .cypress estscypressintegrationexamplesactions.spec.js

If you want to get started with a basic test spec that you can use with Cypress, the basic installation comes with a set of pre-defined scenarios. Many of them use the https://example.cypress.io website as a target to learn and ramp up with the frameworks and their APIs. Here is basic test code that you can use:

describe('My First Test', () => {

  it('Gets, types and asserts', () => {

    cy.visit('https://example.cypress.io')

    cy.contains('type').click()

    // Should be on a new URL which includes

    // '/commands/actions'

    cy.url().should('include', '/commands/actions')

    // Get an input, type into it and verify that the value

    // has been updated

    cy.get('.action-email')

      .type('[email protected]')

      .should('have.value', '[email protected]')

  })

})

To learn more about the basic steps that you can do with Cypress, visit (https://docs.cypress.io/guides/getting-started/writing-your-first-test#Add-a-test-file).

Now that we've covered the basics of Cypress, let's briefly explore some additional capabilities of this framework. Later in the book, we will dive much deeper into using the advanced features of the Cypress framework.

Important features of Cypress

A unique and very powerful feature that Cypress offers developers as part of the debugging and real-time feedback from the web application is Time Travel.

Time travel allows developers to hover their mouse over commands on the left panel of the test runner and view in real time what happened on the web application, including DOM snapshots and other debugging insights. Such a capability allows developers to debug their web applications step by step and fix issues in real time. The following is a screenshot of a mouse hover action on a test step in the left panel of the test runner, with synchronized action visibility on the right side, as it occurred on the web application under test:

Figure 3.10 – Cypress time travel example (mouse hover action in sync with web DOM tree)

Developers who wish to expand the abilities of Cypress and run tests in parallel, examine flaky tests, and track test execution history, can easily connect their Cypress framework to the web-based Cypress dashboard, and more. From the Cypress UI test runner, developers would sign in with email or GitHub credentials and be able to get started. Cypress offers through its web portal multiple types of reporting dashboards that are add-ons and require a paid license as opposed to the free open source framework covered earlier.

Figure 3.11 – Cypress dashboard user interface and capabilities

Now that we've covered the basics of Cypress, let's briefly explore some of the additional capabilities of this framework.

Cypress offers developers the ability to create component test scenarios (https://docs.cypress.io/guides/component-testing/introduction#Getting-Started). Such tests are the middle layer between unit and integration tests that allows us to test the quality of a single component within a web application. The capability of component testing is still being built by Cypress, but developers can already start using it by running Cypress with the relevant command-line option, as follows:

npx cypress open-ct

Do note, that prior to running the preceding command, a project needs to be set up according to the Cypress guidelines. The following image shows the Cypress component testing screen that you can access after running the preceding command:

Figure 3.12 – Cypress component testing screen

As mentioned previously, the Selenium framework comes with a basic Selenium IDE recorder to help users get started with their first Selenium scripts. Similarly, Cypress is developing its own version of such a recorder under the name of Cypress Studio (https://docs.cypress.io/guides/core-concepts/cypress-studio). This solution will allow new users of the technology to record Cypress scripts through a GUI tool. The recorded tests will generate a JavaScript test code that can then be modified and enhanced. In addition, there is also a good list of browser extensions, such as this Chrome Cypress recorder: https://chrome.google.com/webstore/detail/cypress-recorder/glcapdcacdfkokcmicllhcjigeodacab, that are worth exploring.

Later in this book, when we cover the advanced capabilities of Cypress, we will touch upon the mocking and network control abilities of Cypress, using various plugins with Cypress, automating Cypress for react native applications, and more.

Now that we have covered the basic abilities of Cypress, let's proceed to Google Puppeteer.

Getting started with the Google Puppeteer framework

Puppeteer is an open source node library and a framework developed by Google that is based on the CDP. It allows web application developers to control headless or headed chrome browsers through high-level APIs over the above mentioned CDP or DevTools protocol (https://chromedevtools.github.io/devtools-protocol/). As described earlier in this chapter, Puppeteer and Playwright share the same architecture and CDP to create and run tests.

Unlike Selenium and Cypress, Puppeteer only supports Chrome and Chromium-based browsers, which means that testing end-to-end across all other browsers, such as WebKit Safari and Mozilla Firefox, isn't supported.

Among the core capabilities of Puppeteer is the generation of screenshots and PDFs of websites' pages, crawling a single-page application, and generating pre-rendered content, automating form submission, UI testing, and user gestures such as keyboard inputs. With the support of the CDP, developers can test Chrome extensions, as well as capture a timeline trace of the site for performance monitoring.

To get started with Google Puppeteer, you should obviously have Node.js installed on your local machine, and then run the install command as follows:

npm install puppeteer

Keep in mind that Google provides two versions of Puppeteer: a standard installation, which is done using the preceding command, and a puppeteer-core version that by default does not download any of the Chrome browsers to your local machine. The standard installation downloads the latest Chrome browser to the local machine and uses it as the target browser for testing. Puppeteer supports both the headed and headless execution modes. Scripting with Puppeteer is done with JavaScript, which makes the framework very appealing for frontend web application developers. As seen in Chapter 2, Challenges Faced by FrontEnd Web Application Developers, a developer can easily create a Puppeteer script that takes a full website screenshot, monitors performance, or even validates the accessibility of a website that is under development.

A website HTTP archive (HAR) file can be easily generated using Puppeteer. Using a generated HAR file, developers can review the entire traffic within their website and get performance and security insights for each of the transactions.

The following code snippet will navigate to the Packt website and generate a HAR file for review (for that, you need to install the npm package puppeteer-har):

const puppeteer = require('puppeteer');

const PuppeteerHar = require('puppeteer-har');

(async () => {

  const browser = await puppeteer.launch();

  const page = await browser.newPage();

  const har = new PuppeteerHar(page);

  await har.start({ path: 'book_demo.har' });

  await page.goto('https://www.packtpub.com/');

  await har.stop();

  await browser.close();

})();

On running the preceding test code, a new HAR file under the name book_demo.har will be generated:

node [filename.js] //depends on the JavaScript file name given above

Opening the generated HAR file with the Google HAR analyzer web tool will show the following output that can be examined by the frontend developers for web traffic issues, performance issues, and more.

Figure 3.13 – Puppeteer-generated HAR file example

To get more examples, it's highly recommended to bookmark the Google GitHub repository for Puppeteer (https://github.com/puppeteer/puppeteer).

Later in the book, we will cover more advanced features of the Puppeteer framework.

Now that we have covered the basics of Puppeteer, let's proceed with Microsoft's Playwright framework.

Getting started with the Microsoft Playwright framework

Playwright is one of the newest but fastest-growing cross-browser test automation frameworks. Built by the same team that built Puppeteer, Playwright is also a CDP-based framework. As opposed to Google's Puppeteer, Playwright supports multiple languages and most leading browsers. With Playwright, developers can script in JavaScript, TypeScript, Python, Java, and .NET, and execute across most leading browsers including Chrome, Firefox, Edge, and WebKit Safari. The Playwright framework can be executed in headed or headless mode and can also support mobile viewport emulation (https://playwright.dev/docs/emulation#devices) with various user agents, as opposed to Cypress, for example.

From an architecture perspective, Playwright uses the CDP to interact with the browsers, drive actions such as browsers' pages inputs, scan for security, monitor network processes, and much more. A thorough architectural review is presented by Microsoft in this online session: https://www.youtube.com/watch?v=PXTspGn1im0.

To get started with Playwright, simply install the Node.js package through the following commands:

npm i -D @playwright/test

npx playwright install

Note that Playwright automatically downloads the Chromium, WebKit, and Firefox browsers to a local folder on your machine. You can learn more about the installation of the framework here: https://playwright.dev/docs/intro#installation.

Once installed, simply create a basic JavaScript test file, save it with the extension of *.spec.js, and place it in the tests folder:

const { test, expect } = require('@playwright/test');

test('basic test', async ({ page }) => {  

await page.goto('https://github.com/login/');  

await page.fill('input[name="login"]', ' [user name]');

await page.fill('input[name="password"]', '[password]');

await page.click('text=Sign in');

});

The preceding code snippet will navigate to the GitHub website and perform a simple sign-in.

To run the test with the browser in a headed mode, run the following command:

npx playwright test --headed

If the preceding command is not specified, the execution will run in headless mode.

Figure 3.14 – Playwright command-line options

If you wish to run the previous test on a specific browser, you'd need to use "--browser=webkit" as an example.

When using Playwright, unlike using Cypress, developers cannot utilize the time travel functionality for immediate debugging; however, Playwright does support some very valuable features including geolocation testing, mocking, file handling, using parameters, and iFrames, which are specifically unsupported by Cypress at the time of writing this book.

iFrame

An iFrame is an HTML document that is embedded inside another document on a website. It is typically used to insert content taken from another source or website.

While for most of the frameworks that we cover in this book, there is available integration with cloud vendors such as Perfecto, Sauce Labs, and BrowserStack, Playwright offers built-in parallelization and sharding to drive the tests faster and at scale.

Playwright uses the term worker to run tests in parallel. From a practitioner standpoint, there are two methods to run tests in parallel.

The first one is through an additional command-line option that should be added to the execution:

npx playwright test -–headed -–workers 4

The second option is to edit the playwright.config.js file that comes with the framework installation, and add the following line:

Const config = {

Workers: process.env.CI ? 4: undefined,};

Module.exports = config;

There are additional useful features within Playwright, such as stopping the entire test execution upon a certain amount of test failures. For example, if within a large test execution of a suite, the failure number (using --max-failures=10) reached 10 failures, the entire test run will stop at that point to avoid running the full suite.

Playwright can also help perform visual validations, monitor network traffic, and help test complex scenarios. We will discuss some of these capabilities later in the book.

Lastly, Playwright integrates with most of the continuous integration tools, as well as with Docker, and other third-party test runners, such as Jest and Mocha.

Summary

As highlighted in this chapter, web application developers have a wide range of technology selections when it comes to their frontend web application testing. There are some commonalities between the frameworks; however, there are also some material differences between them. Selecting what makes the most sense for the short term, as well as for the long term, is a tough challenge that we will cover later in this book.

In this chapter, we covered the fundamentals of the top four JavaScript test automation frameworks for web developers.

We provided an overview of the core benefits of the frameworks together with some getting started instructions, so that developers can get a sense of what it takes to use these frameworks for a new project.

As we move forward in the book, we will provide a deeper dive into the advanced capabilities of each framework, as well as some recommendations on when it's best to use one framework over another.

In the following chapter, we will cover tips and recommendations to help developers and other testing personas within the project better match the test framework to their objectives and use cases.