Chapter 6: Map the Pillars of a Dev Testing Strategy for Web Applications

This chapter opens the second part of the book and is aimed at addressing the testing strategy aspects of a web application development project, from the requirements and objectives stages up until the measurements of success.

Building a testing strategy for a given web application depends on the product requirements, the quality acceptance criteria, the available skills and resources within the team, and the target markets (end users) for which the application is intended. In this chapter, you will learn how to combine the relevant considerations into a testing strategy that covers all quality aspects and continuously meets the end user experience. In addition, you will learn about some key metrics that can be used to monitor and measure the success of a strategy.

The chapter is designed to cover the following topics:

• The key pillars of a web application testing plan and strategy
• Measuring the success of your continuous testing strategy
• A case study – a real-life web application testing strategy

The key pillars of a web application testing plan and strategy

In the first part of the book, we covered both the leading frontend test automation frameworks available for web developers, as well as the leading web application development frameworks. We learned that each development or testing framework comes with pros and cons. Knowing this upfront is an advantage, but this alone does not make up the main element within a testing strategy. These frameworks are the tools and, most likely, the enablers to drive a solid test plan from A to Z.

How and which tool to use needs to be driven by the testing plan and strategy itself, and to build such a plan, software leaders must rely on the core fundamentals of the testing pyramid matched to the product business requirements.

Let's start with the types of testing that any web application needs to undergo prior to being released to production. In Chapter 1, Cross-Browser Testing Methodologies, we covered the main web application testing types, which consist of functional, non-functional (performance, security, accessibility, and so on), APIs, visual, and so on. Illustrated at a high level in Figure 6.1 is the basic testing pyramid:

Figure 6.1 – The basic software testing pyramid (source – "The test automation pyramid.png" uploaded by Croncal, licensed under the CC BY-SA 4.0 license (https://creativecommons.org/licenses/by-sa/4.0/deed.en) at https://commons.wikimedia.org/wiki/File:The_test_automation_pyramid.png)

When you build a web application, you ought to consider each aspect of the product from an end user experience perspective. The end user cares about the look and feel of a website, how fast it performs, whether the site is available across all digital platforms (mobile and web), and how the services and third parties that the app uses work for them.

It's a recommended practice to spread all testing types throughout the pipeline. When doing so and providing frontend developers with a proper feedback loop, it allows them to fix issues faster and reduce the number of escaped defects that seep through to production.

In addition, partnering with the SDETs within the agile teams early in the requirements phase is imperative to allow them to develop test automation scenarios in time for the End-to-End (E2E) testing cycles.

In this section, we are going to discuss six pillars or considerations for building a reliable and robust web application. But all development must start with recognizing and defining the target users of the application.

Know your target users

When releasing a web application version to the public, knowing more about your end user traits can serve as a huge advantage from both development and testing perspectives. If the release is a brand-new application that doesn't carry any user analytics history, it is important to get clear targets from the product managers and business units so that a tester knows which mobile and web platforms, including versions and geographies, the application supports. If the release is a follow-up on top of an existing web application, web traffic analytics can be leveraged to figure out user journeys, the most used mobile and web platforms, regression defects from previous releases, and so on. This process can help to tailor the scoping of the testing plan and allow a timely release, optimize testing activities, and reduce risk.

When you consider your audience, of course, it should also be very industry- and vertical-specific. End users across verticals differ in many ways: age, expectations, the mobile and web platforms used, accessibility focus, the languages known, compliance requirements, and so on. So, knowing about all these traits pertaining to target users is essential to build a proper test plan.

Building a test plan

By knowing your target audience and understanding the product requirements for an upcoming release, test managers and developers can realize the scope of testing that needs to be part of the release definition of done.

Such a test plan needs to properly cover all the website workflows and end user journeys, as well as consider every dependency on the web application that can be an issue from a quality perspective. The test plan must cover all the relevant testing types and the most critical mobile and web configurations. Test data and test environments for both the development stages and testing stages must be part of the plan and become available prior to initiating the testing process.

We won't define again what each testing type means, since we did a lot of this in Chapter 1, Cross-Browser Testing Methodologies; however, as a reminder, we covered testing all internal and external links, the functionalities of each web page and user flow, usability and accessibility, localization and internationalization, coverage of the legacy, the latest and beta versions of web and mobile, layouts and views, and so on.

Make sure that you outline the preceding and check all boxes from a testing type perspective as part of the plan. As already highlighted in previous chapters, it takes a village to build and test a modern web application, so, ensuring that there are sufficient resources and tools and acquiring licenses for the project are other key requirements to address upfront.

Prep your tool stack and environments

Test development and execution do not happen in a void and require solid and up-to-date environments for both the creation, maintenance, and execution of each of the aforementioned testing types. Ensure as part of your plan that each resource from your team has valid access to the tools and environments that they need to accomplish their goals. In many cases, these practitioners will need a mocking services environment and pre-production test datasets to create and validate their testing. Development and testing leaders should be responsible for the tools and environments that their team members will use throughout the pipeline.

Set quality criteria and objectives

This is a crucial part of the test plan and requires handshakes between testing, development, product, and business owners. To know with confidence that a release is ready from a quality perspective, it needs to adhere to key metrics such as test coverage, defects opened with severity/priority in the system, platform coverage, functional and non-functional quality criteria, performance and availability measures, proper product release notes, available documentation, and so on.

Build a timeline and a schedule

Based on the available resources, tools, environments, and testing requirements, there needs to be an agreed-upon timeline that illustrates the different phases of the development life cycle, a definition and criteria that allows movement from one stage to another, and of course, a target release date with quality criteria and metrics to measure the release readiness. Orchestrating the release pipeline across the different practitioners is an art and requires communication and discipline across the different parties. Bear in mind that times and schedules have their own external and internal dependencies; hence, ensure that you put buffers (some would recommend +25% extra time added to the original estimates) into the planning to accommodate the unexpected.

Execute, monitor, measure, and document

This might be among the most critical and challenging phases of a plan, since it involves different personas, different objectives, and different goals that need to be tracked, monitored, and measured to provide decision-makers with the right data to make a go/no-go release decision.

The execution upon the agreed timeline and schedule together with the evidence of quality metrics and criteria are the ingredients of a successful release. Confidence is derived from data and test analysis, which needs to be present prior to the release date.

Making all the six pillars come together is dependent on the three well-known agile components – people, process, and technology:

Figure 6.2 – People, process, and technology enable release success

It is important to consider all the preceding aspects as part of your development of the testing plan as well as measuring the test deliverables coming from the different personas. Note that some of the preceding items need to be maintained over time as the product changes, the market evolves, and the end user's behavior changes.

Now that we've covered the building blocks of a solid test plan, including the technology stack, customer profiles, and processes within the software development life cycle, let's understand how to measure the preceding plan and determine whether a software release is on track and meets the criteria of success or not.

Measuring the success of your continuous testing strategy

Now that we've defined at a high level the steps and scoping for a generic test plan, let's look at some common and important metrics and KPIs that can help assess the level of quality of your web application. These KPIs should be part of the Definition of Done (DOD) (to learn more about DOD, see https://www.scruminc.com/definition-of-done/) for a scoped software release. In addition, quality metrics should be part of the quality criteria section within any test plan document.

Success, like quality, is a moment in time; therefore, it needs to be well monitored and structured in a way that decision-makers can analyze on demand.

There are various types of metrics that can be used, based on product requirements, the market vertical that is being served, historical data, and any other category of metrics that is relevant to the business.

The following are 27 suggested metrics by categories that can be adopted by web application teams and used as a basis to add more specific organizational metrics.

I decided to break these metrics into three categories:

• Speed
• Quality
• Cost:

Figure 6.3 – The recommended continuous quality metrics for agile teams

As you can see in the preceding figure, each category focuses on a challenge or an objective and offers only a few items for measurement. Teams can pick all of them or a subset, as well as add to these, but successful projects are the ones that cover all three aspects over time.

To highlight a few of the preceding metrics, starting with quality, we can focus on the Mean Time to Detect (MTTD). The MTTD refers to the time it takes to identify a defect in the code based on the total test execution time. Obviously, the shorter it takes to uncover a defect and then resolve it, the better. It also reflects on the effectiveness of the test code. Another example from the preceding metric is the defect distribution by different considerations – priority, severity, functionality, and so on. Knowing the volume of defects and the priority attributed to each testing type and web application area helps to determine the current quality plans, as well as the future testing scope. From a speed and velocity perspective, knowing within the testing suites the percentage of false-positive and blocked tests provides visibility into the value of your testing suite and can eliminate waste within the pipeline. From a cost perspective, looking at the cost of test execution can provide insights to decision-makers regarding tools and testing lab utilization, how well parallel testing is being implemented within the teams, and so on.

Make sure that you use the preceding set of metrics and include the relevant ones within your testing plan and strategy, as well as leveraging the preceding metrics as part of your evaluation process for tools that you have within your DevOps stack.

As a last tip from a measurement and continuous improvement perspective, it is always a great idea to benchmark your own web application and compare it to previous historical industry incidents, such as security issues, functionality, performance, and other quality-related pitfalls.

There is a useful list of common performance defects published on the iLabquality website (https://www.ilabquality.com/the-10-most-common-web-app-performance-problems/), highlighting things such as unoptimized databases, poorly written code that does not follow best practices, poor load distribution, and troublesome third-party services as common root causes for performance problems in web applications. From a functional standpoint, common bugs that keep coming up across web applications involve poor site navigation, poor mobile device optimizations, inconsistent user experience across platforms, navigating back and forward within a website, negative input into textboxes, and so on.

Functionality and performance are only a subset of an entire test plan and strategy, so learning from others' mistakes around different types of defects can save a great amount of time, resources, and money throughout the life cycle of your project.

The preceding guidelines should help frontend developers and test engineers to properly define metrics to measure their success in a given software release. Now that we have provided in the preceding two categories of functional and non-functional quality both a structured way of building a test plan and measuring its success, let's take a real-time web application and design a test plan that covers the aforementioned building blocks.

A case study – a real-life web application testing strategy

Let's take a specific website and design a testing strategy that will fit end user expectations. For this exercise, I used the Miro tool to create a mind map of the E2E website application flows and scoping of the tests.

The following illustration is based on the Barclays website application, serving mainly the UK. I placed the major testing types with some real-life navigation options that are available for the customers of this bank from any mobile device or desktop OS:

Figure 6.4– A mind map example of web testing for the Barclays website, drawn with Miro boards

As you can see from the preceding mind map, the website testing plan covers all types of test categories to ensure the high quality and performance of the web application. The mind map focuses on tests such as API testing, functional testing broken by the navigation flows within the website, the usability of the web application, performance, and availability. In addition, it covers more advanced aspects of the web application, including accessibility and chatbots, and considers full platform coverage for the types of mobile and desktop systems that will be used within the testing lab. Lastly, the mind map covers production testing for both performance and availability of the web application, as well as service monitoring.

The preceding mind map is not complete; it is just a subset, since each branch continues to an inner link or page, which includes usability, accessibility, performance, security, and many other testing items that fall into the testing plan. However, the mind map provides an understanding of what a real-life testing plan and scoping exercise should look like, and how many complex considerations ought to be planned throughout the teams.

Since agile and DevOps teams are issuing daily to weekly releases, the mind map and testing activities must be part of a strategy that does some level of incremental and differential analysis of the changes between the previous and current version of the web application so that teams can better optimize and scope the entire testing throughout the pipeline, within and outside of the CI process. In addition, even if there is no change to the website, as we learned earlier in the book, every few weeks a new web platform or a mobile platform is introduced and needs to be fully covered and supported, and that is what continuous testing is all about – automating and running continuously tested subsets to ensure high-quality production websites.

Summary

This chapter offered an experience-driven testing plan and strategy so that developers and test managers can scope properly their web application projects from one release to the next.

We then unfolded the scoping of a test plan into three categories of metrics and KPIs to help decision-makers better assess, using data, the quality of a given application. We focused on quality, speed, and cost to categorize these metrics.

Lastly, we took a real-life production website from the financial industry and created a thorough mind map that scopes most testing types across the different testing types: functional, performance, security, APIs, usability, and so on.

We clarified that scoping and planning is always a moment in time, especially for agile and DevOps teams; hence, they need to be always validated and modified to be relevant at the right production moments.

That concludes this chapter! In the following chapter, we will provide a deeper consideration matrix between the top four JavaScript test automation frameworks.