Introduction
Software testing is “an empirical technical investigation conducted to provide stakeholders with information about the quality of a product or service under test” [1]. Testing is a critical phase of the software testing lifecycle, paramount to guaranteeing that the system under test meets quality standards, requirements, and consumer needs. While the field has existed for over half a century, software testing is not without challenges, difficulties, and flaws. Some are brought on by new and evolving technologies and methodologies while others are timeless, existing since the field’s inception.
1.1 SOFTWARE TESTING CHALLENGES
As software systems grow in both size and complexity, quality (defined as having “value to some person” [1]) becomes significantly more difficult to ensure. These increasingly complex systems make existing testing issues more prominent and cause new issues to arise. Testing has not been ignored by the research community, but traditional research topics focus on only a small portion of these challenges, many of which may or may not be the most crucial issues for the industry as a whole.
Donald Firesmith of Carnegie Mellon University’s Software Engineering Institute (SEI) detailed a series of recurring problems that he identified during his three decades of software development and independent technical assessments of development projects. Originally published in the SEI Blog [5, 6] and related presentations [7] and culminating in a recent book [9], he analyzed his most commonly occurring software testing problems as derived from both his own experienced and feedback from 29 members of various LinkedIn groups and the International Council on Systems Engineering (INCOSE), incorporating their findings and weighing them against his own.
Firesmith separated these findings into two primary categories: general testing problems and test type-specific problems. General testing problems were further divided into eight sub-categories: test planning and scheduling problems, stakeholder involvement and commitment problems, management-related testing problems, test organizational and professionalism problems, test process problems, test tools and environments problems, test communication problems, and requirements-related testing problems.
Although the identified problems are common to some or many projects, Firesmith’s work was not prioritized by frequency or severity. Furthermore, several of his recommendations detailed in his presentation [8] oversimplify the problem and, in some cases, provide solutions that would not be achievable in many environments. For example, if the test team has inadequate experience, his suggestion is to hire professional testers and provide more training—both of which can be difficult or impossible to achieve with budgetary and time constraints.
Our approach to identifying and quantifying the major software testing challenges currently facing the industry is somewhat different. The software testing challenges were obtained from industry professionals through the Hard Problems in Software Testing (HPST) series of events [3], personal interviews, and an online survey [4]. The survey queried respondents about issues they have faced in their work and on which areas they believed researchers should focus their attention. Results were gathered and analyzed quantitatively for recurrences. The responses were analyzed both individually and in categories of similar statements. More details on the HPST project are provided in Chapter 2.
1.2 SERVICE-ORIENTED SOLUTIONS
Many tool vendors are promoting a service-oriented solution, commonly known as Testing as a Service (TaaS), to address software testing challenges. TaaS is a model of software testing that leverages the vast and elastic resources of the cloud to offer easily accessible services that handle a variety of testing activities for consumers on a pay-per-test basis [2]. The move toward TaaS is driven by several recurring issues in software testing, including increasing costs and effort, limited time, and difficulty handling a large variety of testing.
In theory, TaaS could resolve some of the major issues plaguing the industry. In practice, it is unclear where TaaS is applicable and how effective it can be for various software testing problems. Leah Riungu-Kalliosaari, a researcher and doctoral student at Finland’s Lappeenranta University of Technology (now at the University of Helsinki), studied software testing in the cloud [8]. Specifically, she focused on many of the socio-economic factors that influenced the adoption and use of cloud-based testing in a variety of organizational contexts in order to provide a better framework of cloud-based testing and propose strategies that could assist organizations in adopting these cloud-based testing technologies. Beginning in 2010, she and her collaborators, Ossi Taipale and Kari Smolander, conducted a series of interviews with members of the industry focused on Testing as a Service.
These interviews were performed with individuals working in consumer and provider roles in TaaS at various companies and served to explore conditions affecting the adoption of TaaS [11], elicit research issues [12], and discern the effects of cloud-based testing [13]. The interviews consisted of semi-structured questions. All responses were recorded and analyzed using ATLAS.ti, a qualitative data analysis tool. Ultimately, Riungu-Kalliosaari published a series of papers providing benefits, challenges, and needs of TaaS, as well as a roadmap for organizations considering the feasibility of TaaS migration. The results of this research are included in Chapter 3.
While Riungu-Kalliosaari’s work is important to those adopting or improving TaaS technologies, her work did not directly answer the question proposed by this book: How software testing challenges can be resolved by utilizing TaaS. The work on TaaS adoption aligned with several of the challenges identified by the HPST survey, but this was, in many cases, coincidental.
Our approach to determining the applicability and efficacy of TaaS to today’s testing challenges is built upon a literature study of the academic research and commercial offerings of TaaS tools and techniques. This survey provides a baseline of understanding of current TaaS solutions. Details of the survey are provided in Chapter 3.
1.3 GAP ANALYSIS BETWEEN HPST AND TaaS
To truly ascertain if TaaS is an appropriate solution to a specific testing problem, more information is needed. We obtain this information by performing a gap analysis between the requirements implicit in the hard problems in software testing and available TaaS solutions. The gap analysis is primarily qualitative in nature, but it does offer insight into the possible solution space provided by TaaS.
Chapter 4 presents a case study of a web application and discusses its testing needs with three representative TaaS solutions. The case study embodies several hard problems in software testing. General observations are cautiously made from the case study as to the broader applicability of TaaS to HPST.
The gap analysis also identifies possible improvements to TaaS. These unmet requirements are suitable starting points for future research. For the practitioner, these unmet requirements are warnings that very few hard problems have single-point solutions. We expect that it will always be axiomatic in software testing that a federation of tools and techniques are needed to fully address hard problems.