Summary
Software testing is a critical phase of the software development lifecycle, responsible for helping assure system quality. However, testing suffers from a plethora of challenges, some timeless and some new, causing quality to become increasingly difficult to ensure. TaaS is an emerging method of testing software that leverages the vast and elastic resources of the cloud to offer accessible services to handle testing activities on a pay-per-use basis. TaaS promises to alleviate many of the issues of traditional testing, providing more efficient and effective testing with less effort.
The research reported in this book used the HPST series of events and a survey on problems in software testing to help determine the most challenging issues that are plaguing the industry today. An evaluation framework was then developed to compare these problems against the state of TaaS, to ascertain where a TaaS solution would be appropriate. The academic and commercial aspects of industry were both considered in detail. Ultimately, a case study was performed using three TaaS tools and a gap analysis was performed. The resulting gap was used to identify areas in need of additional work.
Significant academic work has been done in software testing over the years, but rarely are industry concerns truly considered. This work was an attempt to inform an applied research agenda based on real-world needs as described by leading practitioners. The evaluation of TaaS offerings as possible solutions to the hard problems identified is one step toward leveraging modern tools and techniques to solve persistent testing challenges.
5.1 SUMMARY OF RESULTS
The results of the HPST project are summarized according to two criteria: how well the original research objectives were satisfied, and what new contributions this work has made to the software testing community.
5.1.1 RESEARCH OBJECTIVES
There were five objectives for this work when the project began. It is instructive to examine how well each of these objectives was satisfied.
1. Objective: To identify and quantify recurring challenges in software testing as defined by industry professionals.
A survey was conducted with industry professionals to elicit hard problems in software testing. The responses were quantified in Chapter 2, allowing the top issues to be identified. The resulting top challenges were used to create an evaluation framework for use against TaaS tools.
2. Objective: To review and analyze the academic and commercial states of Testing as a Service (TaaS).
The academic and commercial literature on TaaS was surveyed and analyzed in Chapter 3. Academic research focused on architecture and frameworks, types of testing, benefits and challenges, and more. The commercial aspect described the current industry offerings.
3. Objective: To develop an evaluation framework for TaaS tools based on the previously identified hard problems in software testing.
An evaluation framework, presented in Chapter 4, was developed with five hypotheses, based on the top five challenges identified by the survey. Related concerns and common tool criteria were tied to the framework, building up the evaluation criteria for each hypothesis.
4. Objective: To apply the framework against a selection of TaaS tools with the goal of evaluating their ability to overcome these challenges.
The evaluation framework was applied to three TaaS tools: Sauce Labs, SOASTA CloudTest Lite, and BlazeMeter. The lacking areas became obvious as the evaluation was filled out for each tool. Furthermore, aspects of the academic research were applied to the tools, noting additional concerns that should be considered.
5. Objective: To perform a gap analysis between the current state of TaaS and the needs of the industry and to analyze that gap to recommend improvements that could benefit the industry.
While the tools supported many of the most important needs of the industry, a gap still exists between the described challenges and the current state of TaaS. Only one of the tools offered fully automated test case generation, the tools had limited compatibility with external tools and restricted extensibility, and they should offer more forms of testing, especially when recalling the need for regression and security testing. Additionally, concerns from the academic research still exist, specifically in the area of security and the need for transparent pricing.
The research contributions presented in this book can be summarized in two parts: quantifying the hard problems in software testing as defined by members of the industry, and evaluating TaaS tools against the user-defined needs.
Quantifying Hard Problems: This book quantified the top hard problems encountered in software testing, as identified by industry professionals. While testing issues are frequently cited, rarely is the industry surveyed as a whole. While the HPST survey was limited in range, respondents came from a variety of backgrounds, providing necessary diversity. The trends were analyzed and quantified, providing a better look at the challenges plaguing the industry and the severity and impact of each.
Evaluating TaaS Tools: TaaS is a relatively new aspect of software engineering that has been touted as overcoming many of testing’s current shortcomings, but limited work has been done on evaluating tools in real-world settings. Academic and commercial literature was surveyed, providing a thorough background on the field. Three tools were selected for evaluation: Sauce Labs, SOASTA CloudTest Lite, and BlazeMeter. Each was examined in depth before an evaluation framework based on the HPST survey was applied to determine whether the tools met the top needs of the industry. While the tools met the majority of the criteria in the evaluation, a gap was revealed. Shortcomings were discussed to provide recommendations for improvements to TaaS solutions.
5.2 FUTURE WORK
Based on the gap analysis results, future work needs to be done on fully automated test case generation. Of the tools examined, only BlazeMeter offered fully automated load and performance testing. While offering script recording does help those with limited testing knowledge build a test suite, the ability to automatically generate test cases can simplify testing and soften the learning curve for TaaS tools. Furthermore, generated test cases should reduce testing time, improve the quality of the system under test, and ease the burden on testers with limited experience or knowledge.
Additionally, more tools should be evaluated against the framework. Many of the tools offered limited information on their features and lacked free trials or tiers, preventing them from being utilized in the case study. Examining more TaaS tools may reveal more gaps that need to be filled.
Finally, although some software testing challenges seem invariant (e.g., education and training), some of them change over time due to the introduction of new technologies and methodologies. When the existing problems are corrected or if new, higher priority issues are introduced, the framework may be rendered inadequate for tool evaluation. For this reason, the survey should be introduced periodically to reevaluate results.
Software testing will never be entirely free of challenges. However, many of the current and, frequently, timeless issues can be mitigated or their impacts reduced by focusing research on the needs of the industry and applying new methodologies where applicable. For example, TaaS promises to improve testing by resolving many of these existing problems, while promoting dynamic solutions which are easily capable of adapting to future needs.
With industry support, TaaS can truly revolutionize the future of testing, simplifying the creation and runtime of diverse and manageable test suites with little cost and effort on the consumers. However, to maintain this path, hard problems in software testing must be reevaluated every few years, redirecting the tool requirements as necessary to develop and maintain strong products that consider overcoming practical testing challenges a priority.