Software testing is a crucial phase of the software engineering lifecycle, responsible for assuring that the system under test meets quality standards, requirements, and consumer needs. Unfortunately, as software systems grow in size and complexity, testing becomes commensurately challenging and quality becomes significantly more difficult to ensure.
Some of the hard problems in software testing are timeless, while others are brought on by the introduction of new technologies and methodologies. For example, cloud computing offers elastic resources that can be leveraged for software testing, but cloud-based applications introduce new intricacies into the testing process. Similarly, agile development places unique strains on testing activities—particularly in terms of compressed schedules.
The Hard Problems in Software Testing (HPST) project addressed this issue by asking real testers a relatively simple question: “Where are your pain points?” We hoped that their candid answers would help us (researchers) help them (practitioners) by directing our efforts on issues that truly matter to them. It is our sincere belief that working together, we can make headway toward solving some of the grand challenges in software testing.
WHAT IS UNIQUE ABOUT THIS BOOK?
We began the HPST project in Fall 2011 with an interactive session held as part of the 3rd International Workshop in Software Testing in the Cloud at the IBM CASCON conference in Toronto, Canada. We then hosted an invitational symposium at the Florida Institute of Technology and organized a workshop at the Software Testing Analysis & Review (StarEast) conference in Orlando in Spring 2012.
In addition to the HPST series of events, we continually gathered data using an online survey and selected personal interviews. The project’s goal was to identify hard problems in software testing as voiced by leading practitioners in the field. These problems had to be sufficiently challenging to warrant focused attention for the next five years. They also had to be sufficiently important to the community such that if measurable progress were made in solving these problems, we would advance the state of the practice in software testing to everyone’s benefit.
Software testing is both practically relevant and academically interesting. It is a somewhat unique software engineering activity area that is both very applied and well-grounded in sound computing theory. Unfortunately, there has traditionally been a schism in testing between new approaches espoused by researchers and the timeless problems that practitioners struggle with on a daily basis. Such disconnects can hinder communication between the two camps and thereby limit the adoption of promising new testing tools and techniques.
WHO SHOULD READ THIS BOOK?
This book represents the culmination of a three-year effort. In addition to summarizing our findings of the current hard problems in software testing, this book also examines where Testing as a Service (TaaS) may offer a viable solution to these important problems. TaaS is a relatively new model of software testing that relies on cloud computing and service orientation to provide testing capabilities unavailable to many organizations. Researchers may benefit from an introduction to TaaS and glean insights as to where this new approach to testing could aid in some of their own projects.
Software testing practitioners (including test managers) may find solace that many of the hard problems they struggle with are shared with others in multiple disciplines. The evaluation of representative TaaS tools could aid in their own decision making process as they consider alternate solutions to their testing challenges.
Students are always looking for thesis and dissertation topics. This book is full of possible research themes, as evidenced by the hard problem summary. The gap analysis between perceived testing need and current TaaS capabilities also provides a number of promising avenues of future work. There is certainly no lack of topics to choose from.
OUTLINE OF THE BOOK
The book is structured as three main parts: a summary of the hard problems in software testing, an examination of testing as a service, and case study with gap analysis. Chapter 2 focuses on the Hard Problems in Software Testing project, explaining its background, the process, the results, and some conclusions. Chapter 3 details the field of Testing as a Service and explores the academic and commercial state of the industry. Chapter 4 offers a case study, evaluating a selection of TaaS tools against criteria developed from the software testing challenges, and analyzes the gap between the current state of the industry and the needs of testing professionals.
Scott Tilley and Brianna Floss
Melbourne, FL
July 2014