CHAPTER 3

Testing as a Service (TaaS)

As software systems grow in size and complexity, testing is becoming an increasingly costly endeavor in terms of time, labor, and other resources. Despite the best efforts of software testers, defects continue to elude standard testing techniques, resulting in production environments where errors cost significantly more to repair.

Testing as a Service (TaaS) is a relatively new form of testing that “leverages the vast resources of cloud environments to offer testing services to consumers on an as-needed basis” [2]. TaaS providers tout its potential to revolutionize the industry, promising to greatly reduce efforts while costing less than traditional testing. Nevertheless, TaaS, like any new technology, is not without risks and challenges of its own.

This chapter provides an overview of the current state of TaaS from both academic and commercial standpoints. Chapter 4 compares the capabilities of the current state of TaaS solutions to the testing needs identified by the HPST survey described in Chapter 2.

3.1    KEY COMPONENTS OF TaaS

This section provides background on two key components of TaaS: cloud computing and service-oriented architecture. The cloud provides the test execution environment for the system under test and for the testing tools themselves. Service orientation provides a programmatic interface to the cloud-based testing infrastructure.

3.1.1   CLOUD COMPUTING

According to the National Institute of Standards and Technology (NIST), cloud computing is “a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable resources … that can be rapidly provisioned and released with minimal management effort or service provider interaction” [15]. Cloud computing must offer on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service that can be deployed in four separate models:

•  The private cloud is owned by a single organization that maintains sole access.

•  The public cloud is available for general consumption.

•  The community cloud is accessible by a specific set of consumers.

•  The hybrid cloud is composed of a distinct and independent set of cloud infrastructures (private, community, and/or public).

NIST also defines three models of service, where cloud providers offer a set of underlying infrastructure to be utilized by consumers:

•  Software as a Service (SaaS): Consumers run applications in the cloud, which are accessible through the web. SaaS providers supply the operating systems, storage, networking, and servers. Examples include Google Docs and Microsoft Office 365.

•  Platform as a Service (PaaS): Consumers use provided languages, libraries, and tools to deploy applications to the cloud, offering them greater control over applications and their configurations. PaaS providers supply the operating systems, storage, networking, and servers. Examples include Google App Engine and Amazon Web Services (AWS) Elastic Beanstalk.

•  Infrastructure as a Service (IaaS): Consumers deploy their own operating systems and applications and have access to provider-provisioned processing, storage, networking, and other resources. IaaS providers have some control over the operating systems, storage, and, in some cases, networking. Examples include Amazon EC2 and Windows Azure.

3.1.2   SERVICE-ORIENTED ARCHITECTURE

Service-oriented architecture (SOA) is an architectural style of distributed computing in which functionality is defined by loosely coupled components that are responsible for realizing one or more capabilities, called services [16]. SOA is not restricted to any one technology, but web services are commonly used as a means to provide interoperability. Web services utilize a machine-processable interface written as an XML-based web services description language (WSDL) and communicate with external systems using SOAP messaging, an XML-based protocol containing structured information. TaaS uses services to perform testing responsibilities and report results. Aspects of SOA are mentioned in later sections, particularly in describing TaaS product features in Section 3.3.2.

3.1.3   TESTING AS A SERVICE (TAAS)

Since the NIST’s definition of cloud computing was written in 2011, the usage has expanded to include additional types of services with distinctly new purposes. Combining elements of cloud computing and service-oriented architecture, TaaS “offers the ability to test software in the cloud as a competitive and easily accessible web service, allowing tests the ability to harness the vast and elastic resources provided by the cloud infrastructure” [2] [17]. To perform a test, an individual or automated tool submits the request via a well-defined service. The test is executed per request and all test findings and metrics are returned in the form of a detailed report.

The actual test execution may be handled in three forms:

•  The tests can be executed in a fully automated manner, based on a set of pre-defined parameters included in the submitted request.

•  The tests can be human-backed, run in a similar manner to outsourcing where a request is submitted and human intervention in the form of a third party is required to complete the request.

•  The tests can be completed through a hybrid mixture of human-intervention and automation.

The goal of TaaS is to offer highly available and highly accessible testing services at a low cost, using a pay-as-you-test pricing model. TaaS aims to provide more thorough testing performed in less time than traditional testing, promising safer and more reliable applications [17]. Various types of testing have already been incorporated into testing services and are described in the following sections.

3.2    ACADEMIC STATE OF TAAS

In academia, various aspects of TaaS have been studied and evaluated. Research into architecture, examination of specific types of testing, and case studies are detailed in the following sections. Benefits, challenges, and needs as discussed by academics are reviewed. Other related research, with the exception of papers overviewing and surveying existing work, is briefly mentioned.

3.2.1   ARCHITECTURE

According to academic research, TaaS utilizes a layered architecture, typically consisting of four to five layers [18] [19]. The actual layers vary to some degree, but most follow the same general flow described by Lian Yu and his associates from Peking University and the IBM China Research Center in 2010 [20]. Figure 3.1 shows Yu’s TaaS architecture.

Figure 3.1: Yu’s TaaS architecture.

The top layer supports customer interaction, with services being accessible over the internet via various tools and applications as permitted by the service provider. Service requests are submitted by users with any necessary parameters and data. Yu divides the top layer, referring to it as the Test Tenant and Test Service Contributor layer, where the contributor side allows providers to publish new services directly to the platform.

The remaining layers handle test and resource management, test execution, and, in some cases, a database layer that stores tasks, code, testing services, bugs, and other relevant information. The second layer is the Test Task Management layer, which acts as a testing service bus and is responsible for checking capabilities; clustering, scheduling, and dispatching tests; monitoring tasks; and registering, reposing, and publishing services. The Test Resource Management layer monitors the physical and virtual resources, allocating them as required. The final layer is the Test Layer, which handles the actual test execution and aggregation of test results. When providers offer test case generation services, the Test Layer is also responsible for that functionality.

3.2.2   TYPES OF TESTING

Several academic researchers have focused their efforts on describing or creating testing services that offer the functionality of a specific type of testing. A survey of this research is described in the following subsections.

Predicate and Certification Testing

In 2010, George Candea, Stefan Bucur, and Christian Zamfir of École Polytechnique Fédérale de Lausanne in Switzerland presented three different potential testing services [17], each intended for a unique stakeholder. Their research endeavored to promote TaaS as a powerful tool, permitting complex testing with minimal upfront investments. Each service offered a specific type of testing necessary to guarantee quality to the stakeholder, while demanding little to no knowledge of software testing.

TaaS_D for Developers (“Sidekick”) acts as a continuous predicate testing service, automatically pulling the latest code from the source code repository to test against a series of vendor- and developer-defined predicates, described respectively as universal and application-specific predicates. The service walks through execution paths, creating assertions based on the predicates. Results are returned to the developers to promote better software development. TaaS_D should be run prior to submitting code for test.

TaaS_H for End Users (“Home Edition”) is a separate predicate testing service intended for end users that want to verify the quality of a software application on their personal system with minimal effort. Users simply upload the application in binary or bytecode form and the service examines it for common bugs, such as buffer overflows, deadlocks, and race conditions. The service returns a webpage containing all results in a readable and easily understood format, including any identified issues ranked by severity. TaaS_H is less flexible than TaaS_D, but offers quality assurance to stakeholders with no knowledge of software testing.

TaaS_C Certification Services is a public certification service that provides an objective assessment and rating of the quality of an application under test. Based on the funding government or industry, the application can be tested and certified on varying levels, differing in the predicates against which it is examined. A complete report is automatically generated for internal use or for publishing statistics on the software’s quality and reliability. TaaS_C can be rerun with each subsequent release, quickly recertifying the system.

Optimized Regression Testing

Sheng Huang and his associates from Fudan University, Peking University, and IBM China Research Lab described a selective regression testing platform called RTaaS in 2011 [21]. RTaaS utilizes their optimized regression test selection tool, ORTS, to seek out the minimal set of regression tests affected by new or modified code, submitted in the form of a project binary. Impacted regression tests are then executed in the cloud environment, saving time and effort by reducing the number of tests being rerun and by taking advantage of the cloud’s resources. RTaaS’s architecture is shown in Figure 3.2. Experiments on the RTaaS platform are described in the section on academic case studies.

Figure 3.2: RTaaS architecture.

Web Service Load Testing

In 2012, Minzhi Yan and fellow researchers from Beihang University in Beijing, China, identified challenges in pre-existing web service load testing environments and laid out four requirements for a better, TaaS-based system [22]. Their WS-TaaS implementation, based on their Service4All PaaS platform, needed the following:

•  Transparency, negating the tester’s need for knowledge about the hardware and software configuration;

•  Elasticity, allowing the test capabilities to scale appropriately with demand;

•  Geographical distribution, simulating real world runtime scenarios of user access from varying locations across the globe; and

•  Massive concurrency and sufficient bandwidth, offering the ability to manage wide test loads concurrently and with sufficient bandwidth to complete the request.

Yan’s WS-TaaS layered architecture, similar to Yu’s in execution, is shown in Figure 3.3.

Figure 3.3: WS-TaaS architecture.

WS-TaaS has three different test modes [23]. A static test examines the performance of the web service using a pre-defined load parameterized by the user. A step test studies the web service’s ability to cope with a specific load span, beginning with a start number, step size, and end number of concurrent requests. A maximal test determines the web service’s load limit, stressing the service until more than 10% of the concurrent requests fail. Various experiments were performed against WS-TaaS and are described in the academic case studies section.

Structural Unit Testing

In 2013, Nikolai Kosmatov and his colleagues from the CEA LIST Institute’s Software Reliability Laboratory in Gif-sur-Yvette, France, discussed their work on structural unit testing with Path-Crawler-online.com and its potential as a TaaS application [24]. PathCrawler is an online service that accepts source code written in C and uses concolic or dynamic symbolic execution to generate test case inputs that guarantee structural coverage of the application. PathCrawler is not currently deployed in the cloud for test execution, but is actively under development and has been considered a good candidate for becoming a TaaS application. The authors describe the need for unit testing in the cloud and the potential role that PathCrawler could play as a developer’s sidekick.

PathCrawler works by compiling the submitted C source code into an executable test harness, called a Launcher. The Launcher’s executable paths are explored by the Path Explorer, which sends partial paths to the finite-domain constraint Solver. The Solver returns the test inputs for that path to the Launcher, which can then execute the system under test with the data. The process is cyclic, allowing the Path Explorer to continue down the next partial path. When the Solver is no longer able to generate a test case or a timeout occurs, the Path Explorer seeks an alternate path. This process is illustrated in Figure 3.4. When complete, PathCrawler generates and returns all test cases, along with coverage and path statistics.

Figure 3.4: PathCrawler method.

3.2.3   CASE STUDIES AND EXPERIMENTS

Many of the academic researchers conducted case studies and experiments with their TaaS systems. The results provide useful insight into the state-of-the-art in the area.

Performance of Local and Remote Systems

Yu and his colleagues built their TaaS platform on Apache Tuscany using Java and C++ [18]. They selected two white box and two black box services for the generation of test cases: BP (Basic Path), BRO (Branch and Relational Operator), GEO (Generalized Extremal Optimization), and Pairwise. Their first test evaluated the difference in service invocation times between local and remote systems using all four services. Figure 3.5 shows that an additional 2.5–3.5 seconds was required for remote invocation. The increase was attributed to the necessary time required to communicate between test tenants and the TaaS platform.

Figure 3.5: Local vs. remote invocation times.

Their second experiment used BP and BRO and demonstrated the performance of test case generation while increasing the size of the system under test, taking the average time from ten trials at each size. Figure 3.6 shows the runtime when increasing the lines of code in the application; Figure 3.7 shows the runtime when increasing the number of conditional branches in the application. When increasing the lines of code, additional time was only necessary for parsing the source. On the other hand, the tools were sensitive to increases in complexity. Significant increases in time were required when the number of conditional branches increased beyond twenty, but was considered acceptable otherwise.

Figure 3.6: Runtime with increasing lines of code.

Figure 3.7: Runtime with increasing number of conditional branches.

Experiments with RTaaS

Huang et al. deployed their RTaaS tool to a software park in China for study [21]. Three groups participated in the case study. Project A was a small project under IBM China that required only minor function updates or occasional bug fixes. Projects B and C were both under development by independent software vendors that were struggling with the time required for regression testing. Table 3.1 offers the basic information on each project, including number of files, source lines of code (in thousands), programming artifacts, test iterations, and test cases. Programming artifacts include any external files required for execution, including software libraries (Java, JSP, JavaScript) and configuration files. For the purpose of this case study, all projects used comparisons between the optimized regression test selection and rerunning all regression tests.

The regression test selection (RTS) time was studied across all projects. As package unzipping occurred offline, only the change identification and test selection were considered in the RTS time. The results are shown in Figure 3.8.

Figure 3.8: RTS cost across project iterations.

Additionally, Huang and his associates examined the effectiveness of the selection strategy, which they referred to as its safety. Rerunning the entire regression test suite is considered very safe, but costs too much in terms of time consumed. In each project iteration, the rerun all and RTS methods found the same number of defects, suggesting they shared the same high safety for these cases. The results of this experiment are shown in Figure 3.9.

Figure 3.9: Safety of RTaaS.

Experiments with WS-TaaS

Yan et al. performed experiments to compare the performance of WS-TaaS against traditional single node JMeter load tests [23]. Their first test compared the average response time (ART) of three services under varying loads. Figure 3.10 shows that the ART of the JMeter tests increased significantly with an increasing number of concurrent requests, while WS-TaaS tests’ ART remained relatively consistent despite load increases.

Figure 3.10: ART comparison with increasing load.

Another test analyzed the error ratio of JMeter and WS-TaaS when running load tests against a web service that validates email addresses. As shown in Figure 3.11, the load limit for WS-TaaS was significantly less than JMeter.

Figure 3.11: Error ratio over increasing loads.

3.2.4   BENEFITS, CHALLENGES, AND NEEDS

A significant amount of early research into TaaS focused on its potential, exploring anticipated benefits and migration feasibility, and its risks, seeking challenges that must be addressed. These items are discussed below.

Benefits

One of the most commonly cited benefits of TaaS in academic literature is reduced cost. Software development has become a costly endeavor when considering employee salaries and benefits, physical building space and utilities, equipment upgrades and maintenance, and software licenses [2]. Each of these assists in driving the cost of the final product up, which must then enter a competitive market where prices frequently affect success. TaaS promises an overall reduction in the costs associated with testing by offering testing services at a pay-as-you-test rate. The base rate includes a large variety of environments and hardware configurations with licensed software, including operating systems, preconfigured and available for use. Not only does this reduce the cost of test lab machines, maintenance, software, and space—TaaS reduces the amount of overall labor cost by minimizing the number of testers required for the project. With access to the cloud’s vast and elastic resources, virtualized servers and on-demand systems are offered at a price point that is simply unachievable with traditional test labs [13].

Another major benefit of TaaS is a more efficient and less time-consuming testing process. In traditional testing, large test suites may take hours or days to run, sometimes on just one of the several environments under test. TaaS offers the option to run these tests with more resources and in more environments and configurations simultaneously, thereby eliminating the restrictions imposed by traditional testing’s physical labs and tight schedules. The services handle all of the resource provisioning, reducing setup time and allowing for a more fluid testing process. Results are returned to the testers after test completion and can be rerun as necessary, providing the continuous regression testing availability necessary for agile development [11]. With much greater access to environments and configurations, testers have a better indicator of the system under test’s quality and reliability.

Additionally, the testing services are highly consumable, allowing testers to quickly learn the TaaS tool with minimal effort; the complicated aspects of testing are handled by the provider, out of the tester’s sight. In Huang’s case study, users mastered the basic steps of the RTaaS tool within hours [21].

Challenges

As expected with any new technology, the benefits are balanced by issues and challenges. Perhaps the most significant challenge is maintaining constant global availability. Due to the nature of software development, test services must be accessible by end users at all times, providing flexibility and enhancing agility in testing [13]. While maintaining availability may be a difficult task on the provider’s side, the failure to do so could have crippling consequences on a consumer by causing lost time and data. Additionally, customer and technical support must be equally available to assist testers, especially when working in international markets. These staff members must maintain skills that match or exceed the needs of their clients to provide the best customer experience.

Another major challenge that exists across online services is security. Consumers must be confident that their proprietary information is secure and that any system flaws remain confidential. In addition to application security, any issues with storage or mismanagement of test data could compromise a system [13]. One option is to provide mock data that excludes sensitive information. At the same time, adequate quality measures sometimes depend on real production data [11]. Non-disclosure agreements may help mitigate some of these security concerns.

Existing software components that lack a standardized API for the cloud infrastructure may have connectivity problems, incurring increased integration costs [25]. Any interoperability challenges between the system under test and the cloud or TaaS tool could have disastrous effects on the testing process. Interoperability risks and other unsatisfied non-functional requirements could dissuade potential customers from adopting TaaS technologies.

As affordable testing services become available to the public, end users with less benevolent intentions could adopt TaaS tools to locate and exploit weaknesses in software. Candea et al. acknowledge these concerns, but believe that they may compel software companies to perform additional testing to mitigate risks and release better quality products [17].

Needs

Beyond the aforementioned challenges, TaaS has various needs that must be satisfied. Pricing models must be transparent and descriptive to educate customers, allowing them to make informed estimations of their expenses [12]. The description of services must be equally compelling, providing detailed information on service definitions and reportable metrics.

Standards for TaaS must also be created. Both the consumers and providers must have well-defined service standards that illustrate testing and quality details to assist in the development of fair Service Level Agreements (SLAs) [25]. At the same time, standards must not be overly restrictive, as that could lead to more predictable testing [11].

TaaS cannot succeed without innovative test methods and solutions, both to increase demand and to maintain its relevance in the future. Additionally, dynamic test platforms and tools must be used to provide powerful test simulators, develop flexible interfaces for existing tools and solutions, offer seamless integration, validate scalability and performance, and more [25].

Finally, as stated by Riungu-Kalliosaari et al., “cloud-based testing is more efficient and effective than traditional testing methods but organizations must understand and trust it before it sees widespread adoption” [13]. TaaS must prove itself to the community prior to being fully utilized.

3.2.5   OTHER RESEARCH

There are a few other areas of research that can affect the future of TaaS. These include mobile testing platforms, knowledge management systems, and a new type of fault injection testing technique called “failure as a service.”

Mobile TaaS Framework

Oleksii Starov and Sergiy Vilkomir of East Carolina University proposed a mobile TaaS platform called Cloud Testing of Mobile Systems (CTOMS) [26]. In 2013, they published a paper discussing the system, which consists of a cloud of devices, a static analysis engine, and a statistics sub-system. Testers can select from this pool of mobile devices for functional, performance, or manual testing. In their research, Android was the focal platform due to its widespread use and open-source platform.

CTOMS was developed as a master application deployed to the cloud with slave servers acting as nodes. The master application is responsible for organizing the core functionality of the system by collecting information, organizing and distributing work, and gathering the results for users. The nodes perform the actual testing, manage the devices, and collect statistics. The implementation is depicted in Figure 3.12.

Figure 3.12: CTOMS implementation.

Knowledge Management System

Rusli Abdullah of the University Putra Malaysia described the need for a Collaborative Knowledge Management System (CKMS) for TaaS [27]. His work rightly promotes the necessity of disseminating acquired knowledge to the community to assist in an overall reduction in errors and an improvement in quality of TaaS systems. The repository of knowledge would be structured as detailed in Figure 3.13. The repository is generic enough to be useful for many situations beyond just TaaS knowledge.

Figure 3.13: Structure of CKMS Repository.

Failure as a Service

Haryadi Gunawi (now at the University of Chicago) and his colleagues at the University of California, Berkeley, researched another aspect of system quality. Their work proposed a new cloud service that permits routine large-scale failure drills, allowing testers to pinpoint problems preventing a normal recovery following a service outage [28]. In many high profile cloud software outages, the issues were prolonged due to a failure to consider all potential complications after the system went temporarily offline. While not directly related to TaaS, Gunawi et al.’s Failure as a Service (FaaS) tool offers an additional means to verify quality in a software vendor’s cloud application.

3.3    COMMERCIAL STATE OF TAAS

Numerous companies are already providing TaaS tools and solutions. Twelve different companies are discussed in this section. Of them, three are discussed in much greater detail during the case study provided in the following chapter. Table 3.2 lists the vendors currently involved in TaaS along with their website. The company name is located in the first column and is emphasized with bold characters. If the product name differs, it is listed below the company name in the normal font.

As TaaS is a relatively new aspect of computing, commercial offerings are subject to change. Furthermore, it is important to note that the information contained in the remainder of this chapter is based on advertised features that were not, in the majority of cases, validated during the course of this research for availability, functionality, or quality. Many tools, especially the larger commercial products by CA, HP, IBM, and Oracle, were not available for trial or limited use. Additionally, absent information may not indicate that a feature is not provided; rather, the feature simply is not advertised by the vendor.

3.3.1   TYPES OF TESTING

The aforementioned companies deliver a wide variety of testing services. An overview of comparable services is displayed in Table 3.3. The forms of testing offered by TaaS vendors are defined below. Unless otherwise specified, the definitions are based on IEEE Standard 61012-1990 [29] and/or ISO/IEC/IEEE 29119-1 [30, 31, 32].

•  Concurrency Testing: A form of testing that evaluates the behavior of two or more components when run within the same time interval.

•  Domain Testing: A form of testing focused on systematically covering different input situations based on the input domain, negating the need to exhaustively cover all possible input values [33].

•  Functional Testing: A form of testing that ignores the internal workings of the component or system and instead evaluates the outputs generated in response to inputs and execution conditions.

•  Infrastructure Testing: A method of testing the system’s underlying infrastructure.

•  Integration Testing: A form of testing that evaluates the interaction between software and hardware components.

•  Load Testing: A form of performance efficiency testing used to evaluate how a system behaves under conditions of varying load, as anticipated during low, typical, and peak usage.

•  Localization Testing: A type of testing that verifies the quality of a system’s translation of its interface for suitability in another region [34].

•  Performance Testing: A form of testing that evaluates the ability of a system or component to accomplish designated functions with time and/or resource restrictions.

•  Regression Testing: A method of selectively re-testing previously evaluated behaviors of a system or component following modifications to ensure that it still complies with the original requirements and that the change had no unintended consequences.

•  Security Testing: A form of testing that evaluates the ability of a system to protect a test item and its associated data from unauthorized persons or systems while allowing access for authorized persons or systems.

•  Stress Testing: A variation on load testing that evaluates how a system behaves under conditions of higher than anticipated or specified load or with below specified resource availability.

•  Unit Testing: A form of testing that scrutinizes the behavior of individual or related software units.

•  Usability Testing: A form of testing that judges how easily a system or component can be understood and operated by a user.

In addition to the types of testing listed in the above table, several vendors have unique services available to consumers. Some of these unique services include:

•  Applause: Localization and Usability Testing

•  BlazeMeter: Stress Testing

•  CA LISA: Concurrency and Regression Testing

•  CSC: Domain Testing

•  IBM: Infrastructure and Integration Testing

3.3.2   PRODUCT FEATURES

The advertised product features of each tool are displayed in the next three tables. Table 3.4 shows the standardized protocols and system types evaluated by each tool. Table 3.5 lists the supported test script languages and notes whether tests can be recorded and played back rather than written from scratch. Table 3.6 catalogs compatible tools for version control, driving builds, continuous integration, and defect tracking. Unfortunately, several of the vendors offer little information on their tools and services, which accounts for many of the blank cells in the following tables.

The following describes the protocols that are employed by vendors. Several of these web service descriptions are based on CMU/SEI-2010-TR-011 [35]:

•  HTTP (Hypertext Transfer Protocol): Services are tested for conformance over HTTP.

•  HTTPS (HTTP Secure): Same as HTTP, but the services require verification for security, such as checking for the validity of an SSL certificate.

•  XML-RPC (Extensible Markup Language-Remote Procedure Call): An XML-based protocol that transfers information over HTTP. XML-RPC proceeded SOAP.

•  SOAP (Simple Object Access Protocol): An XML-structured protocol for exchanging information in web services. Can be used in HTTP and non-HTTP situations, such as TCP and named pipes.

•  REST (Representational State Transfer): A more recent and lightweight standard that utilizes HTTP to transfer information in web services.

•  WSDL (Web Services Description Language): An XML-based description of a web service, often used in combination with SOAP to provide web services.

Additionally, databases and applications are included in Table 3.4 and are used in the following context:

•  Databases (DB): The database information can be accessed for verification via SQL (MSSQL, MySQL, Oracle, etc.).

•  Applications (Apps): The types of applications that are testable by the tool.

Note: By integrating with Jenkins CI, consumers gain access to the Jenkins plugin library. This library consists of version control systems, drivers, defect tracking tools, and more.