15.2.1 CppTest

CppUnit is a unit-testing framework for the C++ programming language, described as a C++ port for JUnit. The library can be compiled for a variety of portable operating systems interface (POSIX) platforms, allowing unit testing of “C” sources as well as C++ with minimal source modifications.

Test scripts are written in the syntax of a C++ class and include specialized test_add() statements, where each test_add() statement refers to separately defined routines that invoke tests in the form of logical assertions. Each such a routine is declared as a void C++ method that includes assertions of the form TEST_ASSERT(), which takes logical assertions as parameters. Using these statements and structures, the tester can define arbitrarily complex test scripts.

In order to run a test script, the user has to define an output handler, which records and organizes the output produced by the TEST_ASSERT() statements; a number of formats are available, including a text format, that may be shipped to the stdout stream and an html format, among others.

Among the strengths of CppTest, we cite that it puts the power of C++ at the disposal of the tester, to define arbitrarily sophisticated test scripts; in particular, the tester may use the exception-handling mechanisms of C++ in conjunction with the assert statements of CppUnit to handle any exceptions that may arise during the test in a controlled manner.

15.2.2 SilkTest

SilkTest is a suite of related test tools that offer a range of interrelated functionalities, including the following:

• SilkTest Classic, which uses a domain-specific scripting language inspired from C to support the development of test scripts.
• Silk4J, which uses Java as a scripting language to support the derivation of test scripts in Eclipse.
• Silk4Net, which uses Visual Basic or C# as a scripting language to support the derivation of test scripts in Visual Studio.
• SilkTest Workbench, which uses VB.NET as a scripting language to support test automation.

In order to operate SilkTest, we open a new project, give it a name, associate the project name to the application we want to test (possibly a web-based application), configure SilkTest to a particular browser, then record the sequence of tests we want to run. This sequence of tests defines the test script; once this script is saved, it can be run by merely calling the project name and invoking its execution, possibly unattended.

Among the strengths of this tool, we cite its ability to support integrated testing of graphical user interface (GUI) applications, its ease of use (notably that it does not require any programming knowledge, hence can be operated by the end user of the application), and the availability of test templates that provide standard test frameworks. Silk has two drawbacks: first, it can only support two browsers, namely, Internet Explorer (© Microsoft) and Firefox (© Mozilla). Also, it does not support migration between the two browsers, that is, a script composed on one browser cannot be run on the other.

SilkTest can be seen as a scripting tool but can also be seen as a save-and-replay tool, to the extent that it stores tests by recording an interactive session on the web-based application. The tools of the next section fall exclusively in the latter category.

15.3.1 TestComplete

TestComplete is an automated testing tool that aims to enable testers to create, record, edit, and save complex test scripts; it also saves failure logs that correspond to test scripts.

Sample example of use: We use this tool to edit, save, then replay a test sequence.

• First, we click on the button labeled “record a new test” on the TestComplete start page.
• This brings up the window to create a new test project and allows us to give it a name and to decide where to save it; also, it allows us to associate the project to an application that we want to test.
• Upon this, the TestComplete window minimizes down to a small recording toolbar; from now on and until we click on the stop button, everything we do with the selected application is recorded in the test script.
• Once the test script is saved, it can be reexecuted by clicking on the button labeled “run the test” in the window that corresponds to the TestComplete project.

Among its strengths, TestComplete offers the ability to record arbitrary interactions between the tester and the application, can run unattended, and does not require any programming or scripting skills. Nevertheless, for users who have adequate programming skills, TestComplete offers the ability to integrate sophisticated testing scenarios written in a wide variety of scripting languages, such as VBScript, JScript, C++Script, C#Script, or DelphiScript.

15.3.2 Selenium IDE

Selenium interactive development environment (IDE) is a Mozilla Firefox add-on that provides an interface for running test cases or complete test suites via a simple mechanism of record and playback of browser transactions. It can be applied to any application whose user interface is supported by a browser; this includes virtually all web applications.

Sample example of use: We can start an experiment by opening a tab of Firefox and running the Selenium IDE through its Tools tab; once Selenium IDE is on, we push the record button to start the recording process. From then on, Selenium IDE records any operation we do on the browser, such as visiting URLs, navigating through Web pages, and searching keywords on Web pages. When we initiate the replay, Selenium runs all the interactions that were recorded and displays the reactions of the web application under test.

Among the strengths of Selenium IDE, we cite that it requires no programming experience, hence can be run by an end user; that it provides for extensions and customizations; and that its scripts can be converted to a variety of programming languages, such as Java, Ruby, C#, and Python. The main weakness of Selenium IDE is, of course, that it can only run on Mozilla Firefox.

15.4.1 LoadRunner

LoadRunner is a testing tool for performance and load testing of software applications; it proceeds by simulating arbitrary levels of workload (in the form of concurrent users) on an application under test and observing its behavior.

Sample example of use: The main artifact that LoadRunner deals with is that of a load scenario; a load scenario is defined by a set of virtual users, along with characteristics of their demands on the system under test. LoadRunner enables the tester to create, edit, save, and invoke load scenarios. When a load scenario is invoked on an application, it enables the tester to observe and analyze the behavior of the system under the workload imposed by the load scenario. Once the scenario has completed its execution, Load Runner delivers a table of statistics, which may include such details as the following:

• Tables showing the number of active virtual users, the number of processed transactions, and the number of failed transactions.
• Graphs plotting a variety of parameters as a function of time, including number of virtual users, response time, number of hits per second, and system resources.
• Statistics (such as Max, Min, Average, and Standard deviation) of relevant parameters, such as rate of occupancy of processor time, file data operations per second, processor queue length, and page faults per second.

The main strength of LoadRunner is clearly its ability to simulate arbitrary levels of workload, along with its ability to capture detailed statistical information on the behavior of the program under test; this information is a great resource to a tester who wants to analyze the operational properties of a software application.

15.4.2 Grinder

Grinder is a desktop application that enables the tester to test a software application under controlled workload conditions; to this effect, Grinder uses a specialized scripting language called Jython, which is derived from Python.

Sample example of use: When Grinder is called, a window opens with four tabs, labeled Graphs, Results, Processes, and Scripts. Under the Processes tab, we configure the agent processes that act upon the application under test. Under the Script tab, we are given the opportunity to edit test scripts or to invoke test scripts that are already written in Jython. We start the test codified in the selected script by pressing the button labeled “start process.” Upon completion of the test, we obtain statistics about the performance of the target application. For a transaction processing application, these may include such details as the following:

• Number of tests, divided into number of successful tests and number of failures.
• Mean number of transactions per second and peak number of transactions per second.
• Mean response time, divided into mean time to resolve host, mean time to establish connection, and mean time to access the first byte.
• Mean response length and number of response bytes per second.

Among the strengths of the Grinder, we can cite its ability to load test any application that has a Java API, its ability to support detailed test scripts written in Jython, and its ability to deploy a wide range of distributed load injectors (i.e., processes that simulate the workload on the system).

15.4.3 QF-Test

QF-Test is a tool for the creation, execution, and management of automated system and load tests for Java and Web applications with a GUI. It is compatible with a wide range of development technologies and development environments, including Swing, JavaFX, and SWT and supports cross-browser testing of web GUI. It can test static html web pages as well as dynamic web applications developed in Ajax technologies.

Sample example of use: QF-Test has a capture-and-replay function that enables it to run individual tests or complete test suites by merely starting the recording function then manually executing whatever test script the tester wants. Whenever the test script is invoked, it executes the test on the application and produces a configurable report of the execution including statistics about possible errors. Because of its scalability, this tool can be used for system-level testing.

15.4.4 Appvance PerformanceCloud

Appvance PerformanceCloud is a tool that specializes in testing the performance and load capabilities of web applications. It is compatible with Ajax development environments, supports end-to-end testing to a great level of detail through all the steps of the application, can simulate millions of simultaneous users using a wide variety of web browsers, down to fine detail such as every keystroke and every message transfer, and supports pervasive agile integration. In addition, it requires little or no coding and supports customizable report generation, thereby providing faster test development.

For a simulated user load, the tool provides a detailed analysis of system performance, including information about the performance of individual transactions, their rate of success, their resource usage, their execution path, and so on. It also offers a large menu of measures one can collect on the application under test, such as its overall resource usage, its failure history, its performance scalability, or its throughput.

15.4.5 JMeter

Apache JMeter enables the tester to analyze the performance of an application under controllable workload conditions; it can be used to evaluate system performance on static and dynamic resources and to simulate a transaction load on a server, a group of servers, or a network. It provides performance data under a wide variety of textual and graphic formats, under arbitrarily controllable heavy workloads.

Sample example of use: The basic artifact in JMeter testing is the test plan, which is a sequence of steps that JMeter executes whenever it is invoked. JMeter supports the tester in editing a test plan, which may contain any combination of the following elements:

• One or more thread groups, where each thread group is a collection of concurrent processes submitting simultaneous queries to the system under test.
• Logic controllers, which coordinate the deployment of the thread groups by means of control constructs (similar to programming language constructs).
• Sample-generating controllers, which generate requests (e.g., http requests) that each thread group will produce when they are activated.
• Listeners, which are JMeter functions that collect performance data and store it as part of the application performance analysis.
• Assertions, which are test scripts that enable the tester to check whether the application under test is returning the expected results.

The output of JMeter depends on the format the tester has specified in the Listener and may include detailed information about the execution of the system under test (functions invoked in the system, number of http queries, number of database queries, number of ftp requests, number of active and sleeping tasks, etc.) as well as the resource usage of the system (CPU usage, memory usage, buffer usage, communication infrastructure bandwidth, etc.).

Among the strengths of JMeter, we may mention the following:

• Because JMeter is an open-source tool, a user may adapt to her/his needs and customize it as needed.
• One may build a relatively sophisticated test plan without prior knowledge of test scripts or scripting languages; also, the process of constructing test plans is fairly simple, thanks to a user-friendly GUI.

15.5.1 JUnit

JUnit is an open-source framework for writing test oracles in Java to perform unit testing on Java code. It comes with a .jar executable file that must be added as an external jar file to libraries in the user’s project. It enables the user to deploy predefined functions to perform unit testing. Oracles are modeled as Java classes that compute Boolean assertions.

Sample example of use: JUnit oracles can be composed by means of Java classes of type @Test and can be arbitrarily hierarchical (higher-level oracles calling more elementary oracles). Once a JUnit class is written, it can be compiled in Java; its execution yields a report in the form of a tree structure showing each assertion of the hierarchy, and whether it returned true or not.

The main strength of JUnit is that it enables the user to write arbitrarily complex test oracles in a structured, hierarchical manner; also, it supports a neat separation between production code and test code and makes it possible for developers and testers to work independently.

15.5.2 TestNG

TestNG is a testing framework for the Java programming language; it is designed to cover a wide range of testing categories, including unit testing, integration testing, system testing, and so on. TestNG is an annotation-based testing framework that aims to make up for some of the shortcomings of JUnit by offering a higher level of flexibility; whereas JUnit tests are structured by means of the class hierarchy of Java, TestNG tests are composed from elementary test classes by means of an XML file structure.

Sample example of use: In order to build a test suite in TestNG, the tester must write elementary tests in Java classes; then he must compose these elementary tests into suites by means of an XML file structure. Execution of the test suite returns a report on which elements of the suite were successful and which elements were not; whereas in JUnit a test that fails causes its dependent tests to fail, in TestNG, when a test fails, its dependent tests are skipped, and the remainder of the test suite is allowed to proceed.

15.6.1 Rational Purify

Rational Purify is a testing tool that checks for run-time memory-related exceptions, such as out-of-bounds array references, pointer references, uninitialized memory references, buffer overflow, inappropriate deallocation of memory, unauthorized write operations, and so on. It supports a diverse set of languages (Java, C++, and .NET) and a wide range of development environments (Visual Studio, .NET, and Eclipse).

Sample example of use: Rational Purify requires no setup in the way that other tools do; all we need is to run the tool on the source code of the program under test; the tool returns its diagnosis in the way a compiler does, that is, by pointing to the offending statement and posting an appropriate error message. This tool operates on intact source code; no modifications are needed, provided it is written in one of the supported programming notations.

15.6.2 Astree

Astree takes its name from the French translation of Real-Time Embedded Software Static Analyzer. It is a static analyzer of C code whose purpose is to check C programs for a range of run-time errors. The operation of this tool is based on the theory of abstract interpretation, which reasons about program semantics by means of abstractions. The abstractions must be selected in such a way as to be sufficiently high to be computationally tractable yet sufficiently detailed to enable effective reasoning.

Astree is deployed on source code and produces a report in the form of possible warnings or error messages pertaining to run-time errors such as arithmetic overflow or underflow, array references out of bounds, division by zero, as well as any user-defined assert statements. Astree is guaranteed to signal all possible errors, though it can also signal errors that cannot happen (i.e., false alarms).

15.7.1 FitNesse

FitNesse is a collaborative tool whose purpose is to serve as a medium for communication between the various stakeholders of testing, namely, programmers, testers, customers, verification and validation (V&V) team members, and so on. It is essentially a wiki that enables stakeholders to share information and artifacts pertaining to the testing activities of an ongoing software development and evolution process.

Sample example of use: Imagine that we want to test a division function. FitNesse offers the means to develop a routine to check the operation of this function by providing a set of test data and a test oracle. Regardless of which stakeholder provides which information, the fact that this information is shared in FitNesse ensures that it is double checked and vetted by all relevant stakeholders.