INTRODUCTION

As seen in earlier chapters there are many tasks and activities that need to be performed during the testing process. In addition, other tasks need to be performed to support the testing process.

In order to assist in making the testing process easier to perform and manage, many different types of test tools have been developed and used for a wide variety of testing tasks. Some of them have been developed in-house by an organisation’s own software development or testing department. Others have been developed by software houses (also known as test-tool vendors) to sell to organisations that perform testing. More recently, open source tools have been developed that can be reused and enhanced. Even within the same type of tool, some will be home-grown while others will be developed as open source tools or by test-tool vendors.

This chapter discusses the potential benefits and pitfalls associated with test tools in general. It then describes the most commonly used types of test tools and concludes with a process for introducing a tool into a test organisation.

Learning objectives

The learning objectives for this chapter are listed below. You can confirm that you have achieved these by using the self-assessment questions that follow the ‘Check of understanding’ boxes distributed throughout the text and the example examination questions provided at the end of the chapter. The chapter summary will remind you of the key ideas.

The sections are allocated a K number to represent the level of understanding required for that section; where an individual topic has a lower K number than the section as a whole, this is indicated for that topic; for an explanation of the K numbers, see the Introduction (page 2).

Test tool considerations (K2)

Effective use of tools (K1)

Self-assessment questions

The following questions have been designed to enable you to check your current level of understanding for the topics in this chapter. The answers are at the end of the chapter.

Question SA1 (K2)

Which of the following pairs of test tools are likely to be most useful during the test analysis stage of the test process?

Question SA2 (K1)

Which of the following is most likely to cause failure in the implementation of a test tool?

Question SA3 (K2)

What benefits do static analysis tools have over test execution tools?

WHAT IS A TEST TOOL?

Definition of a test tool

The ISTQB Glossary of Testing Terms defines a test tool as:

A software product that supports one or more test activities, such as planning and control, specification, building initial files and data, test execution and test analysis.

Therefore, a test tool can be thought of as a piece of software that is used to make the testing process more effective or efficient. In other words, anything that makes testing easier, quicker, more accurate and so on.

This book will focus on those test tools that are listed in the 2018 syllabus. These are listed in Table 6.5 on pages 227 to 232 and are, generally, the test tools that are most commonly used in the testing process. Other test tools that have been removed from the 2018 syllabus (such as Test Comparators) are also discussed to help understand newer tools and for completeness. But they do not need to be studied for the exam and are marked with [Not in Syllabus].

Benefits and risks of using any type of tool

Let us consider the building of a new hotel and examine the similarities with the introduction and use of test tools. Test tools need to be thought of as long-term investments that need maintenance to provide long-term benefits. Similarly, building a hotel requires a lot of upfront planning, effort and investment. Even when the hotel is ready for use, there is still a continual long-term requirement for the provision of services such as catering, cleaning, building maintenance, provision of staff to provide ad hoc services to customers and so on and, from time to time, the need for upgrades to infrastructure to keep up with new technology and customer demands. The long-term benefit is that this upfront investment and ongoing maintenance and support can provide substantial income in return.

In addition, there are risks that, over a period of time, the location of the hotel will become less attractive, resulting in lower demand, lower usage and a maintenance cost that is greater than the income received. Therefore, the initial investment is wasted because the ongoing need/requirement did not exist.

The graph in Figure 6.1 demonstrates a typical payback model for implementing a test execution tool. The same principle applies to the majority of test tools. Note that there is an ongoing maintenance cost of using the tool, but this ongoing maintenance cost needs to be less than the cost of performing testing activities without the tool if the investment is to be worthwhile.

Figure 6.1 Test tool payback model

The same advantages and disadvantages apply to the use of most types of test tool. However, there are exceptions to this generalisation (and to the same generalisation made in the ISTQB syllabus). Some tools, such as comparators, can be used virtually straight out of the box. A comparator can check whether one large test file is the same as another. If it is different, it can identify and report on the differences. This is very difficult and time-consuming to do manually. In addition, defect management tools are fairly intuitive and easy for both experienced and novice testers to use. They are also likely to provide a ‘quick win’.

Other tools can be built by developers in-house as the need arises. For instance, test harnesses, test oracles or test data preparation tools may be relatively easy to produce for developers with a good understanding of the tool requirements and the systems and databases in the test environment. More recently, open source tools have allowed developers and testers to use freeware tools as the building blocks for developing in-house tools to meet specific needs. In addition, test tools have been developed by the UK Financial Conduct Authority for use by banks and building societies that participate in the Faster Payments and Account Switcher schemes.

Benefits

The main benefit of using test tools is similar to the main benefit of automating any process. That is, the amount of time and effort spent performing routine, mundane, repetitive tasks is greatly reduced. For example, consider the time and cost of making consumer goods by hand or in a factory.

This time saved can be used to reduce the costs of testing or it can be used to allow testers to spend more time on the more intellectual tasks of test planning, analysis and design. In turn, this can enable more focused and appropriate testing to be done – rather than having many testers working long hours, running hundreds of tests.

Related to this is the fact that the automation of any process usually results in more predictable and consistent results. Similarly, the use of test tools provides more predictable and consistent results as human failings, such as manual-keying errors, misunderstandings, incorrect assumptions, forgetfulness and so on, are eliminated. It also means that any reports or findings tend to be objective rather than subjective. For instance, humans often assume that something that seems reasonable is correct, when in fact it may not be what the system is supposed to do.

The widespread use of databases to hold the data input, processed or captured by the test tool, means that it is generally much easier and quicker to obtain and present accurate test management information, such as test progress, defects found/fixed and so on (see Chapter 5). The introduction of web-based tools that have databases stored in the cloud means that such information is available to global organisations 24 hours per day, seven days per week. This facilitates round the clock working and can reduce elapsed times to analyse, fix and retest defects.

Risks

Most of the risks associated with the use of test tools are concerned with over-optimistic expectations of what the tool can do and a lack of appreciation of the effort required to implement and obtain the benefits that the tool can bring.

For example, consider the production environments of most organisations thinking about using test tools. They are unlikely to have been designed and built with test tools in mind. Therefore, assuming that you want a test environment to be a copy of production (or at least as close to it as possible), you will also have a test environment that is not designed and built with test tools in mind.

Consider the test environments used by vendors to demonstrate their test tools. If you were the vendor, would you design the environment to enable you to demonstrate the tool at its best or to demonstrate the shortcomings it may encounter in a typical test environment?

Therefore, unless detailed analysis and evaluation is done, it is likely that test tools will end up as something that seemed a good idea at the time but have been largely a waste of time and money. A process for avoiding such problems when introducing a tool into an organisation is described later in this chapter.

After a test tool has been implemented and measurable benefits are being achieved, it is important to put in sufficient effort to maintain the tool, the processes surrounding it and the test environment in which it is used. Otherwise there is a risk that the benefits being obtained will decrease and the tool will become redundant. Additionally, opportunities for improving the way in which the tool is used could also be missed.

For example, the acquisition of various test tools from multiple vendors will require interfaces to be built or configured to import and export data between tools. Otherwise much time may be spent manually cutting and pasting data from one tool to another. If this is not done, then data inconsistencies and version control problems are likely to arise. Similar problems may arise when testing with third-party suppliers or as a result of mergers and acquisitions. The increase in common standards for interfaces such as Extensible Markup Language (XML) means that the capability for developing successful interfaces is greater, but substantial time and effort are often still required.

Maintenance effort will also be required to upgrade and reconfigure tools so that they remain compliant with new platforms or operating systems.

TEST TOOLS

Types of tool

There are several ways in which test tools can be classified. They can be classified according to:

In this book, test tools will be classified according to the type of activity they support (as in the ISTQB Foundation Level syllabus).

Tool support for management of testing and testware

Test management tools and application life cycle management tools Test management tools and application life cycle management (ALM) tools provide support for various activities and tasks throughout the testing process. The main difference between a test management tool and an ALM tool is that:

In the remainder of this section any service/function provided by a test management tool will also be met by an ALM tool unless otherwise stated.

Consequently, standalone test management tools tend to be cheaper than ALM tools.

The diagram in Figure 6.3 shows how a test management tool (or ALM tool) is the hub or centre of a set of integrated test tools.

Test management (and ALM) tools provide an architecture for creating, storing and editing test procedures. These may be linked or traced to requirements, test conditions and risks. Such test procedures can then be prioritised or grouped together and scheduled so that they are run in the most effective and efficient order. Some test management tools allow dependencies to be recorded so that tests that will fail owing to a known defect can be highlighted and left unexecuted. This allows testers to be redirected to run the highest priority tests available rather than waste their time and the test data they have prepared on tests that are certain to fail.

Figure 6.3 An integrated set of tools

Tests can be recorded as passed or failed and usually a test management (or ALM) tool provides an interface to a defect management tool so that a defect can be raised if the actual and expected results differ.

Test management (or ALM) tools can provide management information and reports on test procedures passed or failed. The amount of integration with other specialist tools is significant here. For instance, integration with requirements management tools allows reports to be produced on test progress against one or more requirements. Integration with incident management tools also allows reports to include analysis of defects against requirements.

Test management tools generally hold data in a database. This allows a large amount of reports and metrics to be produced. The metrics produced can be used as inputs to:

Test management information reports should be designed to meet the needs of project managers and other key users. It may be necessary to export/import data in appropriate formats to other tools such as:

A test management (or ALM) tool can also enable reuse of existing testware in future test projects.

Defect management tools Defect management tools (also known as incident management tools) are one of the most widely used types of test tool. At a basic level, defect management tools are used to perform two critical activities: creation of a defect report; and maintenance of details about the defect as it progresses through the defect life cycle.

The level of detail to be captured about the defect can be varied depending on the characteristics of the tool itself and the way in which the defect management tool is configured and used by the test organisation.

For example, the defect management tool can be configured so that lots of mandatory information is required in order to comply with industry or generic standards such as IEEE 1044. In addition, workflow rules may also be applied to ensure that the agreed defect life cycle is strictly adhered to, with defects only able to be assigned to particular teams or users. Alternatively, the tool can be configured to require very limited mandatory information, with most fields being free format.

Defect management tools also use a database to store and manage details of defects. This allows the defect to be categorised according to the values stored in appropriate fields. Such values will change during the defect life cycle as the defect is analysed, debugged, fixed and retested. It is often possible to view the history of changes made to the defect.

The database structure also enables defects to be searched and analysed (using either filters or more complex Structured Query Language (SQL)-type queries). This provides the basis for management information about defects. Note that as the values held against each defect change, the management information will also change. Therefore, users need to be aware of the danger of using outdated reports.

This data can be used in conjunction with data held in test management tools when planning and estimating for future projects. It can also be analysed to provide input to test process improvement projects.

Fields in the database structure normally include:

This allows management information to be produced from the defect management database about the number of high-priority defects with a status of Open or Reopen that are assigned to, say, Peter Morgan, compared with the number assigned to Brian Hambling.

ALM tools typically include fully integrated defect management tools as part of their core product, while other defect management tools can be integrated with test management, requirements management and/or test execution tools. Such integration enables defects to be input and traced back to test cases and requirements.

Requirements management tools Requirements management tools are used by business analysts to record, manage and prioritise the requirements of a system. They can also be used to manage changes to requirements – something that can be a significant problem for testers, as test cases are designed and executed to establish whether the delivered system meets its requirements. Therefore, if requirements change after tests have been written then test cases may also need to change. There is also a potential problem of changes not being communicated to all interested parties, thus testers could be using an old set of requirements while new ones are being issued to developers.

The use of a traceability function within a requirements tool (and/or integrated with an ALM or test management tool) enables links and references to be made between requirements, functions, test conditions, defects and other testware items. This means that as requirements change, it is easy to identify which other items may need to change.

Some requirements management tools can be integrated with test management tools, while ALM tools typically enable requirements to be input and related to test cases within the ALM tool.

Requirements management tools also enable requirements coverage metrics to be calculated easily, as traceability enables test cases to be mapped to requirements.

As can be seen, traceability can create a lot of maintenance work, but it does highlight those areas that are undergoing change.

Configuration management tools and continuous integration tools Configuration management tools are designed primarily for managing the versions of different software (and hardware) components that comprise a complete build of the system; and various complete builds of systems that exist for various software platforms over a period of time.

Continuous integration tools have been developed more recently and can be used in conjunction with configuration management tools to ensure that the correct versions of different programs are integrated into the Daily Build that is deployed into the test environment. This is particularly advantageous for Agile developments where it is important to produce builds automatically and quickly.

A build is a development activity where a complete system is compiled and linked (typically daily) so that a consistent system is available at any time including all the latest changes.

The amount of benefit to be obtained from using configuration management tools and continuous integration tools is largely dependent on:

For example, a software house selling different versions of a product to many customers who run on a variety of operating systems is likely to find configuration management tools more useful than an internal development department working on a single operating system for a single customer. However, an internal development department using an Agile approach will find continuous integration tools almost essential for managing frequent deployments (Daily Builds) into the test environment.

The use of configuration management tools allows traceability between testware and builds of an integrated system and versions of sub-systems and modules. Traceability is useful for:

Tool support for static testing

Tools that support reviews Tools that support reviews (also known as review tools or review process support tools in previous versions of the syllabus) provide a framework for reviews or inspections. This can include:

Review tools can interface with configuration management tools to control the version numbers of a document under review.

If reviews and inspections are already performed effectively, then a review tool can be implemented fairly quickly and relatively cheaply. However, if such a tool is used as a means for imposing the use of reviews, then the training and implementation costs will be fairly high (as is the case for implementing a review process without such tools). These tools support the review process, but management buy-in to reviews is necessary if benefits from them are to be obtained in the long run.

Review tools tend to be more beneficial for peer (or technical) reviews and inspections rather than walkthroughs and informal reviews.

Static analysis tools Static analysis tools (also known as static code analysers) analyse code before it is executed in order to identify defects as early as possible. Therefore, they are used mainly by developers prior to unit testing. A static analysis tool generates lots of error and warning messages about the code. Training may be required in order to interpret these messages and it may also be necessary to configure the tool to filter out particular types of warning messages that are not relevant. The use of static analysis tools on existing or amended code is likely to result in lots of messages concerning programming standards. A way of dealing with this situation should be considered during the selection and implementation process. For instance, it may be agreed that small changes to existing code should not use the static analysis tool, whereas medium to large changes to existing code should use the static analysis tool. A rewrite should be considered if the existing code is significantly non-compliant.

Static analysis tools can find defects that are hard to find during dynamic testing. They can also be used to enforce programming standards (including secure coding), to improve the understanding of the code and to calculate complexity and other metrics (see Chapter 3).

Some static analysis tools are integrated with dynamic analysis tools and coverage measurement tools. They are usually language specific, so to test code written in C++ you need to have a version of a static analysis tool that is specific to C++.

Other static analysis tools come as part of programming languages or only work with particular development platforms. Note that debugging tools and compilers provide some basic functions of a static analysis tool, but they are generally not considered to be test tools and are excluded from the ISTQB syllabus.

The types of defect that can be found using a static analysis tool can include:

Tool support for test design and implementation

Test design tools Test design tools are used to support the generation and creation of test cases. In order for the tool to generate test cases, a test basis needs to be input and maintained. Therefore, many test design tools are integrated with other tools that already contain details of the test basis such as:

The level of automation can vary and depends on the characteristics of the tool itself and the way in which the test basis is recorded in the tool. For example, some tools allow specifications or requirements to be specified in a formal language. This can allow test cases with inputs and expected results to be generated. Other test design tools allow a GUI model of the test basis to be created and then allow tests to be generated from this model.

Some tools (sometimes known as test frames) merely generate a partly filled template from the requirement specification held in narrative form. The tester will then need to add to the template and copy and edit as necessary to create the test cases required.

Tests designed from database, object or state models held in modelling tools can be used to verify that the model has been built correctly and can be used to derive some test cases. Tests derived can be very thorough and give high levels of coverage in certain areas.

Some static analysis tools integrate with tools that generate test cases from an analysis of the code. These can include test input values and expected results.

A test oracle [Not in Syllabus] is a type of test design tool that automatically generates expected results. However, these are rarely available because they perform the same function as the software under test. Test oracles tend to be most useful for:

However, test design tools should be only part of the approach to test design. They need to be supplemented by other test cases designed with the use of other techniques and the application of risk.

Test design tools could be used by the test organisation in the scenario but the overhead to input the necessary data from the test basis may be too great to give any real overall benefit. However, if the test design tool can import requirements or other aspects of the test basis easily, then it may become worthwhile.

Test design tools tend to be more useful for safety-critical and other high-risk software where coverage levels are higher and industry, defence or government standards need to be adhered to. Commercial software applications, like the hotel system, do not usually require such high standards and therefore test design tools are of less benefit in such cases.

Model-based testing tools (also known as modelling tools) Model-based testing tools are used primarily by developers during the analysis and design stages of the development life cycle. The reason modelling tools are included here is because they are very cost-effective at finding defects early in the development life cycle.

Their benefits are similar to those obtained from the use of reviews and inspections, in that modelling tools allow omissions and inconsistencies to be identified and fixed early so that detailed design and programming can begin from a consistent and robust model. This in turn prevents fault multiplication that can occur if developers build from the wrong model.

For instance, a visual model-based testing tool using Unified Modeling Language (UML) can be used by designers to build a model of the software specification. The tool can map business processes to the system architecture model, which, in turn, enables programmers and testers to have a better and common understanding of what programs should do and what testing is required.

Similarly, the use of database, state or object models can help to identify what testing is required and can assist in checking whether tests cover all necessary transactions. Integration with test design tools may also enable modelling tools to support the generation of test cases.

Test data preparation tools Test data preparation tools are used by testers and developers to manipulate data so that the environment is in the appropriate state for the test to be run. This can involve making changes to the field values in databases, data files and so on, and populating files with a spread of data (including depersonalised dates of birth, names and addresses, etc. to support data anonymity).

TDD, ATDD and BDD tools As technology has evolved and processes have improved, new test tools have been developed to support TDD and BDD approaches (outlined in syllabus sections 1.4.2 and 2.2.1). Readers should be aware that tools exist to support these processes (although many readers may not have used these processes or such tools because they are configured and used primarily by developers during component testing).

TDD, ATDD and BDD tools can be further integrated with other types of test tools to design, implement and execute test cases. This integration enables code changes to be compiled quickly, accurately and regularly, and the test suite executed automatically.

TDD tools have been built to support the test-driven development process and these tools enable developers to design test cases and test procedures at component level. These tools usually integrate with or provide interfaces to types of test execution tools.

ATDD is an extension of TDD and, consequently, ATDD tools are an extended version of TDD tools.

BDD tools are also an extension of TDD tools and provide an interface to allow acceptance testers to define user stories or test cases in their own business language (Domain-Specific Language – DSL). The BDD is then configured by the developer to interpret the user story and produce automatically executable test procedures (scripts).

Developers can configure ATDD and BDD tools to enable users to define test cases in a structured format/template that enables such test cases to be captured by ATDD and BDD tools.

Figure 6.4 Testing of daily builds using a set of test tools

ATDD and/or BDD templates can be used by acceptance testers to define test cases in a structured version of their own hotel business language. These structured test cases can then be automated and run by test execution tools/unit test framework, which enables the automated part of acceptance testing to be carried out more efficiently.

Many of the tools discussed in the section above (Tool support for test design and implementation) are also used for test execution (or can interface with test execution tools).

Tool support for test execution and logging

Test execution tools cover a wide range from basic test comparators to ATDD tools (see above) that convert acceptance test cases into executable scripts and then report upon whether they have passed or failed.

Test comparators [Not in Syllabus] Test comparators compare the contents of files, databases, XML messages, objects and other electronic data formats. This allows expected results and actual results to be compared. They can also highlight differences and thus provide assistance to developers when localising and debugging code.

They often have functions that allow specified sections of the file, screen or object to be ignored or masked out. This means that a date or time stamp on a screen or field can be masked out because it is expected to be different when a comparison is performed.

Table 6.2 shows an extract from the transaction table in the hotel chain database for data created on 20/10/2018.

Table 6.2 Hotel system extract (20/10/2018)

A regression test was run on 5/11/2018. Table 6.3 shows an extract from the transaction table for this data.

Table 6.3 Hotel system extract (5/11/2018)

The Transaction ID and Trans_Date fields contain different values. But we know why this is the case and we would expect them to be different. Therefore, we can mask out these values. Note that some automated test comparators use test oracles while others provide functions to add on values to take into account known differences (e.g. 15 days later) so that the actual results and expected results can be compared.

Comparators are particularly useful for regression testing since the contents of output or interface files should usually be the same. This is probably the test tool that provides the single greatest benefit. For instance, manually comparing the contents of a database query containing thousands of rows is time-consuming, error prone and demotivating. The same task can be performed accurately and in a fraction of the time using a comparator. Comparators are usually included in test execution tools, which is why they are no longer specified in the syllabus.

Test execution tools Test execution tools allow test scripts to be run automatically (or at least semi-automatically). A test script (written in a programming language or scripting language) is used to navigate through the system under test and to compare predefined expected outcomes with actual outcomes. The results of the test run are written to a test log. Test scripts can then be amended and reused to run other or additional scenarios through the same system. Some tools offer GUI-based utilities that enable amendments to be made to scripts more easily than by changing code. These utilities may include:

Record (or capture playback) tools Record (or capture playback) tools can be used to record a test script and then play it back exactly as it was executed. However, a test script usually fails when played back owing to unexpected results or unrecognised objects. This may sound surprising, but consider entering a new customer record onto a system:

Also note that expected results are not necessarily captured when recording user actions and therefore may not be compared during playback.

The recording of tests can be useful during exploratory testing for reproducing a defect or for documenting how to execute a test. In addition, such tools can be used to capture user actions so that the navigation through a system can be recorded. In both cases, the script can then be made more robust by a technical expert so that it handles valid system behaviours depending on the inputs and the state of the system under test.

Data-driven testing Robust test scripts that deal with various inputs can be converted into data-driven tests. This is where hard-coded inputs in the test script are replaced with variables that point to data in a data-table. Data-tables are usually spreadsheets with one test case per row, with each row containing test inputs and expected results. The test script reads the appropriate data value from the data-table and inserts it at the appropriate point in the script (e.g. the value in the Customer Name column is inserted into the Customer Name field on the input screen).

Keyword-driven testing A further enhancement to data-driven testing is the use of keyword-driven (or action word) testing. Keywords are included as extra columns in the data-table. The script reads the keyword and takes the appropriate actions and subsequent path through the system under test. Conditional programming constructs such as IF ELSE statements or SELECT CASE statements are required in the test script for keyword-driven testing.

Technical skills Programming skills and programming standards are required to use the tool effectively. It may be that these can be provided by a small team of technical experts within the test organisation or from an external company. In data-driven, and particularly keyword-driven, approaches, the bulk of the work can be done by manual testers, with no knowledge of the scripting language, defining their test cases and test data and then running their tests and raising defects as required. However, this relies on robust and well-written test scripts that are easy to maintain. This takes much time and effort before any sort of payback is achieved.

Maintenance It is essential that time (and consequently budget) is allowed for test scripts to be maintained. Any change to a system can mean that the test scripts need to be updated. Therefore, the introduction of a new type of object or control could result in a mismatch being found between the previous object type and the new one. The relevant level of technical skills and knowledge is also required to do this.

Effective and efficient use The efficiency and effectiveness benefits that come from the use of a test execution tool take a long time to come to fruition. First, the selection and implementation process needs to be planned and conducted effectively (a generic process for selecting and implementing any type of test tool is detailed later in this chapter). However, there are certain issues that are particularly relevant to test execution tools and these are described below.

The long-term benefits of test execution tools include:

Note that benefits come primarily from running the same or very similar tests a number of times on a stable platform. Therefore, they are generally most useful for regression testing.

The graph in Figure 6.5 shows how the benefits of using test execution tools take some time to pay back. Note how in the early stages the cost of using automated regression testing is greater than the cost of manual regression testing. This is due to the initial investment, implementation, training, initial development of automated scripts and so on. However, the cost each additional time the test is run is less for automated regression testing than it is for manual regression testing. Therefore, the lines on the graph converge and at a point in time (known as the break-even point) the lines cross. This is the point at which the total cost to date for automated testing is less than the total cost to date for manual regression testing.

This is clearly a simplistic view, but it demonstrates how an initial investment in test execution tools can be of financial benefit in the medium to long term. There are other less tangible benefits as well. However, to get this financial benefit you will need to be sure that there is a requirement to run the same (or very similar) regression tests on many occasions.

Test harnesses and unit test frameworks A test harness is a test environment comprised of stubs and drivers needed to execute a test. It is used primarily by developers to simulate a small section of the environment in which the software will operate. Test harnesses are usually written in-house by developers to perform component or integration testing for a specific purpose. Test harnesses often use ‘mock objects’ known as ‘stubs’ (which stub out the need to have other components or systems by returning predefined values) and ‘drivers’ (which replace the calling component or system and drive transactions, messages and commands through the software under test).

Figure 6.5 Test execution tools payback model

Test harnesses can be used to test various systems or objects ranging from a middleware system (as in Figure 6.6) to a single or small group of components. They are frequently used in Agile development so that existing tests can be rerun as regression tests to establish whether existing functionality is adversely impacted by the changes made.

A unit test framework is generally more robust and reusable than a standalone test harness and is typically able to support multiple test harnesses for related purposes. It may also provide additional support for the developer, such as debugging capabilities.

There are similarities with the principle behind data-driven testing using test execution tools because the harness allows many different test cases to be designed and run without the time-consuming process of keying them manually. This raises the question of how much benefit can be obtained from using a test execution tool when a test harness can be used instead. As usual, it depends on the circumstances, the risk, the purpose and the level of testing being performed.

Coverage tools Coverage tools measure the percentage of the code structure covered across white-box measurement techniques such as statement coverage and branch or decision coverage. In addition, they can also be used to measure coverage of modules and function calls. Coverage tools are often integrated with static and dynamic analysis tools and are primarily used by developers.

Coverage tools can measure code written in several programming languages, but not all tools can measure code written in all languages. They are useful for reporting coverage measurement and can therefore be used to assess test completion criteria and/or exit criteria.

Coverage measurement of requirements and test cases/scripts run can usually be obtained from test management tools. This function is unlikely to be provided by coverage tools.

Coverage measurement can be carried out using intrusive or non-intrusive methods. Non-intrusive methods typically involve reviewing code and running code. Intrusive methods, such as ‘instrumenting the code’ involve adding extra statements into the code. The code is then executed and the extra statements write back to a log in order to identify which statements and branches have been executed.

Instrumentation code can then be removed before it goes into production.

Intrusive methods can affect the accuracy of a test because, for example, slightly more processing will be required to cope with the additional code. This is known as the probe effect and testers need to be aware of its consequences and try to keep its impact to a minimum.

Tool support for performance measurement and dynamic analysis

Dynamic analysis tools Dynamic analysis tools are used to detect the type of defects that are difficult to find during static testing. They are typically used by developers, during component testing and component integration testing, to:

They are generally used for safety-critical and other high-risk software where reliability is critical.

Dynamic analysis tools are often integrated with static analysis and coverage tools. For example, a developer may want to perform static analysis on code to localise defects so that they can be removed before component test execution. The integrated tool may allow:

Dynamic analysis tools are usually language specific, so to test code written in C++ you need to have a version of a dynamic analysis tool that is specific to C++.

The tool could then:

Performance testing tools Performance testing is very difficult to do accurately and in a repeatable way without using test tools. Therefore, performance testing tools have been developed to carry out both load testing and stress testing.

Load testing reports on the performance of a system under test, under various loads and usage patterns. A load generator (which is a type of test driver) can be used to simulate the load and required usage pattern by creating virtual users that execute predefined scripts across one or more test machines. Alternatively, response times or transaction times can be measured under various levels of usage by running automated repetitive test scripts via the user interface of the system under test. In both cases output will be written to a log. Reports or graphs can be generated from the contents of the log to monitor the level of performance.

Performance testing tools can also be used for stress testing. In this case, the load on the system under test is increased gradually (ramped up) in order to identify the usage pattern or load at which the system under test fails. For example, if an air traffic control system supports 200 concurrent aircraft in the defined air space, the entry of the 201st or 205th aircraft should not cause the whole system to fail.

Performance testing tools can be used against whole systems, but they can also be used during system integration test to test an integrated group of systems, one or more servers, one or more databases or a whole environment.

If the risk analysis finds that the likelihood of performance degradation is low, then it is likely that no performance testing will be carried out. For instance, a small enhancement to an existing mainframe system does not necessarily require any formal performance testing. Normal manual testing may be considered sufficient (during which poor performance might be noticed).

There are similarities between performance testing tools and test execution tools in that they both use test scripts and data-driven testing. They can both be left to run unattended overnight and both need a heavy upfront investment, which will take some period of time to pay back.

Performance testing tools can find defects such as:

The cost of some performance tools is high, and the implementation and training costs are also high. In addition, finding experts in performance testing is not that easy. Therefore, it is worth considering using specialist consultancies to come in and carry out performance testing using such tools.

Monitoring tools Monitoring tools are used to check whether whole systems or specific system resources are available and whether their performance is acceptable. Such tools are mainly used in live rather than test environments and are therefore not really testing tools. They tend to be used for monitoring ecommerce, ebusiness or websites, as such systems are more likely to be affected by factors external to the organisation and the consequences can be severe in terms of business lost and bad publicity. Generally, if a website is not available, customers will not report it but will go elsewhere. For example, it was reported in 2003 that a well-known online retailer would lose sales of $660,000 per hour if it were offline during the US trading day. The advent of websites that continually check the status of popular websites means that such information is easily available to the general public.

The use of monitoring tools is generally less important for internal systems because failure is more likely to be noticed only within the organisation and contingency plans may also exist. However, the availability of monitoring tools on mainframes, servers and other forms of hardware means that it is relatively easy to monitor the majority of an organisation’s infrastructure.

Tool support for specialised testing needs

In addition to tools that support the general test process, there are many other tools that support more specific testing issues.

Data quality assessment tools Data quality assessment tools allow files and databases to be compared against a format that is specified in advance. They are typically used on large-scale, data-intensive projects such as:

Data quality assessment tools are not specifically designed for testing purposes. They are used primarily for the migration of production data, but typically the development and testing of a migration project will also use these tools.

Data conversion and migration tools Data conversion tools are used to map data from the data structures in the source system into the data structures required for the receiving system.

Security testing tools Security testing tools are used to test the functions that detect security threats and to evaluate the security characteristics of software. The security testing tool is typically used to assess the ability of the software under test to:

Security testing tools are mainly used to test ecommerce, ebusiness and websites. For example, a third-party security application such as a firewall may be integrated into a web-based application.

The skills required to develop and use security testing tools are very specialised and such tools are generally developed and used on a particular technology platform for a particular purpose. Therefore, it may be worth considering the use of specialist consultancies to perform such testing. For example, specialist consultancies are often engaged to carry out penetration testing. This type of testing is to establish whether malicious attackers can penetrate the organisation’s firewall and hack into its systems.

Security testing tools need to be constantly updated because there are problems solved and new vulnerabilities discovered all the time – consider the number of Windows security releases to see the scale of security problems.

Usability testing tools (including accessibility and localisation test tools) Usability test tools typically record the mouse clicks made by remote usability testers when carrying out a specified task. Some tools also enable other data to be captured such as screenshots, written comments and voice recordings of verbal comments. This recorded data is generally stored in a database so that it can then be analysed easily by staff at the organisation commissioning the testing.

Note that the usability testing tool market is changing very quickly, and new types of usability tools may appear over the next few years. Recent developments include:

Portability testing tools Portability is defined as the ease with which the software product can be transferred from one hardware or software environment to another. Portability test tools are generally used by Portability testing specialists.

Other tools that are not designed specifically for testers or developers can also be used to support one or more test activities. These include:

For example, in the absence of test management or defect management tools, defects could be recorded on word processors and could be tracked and maintained on spreadsheets. Tests passed or failed could also be recorded on spreadsheets.

Summary of test tools

Table 6.5 summarises the types of test tools discussed above. It includes the definition given in the ISTQB Glossary of Testing Terms v3.2 and gives a guide to:

INTRODUCING A TOOL INTO AN ORGANISATION

There are many stages in the process that should be considered before implementing a test tool.

Analyse the problem/opportunity

An assessment should be made of the maturity of the test process used within the organisation. If the organisation’s test processes are immature and ineffective then the most that the tool can do is to make the repetition of these processes quicker and more accurate – quick and accurate ineffective processes are still ineffective.

It is therefore important to identify the strengths, weaknesses and opportunities that exist within the test organisation before introducing test tools. Tools should only be implemented that will either support an established test process or support required improvements to an immature test process. It may be beneficial to carry out a TPI (Test Process Improvement) or CMMI (Capability Maturity Model Integration) assessment to establish the maturity of the organisation before considering the implementation of any test tool.

Table 6.5 Types of test tool

Generate alternative solutions

It may be more appropriate and cost-effective to do something different. In some organisations, performance testing, which may only need to be done from time to time, could be outsourced to a specialist testing consultancy. Training or recruiting better staff could provide more benefits than implementing a test tool and improve the effectiveness of a test process more significantly. In addition, it is more effective to maintain a manual regression pack so that it accurately reflects the high-risk areas than to automate an outdated regression pack (that is no longer relevant) using a test execution tool.

An early investigation of what tools are available is likely to form part of this activity.

Constraints and requirements

A thorough analysis of the constraints and requirements of the tool should be performed. Interested parties should attend workshops and/or be interviewed so that a formal description of the requirements can be produced and approved by the budget holder and other key stakeholders.

A failure to specify accurate requirements (as with a failure to specify accurate requirements for a piece of software) can lead to delays, additional costs and the wrong things being delivered. This could lead to a review tool being implemented that does not allow access across the internet, even though there is a need for staff from many countries to participate in reviews. Any financial or technical constraints (e.g. compatibility with particular operating systems or databases) should also be considered.

It is useful to attach some sort of priority or ranking to each requirement or group of requirements.

Training, coaching and mentoring requirements should also be identified. For example, experienced consultants could be used for a few weeks or months to work on overcoming implementation problems with the tool and to help transfer knowledge to permanent staff. Such consultants could be provided by the vendor or could be from the contract market.

Requirements for the tool vendor should also be considered. These could include the quality of training and support offered by the vendor during and after implementation and the ability to enhance and upgrade the tool in the future. In addition, their financial stability should be considered because the vendor could go bankrupt or sell to another vendor. Therefore, using a small niche vendor may be a higher risk than using an established tool supplier.

If non-commercial tools (such as open source and freeware) are being considered then there are likely to be risks around the lack of training and support available. In addition, the ability or desire of the service support supplier (or open-source provider) to continue to develop and support the tool should be taken into account.

Evaluation and shortlist

The tools available in the marketplace should be evaluated to identify a shortlist of the tools that provide the best fit to the requirements and constraints. This may involve:

It may also be useful for the test organisation to send a copy of its list of requirements and constraints to tool vendors so that:

The outcome of this initial evaluation should result in a shortlist of perhaps one, two or three tools that appear to meet the requirements.

Detailed evaluation/proof of concept

A more detailed evaluation (proof of concept) should then be performed against this shortlist. This should be held at the test organisation’s premises in the test environment in which the tool will be used. This test environment should use the system under test and other software, operating systems and hardware with which the tool will be used. There are several reasons why there is little benefit from evaluating the tool on something different. For example:

In some cases, it may be worth considering whether changes can be made to the organisation’s test environments and infrastructure, but the costs and risks need to be understood and quantified.

(Note that if there is only one tool in the shortlist then it may be appropriate to combine the proof of concept and the pilot project.)

After each proof of concept the performance of the tool should be assessed in relation to each predefined requirement. Any additional features demonstrated should be considered and noted as potential future requirements.

Once all proofs of concept have been carried out it may be necessary to amend the requirements as a result of what was found during the tool selection process. Any amendments should be agreed with stakeholders. Each tool should then be assessed against the finalised set of requirements.

There are three likely outcomes at this stage:

Negotiations with vendor of selected tool

Once a tool has been selected discussions will be held with the vendor to establish and negotiate the amount of money to be paid and the timing of payments. This will include some or all of the following:

Discussions should establish the amount to be paid, first, for a pilot project and, secondly (assuming the pilot project is successful), the price to be paid for a larger scale implementation.

The pilot project

The aims of a pilot project include the following:

The pilot project should report back to the group of stakeholders that determined the requirements of the tool.

If a decision is made to implement the tool on a larger scale then a formal project should be created and managed according to established project management principles.

Key factors in successful implementations of test tools

There are certain factors or characteristics that many successful tool implementation projects have in common:

Summary of test tool implementation process

The diagram in Figure 6.7 outlines the process for selecting and implementing a test tool in an organisation. This shows that there are several points at which a decision could be made not to introduce a tool. It also demonstrates that the activities during the evaluation and negotiation stages can follow an iterative process until a decision is made.

SUMMARY

We have seen that the main benefits of using test tools are generally the same as the benefits of automating a process in any industry. These are: time saved and predictable and consistent results.

However, we have also seen that there can be considerable costs in terms of both time and money associated with obtaining such benefits. The point at which the use of tools becomes economically viable depends on the amount of reuse, which is often difficult to predict.

Other risks include over-optimistic expectations of:

We have seen that there are many types of test tools and that they provide support to a variety of activities within the test process. We have also seen that tools are used by a variety of staff in the software development process and that some are of greater benefit to developers than testers.

We have looked at the different scripting techniques that can be used with test execution tools. This ranges from the simple record–playback to data-driven and keyword-driven scripts.

We identified a process for selecting and introducing a test tool into an organisation. This involves understanding the interactions between activities within the process and examining the purposes of a proof of concept and of a pilot project. We also examined the problems likely to be encountered when implementing a tool and looked at actions that can be taken in an attempt to overcome or avoid such problems.

Figure 6.7 Test tool implementation process

We also noted that a decision not to introduce a tool could well be a valid decision at several stages within the process.

Example examination questions with answers

E1. K1 question

A project requires test tools to support both requirements management and usability testing. Which two of the following classes of test tool are appropriate to select tools from?

E2. K2 question

Which of the following correctly identifies a benefit of test automation?

E3. K1 question

Which of the following is a special consideration for test execution tools?

E4. K2 question

Which of the following is not an important principle for test tool selection?

E5. K1 question

Which of the following is a typical objective for a pilot project for introduction of a test tool into an organisation?

E6. K1 question

Which of the following is a typical positive success factor for implementation of a test tool within an organisation?

Answers to questions in the chapter

SA1. The correct answer is d.

SA2. The correct answer is c.

SA3. The correct answer is a.

Answers to example examination questions

E1. The correct answer is a.

The correct classes of tool are tool support for management of testing and testware (requirements management tool) and tool support for specialised testing needs (usability testing). The only answer that identifies this combination is a.

E2. The correct answer is b.

Option a is incorrect; neglect of version control is listed in section 6.1.2 as a potential risk. Option c is incorrect; vendors going out of business or discontinuing support of a tool are potential risks. Option d is incorrect; regression testing may be easier and quicker to do but will still be required. Option b is specifically listed as a potential benefit of test automation and is the correct answer.

E3. The correct answer is a.

Option a is correct. At least one member of a testing team will need to be able to develop test scripts using a scripting language, so expertise by some of the test team is required – but this does not mean every tester needs to be able to read/write in any scripting language(s). Option b is incorrect; this is a special consideration for test management tools but not for test execution tools. Option c is incorrect; although capturing manual tests is not the optimum approach to automated test execution, it can be done. Option d is incorrect; at least one tester will need to be trained to use the tools, but not necessarily the entire team.

E4. The correct answer is d.

Options a, b and c are all key principles listed in the syllabus, section 6.2.1. Option d contradicts option c and is generally considered a dangerous approach that could significantly impair the benefits of using a tool and incur significant costs.

E5. The correct answer is b.

Option a is incorrect; this needs to be done before selecting the tool. Option c is incorrect; changes may be needed to processes and practices, but changes to a tool are expensive and risky. Option d is incorrect because the anticipated benefits would have been determined earlier to support the selection process; at this stage the aim is to determine if the expected benefits can be achieved. The correct option is b, which is listed in section 6.2.2 of the syllabus.

E6. The correct answer is d.

Option a is incorrect; section 6.2.3 of the syllabus suggests that tools should be rolled out incrementally. Option B is incorrect because changes to a tool are potentially expensive and risky; any changes to accommodate the tool should be made to processes and practices. Option c is unlikely to be successful because no tool is likely to be successful in all cases or in all types of project; the syllabus suggests defining guidelines for the use of the tool so that users can best decide where it will add most value. Option d is correct and is listed in the syllabus, section 6.2.3.