The Data-Driven Approach
Kit defines three generations of automated software testing (1):
The first generation is basically GUI testing. The use of capture/playback tools to develop automated test scripts is limited to recording user actions at the level of the GUI, editing the resulting test scripts, and replaying the edited test scripts. The resulting test scripts are unstructured, undocumented, and not maintainable.
In the second generation, the scriptwriters have developed the ability to “build well-structured, documented, robust, maintainable tests.” The testing project becomes an engineering project at this level; test scripts include error capture and recovery logic, and a key characteristic is reusability of test script components.
Kit characterizes the third generation of test automation as being in control of the test resources. At that level, test design and test automation are seen as separate activities.
The key to third-generation capture/playback use is in the nature and quantity of the test scripts and, in particular, the relationship between test scripts and test cases. First- and second-generation capture/playback users typically have one main script for each test case. This results in many scripts, all of which must be maintained. Third-generation users have discovered a way to use scripting in a different, novel way—a way that lets a single script process every test case.
What is the data-driven framework and where does it sit with the generations of capture/playback tool use? There are several opinions as to what constitutes data-driven testing. In fact, a heated and lengthy discussion appeared in the SQA users group (www.dundee.net/sqa) a couple of years ago. See Appendix B for the complete text of the discussion as it was captured by Nagle of SAS Institute, synthesized, and posted on the Web site (groups.yahoo.com/group/RobotDDEUsers) devoted to data-driven testing and the DDE approach. This approach is implemented in the SQABasic language that is the native language used in Rational Software's Robot product. For Mercury Interactive customers who use WinRunner, Keith Zambelich of Automated Testing Specialists (www.auto-sqa.com) has developed a toolkit for WinRunner that supports a data-driven testing framework implementation very similar to Nagle's DDE. These approaches, as well as the CSDDT framework, are discussed below.
So, what is data-driven testing? It is a commonly used term these days, but most think of data-driven testing as simply using external data as the source of input into the AUT; for example, cases in which data pools are used. Here is our definition:
Data-driven testing is testing where data contained in an input test data file control the flow and actions performed by the automated test script.
An input test data record is read from an external file or spreadsheet and is developed independently from the test script program. This idea has been further enhanced and modified by the Archer Group and others. The methodology introduced here also employs the use of control data to determine what actions are taken and the order in which they occur.
Data-driven testing is the use of archived test data, usually in the form of simple CSV text files, to drive the automated testing process. Data-driven testing can be expanded to include control data as well as test data. The test data test both GUI and server-level data validation rules that represent an application's functionality. The control data drive the test script by directing it to the appropriate location in the application to execute the test and by indicating what type of test or action to execute.
The characteristics of data-driven test scripts include the following:
They use simple input text files.
They are highly maintainable.
They are easier for nonprogrammers to use.
They document the tests that are being executed.
They allow dynamic data input via placeholders.
They use input data to control the test execution.
The content of data-driven tests can include (18, 26):
Parameters that you can input to the program
Sequences of operations or commands that you use to make the program execute
Sequences of test data through which the program is driven
Placeholders that trigger the test script to create a dynamic data value at runtime
Documents that you have the program read and process
The test data should be designed using the techniques described in Chapter 3 in the section discussing Black Box (requirements-based) and White Box (code-based) approaches.
The advantages of data-driven test scripts are
It is not necessary to modify the test script when adding additional test data because the test data is appended to the existing text file.
It is easy to modify the data records.
Multiple test data records can be developed as functional variants.
Multiple input data files can be created and used when required.
One thing we can be sure of: Data-driven testing fits Kit's definition of third-generation capture/playback test tool use. It satisfies all of his criteria.