This chapter explains the details of creating and using the ordered and generic tests. Previous chapters explained the creation of many tests including unit tests, manual tests, web performance tests, coded web performance tests, and load tests. Visual Studio provides the additional feature of grouping and ordering some or all of these tests and then execute them in order; this is called an ordered test. The main advantage of creating ordered tests is that it enables us to execute tests in an order based on the dependencies. For example, a web test may be dependent on the results produced by executing the unit test. So the unit test needs to be executed before executing the web performance test.
Generic tests are just like any other tests except that they are used for running the existing third-party tool or program that can also be run using the command line. Let us create the sample tests in this chapter and see the usage of both generic and ordered tests. This chapter covers the following sections:
- Creating an ordered test
- Executing an ordered test
- Creating a generic test
- Results file for a generic test
The following screenshot shows the list of all the tests that are created under the Test Project. The tests are independent and there is no link between the tests, but the output of one test may be required for the other test to start.
There are different types of tests, such as unit test and web performance test, under the Test Project. Creating an ordered test and placing some of the dependent tests in an order will help the test execution to happen in that order, without breaking any dependencies.
To create an ordered test, follow these steps:
- Select the Test Project from Solution Explorer.
- Right-click and select Add Ordered Test.
- Select an ordered test from the types of tests listed.
- Save the ordered test by choosing the Save option within the File menu.
The ordered test will get created under the Test Project and the ordered test window is shown, where we will select from the existing tests and put them in order. The following screenshot shows different options for ordering the tests:
The first line is the status bar, which shows the number of tests selected for the ordered test.
The Select project to view tests drop-down list has the option to choose any particular Test Project, to display tests within it. This drop-down list has the default value /All Loaded Tests, which displays all available tests under all projects.
The Available tests list displays all the tests from the selected Test Project in the Projects drop-down list.
The Selected tests list contains the tests that are selected from the available tests list, to be placed in order.
The two right and left arrows are used for selecting and unselecting the tests from the Available tests list to the Selected tests list. Multiple tests can be selected by pressing the Ctrl key and selecting the tests.
The up-down arrows on the right of the Selected Tests list are used for moving the test up or down and setting the order for the testing.
The last option, the Continue after failure checkbox at the bottom of the window, is to override the default behavior of the ordered tests, which is to abort the execution on failure of any test. If the option Continue after failure is unchecked and if any test in the order fails, then all remaining tests will get aborted.
An ordered test can be run like any other test. Open the Test Explorer window and select the ordered test from the list, then right-click and choose the Run Selected Test option. Once the option is selected, the test execution starts, working through the tests in the order in which they are placed. After the execution completes, the Test Explorer window shows the status of the ordered tests. If any of the individual tests in the list fail, the ordered test status would be Failed. The summary of Test Run statuses for all tests in the ordered test is shown in following screenshot. The sample application had three tests within the ordered test, but two of them failed and one has passed. Overall, the ordered test's status is failed because of two test failures.
The Test Explorer window also provides detailed information about the tests run. Select a test that is being run in the explorer to see it. This information includes the link to the source code where the test fails, followed by test name and the message from the test method for failure, the elapsed time for the test, and stack trace for the test. The total duration of the ordered test is also shown next to the ordered test failure status under the failed tests section.