The File Comparer – Reports.xlsx workbook contains a sheet called Summary. This sheet holds three test suites designed for the File Comparer application, where their titles describe the underlying purpose. The tests suites are generated using a chosen set of tests previously created, and they constitute our prospective test lists for this chapter.
To create a test list inside Test Studio, go to the Test Lists tab. As shown in the following screenshot, this tab has two read-only panes in which it displays information about the test lists present in the project. The left pane has the created test lists while the right one shows the test entries associated with each of these lists:
The Test lists tab
Test Studio offers two sorts of test lists accessible from the Add tab as shown in the following screenshot. Lists are static test suites that can have their definition changed only through direct manual update. Therefore, if you create a manual list with two tests, it will always have these tests unless it is directly edited. Dynamic lists, on the other hand, are smart, self-updating test suites. They offer greater flexibility and less maintenance cost. The following two sections further discuss the nature of the two lists and demonstrate how to create and manipulate them.
The Test list types
As briefly mentioned before, regular lists are not automatically updated. They are mainly defined by a set of tests manually assigned upon creation, which can be changed in the future. We will map the smoke testing suite showing in the Summary sheet of the File Comparer – Reports.xlsx workbook to a static list by first clicking on the List button of the Add tab. The Create a New Test List window shown in the following screenshot is invoked:
Creating lists
Three types of information need to be supplied in order to complete the creation of the list. Firstly, it receives the title for the test suite in its free input Name field, which is Smoke Testing in our case. Secondly, it allows the specification of the test type that can be any of the following values:
Since the smoke testing suite contains automated tests, choose Automated from the Select Test List Type combobox. Thirdly, it contains a section that allows the addition of the tests that will contribute to the definition of this static list. The project hierarchy is entirely displayed in the left pane called Tests. However, notice how only the tests based on the WPF template are enabled and can therefore be moved from left to right. This is in fact not a problem with the test selection feature but, as discussed before, it is a restriction on the type of tests that can be added under a list with an automated type. Had there been web tests in the project, those tests would also appear in the Tests pane. As identified in the Summary sheet, select the following four tests, add them to the right pane, and then click on OK:
Func-1_FileCompare_Equal_SuccessfulFunc-2_FileCompare_DefaultComparison_SuccessfulFunc-3_FileCompare_MapFolder_SuccessfulFunc-4_History_SingleDate_Successful
Notice how an entry for the list is added to the Test Lists pane with the corresponding tests shown under the Tests pane.
Dynamic lists offer flexibility because they are smart enough to update their own definitions each time they are invoked for execution. Instead of accepting a predefined set of tests, dynamic lists are rather defined using a combination of criteria. The evaluation of these criteria will result in the actual tests that are going to run under the scope of the list. The gained advantage is that knowing how automation and maintainability should be an inseparable pair, and keeping the test lists up-to-date, no longer requires maintenance effort upon the creation of a new test. The newly added test will be automatically appended to the suitable dynamic list whenever its attributes pass the verification. There are many criteria based on which we can filter the tests as we will see shortly.
The Summary sheet contains a test suite called File Info Upload Feature. This suite is supposed to hold the tests responsible for verifying the functionality of the File Info feature. As you may have noticed, File Info Upload suggests changeability in its constituent tests based on the following levels:
- Feature test case generation: The process of generating functional tests for a feature expands in time to some extent. This process contains generation of test cases based on the various test techniques. Hence, tests will keep on emerging until the entire generation activity is completed. Accordingly, each created test needs to be added to the appropriate feature test list.
- Bug identification: The generated test cases for a feature will most likely uncover only a part of the residing bugs. As other bugs are revealed during the application's lifetime, matching tests need to be added to the automated project and therefore to the corresponding feature test list.
- Feature change: As changes are introduced to the feature, new test cases are going to emerge during the regression testing cycles and those need to be added to the feature test list as well.
- Pesticide paradox: This testing principle states that as a test suite is run over and over again, its capacity in finding new bugs will gradually decrease. This is due to the fact that bugs that were already uncovered got fixed by the developers. In order to avoid this situation, test cases are regularly reviewed and could perhaps have their inputs altered. Additionally, new tests can be added, which also need to be appended to the feature test lists.
The preceding four reasons apply to the File Info Upload feature and incur maintenance cost that is alleviated by dynamic lists. This test suite is therefore recommended to be created as a dynamic list.
The criterion we are going to use in order to identify the characterizing tests for this dynamic test list is the
CustomProperty1 variable, which we have seen in the previous chapter. Remember that we conventionally mapped this property to the feature under test. Since the importance of these custom properties is further highlighted in this section, this conventional mapping can now be more particularly adapted to each testing environment. Let's edit the value of CustomProperty1 of the following tests to File Info:
Func-6_FileInfo_Create_SuccessfulFunc-8_FileInfo_Export_Successful
Now that the conditions are satisfied, go back to the Test Lists tab and click on the Dynamic List button of the Add tab. The Add Dynamic Test List window shown in the following screenshot is invoked:
Creating dynamic test lists
As in the regular lists case, we firstly need to fill in the dynamic list title. So, enter File Info Upload inside the Test List Name field.
The second section contains the options for crafting the rules based on which the tests will be filtered. Initially, the Results section displays all the tests present in the project since no rules are added yet. To retrieve only the preceding listed tests, add the rule shown in the following screenshot and then click on the plus button:
Adding dynamic lists rules
More complex rules can be constructed by adding further criteria that will be joined with an AND operator. The additional criteria can be a combination of one or all the following:
CustomProperty1,CustomProperty2,CustomProperty3: They are found and are editable in the Misc section of the test's Properties paneName: They are found in the Attribute section of the test's Properties pane and are editable in the Project Files paneOwner: They are found and are editable in the Attribute section of the test's Properties panePath: They are found in the Attribute section of the test's Properties pane and are editable in the Project Files panePriority: They are found in the Attribute section of the test's Properties pane and are editable in the Attribute section of the test's Properties pane
Click on OK to finish the creation of the dynamic list.
Lastly, we will create a test list of manual type for the security test suite. Click on the List button and create a list similar to the one depicted in the following screenshot:
Creating a security test list