Generic tests are a way to integrate external tests into Visual Studio. There could be applications that use external components or services, which need to be tested as part of the whole system testing. In this case, the external component details are not exposed and the internal logic is also unknown. In order to test these third-party components, generic tests in Visual Studio act as wrappers for testing these external components within the boundary of Visual Studio. Once it is wrapped, the generic tests run just like any other test, through Visual Studio IDE.
The external component test should adhere to the following conditions, to be categorized under generic tests in Visual Studio:
- It must be run from the command line
- The component must return a Boolean value of either
TrueorFalsewhen executed in the command line - It should return detailed results for internal tests within the component
This is similar to any other test in Visual Studio. Right-click on the Test Project in Solution Explorer and add a generic test. A new window opens to set the values or parameters for the generic test.
The new window denotes that all the required values are for executing another test application or function from the command line by passing parameters. For a command-line execution, we may have to set the environment variables and the execution parameters, set the working directory, copy or deploy some files, and set the output directory and the file. All these details can be set using the generic test.
The following table explains the different options and their use:
|
Parameters for Generic Test |
Description |
|---|---|
|
This is the name and path of the application or function to be executed at the command line. There is a browse button to the right of the textbox for finding and selecting the application. | |
|
This is the place to specify the command-line parameters required for the application. These parameters are dependent on the application's expected value. | |
|
Additional files to deploy with this generic test |
In some cases, additional files may be required for the test execution. Add or remove the selected files in the list using the option to the right of the textbox. |
|
Environment variables |
If the application under test uses any environment variables for the execution, set those environment variables here. |
|
This is to set the current working directory in the command line before actually running the test application in the command line. | |
|
While executing the test application, instead of displaying all the results at the command prompt, the results can be redirected to the output file, just as we do during normal command-line commands. | |
|
Exit test run if run duration (in milliseconds) exceeds |
This is to limit the wait time for Visual Studio to move on to the next test in the list or quit. These numbers denote milliseconds and the default is 60 minutes. |
|
Summary results file |
This is helpful in case the third-party test application can write the Test Results to a summary results file, which is an XML file. This is the name and path of the XML file in which the output results should be written. If the number of tests in the application is high, it will be easy to track the result of these individual tests by having the results in XML file; not only the result but also detailed information of the Test Result would be written to this file. |
The following is an example of a generic test that executes the Test.exe application, which is a third-party test application capable of writing the output to the XML file. The command-line parameter for this application is also provided along with the supporting file to be deployed, which is the Readme.txt file. You can see the Output.xml file, which is used to store the output details of the test by Test.exe.
When we execute this generic test, the third-party Test.exe will get executed at the command prompt. The generic test by Visual Studio will get the result back from the third-party Test.exe application, which is a single test. But we do not know how many tests are executed internally within the test, and it is not easy to track the results of all the tests of the third-party application using the generic test. But Visual Studio supports the third-party application with a summary results file, which can be used by the application to write the details of the internal Test Results.
Third-party applications can make use of the class file, which can be generated by using the schema file provided by Visual Studio. The schema file is located at the Visual Studio command line. If Visual Studio is installed in the default C: then the path would be C:Program FilesMicrosoft Visual Studio 10.0XmlSchemasSummaryResult.xsd.
The class file can be generated from this schema file using the xsd.exe utility on any .NET-supported languages. The following code snippet is an example for generating the default SummaryResult.cs class file from an XSD file. The output folder should exist before the command is run. c:AppsSamples is the output folder used in the following sample:
Xsd SummaryResult.xsd /c /l:cs /out:c: emp
The class file is a C# file as we have specified C# as the language in the command-line parameter (as /l:cs). The generated output file would look similar to the following screenshot:
There are two classes as SummaryResult and SummaryResultInnerTest each contain the same methods. SummaryResult collects the overall summary of the Test Run and SummaryResultInnerTest collects the details of the individual tests within the application under test. The following screenshot shows multiple methods within the SummaryResultInnerTest class:
The third-party tool can make use of this class file to write the Test Result details, or the test application should take care of writing the Test Result details into the XML file based on the XML schema used. The results output XML file should look like this:
<?xml version='1.0' encoding='utf-8'?> <SummaryResult> <TestName>Third party test Application</TestName> <TestResult>Failed</TestResult> <InnerTests> <InnerTest> <TestName>Test1</TestName> <TestResult>Failed</TestResult> <ErrorMessage>Test is unsuccessful</ErrorMessage> <DetailedResultsFile>C:TestingTest1Results.txt</DetailedResultsFile> </InnerTest> </InnerTests> </SummaryResult>
In the previous example, the XML file shows the summary Test Result as well as the inner test results. The failed test in the sample writes detailed information about the Test Result to the text file. Writing into the log file should be taken care of by the third-party test application, in the required format.