5.7. UI Testing Patterns
To help with UI testing, there are a number of different patterns which people follow to help organize their testing.
5.7.1. The Test Monkey
A test monkey is a test whose only purpose is randomness. Test monkeys will enter random strings into text boxes and then randomly click on the screen. When using a test monkey, your hope is that it will help flush out those quirky integration issues. Test monkeys may also be called stochastic tests, which is really the same testing pattern, but sounds more technical. Test monkeys are often used for destructive testing. There are two types of test monkey patterns: the dumb test monkey and the smart test monkey.
Dumb test monkey tests are black box tests. They don't know how the application works and they only subject the applications to random user activity. Dumb test monkeys may not realize they caused an error, but are valuable because of their randomness.
Figure 5-22. The entire Selenium suite working together
Smart test monkeys are not as random. Their inputs are generated from data that is reflective of user data. This data is often gathered from log files or data that has been obfuscated from the database.
You may have heard of the infinite monkey theorem: the idea is that a monkey (or other object performing the same task) hitting keys on a typewriter at random for an infinite amount of time has a very good chance of typing a particular text, such as the complete works of William Shakespeare. The term "test monkey" is based on this theorem.
5.7.2. Cassini Testing Pattern
As you may have noticed, the samples shown previously are testing websites via external URLs. You learned in previous chapters that tests should be self-contained, but with the pattern used earlier, are your tests really self-contained? The answer is, not really; the tests are dependent on the external URL functioning correctly. When testing the UI, the URL should not be the production URL. This would mean that your set of UI tests only runs against the production system, which goes against everything we've been talking about thus far. The URL of the application being tested needs to be some type of test machine that you have control over. Most of the time, teams have a system just for this: some call it a "staging system," some call it a "test system," and some teams even bring up a web server on the build/test machine to host the application under test.
This pattern of UI testing raises issues. How do you copy the latest version of your app to the test server? Do you really want to install a web server on the build server? Sure there are solutions to these problems, but these issues can be avoided by using a UI design pattern that brings up a small, lightweight web server for each UI test that is being run.
In Visual Studio 2005 you saw the first version of Cassini. Cassini is the lightweight web server that is used for debugging ASP.NET applications in the IDE. Both Visual Studio 2005 and 2008 come with two Cassini parts: Cassini.exe and Cassini.dll. For your purposes here, you'll only need to work with the dll.
You'll want to bring up an instance of Cassini for each test fixture, run your tests, and then destroy the instance of Cassini. It's important to note that you're only bringing up an instance of Cassini once per test fixture and not per test. This is because Cassini requires a bit of overhead to start up, and you don't really need to bring one up for each test.
The next listing creates an instance of the Cassini web server, tests a web, and then destroys the instance of the Cassini web server. In this example, you'll see that the Cassini web server is created in the TestFixtureSetup method, and destroyed in the TestFixtureTearDown method. The path for where the web application is stored is set in the pathToAppToTest variable:
using System;
using NUnit.Framework;
using Cassini;
using System.IO;
using System.Diagnostics;
using WatiN.Core.Interfaces;
using WatiN.Core;
namespace CassiniDemo
{
[TestFixture]
public class CassiniPatternTests
{
private Server m_webServer;
private string m_serverURL;
[TestFixtureSetUp]
public void Setup()
{
DirectoryInfo appBasePath = new DirectoryInfo(
AppDomain.CurrentDomain.BaseDirectory);
string serverPhysicalPath = appBasePath.Parent.FullName;
int webServerPort = 8090;
string virtualDirectory = "/";
string pathToAppToTest = "AppToTest";
try
{
m_webServer = new Server(webServerPort, virtualDirectory,
serverPhysicalPath);
m_serverURL = String.Format("http://localhost:{0}/{1}",webServerPort,pathToAppToTest);
m_webServer.Start();
Debug.WriteLine(String.Format("Started Port:{0} VD:{1} PD{2}",
webServerPort, virtualDirectory, serverPhysicalPath));
}
catch (Exception ex)
{
Debug.WriteLine(string.Format("Error starting web service {0}",
ex.Message));
}
}
[TestFixtureTearDown]
public void TearDown()
{
try
{
if (m_webServer != null)
{
m_webServer.Stop();
m_webServer = null;
}
}
catch (Exception ex)
{
Debug.WriteLine(string.Format("Tearddown error {0}", ex.Message));
}
}
[Test]
public void Should_Load_The_Default_Page_And_Find_Results_For_AspNet()
{
string expectedText = "The Official Microsoft ASP.NET Site";
string urlToTest = string.Format("{0}/{1}",
m_serverURL, "default.aspx");
using (IBrowser ie = BrowserFactory.Create(
BrowserType.InternetExplorer))
{
ie.GoTo(urlToTest);
Assert.IsTrue(ie.ContainsText(expectedText));
}
}
}
}This example uses WatiN to test the page for a specific test, but other UI testing methods could be used as well, such as parsing the HTML manually.
It's important to note some of the lines found in the previous listing. This is the code that creates the instance of the Cassini web server and then starts the service:
m_webServer = new Server(webServerPort, virtualDirectory, serverPhysicalPath); m_webServer.Start();
What is this testing pattern getting you? Using this pattern will ensure that your tests are self-contained and that the machines where the tests are run will not require that a web server be installed. Using this pattern will also resolve the need to create complex build scripts that copy the application you are testing to another machine. Another nice side effect of this pattern is not having to worry about networking issues. Many times when testing the UI, tests will fail due to network latency or network errors that are out of the developer's hands and are not really considered errors.
Many owners of the build machine believe that it should be kept as clean as possible, meaning that no third-party software should be installed on the machine. This forces developers to include any third-party binary files with their application; if they don't, the build server will not have these files, and therefore the build will break. For some build machine owners, this includes web servers.
5.7.2.1. What Goes into a Good UI Test?
In the previous chapter you learned what makes a good test. Many of these same principles apply, but there are a few key concepts pertaining to UI testing that we haven't covered yet
Tests should be created using the same concepts that an end user would see. When creating UI tests, think in terms of behaviors that encompass the problem domain, instead of thinking of the IDs of text boxes and edit controls. Think of these tests as small unit tests that test the behavior of a feature.
5.7.3. Randomness
You may have already come to the realization that when you are testing UIs, you have to think outside-the-box. Great testers think outside-the-box all the time, which brings us to the topic of randomness.
When creating UI tests, it's important to randomize how the data is entered into the UI. Let's take a step back and think about this for a moment: in Chapter 2 you learned that tests should be repeatable, but if you randomize how the data is entered into the UI, is the test going to be repeatable? The answer is yes. If, eventually, a test fails, that test will be run again in the test cycle, so therefore it is repeatable. Having a log of what the test has done is invaluable for re-creating the random test that failed.
What sort of tasks should be random? The quick answer is behaviors. For example, you could create automated UI tests that drive the user interface into the browser using a mouse, but then also create tests that drive the browser using keyboard navigation. Then, mix up the way they are run. Having both sets of tests will help find those weird integration issues that are extremely difficult to track down later. On smaller systems, this type of random testing may not be needed; if you are only testing a few pages your tests will run fast enough so that you don't need to choose which method of accessing the UI you need. On larger systems, where tests may take hours or even days to run, it's very beneficial to have the ability to test behaviors randomly to help cut the time in which the tests run down.
5.7.4. How Random Should You Be?
User interface bugs have a very bad habit of hiding, meaning that the quirks of the interface are hard to flush out. Most developers have witnessed the weird types of integration issues that users report. Most of the time, these issues are hard to track down mainly because the user neglected to inform you of a key piece of data, such as they were using the tab key to navigate form fields when the error occurred.
There is a level of randomness for each application that is acceptable, and that level is dependent on the application. Testing random characters and mouse vs. keyboard movement randomly is acceptable for most applications, where randomly clicking around on the screen is not for most applications.
5.7.5. Special Characters
As a developer, we're sure you have manually tested web forms by entering garbage text such as "foo," into the fields. Tests such as these do not fully exercise your application. Good UI tests involve entering copious amounts of text, such as Lorem Ipsum, as well as a sample of the full range of alpha numeric characters. Characters such as <>?/_+ are great at breaking applications. You'll test special characters and large amounts of text for edge cases where the application has issues accepting these types of text inputs. You will be surprised at the amount of errors you will find when exercising these types of tests.
The next listing creates a test that enters three paragraphs of Lorum Ipsum into the text search text field on www.google.com. After the text is entered, the tests verify that at least 10 results were found.
Notice the GetLorumIpsum function. This function takes in an integer value of the number of paragraphs you would like to generate. As your application gains more tests, this function could be abstracted into a Test Helper class, because multiple test fixtures should be using this concept:
using NUnit.Framework;
using WatiN.Core;
using WatiN.Core.Interfaces;
using System.Text;
namespace LorumIpsumTest
{
[TestFixture]
public class LorumIpsumTest
{
private const int MAX_PARAGRAPHS = 3;
[Test]
public void Should_Enter_Large_Amount_Of_Text_On_Google_Search()
{
using (IBrowser ie = BrowserFactory.Create(BrowserType
.InternetExplorer))
{
ie.GoTo("http://www.google.com");
ie.TextField(Find.ByName("q")).Value = GetLorumIpsum
(MAX_PARAGRAPHS);
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("Results 1 — 10 "));
}
}
private string GetLorumIpsum(int numParagraphs)
{
StringBuilder lorumIpsum = new StringBuilder();
if (numParagraphs > 0 )
{
lorumIpsum.Append("Lorem ipsum dolor adipiscing elit. Maecenas ");
lorumIpsum.Append("eu nibh. tum in, aliquam at, massa. Maecenas");
lorumIpsum.Append("non sapien et mauris tincidunt cursus. In hac");
lorumIpsum.Append("ante ipsum primis in faucibus orci luctus et");lorumIpsum.Append("vehicula. Lorem ipsum dolor sit amet,");
lorumIpsum.Append("nec felis ultricies venenatis. Ut mollis mi s");
lorumIpsum.Append("eros. Suspendisse felis nunc, malesuada eu, ");
lorumIpsum.Append("dolor at magna.
");
}
if (numParagraphs > 1)
{
lorumIpsum.Append("Fusce mauris enim, semper quis, accumsan eget");
lorumIpsum.Append("rutrum condimentum orci. Duis libero. Suspen");
lorumIpsum.Append("Donec gravida nulla vel felis elementum lobo");
lorumIpsum.Append("ultrices non, ipsum. Fusce et arcu non urna");
lorumIpsum.Append("elementum eu, semper a, mauris. Suspendisse");
lorumIpsum.Append("aliquet. Vestibulum gravida, ipsum id pretiu");
lorumIpsum.Append("euismod neque nunc ut erat. Pellentesque habi");
lorumIpsum.Append("fames ac turpis egestas. Nunc hendrerit elem");
lorumIpsum.Append("interdum mi sit amet justo. Etiam augue. Ph
");
}
if (numParagraphs > 2)
{
lorumIpsum.Append("Praesent id enim. Praesent tortor. Phasellus" );
lorumIpsum.Append("convallis in, imperdiet eu, eleifend vel, nu" );
lorumIpsum.Append("elementum viverra. Integer nec nibh ut erat" );
lorumIpsum.Append("ullamcorper. Praesent porta tellus mauris.
");
}
return lorumIpsum.ToString();
}
}
}The next listing is very similar in concept to the Lorem Ipsum test. If more than one test fixture calls the GetSpecialCharacters function, then this method should be abstracted into a test helper class:
using NUnit.Framework;
using WatiN.Core;
using WatiN.Core.Interfaces;
using System.Text;
namespace SimpleWatinTest
{
[TestFixture]
public class SpecialCharacterTest
{
[Test]
public void Should_Enter_Special_Characters_On_Google_Search()
{
using (IBrowser ie = BrowserFactory.Create(
BrowserType.InternetExplorer))
{
ie.GoTo("http://www.google.com");
ie.TextField(Find.ByName("q")).Value = GetSpecialCharacters();System.Threading.Thread.Sleep(6000);
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("Results 1 — 10 "));
}
}
private string GetSpecialCharacters()
{
return @"""!'@#$%^&*(){}[]-=_+:;<>,.?/|~`";
}
}
}When testing your application for special characters, don't only test characters, but think outside-the-box and also test character patterns. Special character testing is a great time to test for cross-site scripting attacks, as shown next. We will cover cross-site scripting attacks thoroughly in Chapter 10, but for now it's important to know that a cross-site scripting attack is where a user of your application injects malicious HTML or JavaScript into form fields.
using NUnit.Framework;
using WatiN.Core;
using WatiN.Core.Interfaces;
using System.Text;
namespace XSSTest
{
[TestFixture]
public class XSSTest
{
[Test]
public void Should_Check_For_XSS_Attack_Google_Search()
{
using (IBrowser ie = BrowserFactory.Create(BrowserType.InternetExplorer))
{
ie.GoTo("http://www.google.com");
ie.TextField(Find.ByName("q")).Value = GetXSSAttack();
ie.Button(Find.ByName("btnG")).Click();
Assert.IsTrue(ie.ContainsText("Results 1 — 10 "));
}
}
private string GetXSSAttack()
{
return "<script>alert('xss attack'),</script>";
}
}
}This is a very simplistic cross-site scripting attack. There are many different ways to inject scripts into a page and Chapter 10 will cover these methods. For the moment, please note that the test in the previous listing is just a simple example, and more thorough tests encapsulating the techniques discussed in Chapter 10 should be used as true cross-site scripting tests.
5.7.6. Tests Should Be Focused
It's very important to stress (again) the fact that developers and testers should be creating small tests. As a rule of thumb, tests should fit inside your head, meaning the code for the test methods should never be physically longer than the size of your head. Another good example would be that tests should fit on a cocktail napkin.
When using the record/playback method for automated UI testing, tests can become very large. This is a common code smell, leading you to realize that it's time to refactor the test into smaller pieces. Also, it's more than likely that there are sections of the recorded test that can be reused.
Along with test patterns, there are also a number of other tools and frameworks which are designed to help drive the browser.