6.1. Acceptance Testing Terminology
One of the problems you'll face with acceptance testing is that the term has been used for many different techniques and approaches throughout the years. Sometimes the meanings are related and share a common concept. Other times people can be talking about completely different forms of testing. We've provided an overview of the different terminology next:
Acceptance Testing. Also includes customer acceptance testing, user acceptance testing, and functional tests
Executable Specification. The advantage of having acceptance tests as they form a specification which can be run to verify if the implementation matches what the specification defines
Customer. The end-user of the system
System. The application being developed
Acceptance. Meets the functional and nonfunctional requirements
Functional Requirements. Features and actions the system must perform, such as display items or allowing users to log in to the system
Nonfunctional Requirements. Factors around the system, such as performance, scalability, and security
Black Box. Not dependent on internal details or prior knowledge of the system. Data goes in, results come out. That is all the tests should be aware of.
However, although you can define the terminology, we still don't think this gives an accurate picture of how acceptance testing applies to the development lifecycle. Although the concept of acceptance testing has been around for a long period of time, it has gained more attention in recent years, along with Test Driven Development (TDD) which has resulted in improvements in tools, frameworks, and guidance.
Much of the guidance has come from the rise of extreme programming (XP), agile principals, and scrum methodology. There are two main reasons for this. One of the reasons is because acceptance tests focus on the customer and implementing features valuable to the customer. This aligns with the principals of agile development, which also has a strong focus on delivering software that actually meets the customer requirements. The second reason is that by having a set of automated acceptance tests you can ensure that the software constantly meets your customer requirements and ensures you haven't broken anything while implementing new features. In a similar fashion, having your tests automated means you can spend less time manually testing existing features to ensure they still work in later iterations. This means, you can focus on ensuring that the features being developed work as you expect and desire. This allows you to deliver software in a confident and timely fashion.
We feel acceptance testing is most effective when combined with an agile development process. An iterative process allows the team to focus on specific sections of the application and ensure that they are completed, from design to testing, in a single iteration.
During the iteration is the ideal point to write and implement your acceptance tests. The following process is based on scrum and completing a sprint (a scrum iteration) and how acceptance testing fits.
At the start of the scrum, the team accepts work off the product backlog with the highest priority and in a position to be completed. The product backlog should be broken down into user stories. A user story defines a system requirement. The story is a couple of sentences which describes a section of the system which the user requires. An example of a user story is "As a sales administrator, I want to be able to view credit card information so that I can process the payment locally." This describes the viewpoint and user who the story is related to, what they are trying to achieve, and why. This gives you a clear indication of what is required to be implemented.
After a story has been selected, the team should have a good idea about what they are going to implement. This is the stage where they can talk to the customer and product owner to determine what is actually required and to expand on the initial story. Based on this information and other technical discussions amongst the team, tasks are created. These tasks represent the work required to implement the required functionality. This is the stage when you should write the acceptance tests. You know the story you are trying to implement and you should have a good understanding of the tasks required to complete the implementation. As such, you should know how to verify that the application meets the customers' requirements. It's important to note that acceptance tests are not low-level unit tests. They are high-level tests focusing on verifying that the customer requirements, which are defined based on the story, are implemented correctly. We have that found it is useful to define these acceptance tests after selecting a story and before breaking it down into tasks. By having the tests in place you know exactly what you are aiming for. You know everything the system must do and as a result will know when you are finished. You know this because all the tests will pass. This makes the task breakdown more focused on what actually needs to be done.
The team repeats this process until they have enough stories which they feel they can complete within a fixed-length iteration. This then becomes the sprint backlog.
During this planning stage, the customer and product owner should be helping the team define the acceptance tests. The acceptance tests are in place to ensure that the requirements meet their own expectations. The feeling of having this set of tests in place is a great comfort to customers. They can match the story for the sprint, together with a set of tests which describe exactly what will be implemented and how it will work — even before any code has been written. This should also provide a comfort to the development team. The customers have clearly defined the requirements. The team helped the customer come to this conclusion and asked any initial questions while it was still fresh in their minds. The customer should be clear about what they expect and the team should be clear about what they plan to deliver — using the acceptance tests as guidance and verification. However, this doesn't mean that more questions won't arise during development or that incorrect parts of the application won't be developed but the aim is to reduce the cost as much as possible in an attempt to successfully deliver software on time and on budget.
However, customers are generally nontechnical and would be unable to understand the code developed to test the requirements. Instead they would reject the process and either not understand what they were agreeing to, or complain bitterly. Instead of showing customers code, write the tests using plain English and sentences. In a similar fashion to how your story defines a high-level view in a few sentences, acceptance tests should describe the scenario in a few sentences. Even if the customers are technical, you still don't want to discuss the technical details.
There are different styles of how to write the tests. One popular style is the Given, When, Then syntax. For example, your test might be written as follows:
Given a new user account
When it is created on the system
Then the password should default to P@ssw0rd
These can be as complex as required to solve the specific problem you are testing:
Given an overdrawn account
And a valid debit card
When the customer requests cash from an ATM
Then an error message should be displayed
And no cash is returned
And the card is returned back to the customer
Having the tests in this style is how the customers can effectively contribute in the planning session and define the tests, while also being able to verify and sign-off on the details. An important fact around having them in this style is the language and terminology used. It has been our experience that customers are also more enthusiastic about testing when the test cases are defined in this fashion as they feel they are adding real value to the project because of the automatic bond felt by being closer to the project and more involved. This provides a positive feeling about the application from the beginning which hopefully will be carried over when you deliver the software and the customer sees all the tests they helped write passing with working functionally.
If you notice when the test was written there was no mention of technical details, terms, or implementation.
Instead, you should focus on the business problem and scenario, using the same language as the business would use to describe the problem. This has two major benefits. The first is that you don't need a technical background to understand the scenario and the expected outcome. Secondly, if the underlying implementation changes you don't need to go back and update your tests to reflect this change. This is important because the tests are high-level and potentially cover a number of different components of your code base. You don't want them to be dependent on the exact details as over time they will change. By remaining high-level, you allow the flexibility for the system to change without having to affect the tests. Of course, if the scenario is no longer valid because it has been dropped or changed in some way then the test should be updated to reflect this.
Tests which hang around and are not useful are harmful to a project and test suite. The test suite should be as streamlined and focused as possible, enough tests to provide a very high level of confidence, but no wastage. Having redundant tests causes higher maintenance, and as a result, higher costs and frustration. If the test isn't providing any value — remove it.
When you have the tests in place, the next question is how should they be run to ensure they pass? As with many different parts of testing you can either test it manually or you can automate the testing.
Executing these tests manually would be a time-consuming process. Although manual testing might seem the much faster option, in the long run it is more time-consuming as every time you change part of the system you need to re-run your set of manual tests. At the beginning while the system is small this might not be too much of a problem. However, as the system grows this will become more time-consuming and motivation will drop resulting in more mistakes and things potentially being missed. There is also the additional time of converting the scenarios created during planning into manual test cases for them to be executed. As you can see, it doesn't seem as cheap as you think.
The other approach is to automate the tests.
6.1.1. Using Automation
There are many ways to automate customer acceptance tests. One of the most popular approaches is using a tool called FitNesse,
6.1.2. FitNesse
FitNesse is based on Fit. Fit is an abbreviation for Framework for Integrated Testing originally developed by Ward Cunningham. One of the main motivations for Fit is to improve communication and collaboration, allowing customers and business analytics to write and execute tests against the system.
The aim of Fit/FitNesse is collaboration around the system with the result being an executable specification. By using the wiki format for managing the test cases and scripts, it lowers the barrier for accessibility by allowing the entire team, including customers, to collaborate on creating new test cases.
Along with the wiki front end, there is an underlying code base which handles the communication between the wiki and the system under test. The layer provides an abstract view of how to interact with your system which enables you to write simpler, more focused tests for the wiki. Depending on the skill-set of the team, this layer might be written by a developer or technical tester leaving the tests in the wiki to be written by nontechnical members. Having the ability to allocate responsibility to the people with the correct skill-set is important for productivity and motivation.
6.1.3. Setting Up FitNesse Wiki
Setting up a FitNesse wiki for a project is a relativity painless process. After downloading the framework, you need to unzip it into the directory where you want the wiki to be stored. To start the wiki server you simply run the java jar file and define the port number. This will launch the server which you can connect to from any Internet browser, as shown in Figure 6-1:
>java -jar fitnesse.jar -p 8080
FitNesse (v20090513) Started...
port: 8080
root page: fitnesse.wiki.FileSystemPage at ./FitNesseRoot
logger: none
authenticator: fitnesse.authentication.PromiscuousAuthenticator
html page factory: fitnesse.html.HtmlPageFactory
page version expiration set to 14 days.At this point you can start writing your first tests and defining your executable specification for the application.
To create a new page, you should edit the homepage and add links to additional pages which will contain the tests. You do this via the edit box in Figure 6-2. To add a link you need to use a Camel Case name, for example MyFirstTest — the wiki engine will take care of the test:
!1 Welcome to [[FitNesse][FitNesse.FitNesse]]! !2 Example Tests MyFirstTest
Figure 6-1. FitNesse wiki homepage
Figure 6-2. Editing homepage to add a new link
After adding the link you will be taken to the edit page for this new page where you can include your tests. The first part of the test is to set up the environment variables. These tell the FitNesse engine about the environment and where it can find various key bits of information. For example, FitServer is the runner which is capable of running the FitNesse tests:
!define COMMAND_PATTERN {%m %p}
!define TEST_RUNNER {dotnetFitServer.exe}
!define PATH_SEPARATOR {;}After you have defined the generic settings, you need to point the page to the assembly containing your layer between the tests and the actual application:
!path D:UsersBen HallDocumentsTestingASPnet6-AcceptanceSamples FitnesseExampleFitnesseExampleinDebugFitnesseExample.dll
Underneath these settings your FitNesse table is defined:
|Division| |numerator|denominator|quotient?| |10 |2 |5 | |12.6 |3 |4.2 | |100 |4 |24 |
If you break down your FitNesse table, |Division| is the name of the class which handles the translation between the wiki and FitNesse and the application under test.
As such, the class within your test assembly would look like this. In this case, your implementation is in-line with the fixture; however, it could have gone off and called external assemblies or systems. Fundamentally it is just C# with properties and methods. Notice if your table quotient had a question marked. This means that the method should return a value and the result should equal the row's value:
public class Division : fit.ColumnFixture
{
public double numerator = 0.0;
public double denominator = 0.0;
public double quotient()
{
return numerator/denominator;
}
}For FitNesse to work, the column names need to match the property or method. This is the same for the class name. ColumnFixture is one of the many different base classes within Fit and also the most commonly used as it simply maps the properties and methods to the columns.
After saving the wiki page with a correctly formatted table, FitNesse will give you a HTML page with the table rendered in a readable format as demonstrated in Figure 6-3.
After you click the test menu item, the wiki is updated with color coding to indicate if the row passed or failed. As shown in Figure 6-4, the final row failed because the value 25 was returned when 24 was expected.
Figure 6-3. Screenshot of fixture before execution
Figure 6-4. Failing fixture
With this fixture in place, anyone can add additional tests, execute them at any point, and see if they passfail. If you imagine this process when it comes to complex business calculations and rules, anyone can input the values and instantly know if the system supports it or not.
To make the tests a bit more maintainable, you should move your configuration into the root. In the footer of each page there is a link to the root page — this is inherited by every test page and so it is the perfect location for this kind of information. You can now continue to add wiki pages and fixtures into your assembly based on the different logic within your system.
Although ColumnFixture makes it easy to add multiple different test inputs for a particular fixture, it makes it much more difficult to create a script or a story around the test. This is where DoFixture comes in.
DoFixture is much more powerful than ColumnFixture and allows your tables to be more expressive. It takes the same concept of matching column names but allows you to create sentences, injecting values at certain places to make the tests more readable and follow a more natural descriptive language.
The concept is to use various methods on your fixture to form a test script which can be executed against your system. This is the ideal fixture to use to create your executable specification as it provides more freedom to create more complex scenarios and verifications, moving beyond a single scenario with multiple test inputsoutputs. The DoFixture is also suitable for replacing your manual tests, which are discussed in Chapter 7, as they are capable of different steps to verify the behavior.
In the following example, the steps for making tea are described. There are various properties which go into making a great cup of tea. The values and steps taken will determine the outcome. When using a DoFixture, your FitTable would reassemble this:
|Making Tea| |For type|English breakfast|boil kettle And Insert|1|Teabag| |Insert|2|of milk| |Leave tea bag in for|4| |Check|strength is|Normal|
Similar to the ColumnFixture, "Making Tea" is the name of the class. To improve readability, FitNesse will automatically remove the spaces to find the class. You can then define your method calls. Each row represents a method. The columns alternate between parts of the method name and parameters. In this case, the associated method is this:
public void ForTypeBoilKettleAndInsertTeabag(string name, int tea)
{
TeaName = name;
AmountOfTea = tea;
}The first column, For type, is part of the method name with "English Breakfast" being associated with the first argument. Boil kettle and insert is the middle part of the method while 1 is the second argument to the method. Finally, Teabag is the end of the method name. This is how DoFixture finds the correct method to call and the arguments to use.
The second and third rows are the second and third method calls which should be made:
public void InsertOfMilk(int amount)
{AmountOfMilk = amount;
}
public void LeaveTeaBaginfor(int time)
{
AmountOfTime = time;
}The fourth row begins with Check. Check is not part of the method name but a keyword for FitNesse. This simply means that the final column is the value you expect the method to return. FitNesse will then verify the result with the value in the cell. If they are different, it will mark it as an error, showing both expected and actual.
For this example, the method which returns the result is a complex rule system which describes the type of tea you will have as a result of the input:
public string StrengthIs()
{
if (AmountOfTea == 1 && AmountOfTime == 4 && AmountOfMilk == 2)
return "Normal";
if (AmountOfTea == 1 && AmountOfTime == 1 && AmountOfMilk == 6)
return "Milky";
if ((AmountOfTea == 2 && AmountOfTime == 6 && AmountOfMilk == 0) ||
(AmountOfTea > 0 && NoMilk))
return "Black";
if (AmountOfTea == 2 && AmountOfTime == 4 && AmountOfMilk == 2)
return "Strong";
if (AmountOfTea == 0 && AmountOfMilk > 0)
throw new NotTeaException();
return "Water";
}The page is then rendered as shown in Figure 6-5. You can see that the Check column is followed by the StrengthIs method call. This will call the method above. FitNesse expects the results returned from the method to be Normal. If the value is Normal then it passes and you know the system is implemented correctly.
As mentioned, the DoFixture allows you to build more of a story-based model around the scenario you are testing. For example, here you have a separate test but you call the method Without Milk instead of InsertOfMilk. This changes the values in your system and as a result you expect the strength to be 'Black':
|Making Tea| |For type|English breakfast|boil kettle And Insert|1|Teabag| |Without milk| |Leave tea bag in for|4| |Check|strength is|Black|
You can continue to do this for different scenarios, using either different values or calling different methods on your fixture to interact with the system and achieve the result you expect to ensure that everything is working as expected:
|Making Tea| |For type|English breakfast|boil kettle And Insert|2|Teabag|
|Insert|2|of milk| |Leave tea bag in for|4| |Check|strength is|Strong|
Figure 6-5. Making tea story
The resulting page is shown in Figure 6-6 with the multiple different test cases as different tables.
Although it is easy to create tests, it is also important to remember how to structure and organize the tests. FitNesse has the concept of SubWikis. This allows you to have a single page which links to a number of other pages to form a hierarchy of tests. This hierarchy can follow the same structure as your Visual Studio solution for your FitFixtures to keep everything in sync. Another approach is to structure your wiki page based on the different stories and logic parts of your application. By using SubWikis, your tests become much easier to manage as they are logically grouped together.
Creating a hierarchy or SubWiki for your tests is very simple. You can edit your wiki page as you would to add a normal test; however, instead of adding your FitTable, you add another link. By prefixing the link with a > you start to form a hierarchy as done in Figure 6-7.
You can continue to build different levels. When you have finished, you simply have a normal page with your FitTable as demonstrated in Figure 6-8.
To support being able to access C# namespaces, you can include the namespace to the FitFixture. Make sure you include an '!' at the beginning to help the markup. With this in place, the engine will be able to find the class and execute the tests. Figure 6-9 shows the results of a test against a C# class.
Figure 6-6. More story examples
Figure 6-7. FitNesseExample.SubTests
Figure 6-8. FitNesseExample.SubTests.SubStringTest edit page
Figure 6-9. FitNesseExample.SubTests.SubStringTest executed
Having a structured SubWiki is also advantageous when it comes to execution. FitNesse correctly detects that the wiki pages you have created so far are tests because you have ended the names with Test. By doing so, this enables the Test link to appear in the navigation. By having a SubWiki, you can execute a suite of tests. The result is that any test pages attached are executed in a single batch with the results being combined while allowing you to execute all the tests in a much shorter space of time.
To create a suite, you can either end the page name with Suite or Examples. The other option is to manually edit the properties for the page as shown in Figure 6-10.
Figure 6-10. Page Properties
By editing the properties, you can define whether the page should be a standard wiki page, a test, or a suite regardless of its name. After you have selected a suite, the Suite button is enabled. Clicking this will execute all the tests linked to that page and the results will appear in a similar fashion to Figure 6-11.
Although having the page as a wiki is useful for collaboration and a suite allows you to execute a number of different fixtures, you cannot integrate this into any automation process (such as a build step) at this time. Ideally, you want to edit and execute your tests on an ad hoc basis, but also after every successful build or every night.
Richard Fennell from Black Marble (http://www.blackmarble.co.uk/) in the United Kingdom, provides one possible solution. When the wiki executes, it passes various bits of information to the command line runner which does the hard work of executing the tests. The command line runner is very simple to execute as you simply provide the HTML file, the associated assembly, and an output directory where you want the results to appear. For example, to execute the test.html page you would use the following command:
fitnessedotnetFolderRunner.exe -i test.html -a ..FitnesseExample FitnesseExampleinDebugFitnesseExample.dll -o results
Figure 6-11. Results of executing a suite
At this point, the fact that FitNesse is based on a wiki page is irrelevant. The runner expects a constructed HTML page with the HTML correctly formatted. As such, the page can be as simple as shown in Figure 6-12.
Figure 6-12. Basic HTML page
The HTML for the page looks like this. This can be created using any HTML editor:
<html>
<head>
<title>MyFirstTest</title>
</head>
<body>
This defines how we expect division to work<br/><br/>
<table border="1" cellspacing="0">
<tr>
<td colspan="3">Division</td>
</tr>
<tr>
<td>numerator</td>
<td>denominator</td>
<td>quotient?</td>
</tr>
<tr>
<td>10</td>
<td>2</td>
<td>5</td>
</tr>
<tr>
<td>12.6</td>
<td>3</td>
<td>4.2</td>
</tr>
<tr>
<td>100</td>
<td>4</td>
<td>24</td>
</tr>
</table>
</body>
</html>After the command has been executed, a summary of what was executed is outputted to the console and a results directory created. For example, the command would show that two of your rows were correct, while one was incorrect:
2 right, 1 wrong, 0 ignored, 0 exceptions, time: 00:00:00.1904296
In the results directory you have a copy of the associated page which has been updated to reflect the actualexpected results. As you can see from Figure 6-13, this is the same page you would have seen if you were using the wiki-based solution.
At this point you can have your tests run in an automated fashion. Richard took this approach a step further by having each page executed as a separate test. The test would then know if it should pass or fail to represent the result of the FitNesse execution.
For this to execute, simply use the FolderRunner object within the Fit assembly. The run method has the same arguments as the command line and returns an error count after all the tests have been executed:
[Test] public void TestPage()
{
string page = @"test.html";
string assembly = @"FitnesseExample.dll";
fit.Runner.FolderRunner runner = new fit.Runner.FolderRunner(new fit.
Runner.ConsoleReporter());
var errorCount = runner.Run(new[] { "-i", page, "-a", assembly, "-o",
@"results" });
Assert.AreEqual(0, errorCount, runner.Results);
}Figure 6-13. Basic results page
The result is your test results can now be executed and viewed within Visual Studio as shown in Figure 6-14.
Figure 6-14. FitNesse within Visual Studio
By having your FitNesse tests running as unit tests you can combine the test of both worlds. You have the advantage of being able to write your test cases as HTML and in a separate file while being able to execute them as unit tests within Visual Studio or as part of your build system.
Personally, I think FitNesse solves a problem. If you have a complex rule system or accounting package that works with the idea of columns, values, and predictable results then FitNesse will without a doubt help. However, I find the DoFixture and more complex scenario difficult to read and understand and error prone when it comes to attaching the method calls to separated column names. The problem is that some things aren't suited to being stored in a table. As a result, other frameworks have taken the concept of acceptance tests and built their own frameworks capable of having nontechnical members write tests.
6.1.4. FitNesse, WatiN and WroxPizza
To provide some more context about how you could use FitNesse to create customer acceptance tests we wanted to provide an example of how you could combine what we discussed in Chapter 5 with regard to WatiN and UI testing together with FitNesse.
With FitNesse, there is the wiki which provides test inputs and verifies outputs together with the C# fixtures which control the test execution. The hardest part is creating the C# fixtures, which in this case will take input and use WatiN to automate the browser against our website at which point we can return results to the wiki engine.
Following on from Chapter 5, you are going to reproduce the same style of tests you created to verify how the application works using FitNesse to start moving towards a more collaborative approach to writing the test cases. The first test will be to verify that the correct products and categories are listed on the Products Menu page.
As you have done before, you need to create a new wiki page on our FitNesse server. In order to verify the products are being displayed correctly there needs to be a FitNesse table which lists all the products and states if they should be listed on the page or not.
!|WroxFitNesse.ProductsMenuFixture| |product |productListed? | |Thrufropaquentor |true | |p2 |false |
This table assumes that a class called ProductsMenuFixture exists in our assembly, which provides the functionality to verify the website. The fixture needs two parts, one is a variable called product to store the name of the product we want to check to see if it exists on the page, while the other is a method which verifies if that product actually exists. Because we are using the classic FitNesse table, our class will inherit from ColumnFixture. The code to ensure the above table can execute is next:
public class ProductsMenuFixture : ColumnFixture
{
ProductsMenuPage page;
private IBrowser browser;
public ProductsMenuFixture()
{page = new ProductsMenuPage(browser);
}
public string product;
public bool productListed()
{
return page.Products.Contains(product);
}
}The above code should look familiar as we are following the same approach as we took in Chapter 5 to verify the functionality. In Chapter 5 we created a class for each page which modeled how the website was structured, the links, textboxes and general functionality of the site. Because the model was clean and abstract we can simply re-use this set of classes for use within our FitNesse fixture.
Another thing to point out is that our fixture and the productListed( ) method doesn't handle the verification about if it should passfail, instead it just provides a Boolean of the actual result and lets the wiki decide the outcome.
However, before we can execute the wiki page and use WatiN to verify the functionality we have two core parts missing. Previously we used the TestFixtureSetup to configure where the application was deployed to. In FitNesse, we do not have these particular hooks in place. Instead, the Setup and Teardown are separate pages which we can use to configure the application. In this example, we are going to assume that the application is configured and accessible from the FitNesse server via automated deployment scripts to copy the files to an IIS server and not utilize FitNesse. We do this to provide more flexibility and it is much easier to manage via scripts.
Secondly, we used the TestSetup method to start the browser. With FitNesse, this is something that needs to be done via the FitNesse page, either on the page with the tables on or the set-up page. On the page, we would write code such as:
!|WroxFitNesse.WatiNBrowserFactory | |Use Browser|IE|and goto|http://localhost:62201/| !|WroxFitNesse.WatiNBrowserFactory| |Close Browser |
The first table configures the test to use the browser IE and says the website is located at a particular URL. By having these two properties configured by the wiki, we can swap the browser or URL very easily to test in a different way instead of being hardcoded to a particular implementation. Again, abstracting makes life easier when developing our tests.
In order to support the previous tables the tables above, we need a fixture which is capable of controlling the browser. In this case, we are taking advantage of the DoFixture to provide a more natural language approach to writing the tables. The code is just standard C# and stores the information from the wiki in various properties for future use. Notice, we are using static variables so we can access the data from our Test fixtures:
public class WatiNBrowserFactory : DoFixture
{
static IBrowser browser;
public string Url;public static string BrowserName;
public void UseBrowserAndGoTo(string browserName, string url)
{
BrowserName = browserName;
Url = url;
browser = GetBrowser(BrowserName);
browser.GoTo(url);
}
public void CloseBrowser()
{
if (browser != null) browser.Dispose();
}
}The most complex part is the GetBrowser method. This will take the name provided by the wiki page and create a browser instance. By using static methods we can follow the singleton pattern, allowing us to control how many browser instances are open at any particular point in the system:
public static IBrowser GetBrowser(string name)
{
if (browser == null)
{
switch (name)
{
case "IE":
browser = BrowserFactory.Create(BrowserType.
InternetExplorer);
break;
case "FireFox":
browser = BrowserFactory.Create(BrowserType.FireFox);
break;
default:
browser = BrowserFactory.Create(BrowserType.
InternetExplorer);
break;
}
}
return browser;
}With the WatiNBrowserFactory in place, we can update our Test fixture to use the static items to get the browser instance to use for our tests.
public ProductsMenuFixture()
{
browser = WatiNBrowserFactory.GetBrowser(WatiNBrowserFactory.
BrowserName);
page = new ProductsMenuPage(browser);
}There is one more piece of the puzzle to solve before we can execute the tests. If you remember from Chapter 5, in order for WatiN to work, the thread and test runner needs to be set as STA. In order to do this with FitNesse we need to provide a configuration file in our command pattern argument. The file we are creating is called suite.config.xml which is stored within the dotnet folder.
!define COMMAND_PATTERN {%m -c dotnetsuite.config.xml %p}The contents of the file should be as follows:
<suiteConfig>
<fit.Settings>
<apartmentState>STA</apartmentState>
</fit.Settings>
</suiteConfig>More information on the configuration file can be found in the documentation at http://www.syterra.com/FitnesseDotNet/SuiteConfigurationFile.html.
We now have our FitNesse wiki page being able to verify the functionality of our web application using WatiN. Our wiki page has three main parts, the first page tells the WatiNBrowserFactory which browser to use and the base URL. We have our test table which defines our test and input, and finally another table which closes the browser.
Behind the scenes, we have our ProductsMenuFixture which takes in the test input from the wiki. This communicates with the WatiNBrowserFactory to obtain the instance of the browser, and also our page model which we created in Chapter 5 and knows how to interact with the website and return results. Our Suite.config.xml sets the threading model for FitNesse to STA so that WatiN can communicate with IE.
We can now execute our wiki page and see results as shown in Figure 6-15. We can also extend the tests and add more tables such as verifying which categories are displayed:
!|WroxFitNesse.ProductsMenuFixture | |category |categoryListed?| |Pro linguens non trepicandor si quad estis vobis r|true | |category1 |false |
Adding this table means we need to edit our ProductsMenuFixture to include the correct property and method.
public string category;
public bool categoryListed()
{
return page.Categories.Contains(category);
}Figure 6-15. WroxPizza FitNesse Examples using ColumnFixture
While this uses ColumnFixture, a more readable approach could be to use the DoFixture and use a story driven approach when it comes to writing the tests. Personally, we feel this is the more desirable approach. For example, if we wanted to verify that we could add a particular product into the shopping cart a more readability would be as follows:
!|WroxFitNesse.AddItemToCartFixture| |When adding|Thrufropaquentor|into the basket| |For line|1| |Check|name is|Thrufropaquentor| |Check|Quantity is|1| |Check|price is|$718.2393|
We first say we are adding a certain product into the basket. We are then defining the line we are interested in verifying which allows us to perform checks against the particular columns to ensure everything is correct. For us, having this level of control and flexibility is extremely powerful.
The DoFixture implementation for AddItemToCartFixture is as follows. We interact with the page model, which in turn uses WatiN to control IE. We then have multiple methods for name, quantity and price, which uses the value set via the For Line method to obtain the correct shopping cart item from the web page:
public class AddItemToCartFixture : DoFixture
{
private IBrowser browser;
ProductsMenuPage page;
ShoppingCartPage cart;private int lineToVerify;
public AddItemToCartFixture()
{
browser = WatiNBrowserFactory.GetBrowser(WatiNBrowserFactory.
BrowserName);
page = new ProductsMenuPage(browser);
}
public void WhenAddingIntoTheBasket(string productName)
{
cart = page.AddToBasket(productName);
}
public void ForLine(int lineNumber)
{
//Wiki you would want to write 1,2,3 - in code it is 0,1,2
lineToVerify = lineNumber - 1;
}
public string NameIs()
{
List<Cart.Item> lines = cart.Items.Lines;
Cart.Item line = lines[lineToVerify];
return line.Name;
}
public string QuantityIs()
{
List<Cart.Item> lines = cart.Items.Lines;
Cart.Item line = lines[lineToVerify];
return line.Quantity;
}
public string PriceIs()
{
List<Cart.Item> lines = cart.Items.Lines;
Cart.Item line = lines[lineToVerify];
return line.Price;
}
}The resulting wiki page looks as shown in Figure 6-16, which we find readable and straight forward demonstrating the functionality being tested. We can add additional text around the table to help people understand the concepts more, improving readability at each stage.
Other frameworks have also realized that more of a story driven approach as we have described can result in improved tests. One of those frameworks is called Cucumber.
6.1.5. Cucumber
One of the most popular acceptance testing frameworks is Cucumber (http://cukes.info/) which is an open source Ruby framework. The framework takes concepts from BDD, as discussed in Chapter 2, and RSpec, which is another BDD framework for Ruby but with Cucumber focusing on writing automation acceptance tests. When using Cucumber to create customer tests you have two distinct parts. First you have a plain-text file which contains the feature and scenario information which you are verifying. These scenarios are created in plain English using business terminology (instead of technical) to allow the team to create and fully understand what they are covering. Removing ambiguity around technical aspects and focusing on usersusiness requirements reduces the risk of missing tests or being confused by the actual meaning and aim.
Figure 6-16. Shopping Cart Examples using FitNesse DoFixture
The second part of this is how these steps get executed. Each line within a scenario is treated as a step in the process of setting up or verifying the system, using the Given When Then (GWT) we discussed before. Each step is then attached to a Ruby method. The method configures the system in a particular fashion, similar to how the methods in a DoFixture with FitNesse worked.
For example, your feature and scenario file might contain the following:
Feature: Google Search
To find more about Testing ASP.NET
I need to be able to search Google
Scenario: Google Search for Testing ASP.NET
Given that I use Google
When I search for "Testing ASP.NET"
Then I should see "testing"
The feature defines the high-level overview of what the scenarios are covering. This should provide the reader with some context to the scenarios. The scenario then defines the steps and expected result. A feature generally has many different scenarios focusing on different aspects of the behavior for that particular feature.
Each step is a method which is defined in a Ruby file within a step_definitions folder. For example, your given step is defined next:
Given /^that I use Google$/ do pending end
In case you are unfamiliar with Ruby syntax, there are a few key points. First with Ruby method calls, the brackets () around method calls are optional to improve readability. RegEx is also a core part of the language and as such can be defined as a first class citizen, again to improve the readability. Finally, Ruby has the concepts of blocks. A block is a piece of code which can be passed into a method and can be executed as part of the method itself. This is similar to delegates and lambda expressions within C#.
These three concepts are used in the previous method. Given is a method which we are calling. Two arguments are then provided, a key and a block which need to be executed. Under the covers, Cucumber matches the step in your plain text to this method based on the key and executes the method, which executes the block provided.
The same approach is taken for both the When and Then steps as well. A system can have multiple GivenWhenThen blocks, each identified via a unique key. However, having hard-coded strings is not a very effective way to reuse code. In the previous scenario, if you wanted to test two different search terms or results you would need to create different methods. This will increase complexity and maintenance costs.
To cope with this you can include a series of RegEx expressions when you define the RegEx key, which will pull out key parts of the string and pass them into the block as arguments. This has been done with the When and Then blocks as defined next. You can now use these same methods multiple times, providing different strings and verifying different results. This provides amazing power and flexibility:
When /^I search for "([^"]*)"$/ do |arg1| pending end Then /^I should see "([^"]*)"$/ do |arg1| pending end
Within the block for each method you call the pending method. One of the aspects of customer acceptance tests is that they can be written before implementation of the code. Call pending means that you can continue developing the boilerplate scenarios and methods, but when they are executed they will be marked not as failing but as pending. This provides you with an improved indication of the current state of the projects and the tests.
However, you want to replace the pending blocks with actual implementation. In this case, you want to interact with the Google search engine, search for a particular term, and verify the search results containing a particular term. Cucumber provides you with the framework for executing the tests, but you need another framework for interacting with web pages. As discussed in Chapter 5, WatiN is a .NET port of the Ruby framework Watir. You can combine Cucumber with Watir to provide the customer acceptance tests with the ability to test the UI.
In the same file as your step methods you should include the method calls to require the framework as shown in the sample below. This allows you to access the test framework RSpec to provide test-like functionality and Watir to access a browser. Watir has the ability to interact with IE, Firefox, and Safari — this example simply uses IE; however, the principals are the same. In Chapter 5 we covered UI testing and WatiN. Because WatiN is just a port of Watir they are fundamentally the same and as such the same concept applies.
As you want a new browser instance for each scenario, you should use the 'Before' step. This is executed before each scenario is run:
require 'spec' require 'Watir/ie' Before do |scenario| BROWSER = Watir::IE.new_process end
To make sure you don't leave lots of browser windows open, use the "After" step to close the window:
After do |scenario| BROWSER.close end
You can now fill in the implementation of each step and interact with the browser. Your Given block needs to direct the browser to a particular page:
BROWSER.goto("http://www.google.com")Your When block searches for a particular term. The first thing is to set the field name to the value you obtained from the regular expression from your story. After you have entered the search term, click the button:
BROWSER.text_field(:name, "q").set(arg1) BROWSER.button(:name, "btnG").click
Finally, you need to verify that the results include the correct value expected. This is done by obtaining the HTML and using the include? method which returns a Boolean. You can then use the should extension property to verify that it equals true.
BROWSER.html.include?(arg1).should == true
After you have your plain-text scenario and the steps created in Ruby, you can execute it to determine if Google is working as expected. The results of the execution are shown in Figure 6-17.
Figure 6-17. Executing Cucumber
In this case, everything passed as expected. A summary is given of the scenarios executed together and how many steps passed:
- 1 scenario (1 passed)
- 3 steps (3 passed)
If you have a step which failed, there will be more information written to the console, as shown in Figure 6-18.
Figure 6-18. Executing Cucumber with failing scenario
Cucumber also includes the ability to take the plain text story, together with the execution results, and output an HTML report as shown in Figure 6-19. This can be placed on a file share — in the build output — or integrated into your continuous integration dashboard for everyone to access. Because it's readable, everyone should be able to understand the correct state of the project and which parts of the system have not yet been implemented.
Figure 6-19. Cucumber HTML report
To execute, you need to set up Cucumber and Watir. Ruby has a packaging system called RubyGems. To install libraries such as Cucumber and Watir you can use RubyGems to download the library and any dependencies and documentation, and install it into the correct location for you to use. The commands you need to use are here:
gem install cucumber gem install watir
In terms of directory structure, you should generally follow this layout:
/spec
/<storyname>
<name>.feature
/step_definitions
- steps.rbThe <name>.feature file refers to the file containing your plain text with steps.rb.
Finally, within the spec root directory you need a cucumber.yml which stores profiles of different settings. This allows you to store ways of executing it via command line, produce different reports, and execute different sets of tests without having to re-enter the command:
default: —format profile .
Execution is now a case of simply entering "cucumber" from the command line when you're in the spec directory.
When it comes to testing your own ASP.NET application, the fact that Cucumber and Watir are Ruby-based does not hold you back. As demonstrated in Chapter 5, the beauty of ASP.NET is that after it has been rendered by the browser you can use frameworks to interact and verify it is working in exactly the same fashion as you did with Google in this example and WatiN in Chapter 5. Because these are still UI tests, the concepts discussed in Chapter 5 still apply — you are taking advantage of Cucumber and the Ruby syntax to improve how you write the tests and collaborate on the system. Combining Cucumber with automated UI tests is a very effective way to do your customer acceptance tests, taking advantages from both worlds. The fact that you are using it against an ASP.NET application is irrelevant.
However, Cucumber doesn't just need to be used for testing via the browser. It can also test at the object level. However, because your objects are C# based, this is not possible with the same Ruby language. However, Microsoft has created IronRuby which allows you to take the Ruby language but interact with C# objects in a native way. The result is that you can use frameworks such as RSpec and Cucumber to test C# applications. Ben Hall wrote a series of articles for MSDN Magazine which covers this concept in more detail: http://msdn.microsoft.com/en-us/magazine/dvdarchive/dd535415.aspx.
Although Cucumber is Ruby-based and currently the most popular framework of choice, there are alternatives. Ben Hall is working on a C# based acceptance testing framework called xUnit.GWT (http://wiki.github.com/BenHall/xUnit.GWT) which aims to bring some of the advantages Cucumber offers but as C#.
6.1.6. Cucumber, Watir and WroxPizza
As with FitNesse, in order to provide more context around how to use Cucumber, we wanted to describe how the concepts could be applied to the WroxPizza. The first stage is to write the feature and scenario within a file called "list_products.feature" within a directory called features.
The contents of this file are as follows. We first define the feature we are currently testing; in this case we are testing to ensure that the products are displayed correctly on the web page. Underneath the feature we define multiple scenarios which will be converted to steps and executed:
Feature: List products
In order to buy products
I need to be able to see a list of the products on the menu
Scenario: List products on the page
Given I am on the homepage
When I click on 'Menu'
Then I should see a product called "Thrufropaquentor"
And I should not see a product called "p1"The scenario defines how we are verifying the application. We have three main sections; the Given block will tell us what the precondition and setup for the scenario is — in this case we will just be on the homepage. The next step tells us the action we are going to perform — here we are going to click the Menu link. Finally, we have the verification where we say that we expect one problem to appear and don't expect another product.
When we use the console runner to execute the feature, it will say that the steps haven't been executed and provide you with the boilerplate steps to help start the implementation and write the steps. For this scenario, the step outlines was given as the following:
Given /^the homepage$/ do pending end When /^I click the Menu link$/ do pending end Then /^I should see a product called "([^"]*)"$/ do |arg1| pending end Then /^I should not see a product called "([^"]*)"$/ do |arg1| pending end
We can then copy the steps into a file called list_products.rb in the directory featuresstep_definitions. We can then start developing our steps. As we discussed before, we need to include the require statements to access rspec and the Watir framework. We have our Before block to start the browser and then implement each of our steps. Within each of the steps we use a very similar API to the one we discussed in Chapter 5 to interact with a web page:
require 'spec' require 'Watir/ie' Before do |scenario| @browser = Watir::IE.new_process end Given /^I am on the homepage$/ do @browser.goto 'http://localhost:62201' end When /^I click on '(.*)'$/ do |text| @browser.link(:text, link).click end Then /^I should see a product called "([^"]*)"$/ do |name| @browser.div(:class, 'Products').text.should include(name) end Then /^I should not see a product called "([^"]*)"$/ do |name| @browser.div(:class, 'Products').text.should_not include(name) end After do |scenario| @browser.close end
Personally, we find this code very clean and straight forward. For instance to verify the product is displayed correctly we gain access to the div with the class "Products". We then access the text and assert that it should include the name we defined in our scenario definition.
After we execute the feature, we can output the report as html as shown in Figure 6-20 and have a very nice and readable report defining which parts of the system are working as expected.
Figure 6-20. Wrox Pizza tested via Cucumber
We could also follow the similar approach to the one we took with WatiN in Chapter 5 and create models around our pages and structure of the website. With Watir and Ruby, the model would look like this:
class ProductMenu
def initialize(browser)
@browser = browser
end
def goto_homepage(url)
@browser.goto url
end
def click(link)
@browser.link(:text, link).click
end
def products()
@browser.div(:class, 'Products').text
end
def categories()
@browser.div(:id, 'categoryMenu').text
end
endThe implementations of the steps would then interact with the ProductMenu object.
require 'spec' require 'Watir/ie' require 'model/product_menu' Before do |scenario| @browser = Watir::IE.new_process @product_menu = ProductMenu.new @browser end Given /^I am on the homepage$/ do @product_menu.goto_homepage 'http://localhost:62201' end When /^I click on '(.*)'$/ do |text| @product_menu.click text end Then /^I should see a product called "([^"]*)"$/ do |name| @product_menu.products.should include(name) end Then /^I should not see a product called "([^"]*)"$/ do |name| @product_menu.products.should_not include(name) end Then /^I should see a category called "([^"]*)"$/ do |name| @product_menu.categories.should include(name) end Then /^I should not see a category called "([^"]*)"$/ do |name| @product_menu.categories.should_not include(name) end After do |scenario| @browser.close End
In this example, the benefit of having the abstract doesn't add too much extra in terms of readability. The Watir API, together with the added advantage Cucumber brings, means the code is already readable, and the step definition titles provide, context into what they are testing. As a result, having a clean abstraction model is less important. Instead, what is important is having a set of understandable and reusable steps. These form the API when writing your acceptance tests and as such these are what need to be readable, understandable, and most importantly, reusable.
6.1.7. Applying Acceptance Testing to Legacy Code
One of the great advantages of acceptance tests is that they are a lot easier to apply to legacy code than unit tests. Following the approaches we have described, you can start adding these styles of customer acceptance tests and providing some level of automated tests around your application without having large refactoring steps. The fact that it is legacy code is irrelevant as we are working at a much higher layer.
These styles of tests can actually provide an increased benefit for legacy code. By having a solid set of customer acceptance tests you can start refactoring your code and adding unit tests under the covers. The added customer acceptance tests will provide you with an additional safety net to ensure that nothing has broken during the refactoring stage. During the refactoring you should be writing integration style tests around the particular section you're working with; however, these added customer tests provide a good end-to-end coverage and safety net.