Chapter 4. An Introductory Acceptance Test
“If you don’t know where you’re going, you will wind up somewhere else.”
Yogi Berra
An example of an acceptance test is presented, along with four ways that you can execute an acceptance test.
A Sample Business Rule
Here is an example from a previous project where Debbie and Tom created tests in collaboration with the customer. The business representative, Betty, presented the two of them with a business rule for giving discounts that she had obtained from one of the stakeholders. The stakeholder wanted to give discounts to the firm’s customers based on what type of customer they were. Debbie had already completed implementing a previous requirement that determined the customer type. Here’s the rule that Betty gave them:
If Customer Type is Good and Item Total is less than or equal to $10.00,
Then do not give a discount,
Otherwise, give a 1% discount.
If Customer Type is Excellent,
Then give a discount of 1% for any order.
If Item Total is greater than $50.00,
Then give a discount of 5%.
This rule may seem clear. It uses consistent terms, such as Customer Type and Item Total. Debbie and Tom had previously gotten from Betty the definitions of those terms [Evans01]. For example, Item Total did not include taxes or shipping. But even with that consistency, there was an issue. Tom and Debbie looked at the rule and tried to figure out what the discount percentage should be if a customer who is good had an order total greater than $50.00. So Betty, Debbie, and Tom made up a table of examples.1
The answers in this table of examples are going to be used to test the implementation. The first two rows show that the limit between giving a good customer no discount or a 1% discount is $10.00. The “less than or equal to” in the business rule is pretty clear. The tests just ensure that the implementation produced that result. The ?? was put after the 1 in the third example because it was unclear to the triad whether that was the right value. To what type of customer did the last statement in the rule apply?
The fourth row indicates that the discount for an excellent customer starts at the smallest possible Item Total. The fifth and sixth entries show that the discount increases just after the $50.00 point.2
Betty took this table back to the stakeholder. He looked it over and said that the interpretation was correct. He did not want to give a 5% discount to good customers. So ?? from that result was removed from that cell. There was now a set of tests that could be applied to the system. The correct discount amount test is not just a single case but includes cases for all possible combinations.
Tom suggested other possibilities. For example, what if Item Total was less than $0.00? Tom asked Betty whether this would ever happen. She said it might be possible, because Item Total could include a rebate coupon that was greater than the total of the items. So Tom added the following possibilities.
Tom explained that it didn’t seem right to apply a discount percentage that would actually increase the amount that the customer owed. Based on this example, Betty went back to the stakeholder and confirmed that the percentage should be 0% if Item Total is less than 0 for any customer. So the table became as follows.
These examples were the acceptance tests for the system. If Debbie implemented these correctly, Betty would be satisfied. Now it was a matter of how Debbie and Tom were going to use these tests to test the system.
Implementing the Acceptance Tests
Tom and Debbie needed to apply these tests to the implementation they were developing. There were at least four possible ways to do this. First, Tom could create a test script that operates manually at the user interface level. Second, Debbie could create a test user interface that allows her or Tom to check the appropriate discount percentages. Third, Debbie could perform the tests using a unit testing framework. Fourth, Tom and Debbie could implement the tests with an acceptance test framework. Following are examples of how they could use each of these possibilities.
Test Script
In this case, the program has a user interface that allows a customer to enter an order. The user interface flow is much like Amazon or other order sites. The user enters an order and a summary screen appears, such as the one in Figure 4.1.
Figure 4.1. Order Interface
What Tom would have to do is to create a script that either he or Debbie would follow to test each of the six cases in the Discount Calculation table. He might start by computing what the actual discount amount should be for each case. Unless the Order Summary screen shows this percentage, this value is the only output Tom can check to ensure the calculation is correct. Here is an addition to the table that shows the amounts he needs to look for.
The script would go something like this:
- Log on as a customer who has the rating listed in the table.
- Start an order, and put items in it until the total is the specified amount in the Item Total column on the test
- Check that the discount on the Order Summary screen matches Discount Amount in the table
Then the test would be repeated five more times to cover all six cases. Either Tom or Debbie would do this once the discount feature and order features are implemented. This test should be run for all possible combinations. That would have been more difficult if there were more discount percentages for more customer types. There’s another possible way to run these tests.
Test User Interface
To simplify executing the tests, Debbie could set up a user interface that connects to the discount calculation module in her code. This interface would be used only during testing. But having it would cut down on the work involved in showing that the percentage was correctly determined. The interface might be a command-line interface (CLI) or a graphical user interface (GUI). For example, a CLI might be this:
RunDiscountCalculatorTest <item_total> <customer_type>
And when it is run for each case, such as
RunDiscountCalculatorTest 10,00 Good
It would output the result
0
A GUI, such as what’s shown in Figure 4.2, might be connected to the CLI.
Figure 4.2. User Interface for Testing
Regardless of whether it is a GUI or CLI, the user interface has penetrated into the system. It exposes a test point within the system that allows easier testing. Here’s an analogy showing the differences between this method and Tom’s original test script. Suppose you want to build a car that accelerates quickly. You know you need an engine that can increase its speed rapidly. If you could only check the engine operation as part of the car, you would need to put the engine in the car and then take the car on a test drive. If you had a test point for the engine speed inside the car, you could check how fast the engine sped up without driving the car. You could measure it in the garage. You’d save a lot of time in on-the-road testing if the engine wasn’t working properly. That doesn’t mean you don’t need to test the engine on the road. But if the engine isn’t working by itself, you don’t run the road test until the engine passes its own tests.
If you’re not into cars, Figure 4.3 shows a context diagram. The Order Summary screen connects to the system through the standard user interface layer. The Discount Percentage user interface connects to some module inside the system. Let’s call that module the Discount Calculator. By having a connection to the inside, a tester can check whether the internal behavior by itself is correct.
Figure 4.3. Context Diagram
xUnit Test
The next way to perform the testing is to write the tests for the Discount Calculator in a unit testing framework. The framework used is usually in the language that the program is written in. There is a generic framework called xUnit that has versions for many programming languages. Here’s a sample of what these tests look like in Java using Junit [Beck01]. The test would look similar in TestNG [Beust01], but the order of the parameters would be reversed:
Any time there is a change in the examples that Betty and the stakeholder use to explain the business rule, Debbie may want these tests to conform to the changed examples. That’s a bit of waste. The next testing framework can eliminate that waste.
Automated Acceptance Test
Betty, Debbie, and Tom agreed that the examples in the table accurately reflected the requirements and there would be less waste if the table did not have to be converted into another form for testing. Several available acceptance test frameworks use tables. Some examples are in Appendix C, “Test Framework Examples.” With these frameworks, you describe the tests with a table similar to the one for the example.
The following test table works in table-based frameworks, such as the FitNesse and Fit frameworks. A similar style table can be used in narrative-form frameworks, such as Cucumber.3 The table looks practically like the one that Betty presented to the stakeholder.
Now when the table is used as a test, the Fit/FitNesse framework executes code that connects to the Discount Calculator. It gives the Discount Calculator the values in Item Total and Customer Rating. The Discount Calculator returns the Discount Percentage. The framework compares the returned value to the value in the table. If it agrees, the column shows up in green. If it does not, it shows up in red. The colors cannot be seen in this black-and-white book. So light gray represents green and dark gray represents red. The first time the test was run, the following table was output.
With the results shown in the table, it was apparent there was an error in the Discount Calculator. Once it was fixed, Betty saw the passing tests as confirmation that the calculation was working as desired.
An Overall Test
If the discount test is applied using one of the last three forms, there still needs to be a test using the order interface. This ensures that processing an order is correctly connected to the Discount Calculator. The script for an order would be run for a couple of instances. But unless there was a large risk factor involved, the script might just be executed for a few cases, such as the following.
Testing Process
The acceptance test is the original table that Betty, Tom, and Debbie developed to clarify the business rule. This acceptance test can be used at four different levels, as described earlier in this chapter. Because the acceptance test was customer supplied, all four levels are considered acceptance tests in this book. The last two forms are automated by their nature. The second form—an interface to the Discount Calculator—can be automated. The test for an order could also be automated with a little more effort. However, you should still check it manually as well.
Passing the acceptance tests is necessary but insufficient to ensure that the system meets the customer needs. Other tests, such as those for quality attributes and usability (described in Chapter 3, “Testing Strategy”), also need to be passed. See [Meszaros02] for more information.
Summary
• Examples of requirements clarify the requirements.
• The examples can be used as tests for the implementation of the requirements.
• Tests for business rules can be executed in at least these four ways:
• Creation through the user interface of a transaction that invokes the business rule
• Development of a user interface that directly invokes the business rule
• A unit test implemented in a language’s unit testing framework
• An automated test that communicates with the business rule module