22   Automate Acceptance Testing

“All right, so your unit tests verify your code does what you think it should. Ship it. We’ll find out if it’s what the customers want soon enough.”

You’ve worked with the users to develop the features they want, but now you need to make sure the data they’re getting is correct—or at least correct from their point of view.

Andy was on a project a few years ago where their industry standard considered 12 a.m. as the last minute of the day, and 12:01 a.m. as the first minute of the new day (typically, business and computer systems consider 11:59 p.m. as the last minute of the day and 12 a.m. as the beginning of a new day). That small detail made a huge difference when it came to acceptance testing—things just wouldn’t add up otherwise.

Critical business logic needs to be tested independent of the rest of the application, and the users need to approve the results.

But you don’t want to drag the users in to check your results with every unit test. Instead, you want to automate the comparison of your customers’ expectations with your actual implementation.

One wrinkle makes these sorts of acceptance tests different from usual unit tests. You’d like the users to be able to add, update, and modify their data as needed, without having to learn to write code. You have a couple of ways to go about that.

Andy has used a number of schemes that relied on data in flat files that users could edit directly. Venkat recently did something similar using an Excel spreadsheet. Depending on your environment, you may have something that’s already a natural fit with your users (be it data in a flat file, an Excel spreadsheet, a database, or another form). Or, you might consider an existing tool that does most of this for you already.

FIT,^[23] the Framework for Integrated Testing, is a helpful technology that makes it easier to use HTML tables as a mechanism to define and compare test cases and values.

Venkat says:

Getting Acceptance Data

One customer was using a pricing model he had developed in Excel. We wrote tests that compared the output of our project’s pricing code with the output from his Excel spreadsheet and then corrected our logic and formulas as necessary. This gave everyone confidence that critical business logic for pricing was correct and gave the customer the ability to easily modify acceptance criteria.

Using FIT, the customer defines a new feature with examples of its use. The customers, testers, and developers (based on the examples) then create tables that describe the possible input and output values for the code. The developer can then create test fixtures that compare the examples in the FIT tables with the output of the code being developed. The result of the tests—successes and failures—is displayed in an HTML page, making it easy for the users to see.

If your domain experts give you algorithms, calculations, or equations, provide them with a way of testing your implementation in isolation (see ​Attack Problems in Isolation​). Make those tests part of your test suite—you want to make sure you continue to provide the correct answers throughout the life of the project.

Create tests for core business logic. Have your customers verify these tests in isolation, and exercise them automatically as part of your general test runs.

What It Feels Like

It feels like cooperative unit testing: you’re still writing the tests, but someone else is providing you the answers.

Keeping Your Balance

• Not all customers can give you correct data. If they had correct data already, they wouldn’t need the new system.

• You may discover previously unknown bugs in the old system (be it computer or manual) or genuine issues that didn’t exist before.

• Use the customer’s business logic, but don’t get bogged down documenting it extensively.