5.13. How Much Should We Automate?

One of the differences between engine level and UI level tests is the amount you should automate. Developers and testers who are new to creating automated UI tests tend to get carried away, and they may create tests for every little UI element, but not gain very much benefit from these tests. Due to the nature of UI tests, these tests tend to have a higher maintenance cost. There are some tasks that computers can do well, such as creating a large text document and then running that text document through a program to apply styles, all the while trying to break the app that is applying the styles. Then there are certain things that manual testing is more suited for, such as testing whether a particular page looks aesthetically pleasing when it is rendered for printing.

Many of the issues that occur with automated UI testing boil down to the fact that developers and testers test things that should not really be tested. Properly designed applications wrap many business rules into a thick layer, which the user interface will then encompass. To test a business rule in the UI requires a lot of clicking, selecting particular items and entering values, which makes maintenance a nightmare. Which brings us to the next question: what types of behaviors should you automate?

• Session Rules. Many applications have business rules that apply to features that a user session or role will have access to. An example of such a business rule would be that users in the administrator role will have access to the reports section of the website.

• Workflows. Most applications contain some type of workflow that can be tested. On a simplistic level, a workflow could be that a user is prompted for a password before gaining access to an application. On a more complex level, testing that the order of the steps required for accepting a credit card are correct is also considered a workflow.

• Key Business Scenarios. You will get the most value from automated UI testing when you test key business scenarios from end-to-end. Most business applications do not have very many of them, so the amount of key business scenario tests you will have will be small. Take for example an application that enables users to order pizza online. There are two key scenarios that should be tested end-to-end in this application. Test the path that takes the users through the "new user registration" feature, and test the path that allows the user to order the pizza. Automated UI tests that test key business scenarios should be run at least once every day.

You test session rules, workflows, and key business scenarios because the user interface is how the user interacts with your application. A very small CSS error could prevent a user from logging into the application. These types of errors are extremely embarrassing and can be costly, when found by the users. These errors are easily spotted when using automated testing. By only automating these three types of scenarios, it cuts down on the amount of maintenance that will need to be performed on the tests.