5.4. Problems with UI Automation
Testing the UI is a heated topic in testing circles. Many will argue that testing the UI is a waste of time as there are too many problems associated with this type of testing. You have learned that fragile tests increase maintenance tasks by requiring the developer to modify many tests each time the functionality of the system being tested is changed. UI tests are fragile tests and in most projects require the most maintenance, more than all other types of tests. UI testing is considered black box testing; as a tester you will supply the input and only verify the expected outcome without looking at the code that provides the outcome. With black box testing you are not expected to see how the inputs are processed or how the output is generated. There is no way to ensure that each code path is effectively tested (see Figure 5-1).
Figure 5-1. Black box testing
Fragile tests are tests that are very sensitive to the functionality of the system being tested. Fragile tests break often and require a great deal of maintenance. Fragile tests require the most maintenance of any other type of test. Because of the close relationship with the function of the system, and how frequent functionality can change, functional tests are considered to be fragile tests.
Fragility Due to Layout Changes. When a developer learns that automated testing of a UI can be performed, many of them jump right in to creating tests for existing systems that have not been designed with testing in mind. Tools such as Visual Studio Team Test, Selenium, and WatiN have recorders, which allow for creating scripted tests without much thought. The developer may develop hundreds of UI tests using these tools without any issues, but one morning they may come into the office, run their tests, and see a sea of red, meaning many test failures. This situation is common when a designer or another developer makes a change to the UI, such as changing a menu on the master page; suddenly all your tests are broken. You may spend an hour or so fixing your tests to get them running correctly again, only to find the changes to the menu were not finalized and your tests are broken again. Many developers get into this vicious circle and soon abandon the concept of UI testing.
The problem with the previous example could be solved by knowing exactly what is being tested in the UI and how. Is it important to test that the background color for the page is #0C1911, perhaps? It would be important to create a test for this if some function of the application being tested changed the background color to #0C1911. Another time you might want to create a test like this is if there is a process problem. Perhaps the background color of the application used to be #FFFFFF, and a recent feature changes the background to #0C1911 — developers are forgetting to conform to this standard. By creating a test to ensure that the background color is correct, you will save a few calls from the customer.
Timing Problems. There are two types of timing issues that developers and testers should be concerned about. The first type of timing issue occurs on the machine where the test is being run, and has to do with the testing framework and not specifically the system being tested. Functional tests are often not run on hardware that is similar to the hardware the production system is running in. Because many developers do not use a known set of production data for functional tests, the system will not perform as expected and application timeouts will occur that are unexpected. These are issues you may not see in production, but are causing your tests to fail.
The other type of timing issue to be aware of is that web applications perform differently under different connection speeds, and because of this, timing issues will occur. This issue tends to appear most often when you are testing the AJAX functionality of an application that uses AJAX. Because of these connection timing issues, it is essential that developers/testers create tests that account for this issue. It's very common that developers will add "Sleep's" into their tests, but there are better ways which we will explore later in the WatiN and Ajax section of this chapter.
Hard-to-Test UI Elements. As you have probably come to realize, testing a UI is hard but testing some elements are harder than others. Testing when a label appears is much easier than testing that a message on the screen fades in and then fades out at a certain speed. You may find yourself in a situation where you cannot create an automated test to test a certain functionality of the system. UI testing requires the developer/tester to think outside-the-box in many situations. Hard to test UI elements can be overcome with a combination of automated and manual confirmation techniques.
Tight Coupling to Other Parts of the System. For years, numerous sources have been trying to educate developers about separation of concern, but for some reason it is still commonplace for developers to insist on placing business logic inside of UI logic. As we learned in Chapter 2, it's difficult to create tests for systems that are tightly coupled, sometimes impossible without some sort of refactoring. Creating functional tests is just as hard for tightly coupled systems. Developers/testers will find themselves creating tests that should be created at the unit test level.
Unsolicited Events. Web applications can contain logic to raise events that are not based on user interactions. Examples could be when the price of a product reaches a certain amount the product should be displayed differently on the screen. Another example that most web developers have encountered is automatic user logic. In this situation the system will log out the user after a certain time period has expired. Events such as these are difficult to re-create using functional tests.
Many Ways In, Many Ways Out. Because of the nature of web applications, there are many ways to access pages and features inside the application. Did the user access the feature from a menu, a URL, button, or perhaps an external link? How many different ways can a user access a particular feature of your system? Because access to a feature is acyclic, only a finite number of paths exist — should we test each path? Depending on the size of your application, you could spend a great deal of time creating functional tests to test the paths to each function of a system. This topic will be discussed later in the chapter, but will always pose a problem for functional testing.
Infinite Input Sequence. As a general rule-of-thumb, web applications allow users to enter form data in any order they wish. The application will have a default tab order, but the sequence is the choice of the user using the system. As forms become more complex, the number of sequences of how the data can be entered onto the form will increase exponentially. If conditional logic has been applied to fields, such as field 1 must be greater than field 2, then what would happen if field 2 is entered before field 1? As addressed in the previous UI testing issue, testing for these situations can generate a great number of tests that will need to be maintained.
Random Failures. When performing UI tests, it is common to receive random errors that are hard to reproduce. It's common for these errors to be timing issues, or issues with the machine the test is running on. Many developers will ignore these errors and either delete the test or let the test stay red. Not resolving these types of errors make their UI test suite lose value. It's important to track down these random errors and resolve them.
The Pareto principle (also known as the 80–20 rule) states that roughly 80 percent of the effects come from 20 percent of the causes. With automated UI testing, expect 80 percent of the benefit to be obtained from 20 percent of the tests. Don't waste time on complex tests at the expense of high-volume tests.