1.7. When Should Testing Be Automated?

The discussion about whether and when testing should be automated has been going back and forth for many years. When considering this argument, it is important to remember that testing actually covers a number of different techniques, and as such, this question cannot result in a simple yes or no answer. Instead, you need to look at every feature and type of testing which needs to be performed and decide on a case-by-case basis. But how do you decide?

First, you need to consider what you are asking. You are trying to decide if a test you need to perform should be done using scriptcode and made into an automatable, repeatable test or if you want to manually perform the test each time you change something. You then need to consider the type of testing you want to perform and consider the cost of automating the process as opposed to the cost of repeating the test. You need to consider the cost of maintaining the automated test compared to our manual test. Finally, you need to think about the cost of it going wrong and a bug ending up in production. With these factors considered, you can decide if you think the cost of automating is worth the cost and effort compared to manual testing.

With that in mind, let's consider some initial problem domains and the general thought process when faced with the question of whether or not to automate.

When unit testing, our main focus will be on the business logic and process. Arguably, this is the most important part of any application as this defines what happens and how it will happen. If this logic is incorrect, it can cause a lot of pain because it generally will only appear in production. We have already discussed that bugs found in production are the most expensive to fix. Having the production that the unit tests in place provide against bugs making their way into production is definitively an important factor. With business logic, if you have designed your system in a testable fashion then you should find that the tests are easy to write and maintain. If you need to make changes to the underlying logic due to the design of the system being testable, then the cost of maintaining the costs should be minimal, especially when combined with the lower cost of ownership of the core system.

Though your automated tests will verify the business logic is correct, you could also manually test this. However, there are a few major blockers to this idea. Firstly, to test the business logic you would need to go via a UI, either the main applications or the custom wrote UI. This has two possible problems: the main application might have bugs, which blocks you from testing the logic. Or, it hasn't been written yet. Combine this with the fact that after every code change you would need to manually re-enter all these values and edge cases into the UI and verify the result was as you expected based on either a specification or previous experience. As far as we're concerned, this makes automating the business logic essential.

The next major parts of most applications are the data access layer and interacting with a database. The levels of interaction with a database can vary greatly. For example, you could be using the database as a simple data store which your application will pull all the information required from and process within the business logic. The other case is that all your business logic actually takes place within stored procedures and your application simply pulls the data and renders it on-screen. In either case, you have similar issues to what you had with the business logic. If you were to manually test the database, you would need to write some custom queries or UI to execute your logic which is expensive in itself. If you were to automate you could insert the required test data automatically and test your logic, and then verify the results in a more effective fashion.

However, database tests have a series of problems. For starters, the tests are a lot slower than if they were simply running in memory, something that we'll discuss in more depth in Chapter 4. Secondly, if our database schema changes, the tests are more likely to break without re-factoring tools to support us as with business logic. Even with this additional overhead, automating your database tests generally still have a lower cost than manual testing.

The final level is the GUI. This is the area that causes most debate about whether it should be automated or not. The GUI has some interesting problems when it comes to automated testing which we will dive into in Chapter 5. The main problem is that when you are automating the UI you are verifying for correctness; however, because you are already testing the business logic, you are actually only verifying the behavior of the UI. Verifying this behavior is an expensive process because you are trying to replicate the UI's logic as steps within the test code. Although it is possible and certainly can lead to success, it can be expensive. If the behavior changes then it means the tests also need to be manually updated.

If you compare this to manual testing, then you are still performing the behavior verification, and, as before, are also performing usability testing, — verifying the rendering is correct — and checking text for spellings. All this is very expensive to automate, and as such, doing it manually could prove to be more effective as you also generally need to perform it less. There is some benefit to automating the UI behavior tests; it can be made more effective and more appealing, which we will discuss in Chapter 5.

But remember, if you don't automate your tests then you will manually need to perform each test for each release. If you think this will be a painful and costly process, then automate.