Test Suite Maintenance
What most of us probably do not understand is that a suite of automated tests is really a software system itself with the same problems faced by the system it is designed to test. It is prone to errors and extremely sensitive to changes. So, any time a client-server system is tested with automated test scripts, you are really dealing with two systems that have to be maintained. This doubles the maintenance problem.
I worked with one Visual Basic programmer who had previously constructed and maintained an automated suite of test cases for a client-server system. The problem he encountered was that the features were never frozen, and he was constantly updating the test scripts to keep up with all of the changes. Of course, he never caught up. He swears he will never do testing again!
Steve Fuchs, the program manager for Microsoft Test (MS Test is currently a component in Rational Software's arsenal of software development and testing products), sees the macro recorder capabilities of the major test tools as causing several maintenance problems because testers can assume certain things (1):
That the software product will change during its life cycle
That, if it is successful, it will be reproduced in other languages
That the next version of the product will have a better user interface
That there will be less time to test subsequent versions of the product
He argues that the assumption on the part of testers that the product will change leads to scripts that will contain invalid events. He says the effort to isolate and rerecord those portions of the script will be substantial. He goes even further to say that the scripts will contain very little context information about the events, which makes maintenance harder, and that the scripts will contain hard-coded function calls that will require extensive updating for even minor user interface changes. He concludes that even simple changes can affect 50% to 90% of the test cases (2).
Fuchs arguments are valid because new system releases do impact automated test suites. Recorded test procedures will most likely require some updating when new versions of the system include changes to any of the system layers (interface layer, data layer, or function layer). Think of it this way: When we change a software module, we must ask what other modules are affected by this change. Now, when we change a software module, we must also ask what test scripts will be affected by the change. Automated test suites add a new constraint on the system itself. James Bach (in the USENET news group comp.software.testing) says, “Automation systems constructed from scripts are very complex and hard to maintain. The more complex a system is, the more likely it is to fail.”(1) The implication here is that the test cases themselves can be prone to error. So what do we do—write test scripts to test the test scripts?
Another problem is the impact of software and/or hardware platform changes. Again Bach says, “Any change in such a platform will cause wide-spread failure of test automation, unless the change was specifically anticipated.” Bach has experienced problems with test automation stemming from DOS updates and from changes to a Novell network. He was forced to add additional platform-related processes to the test automation in order to make it work. He found that only subtle configuration differences could cause automated test suites to fail.
To be fair to the automated test tool vendors, we must state that their products have come a long way since they were introduced. Many of the conditions that caused earlier test suites to fail are now trapped and logged, and the testing process continues. The test suites are much more robust than before.
An important piece of advice is to use plenty of comments. Insert a comment heading at beginning of the script for complex test procedures. Use a product such as Cyrano for documenting test suites.