3-1. Choose a Proper Set of Tools for Your Needs

In my practice, I met two different types of people. Some tried to solve all their problems with the help of one tool , while others, for each new task, took something new. All of them had their own arguments. The first believed that since they had experience working with one particular tool, it is with its help that they would solve the problem most quickly. Others held the view that for each task there was the most suitable tool, and it was necessary to use it, even if no one in the team had experience with this tool.

As is usually the case, the truth is somewhere in the middle. Each of these types of people faced various difficulties. The decisions of the first were sometimes too bulky and complex, and it was very difficult for anyone, other than the author, to support them. The problem of the others was that it was very difficult to set up the working environment, and it required a lot of time and consultations with the author.

The most obvious is the following solution: find yourself a tool that will cover most of your needs and find one to two additional tools for everything else. Do not arrange a “zoo” of languages, tools, and libraries in your project, but do not try to solve everything with a single tool .

3-2. Do Not Automatically Register Bugs from Scripts

If the automation tool has integration with a bug tracker, one day you might have a wonderful idea: to automatically create a bug if there was an error in the script. Usually this makes no sense, since most errors in automated scripts are due to imperfections of the scripts themselves, or due to random events in the system on which tests are run. Errors also can be from use of the automation tool, due to the fault of the test author, from incorrect configuration, and only occasionally due to a real problem in the application. This is one of the reasons why automatic bug creation is a bad practice in test execution automation.

The second reason to avoid automatically registering bugs is that it is very difficult to write correct steps to reproduce a problem. You can’t just write “run a test like this and at the end look at the error log.” Programmers need clear instructions to reproduce the error that they can do on their computer in debug mode.

There is another solution. You can create tasks in the bug tracker automatically if an error occurs, but the person responsible for the error investigation should be the author of the test. If, for instance, tests are run at night, then in the morning the tester will first look if he has received new bugs during the night that are worth investigating first.

However, in this case, there are hidden pitfalls as well. If we have a problem with the test environment and no tests have passed, then for each of them a separate bug will be created. As a result, in the morning each tester will have a huge number of new tasks created overnight. If this happens on frequent occasions, after a while, people will just start to ignore such notifications. In this case, it makes sense to find the “golden mean”: for instance, generate a template for a new bug, which will then be viewed manually, edited if necessary, and created in a bug tracker.

3-3. Do Not Chase After a “Green Build” in the Prejudice of Quality

Avoid the need to comment out a problem section of test code. Also avoid temporarily changing an expected result just so that a test runs to completion without appearing to fail. Here’s an example of the sort of conversation that can lead to these two mistakes:

• ‘Why do we have red build for three days?’

• ‘Tests fail on a known issue.’

• ‘Then fix it!’

• ‘Not this week, maybe the next one, the problem is not critical there.’

• ‘If the problem is not critical, why is the build red? Make sure it’s green!’

The worst thing that a tester can do in this situation is to comment out a problem verification or worse : to temporarily make its current result an expected one. Often such changes from temporary become permanent, and in a year it is already impossible to understand why a deliberately wrong result is considered correct, or why the necessary verification is commented out.

Certainly, if a build is red for a long time - this is a problem, because people get used to this and feel normal. However, solutions to this problem need to be controlled. For instance, you can add a failing test to be quarantined. Then the results of the build will clearly show that there are quarantined tests. Another option is to disable a failing test in a way that is noticeable in the results. You can specify the number of the bug in the test comment due to which the test is disabled. The bug number can also be used to notify any change of the status of the bug to the person who disabled the test.

If you take the approach of disabling tests , then it makes sense to determine some threshold percentage that you choose not to exceed. For instance, you might consider that having 95% of the tests are working tests as the norm. Then if you noticed that more than 5% of your tests are disabled, you can consider that high number of disabled tests to be a problem.

3-4. Learn the Tool You Work With

Modern development environments provide great capabilities ; however, many autotesters do not even use the main ones. It may be just simple actions we use several times every day or hour. I have repeatedly seen how people use the mouse to select the menu item Edit | Comment Block or commented several lines one by one, instead of pressing a simple key combination and comment out the entire selected block. In other cases, the functionality may be present in the editor, but its use may not be entirely obvious. You may use your IDE for years, having no idea that it contains something you needed from the very beginning.

In addition to simple things (such as working with code), editors can provide more advanced features : automatic code refactoring, connection of additional plug-ins, setting up different types of editor for normal mode and debugging mode, and much more.

Therefore, first of all, learn your tool – try everything you can find in it. And second, if you are missing something, try to find a solution: first in the built-in help system, then on the Internet. It is quite possible that what you need is simply called differently, and you just do not know about it.

3-5. Make Use of Version Control Systems

If you have not used a version control system before, you must begin right now. If you think that such a trifle as automatic tests shouldn’t be stored in version control system, you are mistaken.

You can’t just store your tests on your hard drive, as the tests become more and more time consuming, and one day you can lose them all if the hard disk goes out of order. Simply creating a copy of tests on another computer is a primitive analog of a version control system, so spend one day studying a real version control system and then begin using it, gradually improving your knowledge.

Usually, developers already use some kind of control system, so just start using that same system as well. You can store your tests in the same place where programmers store their source code, or use a separate project for tests. In various projects, this or that approach will be more convenient. Using the same system as your developers means that you can learn from them, and possibly vice versa.

However, keep a close eye on what exactly you put into your version control system. Some programming languages (even scripting languages) can generate binary files to speed up the work. These files are copies of scripts, but they do not need to be saved, since they are generated automatically and for the version control system are just rubbish.

Also, some tools can create separate files in which user settings are stored. Such files should not also be stored in the version control system , since everyone likes to customize the development environment for themselves and storing these settings in the version control system can interfere with other project participants.

3-6. Avoid Custom Forms

In some automation tools there is such a feature as custom forms . Especially often they are found in commercial instruments.

A custom form is a normal window that appears at a certain point during the test run and is intended for entering any information by user. In some projects, these forms are used to enter test parameters that will be used (for instance, server address, user name, etc.).

In most cases, the use of such forms is not recommended as well as any user interaction with automatic scripts. Tests should be run regardless of the person’s presence, so that it can be done automatically, for instance, during nonworking hours (at night or on weekends). To stop the test at the right time, you need to use debug mode and breakpoints.

As for the test startup parameters, they can be stored in configuration files (for instance, ini, xml, and others). Changing these parameters in a file is as easy as entering them into a custom form. If different parameters are used on different computers (for instance, the user name on the current machine), you can use several parameter files, each of which corresponds to the name of the current computer.

3-7. Simplify Everything You Can

You work with a lot of things: tests, code, test environment, virtual machines, SQL queries, test data, reports, and more. It is very important that everything that you work with is simple. This is important both for yourself and for those who will work with it after you. So if something you are working with seems to be complicated, simplify it!

If you have a difficult test and you have to think through for a long time what exactly it does - break it down into several simple tests. If you have complex code and you have to debug it for a long time, simplify this code (for instance, break it into several separate elementary functions). And so on.

Anything that seems difficult or requires a lot of additional actions needs to be simplified. If you solve a problem and find a solution - do not rush to implement it, try to find an easier solution. Often the first way to solve the problem is not the simplest. By thinking for another half an hour, you can save yourself a half-day job.

One of the best ways to find a simpler solution to the problem is to consult with someone who works with you. An outside point of view can give impetus to thinking in a different direction.

For instance, you are looking for a way to automatically close some unnecessary applications on your computer before you start the nightly test run. Instead of writing complex code to find and close all possible applications, you can simply force the computer to restart before running the tests, and then at the time of the test startup, you will not have anything open except for your own tool that runs the tests.

Another example of simplifying the task is described in Chapter 2, in the section “What Should Not Be Automated?” Instead of spending several hours every week on fixing the tests, we simply perform part of the task manually, spending a few minutes.

These examples show that the best solution is not always to improve the existing approach. The best solution can be found if you transfer the problem to a different plane.

3-8. Automate Any Routine

I often meet strange people who work in automation , but can perform other daily tasks in the most inconvenient way. For instance, to start the program they open the Explorer, find the necessary folder in it, and run the program. They do this several times a day, rather than just make a shortcut on the taskbar or start by pressing a certain keyboard combination.

In another case, to start a program with many parameters from the command line, the person opens the console, goes to the desired folder, and manually writes the program name and all its parameters, although the same can be done 10 times faster if you write a batch file.

There are many such examples: people enter huge data sets manually, instead of writing a simple script and executing it; use the mouse where it would be 10 times faster to use the keyboard combination; use simple programs such as Notepad instead of more advanced editors, etc.

Your automation tool is not the only means by which you can automatize routine tasks. Use everything available! Once you have a regular routine task, think about how it can be speeded up or automatized. Using such approaches will not only improve your work speed, but will also improve your knowledge of the operating system you work in.