1-1. Do Not Use Record & Play in Real Projects

Most automation tools (especially commercial ones) have Record & Play functionality: the ability to automatically record certain actions and then play them back just by clicking on the Play button. The seeming simplicity of this functionality is a well-known trap for novice software testers.

Record & Play looks very nice in advertising videos and presentations, but when you actually work, such scripts are not recommended to be used, since they only complicate the process. A recorded script does not use variables, loops, and conditions. Automatic names of created procedures and functions are not usually informative, and all actions are recorded in one function (which can be huge). Very often recorded lines of code are so long that they do not fit on the screen.

Overall, code generated by recording user actions tremendously complicates the support of your tests and their understanding. In the case of a large number of actions, such tests are usually easier to be recorded anew than to be changed even a little bit.

A recorded test must be edited and put in order, and it is even better to write the code manually from scratch. The less automatically recorded code is in your tests, the clearer and easier such tests are to maintain.

There are, however, several cases when the Record & Play can be used to good effect:

• When studying a new automation tool, recording is the easiest and most convenient way to get an idea of how the tool works. While you are studying the capabilities of the tool, Record & Play is your best friend.

• If you have a single task and you are sure that you will never need a script later, feel free to use the recording as well.

• For some controls (for example, custom context menus), it can be difficult to get their identifiers and understand how to work with them. In this case, Record & Play will help you to better understand how to work with this control.

In other cases, it is better to use a framework that will help you to create tests faster and more efficiently than recording them automatically.

1-2. Do Not Use Pauses

When you use any application, there are situations when you need to wait for the end of an action, for example, reading data from the database, searching, loading all page elements, etc.

In such cases, a recorded script “doesn’t know” that it is necessary to wait and tries to perform further actions, although the application is not ready yet, and the result is a test that fails with errors. Junior automation engineers use a pause in such cases:

sleep(5)

The wait time in this case is five seconds, and typically the timings are just hardcoded approximations that are based on the current application speed.

Over time, such delays appear everywhere in the tests. And in that proliferation lies the seeds of trouble.

Imagine that at some point the application starts to spend a little more time, and the automation engineer has to make a huge number of hot fixes in a variety of tests. That process is time consuming and error prone. After several similar, mass hot fixes the tester comes to a new solution. He declares a global variable and uses it everywhere:

WAIT_TIME = 5
...
sleep(WAIT_TIME)

Now we just need to change the delay value in one place so that the wait time is increased everywhere. This approach has one big disadvantage though. If the application starts working faster, our tests will still wait for the same constant time. Since pauses are placed in different places, the total time that the tests work will be much longer than it could be.

The right solution in such cases is to wait for an object or object property. Any action of the application ends with some event: a button that was previously disabled becomes enable (for example, Next button), or a new control appears that was not previously available. Or, on the contrary, some element or window can disappear from the screen (for example, label “Wait for data loading”). Here we have to attach to such events using wait functions . Now the code will look like this:

MAX_WAIT_TIME = 5
...
wait_for_object(object_name, timeout=MAX_WAIT_TIME)

Function wait_for_object can already be provided by the automation tool, or it will have to be written by you. It checks the existence of the object with a certain interval (for example, once per second) and returns true if the object appeared. If the object does not appear within the time specified by the timeout parameter, the function returns false.

Thus, if the object we are interested in appears in a second, then the wait time will be 1 second. If the object appears after 3 seconds, then the function will work only 3 seconds, etc.

Of course, you can’t completely abandon the pause. For example, in wait functions such a pause is mandatory. After each verification of the existence of the object, it is necessary to make a small delay; otherwise our script will use too much of CPU time, thereby slowing down the speed of the application under test. As a result, a wait function will look something like this:

function wait_for_object(object, timeout)
{
 while(timeout)
  {
   if object.exists()
      return true;
    timeout--;
    sleep(1);
  }
return false;
}

Also sometimes there can be situations when an attempt to work with a control immediately after its appearance (or change of its state) leads to incomprehensible errors. In such cases, it is also possible to use small (not more than half a second) delays. For example, in the preceding function you can insert such a delay just before the return true line. However, any pause for a long time is a bad approach when writing tests.

1-3. Provide Exit by Timeout for Loops

Sometimes a test should be continued only after the occurrence of a certain event. For example, we need to wait for a file to be created, so we code as follows:

while not file_exists()
{ /* do nothing*/ }

If for some reason the file is not created, the script will loop forever, which is clearly not desirable. Therefore, in such situations it is necessary to provide a forced exit by a timeout:

timeout = 10;
while not file_exists()
{
  if timeout < 0
  {  break; }
  timeout--;
  sleep(1);
}

In this example, we decrease the value of the timeout by 1 at each iteration and wait for 1 second. After 10 seconds the value of timeout will be less than 0. As a result, we will exit the loop using the break instruction, thus protecting ourselves from the infinite loop.

In addition, using the sleep instruction, we reduce the load on the processor. Verification will be performed once per second, not thousands of times, as in the first example.

Perform an exit by timeout even in those cases when you are 100% sure that there cannot be a hang-up – because most probably you are wrong.

It makes sense to define similar timeouts in the form of constants at the level of the entire project, and not to set each time anew. For example, you can define two types of timeout: short and long. A short timeout (usually equal to a few seconds) will be used for operations that occur almost immediately, but it is necessary to wait for them to finish: for example, creating a file or deleting a directory, an error message appearance when you enter data incorrectly in a text box, and so on. A long timeout (usually equal to several minutes) is intended to wait for the completion of data loading, the appearance of a window, the response from the server, and so on.

1-4. Do Not Consider Test Automation as Full-Fledged Development

Let us be honest: though automation requires programming skills, nevertheless it is not a full-fledged development project. Automation is usually handled by less skilled programmers, since the work is much simpler and is used for the internal needs of the project.

In 99% of cases, you will not need most design patterns. You will not use transactions when working with databases, the volume of your test data is unlikely to approach the volumes of a real application, and you do not have to worry about the visibility scope of functions. You need general knowledge of object-oriented programming (OOP), the ability to write simple SQL queries, and the ability to run tests in debugging mode.

Therefore, there is no need to try every new approach that you learn about software development in general to the practice of writing automated tests. Do not try to show how smart you are. It is better to show the ability to write simple code that even a beginner will understand (it is quite possible that the beginner will support the tests after you).

Even if your project uses an approach such as behavior-driven development (BDD) or keyword-driven testing (KDT) , in which the code is written by one group of people, and the tests are created by another, it is still better to use simpler solutions.

Of course, there are very complex projects where automation is integrated with development, where the qualification of automation engineers is as high as that of the developers. However, even in these cases, automatic tests are used only for internal needs; they don’t impact end users, and they don’t need to be written to the same level as the product being tested.

1-5. Do Not Write Bulky Code

Try to follow the rule of “no more than three nested levels” when using conditions and loops in tests. If you need to write code with a lot of nesting, then step back and think about how to rewrite the code differently, for example, to bring a part of it into a separate function, or to make two separate loops or conditions. Complex and confusing code looks beautiful only when you write it, but not when you try to understand how it works.

Here is an example of code that already has one undesirable nested block:

for tbl in tables
{
  if table.name == "contacts"
    {
      for col in table.columns
      {
        /*enough nesting*/
        if col.name == "first"
        {
          /*this block is redundant*/
          print(col.name[0])
        }
      }
    }
}

Code with the nesting of more than three blocks is very difficult to debug, especially for a person who did not write it. When debugging such code , it is difficult to keep track of the state of all variables. As a result, searching for a simple problem can take a lot of time.

What this man boasted about is actually a bad programming style. Instead of 1 function with 19 levels, you could write 5 functions with 3-4 levels (one function for each type of table, because among all the variety of tables you could clearly distinguish the similar ones). Doing so would lead to a little duplication of code, but this code could be supported.

Sometimes there is a need for more than triple nesting, but always try to think over possible simpler ways of implementation. One of the options for checking the simplicity of the code is the calculation of the cyclomatic complexity of functions (i.e., the number of independent routes that a function can pass through). For popular programming languages, there are ready-made solutions for such tests.

1-6. Verify All Options of Logical Conditions

When you do verifications in tests with several logical conditions such as the following:

if A and B or C

be sure to check such code for each condition (A, B, C in this example) and all probable values (true, false).

Quite often programmers overlook “lazy evaluations,” resulting in incorrect operation of such expressions. Verify each condition, and you’ll catch such problems early.

Also, do not forget to enclose in parentheses those parts of the logical expressions that should be verified together. Do this even if you are 100% sure of the correctness of the computation priority. For example:

if (A and B) or (C and D)

The presence of parentheses significantly simplifies the understanding of the code in the future. It doesn’t matter that you understand the computation priority now. What matters is that the person maintaining the code years into the future understands. Make that person’s job easier by clarifying the order with parentheses.

Verify each of your conditions , and avoid getting caught out by subtle errors like the one my student created.

1-7. Use Coding Standards

Beginners usually don’t use any rules for the names of variables and functions. They write as they like, completely without thinking about the fact that there may be some rules. Nevertheless, for almost every language there are so-called coding standards , which are recommended to be used.

In big projects such standards become mandatory rules. They must be followed by all programmers, since following standards greatly simplifies the understanding of both their own and others' code.

For many languages, such rules have been developed a long time ago and are generally available. You can take them as a sample and supplement them depending on your internal needs. For example, for variables, you can use different prefixes depending on the type of display object this variable corresponds to.

Let’s suppose that you see the following line of code:

Main.Search.Click()

Obviously, there is a click on the Search object in the Main window, but if there is a Search text box, a Search button, and a Search static text in this window, then you have to guess which object is clicked. Guessing is not ideal.

Let’s look at two other examples:

Main.btnSearch.Click()
Main.txtSearch.Click()

or at these:

Main.SearchButton.Click()
Main.SearchField.Click()

These examples are much easier to understand. Even without knowing any special rules, everything is clear intuitively. The use of coding standards improves the readability of code, making it easier for you and others to understand it later.

1-8. Use Static Code Analyzers

For popular programming languages, there are special code analyzers that can report the most common errors or inconsistencies in the standards, for example, pylint for Python language, jslint for JavaScript language, etc.

Typical problems pointed to by similar analyzers are too long of functions, noncompliance with naming standards, too many parameters, incorrect indentation, etc. For many popular programming languages and development environments, there are plug-ins that perform these verifications right in the editor while writing the code.

At first, the use of such tools can be difficult, since they show a lot of errors. However, in a few days you will become so used to the tools that you will write code with practically no such mistakes, and the code will become much more readable.

Even better is to force the use of such analyzers when adding code to the repository. To do so, simply write a script that first calls a static code verification, and only if the verification succeeds, perform a commit.

There may be times when you want to prevent certain rules from being checked by an analyzer. Most provide a mechanism for disabling rules, so that you can always disable a particular rule if it suits your project to do so.

1-9. Add an Element of Randomness to Scripts

Imagine that you are testing standard Notepad application. One of the frequent operations that you have to perform will be the operation of opening the file. However, you can perform this operation in several ways:

• select the File|Open menu item;

• press Ctrl-O hotkey;

• press Alt-F-O key combination;

• press Alt (to activate the menu), then several times the down arrow key, then press Enter;

• pass the path to the file as a command-line parameter;

• drag a file to the Notepad window using the mouse.

There are several ways to achieve the same result, but theoretically any of them may not work. So what’s to be done?

• Should we verify each method separately? This approach is too long.

• Should we use different approaches in different tests? Then there is a possibility of missing some of the approaches.

One way to solve the problem is to choose the method for selecting a menu item in a random manner. For this purpose, we will write a separate function or method that will perform this operation. In this case, we do not have to verify explicitly all the methods, because one way or another each of them will be verified during one of the usual test runs. The more tests that are run and the more often the menu item operation is called, the more often each method will be verified.

At the same time, there is one important nuance. Since the method of action is chosen randomly, we must know exactly in the event of an error which method was chosen in the particular case leading to the error. For this purpose, you need to put in the log the information about which method was chosen, so that if you need to reproduce the problem, you can accurately reproduce the actions of the test.

1-10. Do Not Perform Blind Clicks Against Nonstandard Controls

Everyone sooner or later comes across a situation where some self-made control is not supported by the automation tool. After trying a few ways of testing such a control and not finding anything suitable, people stop at the simplest way of solving the problem: clicking on hardcoded coordinates . However, this is not always the best way.

Imagine a toolbox with several buttons. Let’s assume that our tool does not recognize the buttons on the toolbox and does not see any properties or methods that would allow us to get their coordinates. If we hard-code the coordinates for a click inside the toolbar, then for a while our scripts will work. However, then something can happen (for instance, the order of the buttons or their size changes). In this case, the click will occur elsewhere.

One solution to the problem is to search for an image within another image that is provided by certain tools. We can save the image of the button (just the button rather than the entire toolbar), and then search for the button’s image inside the image of the entire toolbar and click in the found coordinates. On the one hand, searching for an image is a time-consuming operation, but for small images the search will not last too long. On the other hand, even if a button changes its position or size, the search will still work correctly.

If your tool does not support searching for an image inside another one, then at least take such operations to a separate function. In your tests, there should not be such lines:

Window.Toolbar.Click(135, 15)

Instead, there should be a call to a function like the following:

toolbar_click_button_by_image(Window.Toolbar, "Button Caption")

It will be easier in this second case to understand the reason for any error that might occur during your test run. Your code will be clearer and more understandable as well.

1-11. Learn and Use Standard Libraries

Beginners in automation often make the mistake of creating something that has already been provided to them by the capabilities of the programming language or automation tool. Having been faced with some task, the tester starts writing his function or class to solve this task. Usually one’s own version of a given functionality will be worse than the one already available and provided by the language or the tool.

Let’s consider a simple example in Python. Let us suppose that we are working with Windows operating system, and we have a file path and file name in different variables; it is necessary to get the full path to this file. Here is how the newcomer solves this task:

file_path = 'C:\Users\tester'
file_name = 'picture.png'
full_name = file_path + file_name
print full_name

For a while, everything will be fine, but someday the file_path variable will lack a backslash character at the end. Then the tester will rewrite the code a little differently:

full_name = '{0}\{1}'.format(file_path, file_name)

Now we always get the correct file name. Sometimes two slashes will appear between the folder and the file name, but the operating system correctly handles such situations. But do not lull yourself into believing we are done.

Over time, we have a new requirement: to compare the file name obtained by us with the same name in the tested application. As a result, we get an error in those names where we had a double slash between the folder and file name, and again solve our problem by modifying our function, this time with a call to a text replacement function:

full_name = '{0}\{1}'.format(file_path, file_name)
full_name = full_name.replace('\', '')

As you can see, with each new requirement our code becomes more and more complicated, and all these problems could be avoided initially, using the standard Python function:

full_name = os.path.join(file_path, file_name)

First, this approach of using what Python already provides is more simple, convenient, and understandable. Secondly, standard library functions are usually well tested and take into account a lot of small, corner cases that you are likely to miss when creating a new function with the same capabilities by yourself.

Therefore, always look for standard ways of solving problems first, and only if there is no such solution - create your own. Also, when faced with a new library for yourself, look at all of its capabilities. Glance over them, at least. Doing so will save you time in the future.

1-12. Avoid Copy and Paste

Copying and pasting of code in order to reuse it is a common mistake of beginners. Once you need to write a new code, you remember that you already wrote something similar in the past. You find your old code, change it a little, and create a new function.

Then again … again … and again…

Once you find an error in one of your copies, you correct it and understand that in all your copies there is the same problem. You spend the time to find all the places where the mistake was made.

Then it happens again … again … and again…

Once you are faced with a problem that needs to be corrected in the same way in several places, you must make sure that there is exactly one place left. Let it be a function that takes into account all the necessary options, or let it be a function with several parameters, but it must be only one function, with the code in one place.

Do not be too lazy to write code correctly. Be lazy now by copying and pasting, and in the future you will remember what I have said and regret doing so. You will spend an hour and regret that you once grudgingly spent an extra five minutes.

Have pity on yourself, do not copy and paste . If you use a popular programming language, it is likely that the language already has tools for detecting duplicate code. Use them!

1-13. Do Not Use try…catch with an Empty catch Block

Every programmer writes such a construction once:

try{/*some code*/}
catch()
{} /*ignore all exceptions and continue execution*/

Here, we execute certain code, completely ignoring all possible exceptions. This is acceptable for temporary code, when you are experimenting, but it is completely unacceptable in tests that run regularly.

It doesn’t matter how simple the code is in the try block. Even if it’s just one line, there might theoretically be an unexpected error, but you will never know about it. For this reason, the first correction will be that we will add an output message to the catch block :

catch(e)
{print(e.description)}

Now we will have the text of the exception in the log, and we’ll be able to see it later when sorting out the logs and trying to understand what is going wrong.

A similar example is a large try block , and handling only one possible exception in the catch block. This will lead to the same result as in the first case, with the exception of the one processed case that is properly handled.

For such a situation, there is a right approach: intercept only those exceptions that you accurately need to handle, while not ignoring other possible errors. Catch the one error that you care about, and throw the other errors up the chain, like in this example:

catch(e)
{
  if(e.number == 123)
    {/*just ignore this*/}
  else
    {throw e}
 }

If in the future you need to ignore (or handle in a different manner) other exceptions, you simply add a new else block for them.

Sometimes an empty catch block gets into a test accidentally. For instance, you add an empty catch block temporarily, plan to finish it a bit later. To avoid such accidents, just write the catch block right away, without leaving it for later.

1-14. Separate Code from Data

When we write tests, we need a place to store the expected values, which we compare to what the tested application actually produces. When dealing with a single piece of data (one number, text, etc.), we just use variables. For small datasets (for instance, items in the drop-down list), we can use arrays or lists. But what if there is a lot of data (for instance, several dozen values)?

It is inconvenient to store large amounts data along with the code, since the data simply obstructs the code. You can put the data in a separate code file, but in this case it is inconvenient to work with the data, because there is no vertical alignment in text files and, for instance, the lines will have different widths, which makes it difficult to understand the information.

You can solve all these problems by using the data-driven testing (DDT) approach . This is an approach in which data is stored in a separate file in the form of tables. It can be a database of any format, an Excel or comma-separated value (CSV) file .

Modern automation tools usually support working with one or more of these formats. For many programming languages, there are libraries that make it easier to work with such tables. If in your case there is nothing readily available, you can write simple functions to work with formats such as CSV.

Here are some tips for working with data using the DDT approach:

• open data files only in read mode from the tests. Since data files contain model data, changing it automatically during scripts run is potentially dangerous. If you need to change this data, you need to do this manually, knowing exactly what you are changing and why;

• store only one type of data in each column. In case of a database, you will not be able to do otherwise; however, in case of text files or Excel files, you will have to monitor it by yourself. Some Excel drivers can behave strangely if they encounter a value with a mismatched data type. In addition, your tests can also work with errors in case of inconsistency of data types;

• don’t forget to close the connection to the data source file after you have finished reading the data from it. Leaving an open connection can prevent you from connecting to it from other tests;

• in case of using Excel files, make sure that the data is stored as in the database. For instance, if you leave an empty line in the middle of the table, the driver can consider it as the end of the file while reading data. Therefore, closely look after the data in the hidden lines, if there are such in your files.

It doesn’t matter which type of data storage you choose. It is a good practice to keep your data separate from code.

1-15. Learn How to Debug

Debugging is a step-by-step execution of a program in order to detect an error in it. Learning to debug is easy. The basics of debugging can be studied in an hour, but the benefits of using it are huge.

Among novices, there are many such people who simply did not learn debugging, and therefore don’t know that it is possible. That is why debugging is one of the mandatory things that I show during my trainings that I deliver to new people on my team.

Here are the basic concepts that you need to study and begin to use (these capabilities are available in any editor):

• Setting of breakpoints. A breakpoint is the place where execution of the program stops, waiting for action from you.

• Step-by-step debugging. This is what allows you to step forward in code, step backward, execute to a cursor, and watch each step of your program unfold.

• Viewing values of local and global variables. All debuggers provide mechanisms to examine the value of variables.

• Observation of variables and expressions. Debuggers often provide watch list functionality to let you know when the value of a variable or an expression changes.

Typically, there is a detailed description of all the debugging features in the help system of your development environment. Become familiar with that documentation so that you can explore features and take full advantage of what your debugger can offer you.

You can use any programming tutorial to understand the basics of debugging. However, there is one feature specific for automation only. If you are dealing with a variable that corresponds to a display object, the values of its properties will not always match the values during the test run. For instance, the Visible property can be set to False during debugging, because at that moment the editor itself is a top window. However, while the tests are running, the value of this property will be True, because the window with this element will be displayed on the screen, and the editor window will be hidden.

In addition, some actions in the step-by-step mode may not be performed. For instance, when you try to click on a specific object, your tool will not be able to move focus to the desired window. As a result, clicking will not happen.

In all other respects, debugging in test automation does not differ from debugging in programming.

1-16. Do Not Write Code for the Future

There are two ways in which to write tests:

• You write a test and, as you write it, you write the general functions and methods that will be used in this and future tests .

• You first write common functions and methods and then quickly develop a test that uses everything written before.

Each approach has its advantages and disadvantages. Each approach can be effectively used in different situations.

However, there is one approach that does not work: writing a lot of common code for many tests first and then starting using it. The key words here are “a lot”: for example, if you are going to immediately write all the functions and methods for a dozen future tests.

Write for the present. Not for the future.

1-17. Leave the Code Better Than It Was

The Boy Scouts have a rule: “Always leave the place cleaner than you found it.” It is about the cleanliness of the campground: leaving , you need to leave it cleaner than it was before your arrival.

The same goes for someone else’s code. If you stumble upon someone else’s code that you can definitely improve without spending too much time – do it! Make the improvements. Such improvements can be simplification of a clearly tangled piece of code, or simply consist of syntax corrections (adding indents or extra lines to improve readability).

You only need to improve code if you are 100% sure that doing this will not affect the efficiency of the code or the results of its execution, and also if it does not take you more than a few minutes.

If this simple rule is followed by everyone, our tests will constantly improve and reading such code will be much more pleasant. Therefore, teach others the same by your example.

1-18. Choose a Proper Language for GUI Tests

Many automation tools offer a choice of several programming languages you can write tests in. And often people try to choose the same language for tests in which the application under test is written. This approach has a number of advantages:

• We don’t produce programming languages without the need.

• Testers can always ask the programmers for advice.

• In case of necessity, programmers will be able to create and maintain tests.

• API testing is greatly simplified.

However, choosing the same language for tests as for the product being developed is not a strict rule. It is unlikely that programmers will ever look into the test code, so tests can be written in any language that is convenient for testers.

In addition, the tasks of testers are usually much simpler than the tasks of programmers. Writing a commercial application is much more difficult than writing tests for it. For this reason, allowing programmers to design a test engine can lead to complexities that can be avoided. Programmers might initially try to design such a system “with a margin for the future,” trying to foresee what testers will never encounter. Therefore, you should be guided in choosing the programming language for GUI tests , first of all, by what your team will find more convenient and easier to work with, rather than by what your application under which the test is written in.

1-19. Remember to Declare and Initialize Variables

Many automation tools use scripting languages to create tests. In some of the tools, it is allowed to use variables without declaring them. In others you can initialize variables explicitly when they are declared.

For instance:

var myVariable;

or:

notDeclaredVar += 1;

Such simplifications can lead to unpleasant situations. If we do not initialize a variable when we declare it, we can’t know exactly what value will be assigned to the variable. For this reason, any action with it can lead to an error that can’t be reproduced later.

As for the possibility of using undeclared variables , doing so can lead to completely unexpected consequences.

Therefore, if you do not want to encounter such strange mistakes, always follow two rules:

1. Always declare variables.

2. Always initialize them with initial values.

This applies even if declaration or initializing can be skipped in the language you use. Here is an example in JavaScript:

var myVariable = 0;

or:

declaredVar = 0;
declaredVar += 1;

In the first case we initialize a new declared variable , which is not mandatory in JavaScript. In the second example we assign a zero to a previously declared variable before starting using it in order to be sure the variable doesn’t contain an unpredictable value.