2-1. Do Not Duplicate Tested Application Functionality in the Scripts

An application under test performs calculations and outputs a result. How to check that the result is correct? The first option that comes to mind is to calculate the same in the test script and compare the result with what the application gives us! This approach is incorrect for several reasons:

• Calculations can be complex. The programmers have already spent time on them and now you will do the same.

• Formulas for calculations may change later. In this case you will see an error in the report, although the application works correctly. You will have to make corrections, again spending time for this.

• When working with floating-point numbers, the accuracy of calculations may differ in the language that you use for tests and in the language in which the application under test is written. As a result, you will have to artificially customize your calculations so that they coincide with the result of the application.

The correct approach in such situations is to calculate the correct result manually and save it in the script as expected. If one calculation is enough, then write its result directly in the script. If several calculations are needed, use arrays or the DDT-approach.

It is best if the source data and the correct results for them are given to you by competent people (for example, product specialists). Your responsibility is to code the test, but usually it should not be your responsibility to determine what the result should be.

Now if your test has failed, then either the application is not operating correctly, or something has changed in the requirements and you need to update the test data. Either way, you can be assured that the failure is not a problem of the test code!

If, for some reason, it is impossible to completely abandon such calculations, try to simplify and reduce them in tests as much as possible. Also, make sure to put some comments in the script or error message describing what was calculated and why it was done. It may help you or someone else who will be analyzing possible errors in the future.

2-2. Each Test Should Be Independent

Beginners often make an error while automating when one test uses data that was generated in another test. A vivid example of this is the CRUD functionality , when one test creates a record, another one edits it, and the third one deletes the record.

The only advantage of this approach is the time saving; however, several disadvantages appear at once:

• If the test that creates the record fails for some reason, then the rest will also fail.

• In the report you will see not one error, but several, although the functionality of editing and deleting can be correct.

• When the tests are started automatically, it’s not always possible to ensure that they will run in a specific order.

If you want to save a little time and not have to create additional records using the application, you can do it otherwise. For instance, create them using an SQL query to the database, with the application API, or use the database in which the necessary entries are already created. In this case, to verify the independence of the tests from each other you can run the tests in a random order, if this can be implemented for your automation tool.

Another approach to solving the problem is to run first the test, which will create all the test data necessary in the future. However, if this test fails, then running all the others is meaningless, and as a result no test will be started at all.

If you still think that in your case the use of dependencies between tests is the most optimal approach, in each dependent test you should check the results of that test on which they depend, and do not run the dependent test if the previous one did not work correctly .

Another possible way to solve the problem is to combine several dependent tests into one.

2-3. What Should Not Be Automated?

In any project, there are tasks that need to be automated, but there are those that should be left for manual testing only. For each project, of course, such tasks will be different, but you can identify several common areas, which should not be automated.

• Do not automate something that is difficult to maintain. Even if you have already written a test, but you have to regularly fix and restart it, consider deleting it or simplifying it as much as possible.

• Do not automate someone else’s application. For instance, if you need to work with Google mail, do not write complex code that will work with the interface of this mailer (of course, this does not apply to Google employees).

• Automation of the interface (its correctness and ease of use) is possible in principle, but too time consuming and usually incomplete.

• Interaction with any peripheral devices (printers, scanners, etc.) that require human participation is usually either complicated, or requires too much additional manual work.

• Verification of the correctness of various images, graphics, and video is also easier to perform manually.

• Obfuscated applications and individual components cannot be automated, since the names of their properties change at each compilation.

Nevertheless, sometimes it makes sense to combine automation with manual testing.

Do not consider the rules in the prior list to be an axiom. Just keep them in mind and try to find the most optimal solution to your problem. Sometimes the right solution is to break the rules, as I did when commenting out that restart code.

2-4. Ask the Developers for Help

The tests we write are simplified programming, which is necessary to simplify testing. Of course, you can make a complex framework, but in most cases this is not required. The same goes for other areas that programmers come across more often: regular expressions, working with databases, using internal application methods, and more. Therefore, usually programmers are more experienced in such things and can give useful advice in the rare cases when you need to develop more complex testing solutions.

If you are faced with a complex problem that you do not know how to approach, try to ask the advice of developers. Of course, it is not worth running for help with every little thing; it is often much more useful to understand by yourself, thereby increasing your experience.

However, always remember that the view of the programmer of the application is different from yours and not all pieces of advice are equally good. The programmer looks at the application from the inside and knows how it works. You look from the outside and know how it should work.

This example shows the difference between developers’ and testers’ approaches to work with applications. That’s why you better know what and how you should test, and programming issues should be discussed with the developers, thereby improving your skills.

2-5. Cloud Testing

In recent years, cloud services have become popular, providing the ability to run virtual machines in particular. Since these services are quite cheap, many people have a desire to run automatic tests in the clouds. Before making such a decision and starting the implementation , consider several important points:

• To test desktop applications, an open session is mandatory (it can be either a logged-in user or Remote Desktop session). If there is no such session, the operating system simply does not render applications’ GUI, which means that the automation tools will not see a single window or control.

• In particular, the need for an open GUI means that you can’t just configure a remote virtual machine and run tests on it, using, for instance, the command line. You will need to connect to the virtual machine using Remote Desktop and leave the connection window open for the duration of the tests. If during the operation the connection is interrupted for at least a second, the automation tool will not recognize the application elements, and this will lead to unforeseen errors.

• Any automation associated with physical devices is also inconvenient in the cloud, since you have no direct access to the physical devices your cloud provider is allocating to your account.

• Cloud services that provide access to mobile devices are more expensive than usual cloud virtual machines. Therefore, it is better to run automatic tests on local mobile devices, and cloud services should be used only for manual testing (for instance, to verify if the application works on a device that you don’t have).

• Automation of web applications is best carried out in the clouds. It is only necessary not to return the virtual machine to the initial state, if at the end of the tests you need to verify something on this machine .

In general, working with virtual machines in the clouds is not as convenient and fast as with local computers and virtual machines. And given the comparative cheapness of computer hardware, for ordinary tasks you can use only local machines. Cloud services are needed when there are no other options (for instance, for emulating load testing).

2-6. Introduce Automation for Corner Cases

Let’s assume that at a certain point in the test you need to enter a number from 1 to 10 into the field. Testing all 10 cases is quite expensive, so we select a random number and enter only it. If each time while running a test only this number is entered, we will have 9 potential errors that are never checked.

You can choose this number randomly from the range or test several values instead of 1, but in addition, you should always test the values 1 and 10. It is with these corner values there are often associated different errors, which programmers do not take into account during the development or writing of unit tests. Thus, in our automated test we will check at least 3 numbers: 1, 10, and any number in the range from 2 to 9 inclusive.

Of course, we also need to check that the application does not allow entering in this field negative, numeric, and other values that are not suitable for this field, but this already refers to negative testing, and we are talking about a positive one here.

2-7. The Difference Between Error and Warning

There are errors of two types: critical and non-critical. From an automation point of view, this means that if after an error occurs we can continue to run the test, then the error is not critical. It is these non-critical errors that are warnings.

Non-critical errors must be seen in the report in order not to forget about them; however they must be marked with a color different from the color of the critical errors. For instance, critical errors may be marked in red, and non-critical errors may be indicated by yellow.

Unfortunately, many tools generate reports in the JUnit format, which was originally intended for unit tests written by developers. These are small and very fast tests that usually check the correctness of individual functions and methods with different input parameters. Unlike GUI tests, a unit test can’t have an intermediate state; it either passes or fails.

When the unit test engine is used for GUI testing, the test will be stopped at any error that appears. For instance, we verify the sorting in all columns of the table, for which we click on the column headers. If when clicking on one of the columns the data in this column will not be sorted – this is an error. However, this is not a reason to stop the test and not verify the remaining columns.

In order to solve this problem, in tools using JUnit reports, the following approach is used. Performing non-critical verification, in case of an error we simply add an error line to the error list (which is initially empty). At the end of the test, check this list. If it is still empty as it was in the beginning, then there are no errors in the test. If there are errors in the list, we mark the entire test as a failed one and put all the accumulated errors to the log.

If you have an opportunity to use reports that support warnings as one of the variants of the error – use them. It helps to easily distinguish critical errors from those which can be ignored at the moment.

2-8. Use the Appropriate Methodologies

There are a lot of specific methodologies that are used in testing automation: ODT, DDT, KDT, BDD, Page Objects, Model-Based Testing, etc. However, it is not enough to just know them; you need to use them appropriately.

Following are two vivid examples of misuse of methodologies on different projects.

Example 1

In a small project, the team decided to implement automation and employed an experienced automation engineer who previously worked for a large company where the automation was handled by a large department. The experienced automation engineer implements in the new project the same approaches that were used in his previous company; in particular, he uses Keyword-Driven Testing. Since the selected tool does not support this approach, the automation engineer writes it “from scratch” and fully supports it, and also automates existing manual tests.

What is wrong with this approach? It is wrong by the fact that the Keyword-Driven methodology is needed in those projects where the script code is written by one people, and the tests are written by others. If in the project only one person is fully engaged in the automation, then the support for the KDT approach will take 2/3 of the time, and only 1/3 of the time will be spent on creating the test.

In addition, if in the future this person leaves the project, and a less experienced automation engineer comes in his place, it will be very difficult for him to support this approach and eventually he will cease to use it.

Example 2

In the second example, we also deal with a small team with an equal number of programmers and testers. The customer really likes the possibility of test automation and he does everything possible to promote this idea. However, the customer’s knowledge is limited to promotional materials, which are provided by companies that develop automation tools . These materials always say that it is enough to record the actions with the help of the tool, after which the tests will work fine. As a result of such misinformation, the customer requires testers to perform large amounts of work every week. To cope with the load, testers mainly use recording, only occasionally making changes to the recorded tests.

The Downside and How to Avoid It

The disadvantage of the approach taken in the examples is obvious: tests too often have to be changed, and they end up spending a lot of time on it. Sometimes it is easier to just rewrite them again, if the existing functionality has been greatly changed. The efficiency of such automation is very close to zero. Therefore, every time starting a new project or taking on a new task, try to think about what you will need, and what you can refuse.

2-9. Verification of Individual Bugs

In most cases, tests are written to verify some functionality in general and do not verify special cases, for instance, a bug with specific data. However, sometimes it happens that once corrected bug appears again after some time; the reasons for its appearance can be completely different, but the result looks the same. In such cases, it makes sense to write a separate test to verify this particular bug, or a separate test to verify several bugs.

The peculiarity of such bugs is that they do not fit into manual test scripts, and therefore can only be found by chance. Often such errors occur in actively developing projects, where several programmers can work on modifying the same functionality, therefore affecting each other’s changes. If you write such a test, then in the report it is necessary to indicate the number of the existing bug in the bug tracker so that you can see the history of its findings and fixes. You can act the same way when you write an automated test and find an error in the application under test. You register a bug, and specify its number in the comment to the verification in the new test. Once such a bug is fixed, the error simply disappears from the report.

2-10. Make a Pilot Project Before Writing Real Tests

A pilot project is a project that lasts one-two months and is only needed for informational purposes. Pilot projects are used in several cases:

• if you are using a new automation tool for the first time;

• if you automate a new type of project (for instance, you had always worked with web applications, and now move on to testing the desktop or mobile applications);

• when implementing automation in a project where it did not exist before, and it is necessary to quickly show a certain result to the management or the customer.

The purpose of the pilot project is to try the tool, to understand its capabilities, and to study the main advantages and disadvantages. Therefore, expect that at the end of the pilot project you will delete it entirely (or almost entirely), since usually this project does not use the best approaches.

For a pilot project , you usually select a few simple tests and one-two tests of average difficulty. If your application uses complex controls (editable grids, specific controls, etc.), then at least one test should work with such an element. If this is not done right away, then it can later turn out that this tool can’t work with such objects at all, which can significantly complicate the automation process.

Also, pilot projects are useful for training new team members, if they did not work in the field of test automation before. It will help the beginners to try to create their first projects from scratch and learn the basics before starting working with real projects together with other team members.