On an ordinary day, you report to work where you run into your boss down the office coridoor. And there you go! Thirty minutes later, your fine day is struck by the fact that your boss is expecting a list of test cases not automated, yet to be automated, and already automated. All kinds of problems start popping inside your head: "Test workbooks are dispersed all over testers' machines, so how can I possibly get the count of all these test cases? There is a weak traceability between automated and manual scripts, so how can I tell which test case is automated and which one is not?"
If this is your typical testing work environment, you are probably familiar with the trouble you've put into compiling a list of all the tests and producing statistics around certain criteria. In reality, if the manual scripts were saved in Excel workbooks, we would most likely utilize the sheet columns to fill them with values referring perhaps to the automated test ID, incident report ID, feature ID, and so on. We would also use the folders in the filesystem to mimic the hierarchy of projects and their corresponding releases.
This is what the era looks like prior to automating the test management process. With the automation of the manual testing process, the difficulty for maintaining an orderly repository phases out and gets substituted with a smart mechanism capable of promptly responding to testing-specific queries.
This chapter uses features from Test Studio to cope with the preceding problems. It contains examples illustrating:
- A manual test creation
- A manual test integration with MS Excel
- The automated setup and teardown routines for preparing and restoring system states
- A manual test execution
- How to manage manual test case attributes
During the process of converting a manual test into automated scripts, a decision needs to be made concerning which tasks and tests are to be conserved in their manual states, which test cases are suitable for automation, and when would you gain more by keeping a test in its manual state.
There are some assessments that you need to reflect on concerning a test's nature and worth with respect to the project's lifetime before choosing to develop an automated test.
Hence, when a test case is intended for automation, you want to firstly ask yourself, "What is the value behind automating it?"
First of all, you would want to consider the test development effort with respect to the test's lifetime. The prevalence of one of the preceding factors tilts the equation of ROI (return on investment) to one side or another. Ultimately, the less the effort and the longer the usage of the test is, the greater the ROI. For example, to create an automated test involving a user interface surely requires more work than a test calling a web service, given that the former needs to firstly overcome the challenges related to capturing screen objects. If the user interface test will run only once or will deprecate with the next interface remodeling, the durable value for the test is poor and therefore the automation cost is negatively affected this way.
Moreover, consider the test's worth also with regards to its lifetime. As you know, the applications under test gain immunity as fixes are introduced following each test suite execution. So during the early phases, you need to primarily focus on tests that are able to ascertain the appropriateness of a certain functionality and postpone tests in which the underlying bug is fixed and is not likely to recur. For instance, favor automating tests related to the successful submission of a form over a field's input limit. If the field input limit was buggy at some point in time, it is improbable that a developer will deliberately revert it; whereas, in the form submission case, any change elsewhere in the application could create an unforeseen negative impact. Hence, ask yourself, is the test in question adding value for my regression suite?
Secondly, consider the point at which the automation has started with respect to the project's timeline. How far is the project towards its completion? Proper time needs to be accounted for automation, especially if the knowledge in the automation tool is not present yet. For this reason, squeezing in additional tasks in the project schedule could result in delaying feedback on the application's quality state, where this feedback becomes ineffective if not processed in the needed time. This also brings up the question of what to automate versus when to automate? Therefore, you should consider looking at test maintenance effects and start automating those tests with the lowest maintenance cost where updating them is affordable as changes are introduced into the application.
Thirdly, think about the degree of intervention required by the tester. Tests necessitating less human analysis and interference in response to the system events are favored, since more complex the scenarios are, higher is the complexity of the scripts. Eventually, the tests lose their credibility when lots of reported bugs are traced back to the automation code rather than the source code. These situations are called false positives.
All of the preceding points are arguments that support why you should sometimes stick to manual testing. Nevertheless, manually carrying out all the tasks related to manual testing is tedious and devalues its advantages. Common problems with the manual execution process are repository maintenance, subjectivity in test interpretation that requires a prior consent to the extent of procedures' details, reporting results, performing repetitive time-consuming tasks, and others.
Test Studio embeds a solution that starts with creating manual tests and then gradually transitioning them into automated ones. It also provides the possibility to automate fractions of these tests to deliver the tester from the manual handling of tedious tasks. As we will also see, Test Studio's integration with third-party solutions offers versioning and collaboration control over tests.