Even if you have made your tests totally reliable, they will fail occasionally. When this happens, it is often very hard to describe the problem with words alone. If one of your tests failed, wouldn't it be easier to explain what went wrong if you had a picture of what was happening in the browser when things went wrong? I know that when any of my Selenium tests fail, the first thing I want to know is what was on the screen at the time of failure. If I knew what was on the screen at the time of failure, I would be able to diagnose the vast majority of issues without having to hunt through a stack trace for a specific line number, and then go and look at the associated code to try and work out what went wrong. Wouldn't it be nice if we got a screenshot showing what was on the screen every time a test failed? Let's take the project that we built in Chapter 1, Creating a Fast Feedback Loop, and extend it a bit to take a screenshot every time there is a test failure. Let's have a look at how we can implement this with TestNG:
- First of all, we are going to create a package called listeners. Refer to the following screenshot:
- Then, we are going to implement a custom listener for TestNG that will detect a test failure and then capture a screenshot for us by using the following code:
package com.masteringselenium.listeners;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.remote.Augmenter;
import org.testng.ITestResult;
import org.testng.TestListenerAdapter;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import static com.masteringselenium.DriverBase.getDriver;
public class ScreenshotListener extends TestListenerAdapter {
private boolean createFile(File screenshot) {
boolean fileCreated = false;
if (screenshot.exists()) {
fileCreated = true;
} else {
File parentDirectory = new
File(screenshot.getParent());
if (parentDirectory.exists() ||
parentDirectory.mkdirs()) {
try {
fileCreated = screenshot.createNewFile();
} catch (IOException errorCreatingScreenshot) {
errorCreatingScreenshot.printStackTrace();
}
}
}
return fileCreated;
}
private void writeScreenshotToFile(WebDriver driver,
File screenshot) {
try {
FileOutputStream screenshotStream = new
FileOutputStream(screenshot);
screenshotStream.write(((TakesScreenshot)
driver).getScreenshotAs(OutputType.BYTES));
screenshotStream.close();
} catch (IOException unableToWriteScreenshot) {
System.err.println("Unable to write " +
screenshot.getAbsolutePath());
unableToWriteScreenshot.printStackTrace();
}
}
@Override
public void onTestFailure(ITestResult failingTest) {
try {
WebDriver driver = getDriver();
String screenshotDirectory =
System.getProperty("screenshotDirectory",
"target/screenshots");
String screenshotAbsolutePath =
screenshotDirectory +
File.separator + System.currentTimeMillis() + "_" +
failingTest.getName() + ".png";
File screenshot = new File(screenshotAbsolutePath);
if (createFile(screenshot)) {
try {
writeScreenshotToFile(driver, screenshot);
} catch (ClassCastException
weNeedToAugmentOurDriverObject) {
writeScreenshotToFile(new
Augmenter().augment(driver), screenshot);
}
System.out.println("Written screenshot to " +
screenshotAbsolutePath);
} else {
System.err.println("Unable to create " +
screenshotAbsolutePath);
}
} catch (Exception ex) {
System.err.println("Unable to capture
screenshot...");
ex.printStackTrace();
}
}
}
First, we have the rather imaginatively named createFile method, which will try to create a file. Next, we have the equally imaginatively named writeScreenShotToFile method. This will try to write the screenshot to a file. Notice that we aren't catching any exceptions in these methods, because we will do that in the listener.
The last block of code is the actual listener. The first thing that you will notice is that it has a try...catch wrapping the whole method, which initially looks wrong. While we do want a screenshot to show us what has gone wrong, we probably don't want to kill our test run if we are unable to capture or write a screenshot to disk for some reason. To make sure that we don't disrupt the test run, we catch the error and log it out to the console for future reference. We then carry on with what we were doing before.
You cannot cast all driver implementations in Selenium into a TakesScreenshot object. As a result, we capture the ClassCastException for driver implementations that cannot be cast into a TakesScreenshot object and augment them instead. We don't just augment everything, because a driver object that doesn't need to be augmented will throw an error if you try. It is usually RemoteWebDriver instances that need to be augmented. Apart from augmenting the driver object when required, the main job of this function is to generate a filename for the screenshot. We want to make sure that the filename is unique so that we don't accidentally overwrite any screenshots. To do this, we use the current timestamp and the name of the current test. We could use a randomly generated Globally Unique Identifier (GUID), but timestamps make it easier to track what happened at what time.
Finally, we want to log the absolute path to the screenshot out to console. This will make it much easier to find any screenshots that have been created.
As you may have noticed in the preceding code, we are using a system property to get the directory that we save our screenshots in. This will allow you to redirect your error screenshots to any location you desire. We have set a default location of target/screenshots; if you want to override this, we will need to set this system property in our POM.
To give us that ability, we will need to modify the maven-failsafe-plugin section to add an additional property by using this code:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-failsafe-plugin.version}</version>
<configuration>
<parallel>methods</parallel>
<threadCount>${threads}</threadCount>
<systemProperties>
<browser>${browser}</browser>
<screenshotDirectory>${screenshotDirectory}
</screenshotDirectory>
<remoteDriver>${remote}</remoteDriver>
<gridURL>${seleniumGridURL}</gridURL>
<desiredPlatform>${platform}</desiredPlatform>
<desiredBrowserVersion>${browserVersion}
</desiredBrowserVersion>
<!--Set properties passed in by the driver binary
downloader-->
<phantomjs.binary.path>${phantomjs.binary.path}
</phantomjs.binary.path>
<webdriver.chrome.driver>${webdriver.chrome.driver}
</webdriver.chrome.driver>
<webdriver.ie.driver>${webdriver.ie.driver}
</webdriver.ie.driver>
<webdriver.opera.driver>${webdriver.opera.driver}
</webdriver.opera.driver>
<webdriver.gecko.driver>${webdriver.gecko.driver}
</webdriver.gecko.driver>
<webdriver.edge.driver>${webdriver.edge.driver}
</webdriver.edge.driver>
</systemProperties>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Since we have used a Maven variable to make this property configurable, we will then need to set it in the properties section of our POM:
<properties>
<project.build.sourceEncoding>UTF-
8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-
8</project.reporting.outputEncoding>
<!-- Dependency versions -->
<phantomjsdriver.version>1.4.3</phantomjsdriver.version>
<selenium.version>3.5.3</selenium.version>
<testng.version>6.11</testng.version>
<!-- Plugin versions -->
<driver-binary-downloader-maven-plugin.version>1.0.14</driver-
binary-downloader-maven-plugin.version>
<maven-failsafe-plugin.version>2.20</maven-failsafe-
plugin.version>
<!-- Configurable variables -->
<threads>1</threads>
<browser>firefox</browser>
<overwrite.binaries>false</overwrite.binaries>
<remote>false</remote>
<seleniumGridURL/>
<platform/>
<browserVersion/>
<screenshotDirectory>${project.build.directory}
/screenshots</screenshotDirectory>
</properties>
You can see that in our Maven variable definition, we have used a Maven variable that we have not defined before. Maven has a series of predefined variables that you can use; ${project.build.directory} will provide you the location of your target directory. Whenever Maven builds your project, it will compile all of the files into a temporary directory called target; it will then run all of your tests and store the results in this directory. This directory is basically a little sandbox for Maven to play in while it's doing its stuff.
When performing Maven builds, it is generally good practice to use the clean command:
mvn clean verify
The clean command deletes the target directory to make sure that when you build your project, you don't have anything left over from the previous build that may cause problems. This does mean that you will delete screenshots from previous runs if you have not copied them anywhere else before kicking off another run of your tests.
Generally speaking, when we run tests we are only going to be interested in the result of the current test run (any previous results should have been archived for future reference), so deleting old screenshots should really not be a problem. To keep things clean and easy to find, we are generating a screenshot subdirectory that we will store our screenshots in.
Now that our screenshot listener is ready, we just have to tell our tests to use it. This is surprisingly simple; all of our tests extend DriverBase, so we just add a @Listeners annotation to it by using this code:
import com.masteringselenium.listeners.ScreenshotListener; import org.testng.annotations.Listeners; @Listeners(ScreenshotListener.class) public class DriverBase
From this point onwards, if any of our tests fail, a screenshot will automatically be taken.
Screenshots are a very useful aid when it comes to diagnosing problems with your tests, but sometimes things go wrong on a page that looks completely normal. How do we go about diagnosing these sorts of problems?