Well, it's not all bad news. Selenium doesn't explicitly provide support for network traffic; however, it does provide support for proxies. If you want to track your network traffic, what's the best way to do it? Why, a proxy, of course!

There are many proxies available, but we will focus on one in particular: the BrowserMob proxy. The BrowserMob proxy has been written with test automation in mind and integrates very easily with Selenium. Let's look at a basic implementation:

package com.masteringselenium.tests;

import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.core.har.HarEntry;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.remote.CapabilityType;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.Test;

import static org.assertj.core.api.Assertions.assertThat;

public class ProxyBasedIT {

    private static WebDriver driver;

    @AfterSuite
    public static void cleanUpDriver() {
        driver.quit();
    }

    @Test
    public void usingAProxyToTrackNetworkTraffic() {
        BrowserMobProxy browserMobProxy = new  
        BrowserMobProxyServer();
        browserMobProxy.start();
        Proxy seleniumProxyConfiguration = 
        ClientUtil.createSeleniumProxy(browserMobProxy);

        FirefoxOptions firefoxOptions = new FirefoxOptions();
        firefoxOptions.setCapability(CapabilityType.PROXY, 
        seleniumProxyConfiguration);
        driver = new FirefoxDriver(firefoxOptions);
        browserMobProxy.newHar();
        driver.get("https://www.google.co.uk");     
    }
}

Our basic implementation is really quite simple. We are creating an instance of BrowserMobProxy, starting it up and then creating a Selenium proxy configuration using the handy ClientUtil class provided by the BrowserMobProxy team. We then take this proxy configuration and use the FirefoxOptions object to tell Selenium that we want to use it. When Selenium starts up, all network traffic will now be routed through BrowserMobProxy.

If we want to record the traffic, the first thing we need to do is to tell BrowserMobProxy to create a HTTP archive (or HAR) of the network traffic. We then use Selenium to navigate to a website and perform some actions. When we are done, we retrieve the HTTP archive that has been created by BrowserMobProxy.

Earlier in the chapter, we wrote some code to check the HTTP status code for specific resources; we could use a proxy to do the same thing. Let's extend our test to do that and see how usable a solution it is.

First of all, we need to write some code to find a specific HTTP request and return the status code:

private int getHTTPStatusCode(String expectedURL, Har httpArchive) 
{
    for (HarEntry entry : httpArchive.getLog().getEntries()) {
        if (entry.getRequest().getUrl().equals(expectedURL)) {
            return entry.getResponse().getStatus();
        }
    }
    return 0;
}

As you can see, parsing the HTTP archive is quite simple; we get a list of entries and then just iterate through them until we find the entry that we want.

However, this does expose a potential flaw. What if the archive is really big? With our example test here, it's not really a problem because we are only making a single request, so it won't take long to parse the archive. It's worth noting that (at the time of writing) this single request generated 17 entries; obviously this will vary depending on what Google has put on their home page when you try out this code. Imagine how big the archive could get with just one standard user journey, or even a couple.

Now that we have our function to find a status code for a URL, we need to extend our test to use this additional code:

@Test
public void usingAProxyToTrackNetworkTrafficStep2() {
    BrowserMobProxy browserMobProxy = new BrowserMobProxyServer();
    browserMobProxy.start();
    Proxy seleniumProxyConfiguration =  
    ClientUtil.createSeleniumProxy(browserMobProxy);

    FirefoxOptions firefoxOptions = new FirefoxOptions();
    firefoxOptions.setCapability(CapabilityType.PROXY, 
    seleniumProxyConfiguration);
    driver = new FirefoxDriver(firefoxOptions);
    browserMobProxy.newHar();
    driver.get("https://www.google.co.uk");

    Har httpArchive = browserMobProxy.getHar();

    assertThat(getHTTPStatusCode("https://www.google.co.uk/", 
    httpArchive))
            .isEqualTo(200);
}

This is where we see another potential flaw. If you look closely at the test, the URL that we are getting is not the same as the URL that we are asserting on. The one that we are asserting on has an extra slash. The simple solution is to make sure that you specify all base URLs with a trailing slash, but it can easily catch you out.

It must be pointed out that collecting HTTP status codes is not the only thing you can do with network traffic. The fact that it's not a perfect solution for this use case does not mean that it isn't good for other things. There are things that you can only do by tracking network traffic.

For example, if you were writing a checkout application and you wanted to be sure that any transactions in flight were explicitly cancelled when you navigated to a different URL, tracking the network traffic would be ideal. Similarly, if you wanted to check that a specific network request was formatted in a specific way, you would need to scan the network traffic.

There are also other avenues that open up to you if you are using a proxy.

Maybe you would like to simulate a bad network connection. Well, you can do this by configuring your proxy to limit upload and download speeds. How about blocking some content and then taking screenshots of every step in your user journey? You can then quickly and easily view what a flow would look like if the images were not available.

There are many interesting and unusual things you can do if you have access to, and can manipulate, the network traffic.

We have a very basic proxy implementation, but in its current form it doesn't really work well with the test framework that we created earlier in this book. Let's have a look at how we can extend that framework to support proxies.

First of all, we are going to need to tweak our POM a little bit to allow us to set proxy details on the command line. To do this, we first need to add in some additional properties:

<properties>
    <project.build.sourceEncoding>UTF- 
    8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-
    8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
    <!-- Dependency versions -->
    <selenium.version>3.12.0</selenium.version>
    <testng.version>6.14.3</testng.version>
    <assertj-core.version>3.10.0</assertj-core.version>
    <query.version>1.2.0</query.version>
    <commons-io.version>2.6</commons-io.version>
    <httpclient.version>4.5.5</httpclient.version>
    <!-- Plugin versions -->
    <driver-binary-downloader-maven-plugin.version>1.0.17
    </driver-binary-downloader-maven-plugin.version>
    <maven-compiler-plugin.version>3.7.0
    </maven-compiler-plugin.version>
    <maven-failsafe-plugin.version>2.21.0
    </maven-failsafe-plugin.version>
    <!-- Configurable variables -->
    <threads>1</threads>
    <browser>firefox</browser>
    <overwrite.binaries>false</overwrite.binaries>
    <headless>true</headless>
    <remote>false</remote>
    <seleniumGridURL/>
    <platform/>
    <browserVersion/>
    <screenshotDirectory>${project.build.directory}
    /screenshots</screenshotDirectory>
    <proxyEnabled>false</proxyEnabled>
    <proxyHost/>
    <proxyPort/>
</properties>

As you can see, I have not set any default values, but if you have a good idea what the proxy details will be, feel free to add them. I have also set the proxyEnabled property to false by default; you can change this to true if you always want to use a proxy.

The next thing we need to do is to set these as system properties, so that our test can read them in:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>${maven-failsafe-plugin.version}</version>
    <configuration>
        <parallel>methods</parallel>
        <threadCount>${threads}</threadCount>
        <systemPropertyVariables>
            <browser>${browser}</browser>
            <headless>${headless}</headless>
            <remoteDriver>${remote}</remoteDriver>
            <gridURL>${seleniumGridURL}</gridURL>
            <desiredPlatform>${platform}</desiredPlatform>
            <desiredBrowserVersion>${browserVersion}
            </desiredBrowserVersion>
            <screenshotDirectory>${screenshotDirectory}
            </screenshotDirectory>
            <proxyEnabled>${proxyEnabled}</proxyEnabled>
            <proxyHost>${proxyHost}</proxyHost>
            <proxyPort>${proxyPort}</proxyPort>
            <!--Set properties passed in by the driver binary 
            downloader-->
            <webdriver.chrome.driver>${webdriver.chrome.driver}
            </webdriver.chrome.driver>
            <webdriver.ie.driver>${webdriver.ie.driver}
            </webdriver.ie.driver>
            <webdriver.opera.driver>${webdriver.opera.driver}
            </webdriver.opera.driver>
            <webdriver.gecko.driver>${webdriver.gecko.driver} 
            </webdriver.gecko.driver>
            <webdriver.edge.driver>${webdriver.edge.driver}
            </webdriver.edge.driver>
        </systemPropertyVariables>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

We are now ready to start tweaking the rest of our code.  First of all we need  to update our DriverFactory class to read in our proxy settings.  We do this by adding some additional class variables reading in the system properties we configured in our POM:

private final boolean proxyEnabled = Boolean.getBoolean("proxyEnabled");
private final String proxyHostname = System.getProperty("proxyHost");
private final Integer proxyPort = Integer.getInteger("proxyPort");
private final String proxyDetails = String.format("%s:%d", proxyHostname, proxyPort);

Then we will need to modify our instantiateWebDriver method so that it configures our DesiredCapabilities to use a proxy if the proxyEnabled variable is true.

private void instantiateWebDriver(DriverType driverType) throws MalformedURLException {
    System.out.println(" ");
    System.out.println("Local Operating System: " + 
    operatingSystem);
    System.out.println("Local Architecture: " + systemArchitecture);
    System.out.println("Selected Browser: " + selectedDriverType);
    System.out.println("Connecting to Selenium Grid: " + 
    useRemoteWebDriver);
    System.out.println(" ");

    DesiredCapabilities desiredCapabilities = new 
    DesiredCapabilities();

    if (proxyEnabled) {
        Proxy proxy = new Proxy();
        proxy.setProxyType(MANUAL);
        proxy.setHttpProxy(proxyDetails);
        proxy.setSslProxy(proxyDetails);
        desiredCapabilities.setCapability(PROXY, proxy);
    }

    if (useRemoteWebDriver) {
        URL seleniumGridURL = new 
        URL(System.getProperty("gridURL"));
        String desiredBrowserVersion = 
        System.getProperty("desiredBrowserVersion");
        String desiredPlatform = 
        System.getProperty("desiredPlatform");

        if (null != desiredPlatform && !desiredPlatform.isEmpty()) {
            desiredCapabilities.setPlatform
            (Platform.valueOf(desiredPlatform.toUpperCase()));
        }

        if (null != desiredBrowserVersion && 
        !desiredBrowserVersion.isEmpty()) {
            desiredCapabilities.setVersion(desiredBrowserVersion);
        }

        desiredCapabilities.setBrowserName(selectedDriverType.toString());
        webDriver = new RemoteWebDriver(seleniumGridURL, 
        desiredCapabilities);
    } else {
        webDriver = 
        driverType.getWebDriverObject(desiredCapabilities);
    }
}

That's it, the changes are actually very simple.  We now have the ability to specify a proxy on the command line, and we can pre-configure one in our POM. This is great from the point of view of plugging corporate proxy details into your browser; however, it's not ideal if we want to use BrowserMobProxy. When we used BrowserMobProxy in our example test, you may have noticed that we were programmatically starting a BrowserMobProxy instance and interacting with it in our test. To do this, we really want to bake in support for BrowserMobProxy, and we also want to be able to forward on calls to our corporate proxy.

Let's extend our framework again to do exactly this. First of all, we will add a dependency to browsermob-core to our POM:

<dependency>
    <groupId>net.lightbody.bmp</groupId>
    <artifactId>browsermob-core</artifactId>
    <version>2.1.5</version>
    <scope>test</scope>
</dependency>

This will make all of the BrowserMobProxy libraries available for our implementation. Then we need to update DriverFactory so that it can support BrowserMobProxy. We will start off by adding a couple of class variables:

private BrowserMobProxy browserMobProxy; 
private boolean useBrowserMobProxy = false;

We are going to use these later on to hold a reference to any BrowserMobProxy instance we start, and to track whether we are using it or not. The next step is to update our instantiateWebDriver method:

private void instantiateWebDriver(DriverType driverType, boolean useBrowserMobProxy) throws MalformedURLException {
    System.out.println(" ");
    System.out.println("Local Operating System: " + 
    operatingSystem);
    System.out.println("Local Architecture: " + systemArchitecture);
    System.out.println("Selected Browser: " + selectedDriverType);
    System.out.println("Connecting to Selenium Grid: " + 
    useRemoteWebDriver);
    System.out.println(" ");

    DesiredCapabilities desiredCapabilities = new 
    DesiredCapabilities();

    if (proxyEnabled || useBrowserMobProxy) {
        Proxy proxy;
        if (useBrowserMobProxy) {
            usingBrowserMobProxy = true;
            browserMobProxy = new BrowserMobProxyServer();
            browserMobProxy.start();
            if (proxyEnabled) {
                browserMobProxy.setChainedProxy(new 
                InetSocketAddress(proxyHostname, proxyPort));
            }
            proxy = ClientUtil.createSeleniumProxy(browserMobProxy);
        } else {
            proxy = new Proxy();
            proxy.setProxyType(MANUAL);
            proxy.setHttpProxy(proxyDetails);
            proxy.setSslProxy(proxyDetails);
        }
        desiredCapabilities.setCapability(PROXY, proxy);
    }

    if (useRemoteWebDriver) {
        URL seleniumGridURL = new 
        URL(System.getProperty("gridURL"));
        String desiredBrowserVersion = 
        System.getProperty("desiredBrowserVersion");
        String desiredPlatform = 
        System.getProperty("desiredPlatform");

        if (null != desiredPlatform && !desiredPlatform.isEmpty()) {
            desiredCapabilities.setPlatform(Platform.valueOf
            (desiredPlatform.toUpperCase()));
        }

        if (null != desiredBrowserVersion && 
        !desiredBrowserVersion.isEmpty()) {
            desiredCapabilities.setVersion(desiredBrowserVersion);
        }

        desiredCapabilities.setBrowserName(selectedDriverType.toString());
        webDriver = new RemoteWebDriver(seleniumGridURL, 
        desiredCapabilities);
    } else {
        webDriver = 
        driverType.getWebDriverObject(desiredCapabilities);
    }
}

We are now also passing in a Boolean that tells us if we want to use BrowserMobProxy or not.  This lets us configure our RemoteWebDriver object to use a proxy, or a BrowserMobProxy, or both.  This ensures that if we need to use a corporate proxy server to get to the website we are testing we can still use our corporate proxy as well as our BrowserMobProxy.  We have a bit more work to do though, we need to modify our getDriver() method to give us the ability to turn the BrowserMobProxy on or off.

public RemoteWebDriver getDriver(boolean useBrowserMobProxy) 
throws MalformedURLException {
    if(useBrowserMobProxy != usingBrowserMobProxy){
        quitDriver();
    }
    if (null == webDriver) {
        instantiateWebDriver(selectedDriverType, 
        useBrowserMobProxy);
    }

    return webDriver;
}

public RemoteWebDriver getDriver() throws MalformedURLException {
    return getDriver(usingBrowserMobProxy);
}

public void quitDriver() {
    if (null != webDriver) {
        webDriver.quit();

        webDriver = null;
        usingBrowserMobProxy = false;
    }
}

We have a few changes here.  To start off with we have created a new getDriver(boolean useBrowserMobProxy) method that takes in a boolean that allows you to specify if you want to use the BrowserMobProxy or not.  We then compare this boolean to our usingBrowserMobProxy boolean stored in our DriverFactory to see if need to start up, or stop the BrowserMobProxy.

 If we need to start up or stop the BrowserMobProxy we invoke quitDriver() to kill our current WebDriver instance and create a new one since proxy settings need to be specified when instantiating the WebDriver object.

Our old getDriver() method now calls the new one, but it always sets the useBrowserMobProxy value to the value stored internally in DriverFactory so that we continue to use what has already been set.  This can be used by the DriverFactory method without any changes so that our existing code continues to work.

The final change we need to make in DriverFactory is to create a getBrowserMobProxy() method:

public BrowserMobProxy getBrowserMobProxy() { 
    if (usingBrowserMobProxy) { 
        return browserMobProxy; 
    } 
    return null; 
}

This is used to return the BrowserMobProxy instance so that you can give it commands. This is so that you can start collecting traffic, or examine traffic it has already collected.

The final step is to expose this functionality by tweaking our DriverBase class.  First of all we need a way to get hold of a BrowserMobProxy enabled driver instance:

public static RemoteWebDriver getBrowserMobProxyEnabledDriver() throws MalformedURLException {
    return driverThread.get().getDriver(true);
}

Then we need a way to get hold of the BrowserMobProxy object:

public static BrowserMobProxy getBrowserMobProxy() {
    return driverThread.get().getBrowserMobProxy();
}

Now everything is ready for you to use in your tests. Why don't you try writing a test that will check the network traffic (like the one we wrote earlier in this chapter) to try it out?