Running tests from the command line

To test the web performance of http://www.example.com on a WebPageTest public instance using an API key with default configuration:

 webpagetest test http://www.example.com -k API_KEY_GOES_HERE

Or with long parameter names:

 webpagetest test http://www.example.com --key API_KEY_GOES_HERE

Here’s the same test but with the following configuration:

• Run from San Francisco location

• Use latest Chrome

• Use DSL connectivity profile

• Run three times

• First view only (for each run)

• Capture video

• Set “Using WebPageTest” as test label

• Capture DevTools Timeline information

webpagetest test http://www.example.com -k API_KEY_GOES_HERE -l 
SanFrancisco:Chrome -y DSL -r 3 -f -v -L "Using WebPageTest" -M

Or with long parameter names:

webpagetest test http://www.example.com --key API_KEY_GOES_HERE --location 
SanFrancisco:Chrome --connectivity DSL --runs 3 --first --video --label 
"Using WebPageTest" --timeline

Batch jobs can be run in parallel while the response follows the same order as in a given input file. Assuming jobs.txt has the following content:

test http://www.example.com -k API_KEY_GOES_HERE
webpagetest test http://www.example.com -k API_KEY_GOES_HERE -l 
SanFrancisco:Chrome -y DSL -r 3 -f -v -L "Using WebPageTest" -M

Then from the command line, type:

 webpagetest batch jobs.txt

The test command also supports a WebPageTest script file as input instead of a URL. Assuming sample.wptscript has the following content:

logData 0
navigate http://www.example.com/login
logData 1
setValue name=username johndoe
setValue name=password 12345
submitForm action=http://www.example.com/main
waitForComplete

Then from command line, type:

 webpagetest test sample.wptscript

Reading results from the command line

To read results, assuming any of the above test commands returned 150109_DE_ZW7 as testId, type:

 webpagetest results 150109_DE_ZW7

Running tests and reading results from the command line

Since running a test and then reading its results is the most common WebPageTest workflow, the Node.js wrapper provides polling and pingback mechanisms. Here is an example that requests that a WebPageTest public instance test the web performance of http://www.example.com using the default test configuration, and then start polling every five seconds (default interval that can be overriden if a number in seconds is provided for the --poll parameter):

 webpagetest test http://www.example.com -k API_KEY_GOES_HERE --poll

Here’s the same example pinging back to a private instance of WebPageTest, because a public instance wouldn’t be able to pingback localhost:

 webpagetest test http://www.example.com -s wpt-private-server.com --wait

For both of these tests, --timeout could be provided (in seconds) to either stop polling or abandon waiting for pingback.

Running tests and reading results from a Node.js module

All methods are asynchronous; i.e., they require a callback function that is executed when the WebPageTest API response is received with either data or an error. Unlike with the command line, method names on the Node.js module are verbose (e.g., getTestResults versus results) for code readability.

The following example tests the web performance of http://www.example.com on a WebPageTest public instance using an API key with the default configuration, and then polls results every five seconds, getting the first-paint time for first view:

var WebPagetest = require('webpagetest');

var wpt = new WebPagetest('www.webpagetest.org', 'API_KEY_GOES_HERE');

wpt.runTest('http://www.example.com', function(err, res) {
  if (err || res.statusCode >= 400) {
    return console.log(err || res.statusText);
  }
  function results(err, res) {
    if (res.statusCode < 200) {
      console.log('Test', res.data.id, 'not ready yet. Trying again in 5s');
      setTimeout(wpt.getTestResults.bind(wpt, res.data.id, results), 5000);
    } else if (res.statusCode == 200) {
      console.log('First Paint:', res.data.median.firstView.firstPaint);
    }
  }
  console.log('Test', res.data.testId, 'requested. Start polling in 5s');
  setTimeout(wpt.getTestResults.bind(wpt, res.data.testId, results), 5000);
});

This could be simplified using the pollResults option:

var WebPagetest = require('webpagetest');

var wpt = new WebPagetest('www.webpagetest.org', 'API_KEY_GOES_HERE');

wpt.runTest('http://www.example.com', {pollResults: 5}, function(err, res) {
  console.log(err || 'First Paint: ' + res.data.median.firstView.firstPaint);
});

Similarly, pingback coud also be used in the previous example:

var WebPagetest = require('webpagetest'),
    os = require('os'),
    url = require('url'),
    http = require('http');

var wpt = new WebPagetest('wpt-private-server.com');

// Local server to listen for test complete.
var localServer = http.createServer(function(req, res) {
  var uri = url.parse(req.url, true);

  res.end();

  // Get test results.
  if (uri.pathname === '/testdone' && uri.query.id) {
    localServer.close(function() {
      wpt.getTestResults(uri.query.id, function(err, res) {
        console.log(err || 'First Paint: ' + res.data.median.firstView.firstPaint);
      });
    });
  }
});

// Test http://www.example.com.
wpt.runTest('http://www.example.com', {
  pingback: url.format({
    protocol: 'http',
    hostname: os.hostname(),
    port: 8080,
    pathname: '/testdone'
  })
}, function(err, data) {
  // Listen for test complete (pingback).
  localServer.listen(8080);
});

Or to make it even simpler, use the waitResults option:

var WebPagetest = require('webpagetest');

var wpt = new WebPagetest('wpt-private-server.com');

wpt.runTest('http://www.example.com', {waitResults: 'auto'}, function(err, res) {
  console.log(err || 'First Paint: ' + res.data.median.firstView.firstPaint);
});

By setting auto to waitResults, the WebPageTest Node.js Wrapper uses system hostname as the hostname and 8000 as the port, which is incremented by 1 in case the port is in use.

The timeout option is also available for both pollResults and waitResults functions.

To avoid the error-prone hassle of tabs versus spaces, the WebPageTest API Wrapper provides a script builder function named scriptToString:

var script = wpt.scriptToString([
  {logData: 0},
  {navigate: 'http://www.example.com/login'},
  {logData: 1},
  {setValue: ['name=username', 'johndoe']},
  {setValue: ['name=password', '12345']},
  {submitForm: 'action=http://www.example.com/main'},
  'waitForComplete'
]);
wpt.runTest(script, function (err, data) {
  console.log(err || data);
});

Running a proxy from the command line

Assuming a WebPageTest private instance is located at wpt-private-server.com and a local machine named local-machine has bidirectional direct access:

 webpagetest listen --server wpt-private-server.com

This will turn the local machine into a WebPageTest API Wrapper RESTful proxy for wpt-private-server.com. From any other machine in the same network, WebPageTest can be accessed via RESTful proxy, such as:

http://local-machine/help

Displays the WebPageTest API Wrapper help (use http://local-machine/help/<command> to get help for a given command).

http://local-machine/locations

Fetches WebPageTest locations for wpt-private-server.com.

http://local-machine/test/http%3A%2F%2Fwww.example.com.com?first=true

Runs a test for http://www.example.com using default test configuration on wpt-private-server.com.

http://local-machine/results/150109_DE_ZW7

Fetches test results for an existing test with ID 150109_DE_ZW7 on wpt-private-server.com.

If --server or -s is not provided, the WebPageTest API Wrapper first checks the WEBPAGETEST_SERVER environment variable and falls back to the public instance, www.webpagetest.org.

Running a proxy from a Node.js module

The method for running a proxy from a Node.js module is called listen and has one optional port parameter (default 7791).

var WebPageTest = require('webpagetest');
var wpt = new WebPageTest();
wpt.listen(3000);

Examples of overriding assertion comparison

Less-than comparison:

{ "median": { "firstView": {
  "render": 400
}}}

or

{ "median": { "firstView": {
  "render": { "max": 400 }
}}}

Greater-than comparison:

{ "median": { "firstView": {
  "score_gzip": { "min": 75 }
}}}

Range comparison:

{ "median": { "firstView": {
  "requests": { "min": 10, "max": 30 }
}}}

Asserting by MIME type

By either running tests synchronously or just fetching results, it is possible to test by MIME type:

{
  "median": {
    "firstView": {
      "breakdown": {
        "js": {
          "requests": 6,
          "bytes": 200000
        },
        "css": {
          "requests": 1,
          "bytes": 50000
        },
        "image": {
          "requests": 10,
          "bytes": 300000
        }
      }
    }
  }
}

The preceding spec only allows up to 6 JavaScript requests summing up to 200 KB, 1 CSS request up to 50 KB, and no more than 10 images up to 300 KB total.

Asserting by processing breakdown

When runnning tests synchronously in Chrome with the --timeline option, it is possible to test by processing breakdown:

{
  "run": {
    "firstView": {
      "processing": {
        "RecalculateStyles": 1300,
        "Layout": 2000,
        "Paint": 800
      }
    }
  }
}

The preceding spec only allows up to 1,300 ms of recalculate styles, 2,000 ms of layout, and 800 ms of paint time processing. Thus, it avoids rendering regression once these metrics are known by measuring multiple times from previous tests.

Using TAP as test resporter

The Test Anything Protocol (TAP) is a plug-in that can be installed via the Jenkins Plugin Manager. Assuming example.com has the following configuration:

• Staging server: staging.corp.example.com

• Jenkins server: jenkins.corp.example.com

• WebPageTest private instance: wpt.corp.example.com

• WebPageTest location named Default with Chrome browser

• Jenkins has a /specs directory with test specs JSON files, with:

  /specs/homepage.json:

{
  "median": {
    "firstView": {
      "requests": 20,
      "render": 400,
      "loadTime": 3000,
      "score_gzip": {
        "min": 90
      }
    }
  }
}

The build shell command to be executed is:

webpagetest test http://staging.corp.example.com 
--server http://wpt.corp.example.com --first --location Default:Chrome 
--wait jenkins.corp.example.com:8000 --specs /specs/homepage.json 
--reporter tap > homepage.tap

Jenkins (the tool) has a “Post-build Actions” section where users should input homepage.tap as “Test results.” You can see a screenshot at http://bit.ly/wpt-jenkins.

Using JUnit as a test reporter

Using the same TAP example but without plug-ins, Jenkins can report JUnit by default with the following build shell command:

webpagetest test http://staging.corp.example.com 
--server http://wpt.corp.example.com --first --location Default:Chrome 
--wait jenkins.corp.example.com:8000 --specs /specs/homepage.json 
--reporter xunit > homepage.xml

Jenkins postbuild actions should publish a JUnit test result report for homepage.xml.

Encrypting the WebPageTest API key

If you are scheduling your tests to run from public instances of Travis-CI workers, such as from a public GitHub repository, WebPageTest API keys (--key or -k) should be used to prevent abuse, but do not put unencrypted API keys in public files. Fortunately, Travis-CI provides an easy way to do this via secure environment variables, which avoid explicitly passing $WPT_API_KEY in the public .travis.yml file.

Install Travis and go to the repo directory:

gem install travis
...
cd repo_dir

Next, encrypt the WebPageTest API key as a secure environment variable:

travis encrypt WPT_API_KEY=super_secret_api_key_here --add

Note that it must run from the repo directory or use -r or --repo to specify the repo name in the format user/repo—for example, marcelduran/webpagetest-api.

By default, the --add flag will append the encrypted string to the .travis.yml file as:

env:
  global:
    - secure: +_encrypted WPT_API_KEY=super_secret_api_key_here string\_+

In this chapter, we covered how the WebPageTest API can be integrated into your web development pipeline via CI. It helps you leverage the quality of your web pages by preventing key performance metrics from regressing during push cycles. It can also help you track some performance metrics values over time so you can measure the impact of adding new features to the page. Once WebPageTest is integrated into your CI tool, after several pushing cycles you start getting a better idea of the state of perfomance of your pages. Data collected from CI can be used to plot historical information about your pages’ performance. It can catch unexpected regressions in non–performance-related expected changes. Some metrics can be easy to track and catch, such as the number of requests, but some, especially those related to time, can require some tuning to find an optimum range.